How to verify which sqlite version is used within docker?

Node Operators FAQ
,
1

Hi, how can I verify, which sqlite version is used by the storagenode within the docker container on Linux?

CL into the container answers just “sqlite / sqlite3 command not found”.

Outside the container I can confirm, that the latest version is used (3.37.x as far as I remember).

2

I guess SQLite is not installed in the docker container. Instead, it’s probably a GO language dependency that is included in the storage node code directly. Maybe you can find the version in the GitHub source code…

1 Like
3

Thx, so it should use the system version, which is 3.37.2.

4

sqlite is public domain.

The library of db functions is built in to the application.

The “system version” is simply a system installed client application that can read and write sqlite database files. However, typically applications that store information in sqlite databases do not utilize any system installed sqlite client. Rather, the application includes the public domain libraries.

1 Like
5

Thx @anon27637763 @striker43

So there is the option, that my issue described here is a Storj related one, linked to sqlite dependencies within the application, specifically on M1 Macs.

Did I understand that correctly?

6

I trust Stefan Benten to know more than I do about this. He’s already active on the bug report. It may be a machine specific bug as you indicated.

1 Like
7

Thank you for clarification anyway, it helped to understand the issue. Tried to support with updating to the latest sqlite version, but was not aware of that it most probably does not have any effect.

8

It’s more Affirmative… locally installed sqlite absolutely is not called by the storj node software.

The error you reported refers to this function in the go implementation of sqlite… specifically line 1999.

https://github.com/mattn/go-sqlite3/blob/671e666c2e88052a814eeaed6bc8506ef326eb63/sqlite3.go#L1972

I don’t know go… and am a utilitarian programmer…

However, there’s a comment in there that may be related to the bug…

// this is still racy and can be no-op if executed between sqlite3_* calls in execSync.

EDIT:

Please note that this code is not Storj created… Storj software depends on this library, but Storj did not write this library.

9

Thanks, it then makes sense to address it to sqlite, too.

Maybe there’s an alternative for storj to avoid this part of the code in general. Crossing fingers. Currently Mum node crashes up to 3 times a day. Docker restart helps, but this can’t be a solution.

10

I like psql… but that has been hashed out on the forum several years ago.

  • sqlite is what is used for the SNO software.
  • psql is what is (or was - I haven’t looked in a long while) used for the Satellite software.
  • Some sqlite databases were transformed into flat files for processing speed issues.

I imagine someone will figure out the issue you’ve brought up - either on the Storj side or the sqlite side. Any issue with the go implementation of sqlite will affect a large number of other projects too.

1 Like
11

Not sure on that. I’ve seen the latest release of the go-sqlite3 implementation is almost 3 years old:

1.10.0 on 29 May 2019

Anyway, I’ve opened a ticket there, too. :metal:t2:

2 Likes
12

Storj uses v1.14.8. It’s mentioned in the error you posted yourself. At the moment there are version tags up to v1.14.10 on their GitHub. The readme.md even states that v1.14 is the latest version.

V1.14.10 is 24 days old. So there is active development going on for sure.

1 Like
13

Version numbers here are decidedly confusing…

while…

  • go sqlite is on 1.14

What does this mean?

Does it mean that go sqlite is working on implementing sqlite version 1.14 ?

Or does it mean that go sqlite is working on go sqlite version 1.14 ?

What are the feature sets implemented?

OP was asking about the difference in versions between sqlite stand alone package and the “stand alone” package included in storj… However, the question isn’t answerable via comparing versions, since go sqlite is not sqlite implemented as a stand alone package. So, the version number question is actually not the question that is being asked.

However, the version number included in the error is a good for figuring out if that error exists in the current go implementation of sqlite.

1 Like
14

indeed that seems to be promising. the latest release is misleading. :v:t2:
let’s see if there will be any activity on the issue opened today in GitHub.

1 Like
15

Random thoughts from a non-go person…

  1. The select statement in go waits for one of the case situations to be ready, but if more than one case is ready one case is chosen at random… apparently.
  2. The Apple M1 is an ARM processor and executes instructions very efficiently in parallel… someone tested this and posted the results in a Medium Blog
  3. The comment in the sqlite go implementation fully indicates the code may be susceptible to race conditions.

I wonder if a future generation of RPi might run fast enough to show the same error…

Or maybe the error has nothing to do with go sqlite …

1 Like
16

@stefanbenten mentioned, he’s running a node on an M1, too. I would expect him to have the same issue, if it’s a general issue. If not, it might be obvious, that he runs other system settings, like e.g. resources provided to docker (?).

If M1 is calculating too fast, it might help to reduce resources provided to docker. I’ve done that, after my node crashes 3 times a day now, especially when node traffic is very high (compared to the average traffic). I’ve reduced cpu count from 6 to 2 in the docker settings. Not sure if that helps, let’s see.

1 Like
17

Update @anon27637763 @BrightSilence :
Stable for 22h now, with only 2 CPUs instead of 6.
Still crossing fingers. :wink:

2 Likes
18

Update @anon27637763 @BrightSilence :
Still stable - since 59h now. I’ll put this post to “solved”.
Thank you for your support!!

2 Likes