One of five new nodes are not receiving any data

Node Operators troubleshooting
1

Hello friends

I was at a friends house, helping him set up storage nodes for his house.
He has an older system with 5x2TB disks, so we’re using all of them - each for a single node.

Here is the kicker - four of the nodes started up just fine, and are down/uploading data. The last node (well, actually it’s the third) however starts up fine as well, it’s online, the ports are open and everything seems fine, but it does not recieve any new data. An idea of what’s going on?

I’ve included the entire log set to info, since it came online. As you an see, it checks for new versions which has not made it on to it (it’s currently 1.115.5, just like the other four), it starts teh collector every hour and it updates reputation scores. From my point of view, all lights are green. What’s up? :slight_smile:

Other info:

  • OS: Debian 12 server, minimal install
  • Hardware: 2 cores, 4GB RAM (900 MiB used)
  • Docker version: 27.3.1
  • Disk: UUID mounted to folder in the same way all other disks are.
  • I’ve tried moving the entire node to another disk (to eliminate disk issues)
  • I’ve tried recreating the storagenodes identity because I ran out of ideas.

Log:

WARN[0000] /home/docker-01/storagenode-dock01-03/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion
[+] Running 1/1
 ✔ Container storagenode-dock01-03  Recreated                                                                                                                                                                              0.1s
Attaching to storagenode-dock01-03
storagenode-dock01-03  | 2024-11-17 18:25:57,187 INFO RPC interface 'supervisor' initialized
storagenode-dock01-03  | 2024-11-17 18:25:57,187 INFO supervisord started with pid 1
storagenode-dock01-03  | 2024-11-17 18:25:58,190 INFO spawned: 'processes-exit-eventlistener' with pid 38
storagenode-dock01-03  | 2024-11-17 18:25:58,193 INFO spawned: 'storagenode' with pid 39
storagenode-dock01-03  | 2024-11-17 18:25:58,205 INFO spawned: 'storagenode-updater' with pid 40
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "contact.external-address"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.email"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet-features"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.details"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "console.address"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Configuration loaded    {"Process": "storagenode", "Location": "/app/config/config.yaml"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Anonymized tracing enabled      {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Operator email  {"Process": "storagenode", "Address": "[ThisIsHisEmailThatIHaveRemovedNowYes]"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Operator wallet {"Process": "storagenode", "Address": "[ThisIsHisWalletAddress]"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    server  kernel support for server-side tcp fast open remains disabled.  {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    server  enable with: sysctl -w net.ipv4.tcp_fastopen=3  {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode-updater"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Telemetry enabled       {"Process": "storagenode", "instance ID": "12ZuHzihifUrgo97FZqoJzGw9qEYpaprKcQv3Wu6CakYaQ7sdv3"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    Event collection enabled        {"Process": "storagenode", "instance ID": "12ZuHzihifUrgo97FZqoJzGw9qEYpaprKcQv3Wu6CakYaQ7sdv3"}
storagenode-dock01-03  | 2024-11-17T18:25:58Z   INFO    db.migration    Database Version        {"Process": "storagenode", "version": 62}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    preflight:localtime     start checking local system clock with trusted satellites' system clock.        {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17 18:25:59,205 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
storagenode-dock01-03  | 2024-11-17 18:25:59,206 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
storagenode-dock01-03  | 2024-11-17 18:25:59,206 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    preflight:localtime     local system clock is in sync with trusted satellites' system clock.    {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    collector       expired pieces collection started       {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    Node 12ZuHzihifUrgo97FZqoJzGw9qEYpaprKcQv3Wu6CakYaQ7sdv3 started        {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    Public server started on [::]:28967     {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    trust   Scheduling next refresh {"Process": "storagenode", "after": "6h24m22.112381624s"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    pieces:trash    emptying trash started  {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    collector       expired pieces collection completed     {"Process": "storagenode", "count": 0}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    Private server started on 127.0.0.1:7778        {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker starting subprocess     {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker subprocess started      {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.   {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess      trash-filewalker started        {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "dateBefore": "2024-11-10T18:25:59Z"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess      Database started        {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess      trash-filewalker completed      {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully        {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    pieces:trash    emptying trash finished {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "elapsed": "65.788465ms"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    pieces:trash    emptying trash started  {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker starting subprocess     {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker subprocess started      {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess      trash-filewalker started        {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "dateBefore": "2024-11-10T18:25:59Z"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess      Database started        {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess      trash-filewalker completed      {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully        {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    pieces:trash    emptying trash finished {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "elapsed": "44.680614ms"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    pieces:trash    emptying trash started  {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker starting subprocess     {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker subprocess started      {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
storagenode-dock01-03  | 2024-11-17T18:25:59Z   INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess      trash-filewalker started        {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode", "dateBefore": "2024-11-10T18:25:59Z"}
storagenode-dock01-03  | 2024-11-17T18:26:00Z   INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess      Database started        {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:26:00Z   INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess      trash-filewalker completed      {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "bytesDeleted": 0, "numKeysDeleted": 0, "Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:26:00Z   INFO    lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully        {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
storagenode-dock01-03  | 2024-11-17T18:26:00Z   INFO    pieces:trash    emptying trash finished {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "elapsed": "45.772829ms"}
storagenode-dock01-03  | 2024-11-17T18:26:00Z   INFO    pieces:trash    emptying trash started  {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
storagenode-dock01-03  | 2024-11-17T18:26:00Z   INFO    lazyfilewalker.trash-cleanup-filewalker starting subprocess     {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
storagenode-dock01-03  | 2024-11-17T18:26:00Z   INFO    lazyfilewalker.trash-cleanup-filewalker subprocess started      {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
storagenode-dock01-03  | 2024-11-17T18:26:00Z   INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess      trash-filewalker started        {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "dateBefore": "2024-11-10T18:26:00Z"}
storagenode-dock01-03  | 2024-11-17T18:26:00Z   INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess      Database started        {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T18:26:00Z   INFO    lazyfilewalker.trash-cleanup-filewalker.subprocess      trash-filewalker completed      {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode", "bytesDeleted": 0, "numKeysDeleted": 0}
storagenode-dock01-03  | 2024-11-17T18:26:00Z   INFO    lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully        {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
storagenode-dock01-03  | 2024-11-17T18:26:00Z   INFO    pieces:trash    emptying trash finished {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "elapsed": "41.312789ms"}
storagenode-dock01-03  | 2024-11-17T18:26:43Z   INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Total Audits": 0, "Successful Audits": 0, "Audit Score": 1, "Online Score": 1, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
storagenode-dock01-03  | 2024-11-17T18:26:44Z   INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Total Audits": 0, "Successful Audits": 0, "Audit Score": 1, "Online Score": 1, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
storagenode-dock01-03  | 2024-11-17T18:26:44Z   INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 0, "Successful Audits": 0, "Audit Score": 1, "Online Score": 1, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
storagenode-dock01-03  | 2024-11-17T18:26:44Z   INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Total Audits": 0, "Successful Audits": 0, "Audit Score": 1, "Online Score": 1, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
storagenode-dock01-03  | 2024-11-17T19:25:59Z   INFO    collector       expired pieces collection started       {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T19:25:59Z   INFO    collector       expired pieces collection completed     {"Process": "storagenode", "count": 0}
storagenode-dock01-03  | 2024-11-17T19:25:59Z   INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T19:40:58Z   INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
storagenode-dock01-03  | 2024-11-17T19:40:58Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T19:40:58Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
storagenode-dock01-03  | 2024-11-17T19:40:58Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T19:40:58Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode-updater"}
storagenode-dock01-03  | 2024-11-17T19:55:58Z   INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
storagenode-dock01-03  | 2024-11-17T19:55:59Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T19:55:59Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
storagenode-dock01-03  | 2024-11-17T19:55:59Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T19:55:59Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode-updater"}
storagenode-dock01-03  | 2024-11-17T20:10:58Z   INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
storagenode-dock01-03  | 2024-11-17T20:10:59Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T20:10:59Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
storagenode-dock01-03  | 2024-11-17T20:10:59Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T20:10:59Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode-updater"}
storagenode-dock01-03  | 2024-11-17T20:25:58Z   INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
storagenode-dock01-03  | 2024-11-17T20:25:59Z   INFO    collector       expired pieces collection started       {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T20:25:59Z   INFO    collector       expired pieces collection completed     {"Process": "storagenode", "count": 0}
storagenode-dock01-03  | 2024-11-17T20:25:59Z   INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T20:26:00Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T20:26:00Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
storagenode-dock01-03  | 2024-11-17T20:26:00Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T20:26:00Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode-updater"}
storagenode-dock01-03  | 2024-11-17T20:40:58Z   INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
storagenode-dock01-03  | 2024-11-17T20:40:58Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T20:40:58Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
storagenode-dock01-03  | 2024-11-17T21:25:59Z   INFO    collector       expired pieces collection started       {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T21:25:59Z   INFO    collector       expired pieces collection completed     {"Process": "storagenode", "count": 0}
storagenode-dock01-03  | 2024-11-17T21:25:59Z   INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T21:40:58Z   INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
storagenode-dock01-03  | 2024-11-17T21:40:59Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T21:40:59Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
storagenode-dock01-03  | 2024-11-17T21:40:59Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T21:40:59Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode-updater"}
storagenode-dock01-03  | 2024-11-17T21:55:58Z   INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
storagenode-dock01-03  | 2024-11-17T21:55:58Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T21:55:58Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
storagenode-dock01-03  | 2024-11-17T21:55:58Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T21:55:58Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode-updater"}
storagenode-dock01-03  | 2024-11-17T22:10:58Z   INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
storagenode-dock01-03  | 2024-11-17T22:10:58Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T22:10:58Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
storagenode-dock01-03  | 2024-11-17T22:10:58Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T22:10:58Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode-updater"}
storagenode-dock01-03  | 2024-11-17T22:25:58Z   INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
storagenode-dock01-03  | 2024-11-17T22:25:59Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T22:25:59Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode"}
storagenode-dock01-03  | 2024-11-17T22:25:59Z   INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.115.5"}
storagenode-dock01-03  | 2024-11-17T22:25:59Z   INFO    New version is being rolled out but hasn't made it to this node yet     {"Process": "storagenode-updater", "Service": "storagenode-updater"}
storagenode-dock01-03  | 2024-11-17T22:25:59Z   INFO    collector       expired pieces collection started       {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T22:25:59Z   INFO    collector       expired pieces collection completed     {"Process": "storagenode", "count": 0}
storagenode-dock01-03  | 2024-11-17T22:25:59Z   INFO    bandwidth       Persisting bandwidth usage cache to db  {"Process": "storagenode"}
storagenode-dock01-03  | 2024-11-17T22:29:00Z   INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Total Audits": 0, "Successful Audits": 0, "Audit Score": 1, "Online Score": 1, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
storagenode-dock01-03  | 2024-11-17T22:29:01Z   INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Total Audits": 0, "Successful Audits": 0, "Audit Score": 1, "Online Score": 1, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
storagenode-dock01-03  | 2024-11-17T22:29:01Z   INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Total Audits": 0, "Successful Audits": 0, "Audit Score": 1, "Online Score": 1, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
storagenode-dock01-03  | 2024-11-17T22:29:02Z   INFO    reputation:service      node scores updated     {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Total Audits": 0, "Successful Audits": 0, "Audit Score": 1, "Online Score": 1, "Suspension Score": 1, "Audit Score Delta": 0, "Online Score Delta": 0, "Suspension Score Delta": 0}
2

Curious!

To confirm, the other nodes are getting data from all satellites, and this problem node is not getting data from any satellites?

1 Like
3

Hiya @thepaul - Yes, the four other nodes on the same box, on the same IP (different ports, obviously), are getting data from all satellites.

The fifth node in question is not getting data from any satellites. I have remote access to the system and can check if you need more information. It has been online for ~14 hours as of right now, and while it’s all lights green, still sits at 0 usage :slight_smile:

4

Ah, I think I found the problem. That node is reporting to the satellites that it has less than a kilobyte of space available. Could it be a failed mount?

3 Likes
5

WOW, that was fast. I love this community so much.
It’s not entirely impossible I messed something up with the mount.

I did sudo blkid to get the UUID of the disk. I copied the ID into the /etc/fstab file, and mounted it with default permissions like the rest. I wonder why I don’t get any error messages related to lack of space?

Just a sec - I’ll log in and produce a screenshot.

sudo blkid

Sidenote - I would love to use powershells sort-objecton the output to list in sortet fashion. Take node of the UUID of /dev/sdd1

/dev/sdf1: UUID="99a94c81-72b2-4594-88dd-902aca293fd2" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="0eaa4e4e-01"
/dev/sdd1: UUID="8d76a795-634f-4e70-9056-f1a66f09dacb" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="b0de9a32-01"
/dev/sdb1: UUID="350d5d7c-1a45-4854-ad20-897a43e320e4" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="331e1128-01"
/dev/sde1: UUID="62e00658-7d24-41f1-9a56-6e34bcda8f5e" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="eb0acf1f-01"
/dev/sdc1: UUID="74ff54cd-81ed-4a8d-8124-8027e389200e" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="24220a91-01"
/dev/sda5: UUID="46165bf9-2bb1-4a0e-80ad-c67b5a1620c2" TYPE="swap" PARTUUID="c85cda74-05"
/dev/sda1: UUID="abd84b95-da68-4fb6-b182-98e80feac741" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="c85cda74-01"

/etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# systemd generates mount units based on this file, see systemd.mount(5).
# Please run 'systemctl daemon-reload' after making changes here.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda1 during installation
UUID=abd84b95-da68-4fb6-b182-98e80feac741 /               ext4    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=46165bf9-2bb1-4a0e-80ad-c67b5a1620c2 none            swap    sw              0       0
/dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
#
UUID="350d5d7c-1a45-4854-ad20-897a43e320e4" /home/docker-01/storagenode-dock01-01/data ext4 noatime 0 2
UUID="74ff54cd-81ed-4a8d-8124-8027e389200e" /home/docker-01/storagenode-dock01-02/data ext4 noatime 0 2
UUID="8d76a795-634f-4e70-9056-f1a66f09dacb" /home/docker-01/storagenode-dock01-03/data ext4 noatime 0 2
UUID="62e00658-7d24-41f1-9a56-6e34bcda8f5e" /home/docker-01/storagenode-dock01-04/data ext4 noatime 0 2
UUID="99a94c81-72b2-4594-88dd-902aca293fd2" /home/docker-01/storagenode-dock01-05/data ext4 noatime 0 2

Ownership

On the storagenode folders, I individually run recursive chown and chmod, and the ownership of the affected folder looks like so:

docker-01@debian:~/storagenode-dock01-03$ ls -l .
total 16
drwxrwxrwx 6 docker-01 root      4096 Nov 18 12:38 data
-rwxrwxrwx 1 docker-01 docker-01  661 Nov 18 00:04 docker-compose.yml
drwxrwxrwx 2 docker-01 docker-01 4096 Nov 17 19:05 identity
-rwxrwxrwx 1 docker-01 docker-01  279 Nov 17 19:03 setup.commands

Additionally

I did touch somefile.txt; echo "this is just a test yes" > somefile.txt inside of the data directory, that the disk mounts in. I then rebooted to see if the file persisted - it did

6

Ah ah ah, it was me - it’s always the user.

Here is the docker-compose file. I’ve redacted some of the information, but it the smoking gun is just down there. Take a look, dear reader, and see if you can find it yourself

version: '3.3'
services:
    storagenode:
        restart: unless-stopped
        ports:
            - '[PORT]:28967/tcp'
            - '[PORT]:28967/udp'
            - '14003:14002/tcp'
        environment:
            - 'WALLET=[WALLET]'
            - 'EMAIL=[EMAIL]'
            - 'ADDRESS=[IP]:[PORT]'
            - 'STORAGE=1800'
            - 'STORJ_LOG_LEVEL=info'
        user: '1000:1000'
        volumes:
            - './identity:/app/identity'
            - './data:/app/config'
        container_name: storagenode-dock01-03
        image: 'storjlabs/storagenode:latest'

See that? Storage=1800. Just 1800. Not 1800GB, as it was supposed to.

That does raise some questions that @thepaul maybe can answer

  • I have not changed storage2.monitor.minimum-disk-space parameter - why did I not get an error on too low storage?
  • There was no errors in general (as seen by the initial log)
  • Could an alleviation to this issue be to just assume that a user means gigorbytors, if omitting the unit?
3 Likes
7

On a side note - this had nothing to do with the cut undersea cable between Germany and Finland :frowning:

That could have been hella cool, but of course, StorJ (and the internet as a whole, of course) is super resilient to these attacks.

Source:

8
2 Likes
9

Thank you for creating this issue Alexey :slight_smile:

In my opinion, it is of the lowest criticality possible, but it’s nice to have it in the backlog. Have a good day.

1 Like