Got a signal from the OS: "terminated"

After latest update today (v0.27.1) i found in my docker logs:

Those log entries are nearby every minute!!!

2019-12-11T20:50:00.807Z        INFO    piecestore      uploaded        {"Piece ID": "63IFV5URN6JXAJBJQMIX62KIJ3NJASJXVRRZMSBO7PTY7XTTOEJA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"}
2019-12-11T20:50:01.943Z        INFO    piecestore      upload started  {"Piece ID": "B46GKFJCLFXWHP3IQ2UGSMWF2PS263AQGN4NPCB6NS7Z4F5BKEKQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"}
2019-12-11T20:50:02.479Z        INFO    piecestore      upload started  {"Piece ID": "BDE6QSMJPQBOV7M5QQCO6SYZQI5LOQMTFTJXVF7V5XZ4KJAO4UNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-12-11T20:50:02.561Z        INFO    piecestore      upload started  {"Piece ID": "H7HCW4BI3JZDOFTSYN67OOER5M4NBREZN5YFHTSKRAH7AO7YV3SA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"}
2019-12-11T20:50:02.680Z        INFO    Got a signal from the OS: "terminated"
2019-12-11T20:50:02.737Z        ERROR   piecestore:cacheUpdate  error getting current space used calculation:   {"error": "context canceled"}
2019-12-11T20:50:02.740Z        ERROR   piecestore:cacheUpdate  error persisting cache totals to the database:  {"error": "piece space used error: context canceled", "errorVerbose": "piece space used error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceSpaceUsedDB).UpdateTotal:115\n\tstorj.io/storj/storagenode/pieces.(*CacheService).PersistCacheTotals:82\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:68\n\tstorj.io/storj/private/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:63\n\tstorj.io/storj/storagenode.(*Peer).Run.func6:445\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-12-11T20:50:02.839Z        INFO    piecestore      uploaded        {"Piece ID": "6SDEYRW77QK6QP7ELZWQIKLUASOVPB7TCGVNQW5JPVUONVQRD3VA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-12-11T20:50:02.842Z        INFO    piecestore      upload failed   {"Piece ID": "H7HCW4BI3JZDOFTSYN67OOER5M4NBREZN5YFHTSKRAH7AO7YV3SA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:344\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:223\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}

So the node is restarting all the time. There is noone who stops the container. So it seems Storj is shutting down itself, but restarted by the restart-policy.

docker-compose.yml:

version: "3.7"

services:
  storagenode_1:
    container_name: storagenode_1
    image: storjlabs/storagenode:beta
    restart: unless-stopped
    stop_grace_period: 300s
    ports:
      - "28967:28967"
      - "14002:14002"
    volumes:
      - type: bind
        source: /root/.local/share/storj/identity/storagenode_1
        target: /app/identity
      - type: bind
        source: /mnt/storj_data_1
        target: /app/config
    environment:
      WALLET: "XXXXXX"
      EMAIL: "XXXXX"
      ADDRESS: "XXXXX"
      BANDWIDTH: "30TB"
      STORAGE: "8.5TB"
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"

HDD is a dedicated one only for storj.

Runnin as root.
What should i do?

not: storjlabs/storagenode:alpha
only: storjlabs/storagenode:beta

updated, but points anyway to the identical image. problem stays the same :wink:

Same image ID

storjlabs/storagenode     alpha               06be91d6a7c3        8 hours ago         27MB
storjlabs/storagenode     beta                06be91d6a7c3        8 hours ago         27MB

How is your HDD connected ? Is this running on Linux or Windows ?

Hiho.

Storage is a dedicated 10TB SATA HDD connected via 1GBit NFS in the same rack beside the system.

dd if=/dev/zero of=/mnt/storj_data_1/testfile bs=16k count=1024k
17179869184 Bytes (17 GB, 16 GiB) kopiert, 184,817 s, 93,0 MB/s

dd if=/mnt/storj_data_1/testfile of=/dev/null bs=16k
17179869184 Bytes (17 GB, 16 GiB) kopiert, 164,208 s, 105 MB/s

You cant use smb or nfs.
The .db dont like that at all.
The only thing that works is ISCSi. IF this could be the problem

4 Likes