Hi my note have stop growing and start just go back so will really be happy if one like too just view my look or there see some other error ind I just to late to get new data and not some serious error error
whem wrote this I fond bandwidthdb: database is locked should I try move the database out to a ssd ?
2023-08-25T15:32:03+02:00 INFO Configuration loaded {"process": "storagenode", "Location": "/app/config/config.yaml"}
2023-08-25T15:32:03+02:00 INFO Anonymized tracing enabled {"process": "storagenode"}
2023-08-25T15:32:03+02:00 INFO Operator email {"process": "storagenode", "Address": "xxxx"}
2023-08-25T15:32:03+02:00 INFO Operator wallet {"process": "storagenode", "Address": "xxxx"}
2023-08-25T15:32:03+02:00 INFO server kernel support for server-side tcp fast open remains disabled. {"process": "storagenode"}
2023-08-25T15:32:03+02:00 INFO server enable with: sysctl -w net.ipv4.tcp_fastopen=3 {"process": "storagenode"}
2023-08-25T15:32:05+02:00 INFO Telemetry enabled {"process": "storagenode", "instance ID": "1u5CKqKdfwvLH19SkKr9vE2thoWGiBUDaPRFJsRmdYTHEAZ4vs"}
2023-08-25T15:32:05+02:00 INFO Event collection enabled {"process": "storagenode", "instance ID": "1u5CKqKdfwvLH19SkKr9vE2thoWGiBUDaPRFJsRmdYTHEAZ4vs"}
2023-08-25T15:32:05+02:00 INFO db.migration Database Version {"process": "storagenode", "version": 54}
2023-08-25T15:32:10+02:00 INFO preflight:localtime start checking local system clock with trusted satellites' system clock. {"process": "storagenode"}
2023-08-25T15:32:10+02:00 INFO preflight:localtime local system clock is in sync with trusted satellites' system clock. {"process": "storagenode"}
2023-08-25T15:32:10+02:00 INFO Node 1u5CKqKdfwvLH19SkKr9vE2thoWGiBUDaPRFJsRmdYTHEAZ4vs started {"process": "storagenode"}
2023-08-25T15:32:10+02:00 INFO trust Scheduling next refresh {"process": "storagenode", "after": "5h59m49.06551559s"}
2023-08-25T15:32:10+02:00 INFO bandwidth Performing bandwidth usage rollups {"process": "storagenode"}
2023-08-25T15:32:10+02:00 INFO Public server started on [::]:28967 {"process": "storagenode"}
2023-08-25T15:32:10+02:00 INFO Private server started on 127.0.0.1:7778 {"process": "storagenode"}
|2023-08-25T15:32:10+02:00|INFO|pieces:trash|emptying trash started|{process: storagenode, Satellite ID: 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB}|
|---|---|---|---|---|
|2023-08-25T15:32:11+02:00|INFO|lazyfilewalker.used-space-filewalker|starting subprocess|{process: storagenode, satelliteID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs}|
|2023-08-25T15:32:11+02:00|INFO|lazyfilewalker.used-space-filewalker|subprocess started|{process: storagenode, satelliteID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs}|
|2023-08-25T15:32:11+02:00|INFO|lazyfilewalker.used-space-filewalker.subprocess|Database started|{process: storagenode, satelliteID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, process: storagenode}|
|2023-08-25T15:32:11+02:00|INFO|lazyfilewalker.used-space-filewalker.subprocess|used-space-filewalker started|{process: storagenode, satelliteID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, process: storagenode}|
|2023-08-25T15:32:11+02:00|INFO|piecestore|upload started|{process: storagenode, Piece ID: POHTWXCVZFB4DGYBENAZTSGIMCPI5ZVRKH6MTV4LGRVJJEFFAU3Q, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 8848632265082, Remote Address: 5.161.128.79:38108}|
|2023-08-25T15:32:11+02:00|INFO|piecestore|upload started|{process: storagenode, Piece ID: IL6XVYSGKB4M3RPAVG2C6QDPNCEISJZT4NCUJ5T5IIGMXOPQ43AQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 8848632265082, Remote Address: 5.161.128.79:38494}|
2023-08-26T00:38:59+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7.1:737\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:775\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:792\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-08-26T00:38:59+02:00 INFO piecestore downloaded {"process": "storagenode", "Piece ID": "HFGBDAMCHZAMB2D6VEAJN6IOOVYEYEA6NVL4A47ED4VCLGCKDG2Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 9472, "Remote Address": "5.161.123.49:53822"}
2023-08-26T00:38:59+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:532\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-08-26T00:38:59+02:00 ERROR piecestore upload failed {"process": "storagenode", "Piece ID": "FPICLAHHQOXOHHEJIVJHTLNDBG3AUQ6QZ7WG4ZBB22PN3ZULLXIA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT_REPAIR", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:531\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 65792, "Remote Address": "5.161.211.30:46922"}
2023-08-26T00:38:59+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:507\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-08-26T00:38:59+02:00 ERROR piecestore upload failed {"process": "storagenode", "Piece ID": "HPKNXSNPMDIVEAOW2WTPM4T2LTGRPJUWNVAUTBH5XANMSAISZGGQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 17408, "Remote Address": "5.161.143.41:56660"}
2023-08-26T00:38:59+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:507\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-08-26T00:38:59+02:00 ERROR piecestore upload failed {"process": "storagenode", "Piece ID": "3NP7BFNGZJDZ5QZL7CLVJPFAMAPPWCHO6F2557BOJJBICZS6AKOA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 36864, "Remote Address": "5.161.128.79:21388"}
2023-08-26T00:38:59+02:00 INFO piecestore upload started {"process": "storagenode", "Piece ID": "OBHRDTQWGQZOPPKDFRFDRZGJA2ZU4E4IZUA6GYAIKFMWCIV3HIXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 8846090334586, "Remote Address": "5.161.128.79:21388"}
2023-08-26T00:39:09+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:507\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-08-26T00:39:09+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:507\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-08-26T00:39:09+02:00 ERROR piecestore upload failed {"process": "storagenode", "Piece ID": "FPZ7WDLD3BPFPORKLPM2V6C3KAK3XDHN2KXDTRDRDHCIMF765Y3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 36864, "Remote Address": "5.161.128.79:9784"}
2023-08-26T00:39:09+02:00 ERROR piecestore upload failed {"process": "storagenode", "Piece ID": "7YQ5N5WL4TGM25MHELJDMSP6QRS2CNRZGJX5VCEDQUKYL6AJTP7A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 36864, "Remote Address": "5.161.128.79:15474"}
2023-08-26T00:39:09+02:00 INFO piecestore upload started {"process": "storagenode", "Piece ID": "TRPGQWDEELGRTS4MCH5NBWOGHKVHVWIPJVUFH2AUGEL4ZE5AGASQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 8846089716602, "Remote Address": "5.161.128.79:9784"}
2023-08-26T00:39:09+02:00 INFO piecestore upload started {"process": "storagenode", "Piece ID": "EEBKECHKR4MTCIVFEV7A73AY3VPZOLIICELVRPOONW342WKFOUVA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 8846089716602, "Remote Address": "5.161.128.79:15474"}
2023-08-26T00:39:09+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:507\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-08-26T00:39:09+02:00 ERROR piecestore upload failed {"process": "storagenode", "Piece ID": "NTSGMACUNFD6TZU2PJ44I5US6RD3P5ZEZG3YTIWMH6DT6FEBBVNA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 36864, "Remote Address": "5.161.128.79:21372"}
2023-08-26T00:39:09+02:00 INFO piecestore upload started {"process": "storagenode", "Piece ID": "TJNVG6NAZT2BQOIOR6VCLVWIWCDYDBGC75KHMAK63KHDM6JQJY2A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 8846089642106, "Remote Address": "5.161.128.79:21372"}
2023-08-26T00:39:09+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:507\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-08-26T00:39:09+02:00 ERROR piecestore upload failed {"process": "storagenode", "Piece ID": "QWGRWANFH5MQGINRFI7UQMEINYT2BG7ZSLE74WKYD5YCGI2W5NCA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 18688, "Remote Address": "5.161.149.40:42994"}
2023-08-26T00:39:09+02:00 INFO piecestore upload started {"process": "storagenode", "Piece ID": "DLYENP47NPXPG66RQUOT4ZGYUUHCU26P6WN4DM7GFB43YEGZDLKA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 8846089605242, "Remote Address": "5.161.149.40:42994"}
2023-08-26T00:39:12+02:00 INFO piecestore upload started {"process": "storagenode", "Piece ID": "ZTQT5J6WVKRJP6K3W3EDGYVFXVIEVD77TEIS5EV7JAI3RJKJCCDQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT_REPAIR", "Available Space": 8846089586042, "Remote Address": "5.161.50.62:56242"}
2023-08-26T00:39:13+02:00 INFO piecestore download started {"process": "storagenode", "Piece ID": "6QY4L6WLZL2XDW757UWED675IETONFHAY4E55W5HLBAQA7UWFPQA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 5632, "Remote Address": "5.161.197.155:54138"}
2023-08-26T00:39:19+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7.1:737\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:778\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:792\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-08-26T00:39:19+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:532\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-08-26T00:39:19+02:00 INFO piecestore uploaded {"process": "storagenode", "Piece ID": "RIA3I5QZBMCOEJVUZZ5E4NKEV5KGTVGAXJ23MWXZOMNXCI32L3IQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT_REPAIR", "Size": 580096, "Remote Address": "59.127.219.133:50531"}
2023-08-26T00:39:19+02:00 INFO piecestore download canceled {"process": "storagenode", "Piece ID": "DODCU3UTDQBR635OXK5HB7NJUMZWAOZ73CDTKRZ5ISV6ZJGLARLA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 2170624, "Size": 0, "Remote Address": "184.104.224.99:52262"}
2023-08-26T00:39:19+02:00 INFO piecestore download started {"process": "storagenode", "Piece ID": "54TAYSCNFQ6F7QJAPFHFFWZ6ZQGUKSVVMQLJCI3LALBXBNXRTZZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 1726976, "Size": 2304, "Remote Address": "184.104.224.99:52262"}
2023-08-26T00:39:19+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:507\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-08-26T00:39:19+02:00 ERROR piecestore upload failed {"process": "storagenode", "Piece ID": "DSLW6IO5OYME7LZUUGOZSH7JY74NW4AUQALKICH6WP3MVFN3R2UQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 13056, "Remote Address": "5.161.143.41:27976"}
2023-08-26T00:39:19+02:00 INFO piecestore upload started {"process": "storagenode", "Piece ID": "5CWTRP33XAHL5L2RZBMNWP4WVM6MB4MCM6LJTVRH4N6TD7CHJL3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 8846089548666, "Remote Address": "5.161.143.41:27976"}
2023-08-26T00:39:19+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7.1:737\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:775\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:792\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-08-26T00:39:19+02:00 INFO piecestore downloaded {"process": "storagenode", "Piece ID": "E6U47UAAOEUJ4SJDG767RLMBYNDLIFWALQHYV3PPCF6VF33EDMPQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "Offset": 0, "Size": 14592, "Remote Address": "5.161.217.169:57444"}
2023-08-26T00:39:29+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:507\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-08-26T00:39:29+02:00 ERROR piecestore upload failed {"process": "storagenode", "Piece ID": "4EM3UY26NCYWARTIGHOBLXSRVP5PITSTLJZHBGPJ5R62B3MB7J2A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 768, "Remote Address": "50.7.22.66:22918"}
2023-08-26T00:39:29+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:507\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}
2023-08-26T00:39:29+02:00 ERROR piecestore upload failed {"process": "storagenode", "Piece ID": "VVSHKFWADFGLEGVB4CK72DYJSFO3ZYQ5QPIUPIJ4B5KVNHBM7RWQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func6:500\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 32000, "Remote Address": "216.66.40.82:58984"}
2023-08-26T00:39:29+02:00 INFO piecestore upload started {"process": "storagenode", "Piece ID": "T2VPDEKLXZ36EE5EML4PR6MVTAJWB3YYDVDSSBZSKO2TDKGUVGFQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 8846089533050, "Remote Address": "216.66.40.82:58984"}
2023-08-26T00:39:29+02:00 ERROR piecestore failed to add bandwidth usage {"process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:868\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:532\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35"}