Suddenly suspended - no recent changes

[o1eal]

This morning I got this email:

Your Storage Node on the asia-east-1 Satellite was suspended because it produced errors too often during audits.

You were suspended on 2022-06-05 at 12:38 UTC.

You won’t receive any new data on this Satellite until you resolve the issue causing your node to be offline. See here for common solutions.

I cannot think of anything relevant that has changed recently, and I’ve gone through the steps outlined as best I can. My node has been online for nearly a year. Its 4TB drive is approaching “full” but I think not near enough that it would be a factor (I have assigned 3.1 TB to it, and it crossed the 3TB threshold in the last few days.) My dynamic IP is functioning fine, I am connected via SSH to the storj node, over the Internet, right now. No changes to my router. I probably performed an apt upgrade to the Linux OS on the storj node in the last few days, perhaps that was related, but I do that routinely and as far as I know this was not a major upgrade in any way that stood out.

Things I have done in the past for troubleshooting include stopping and deleting the docker container. That’s not a thing suggested on the page linked, so I don’t know whether that’s worth trying.

Suggestions?

[deathlessdd]

Your title says disqualified and the message you got says you were suspended which arent the same. Suspended just means something weird happened which could be many things which is recoverable. Disqualified isnt.

[o1eal]

Thank you, I had missed that nuance. I will edit the title.

[o1eal]

A couple further details.

I do monitor my suspension scores via the dashboard pretty frequently. It has been at 97% to 99% for each satellite in recent days. The only reason it was not 100% is that I knocked the node offline for a few hours, almost a month ago.

Also, I’ve seen that quoting the logs can be helpful (though I know nothing about interpreting this info), so…here:

2022-06-05T21:23:47.501Z	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "7NG7Z5N2Z3FZZO23KR2YVFJ2HNXUMV5XH4YQFRORIRT2V6RINP2Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-06-05T21:23:47.993Z	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "HP6AKLCFOK72K7WJEQHJDVJ2754PHRBDL2RO4ZZLOCG2WUSJD2QQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-06-05T21:23:48.846Z	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "WDIIWKPXOUHUYPFNIE6V4J4A3DSF6HN57NN73VZSG7MSUWI3DQKQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-06-05T21:23:49.054Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "5X55VSTKNB5BK4ITOJR6QBHEAJC5UMTUXLOVJGKVAZNTFXO7DERA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 70629411456}
2022-06-05T21:23:49.263Z	ERROR	piecestore	failed to add bandwidth usage	{"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:437\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-06-05T21:23:49.264Z	INFO	piecestore	uploaded	{"Process": "storagenode", "Piece ID": "5X55VSTKNB5BK4ITOJR6QBHEAJC5UMTUXLOVJGKVAZNTFXO7DERA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 37888}
2022-06-05T21:23:49.339Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "LT4XVY62EECU547PYTOCJMOJSTPLU3EO3X6QVV2EFPBNGRKKFUQQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 70629373056}
2022-06-05T21:23:49.580Z	ERROR	piecestore	failed to add bandwidth usage	{"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:662\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:686\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-06-05T21:23:49.580Z	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "WDIIWKPXOUHUYPFNIE6V4J4A3DSF6HN57NN73VZSG7MSUWI3DQKQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-06-05T21:23:50.015Z	ERROR	piecestore	failed to add bandwidth usage	{"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:437\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-06-05T21:23:50.015Z	INFO	piecestore	uploaded	{"Process": "storagenode", "Piece ID": "LT4XVY62EECU547PYTOCJMOJSTPLU3EO3X6QVV2EFPBNGRKKFUQQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 2319360}
2022-06-05T21:23:50.021Z	ERROR	piecestore	failed to add bandwidth usage	{"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:662\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:686\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-06-05T21:23:50.022Z	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "HP6AKLCFOK72K7WJEQHJDVJ2754PHRBDL2RO4ZZLOCG2WUSJD2QQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-06-05T21:23:50.210Z	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "WDIIWKPXOUHUYPFNIE6V4J4A3DSF6HN57NN73VZSG7MSUWI3DQKQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-06-05T21:23:51.123Z	ERROR	piecestore	failed to add bandwidth usage	{"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:662\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:686\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-06-05T21:23:51.124Z	INFO	piecestore	downloaded	{"Process": "storagenode", "Piece ID": "WDIIWKPXOUHUYPFNIE6V4J4A3DSF6HN57NN73VZSG7MSUWI3DQKQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2022-06-05T21:23:51.254Z	INFO	piecestore	upload started	{"Process": "storagenode", "Piece ID": "ZGCSTIDVFBVLKUDPGVQO4RUZUCGZ4NMYDFZVVRIY2OXTF3LHWDMA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 70627053184}
2022-06-05T21:23:51.335Z	ERROR	piecestore	failed to add bandwidth usage	{"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:407\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-06-05T21:23:51.336Z	ERROR	piecestore	upload failed	{"Process": "storagenode", "Piece ID": "ZGCSTIDVFBVLKUDPGVQO4RUZUCGZ4NMYDFZVVRIY2OXTF3LHWDMA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: mkdir config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/zg: bad message", "errorVerbose": "pieces error: filestore error: mkdir config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/zg: bad message\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:406\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 768}
2022-06-05T21:23:51.485Z	INFO	piecestore	download started	{"Process": "storagenode", "Piece ID": "HP6AKLCFOK72K7WJEQHJDVJ2754PHRBDL2RO4ZZLOCG2WUSJD2QQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}

[littleskunk]

@o1eal

I do monitor my suspension scores via the dashboard pretty frequently. It has been at 97% to 99% for each satellite in recent days. The only reason it was not 100% is that I knocked the node offline for a few hours, almost a month ago.

That is offline suspension. The email you received looks more like too many audit errors. Grep you log for any download failed and you should find them.

[o1eal]

$ docker logs --tail 50 storagenode|grep failed 2022-06-05T22:01:09.275Z ERROR piecestore failed to add bandwidth usage {"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:437\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:09.463Z ERROR piecestore failed to add bandwidth usage {"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:437\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:09.481Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "U4R3O275OZNU3DYDVMVGO27XPN47ITMB3VU3I37TEIBDTQ5NKAXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/u4/r3o275oznu3dydvmvgo27xpn47itmb3vu3i37teibdtq5nkaxq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/u4/r3o275oznu3dydvmvgo27xpn47itmb3vu3i37teibdtq5nkaxq.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/u4/r3o275oznu3dydvmvgo27xpn47itmb3vu3i37teibdtq5nkaxq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/u4/r3o275oznu3dydvmvgo27xpn47itmb3vu3i37teibdtq5nkaxq.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:542\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:09.614Z ERROR piecestore failed to add bandwidth usage {"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:437\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:09.983Z ERROR piecestore failed to add bandwidth usage {"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:437\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:10.012Z ERROR piecestore failed to add bandwidth usage {"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:437\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:10.033Z ERROR piecestore failed to add bandwidth usage {"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:407\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:10.034Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "H73EQBAGEOXJSTOKDASDTNDKMADGJGUF2SCMHQD2BS335IB4IF2A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: rename config/storage/temp/blob-3837355619.partial config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/h7/3eqbageoxjstokdasdtndkmadgjguf2scmhqd2bs335ib4if2a.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: rename config/storage/temp/blob-3837355619.partial config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/h7/3eqbageoxjstokdasdtndkmadgjguf2scmhqd2bs335ib4if2a.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:406\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 181504} 2022-06-05T22:01:10.144Z ERROR piecestore failed to add bandwidth usage {"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:407\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:10.144Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "KW3PVL2EFE6EWBKUYOJ5A4M6FR2Q5TI256VX77GXOCCABGGZF4ZQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: rename config/storage/temp/blob-3355677871.partial config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/kw/3pvl2efe6ewbkuyoj5a4m6fr2q5ti256vx77gxoccabggzf4zq.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: rename config/storage/temp/blob-3355677871.partial config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/kw/3pvl2efe6ewbkuyoj5a4m6fr2q5ti256vx77gxoccabggzf4zq.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:406\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 768} 2022-06-05T22:01:10.836Z ERROR piecestore failed to add bandwidth usage {"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:407\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:10.837Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "FK23NEXWPNAQC3NU5TMD5EC5QGHGQEFGFHK73ZZPOJXRKVOGJTBQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: mkdir config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/fk: bad message", "errorVerbose": "pieces error: filestore error: mkdir config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/fk: bad message\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:406\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 512} 2022-06-05T22:01:10.855Z ERROR piecestore failed to add bandwidth usage {"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:407\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:10.856Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "U3ZXEUHYNW57EGXZPBOIOTVGSI65APTLMTRZ5QUW6QJCLVZ7UL3Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: rename config/storage/temp/blob-772077324.partial config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/u3/zxeuhynw57egxzpboiotvgsi65aptlmtrz5quw6qjclvz7ul3q.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: rename config/storage/temp/blob-772077324.partial config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/u3/zxeuhynw57egxzpboiotvgsi65aptlmtrz5quw6qjclvz7ul3q.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:406\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 768} 2022-06-05T22:01:11.256Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "U4R3O275OZNU3DYDVMVGO27XPN47ITMB3VU3I37TEIBDTQ5NKAXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/u4/r3o275oznu3dydvmvgo27xpn47itmb3vu3i37teibdtq5nkaxq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/u4/r3o275oznu3dydvmvgo27xpn47itmb3vu3i37teibdtq5nkaxq.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/u4/r3o275oznu3dydvmvgo27xpn47itmb3vu3i37teibdtq5nkaxq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/u4/r3o275oznu3dydvmvgo27xpn47itmb3vu3i37teibdtq5nkaxq.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:542\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:11.296Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "RIJ3BMMOSJ2JFIM54HDHXTJUCLX4Y7UNCW4CPB2GDH5YFUVI5H2A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ri/j3bmmosj2jfim54hdhxtjuclx4y7uncw4cpb2gdh5yfuvi5h2a.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ri/j3bmmosj2jfim54hdhxtjuclx4y7uncw4cpb2gdh5yfuvi5h2a.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ri/j3bmmosj2jfim54hdhxtjuclx4y7uncw4cpb2gdh5yfuvi5h2a.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ri/j3bmmosj2jfim54hdhxtjuclx4y7uncw4cpb2gdh5yfuvi5h2a.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:542\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:12.179Z ERROR piecestore failed to add bandwidth usage {"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:407\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:12.180Z ERROR piecestore upload failed {"Process": "storagenode", "Piece ID": "EP2HKJ6A475S4LGQLAWNLT66KTBHIVPVHKJBHBIIEIAJRJFUKGIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: rename config/storage/temp/blob-3531196044.partial config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ep/2hkj6a475s4lgqlawnlt66ktbhivpvhkjbhbiieiajrjfukgiq.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: rename config/storage/temp/blob-3531196044.partial config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/ep/2hkj6a475s4lgqlawnlt66ktbhivpvhkjbhbiieiajrjfukgiq.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*blobWriter).Commit:120\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit.func1:130\n\tstorj.io/storj/storagenode/pieces.(*Writer).Commit:198\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:406\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 181504} 2022-06-05T22:01:13.497Z ERROR piecestore failed to add bandwidth usage {"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:437\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:15.136Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "AN6HQHRC6MXP5HEAJ5PYBG7DUYJF74NVSFY7QBHKNJJJ6GSQNWRQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/an/6hqhrc6mxp5heaj5pybg7duyjf74nvsfy7qbhknjjj6gsqnwrq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/an/6hqhrc6mxp5heaj5pybg7duyjf74nvsfy7qbhknjjj6gsqnwrq.sj1: structure needs cleaning", "errorVerbose": "pieces error: filestore error: unable to open \"config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/an/6hqhrc6mxp5heaj5pybg7duyjf74nvsfy7qbhknjjj6gsqnwrq.sj1\": open config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/an/6hqhrc6mxp5heaj5pybg7duyjf74nvsfy7qbhknjjj6gsqnwrq.sj1: structure needs cleaning\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:75\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:542\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:17.033Z ERROR piecestore failed to add bandwidth usage {"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:662\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:686\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"} 2022-06-05T22:01:17.375Z ERROR piecestore failed to add bandwidth usage {"Process": "storagenode", "error": "bandwidthdb: database disk image is malformed", "errorVerbose": "bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:723\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:665\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:686\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}

[o1eal]

I’ve found this, and will see if I can follow these instructions:

How to fix a "database disk image is malformed"

[littleskunk]

That error message will just block you from seeing bandwidth usage on the storage node dashboard. That error alone doesn’t get you suspended.

I see 5 download failed in your log. That should be your main focus.

Are you using an SMR drive? I would expect it to get unresponsive close to full.

[o1eal]

OK, thank you. I am not sure how to ‘focus’ on them,this level of troubleshooting is kind of over my head, but I’m happy to search on some relevant terms, read in the forums and docs, etc.

It is not a SMR drive, and it’s not as close to full as I had thought. The dashboard reports 2.99 TB used and only 3.1 of the total 4 TB is allocated to Storj node.

Odd issue, though. Just now I issued it a reboot command, and I haven’t been able to reconnect via SSH since then. Hmm.

[deathlessdd]

What kinda system are you running it on?

[o1eal]

Post pictures of your storagenode rig(s)

[My Storj rig] To celebrate my first month I put my Storj rig in a closet in the basement where it’s cooler and out of the way. I’m using a 2007 Mac Mini, running Lubuntu 20.04 LTS. It’s well suited to the task: 1.83 GHz processor, 4 GB RAM (maxed out) and USB 2.0 limit its utility for other purposes. It still has an 80 GB internal HDD, which is a pain to swap out. The only downside: the processor draws more power (24w) than, say, a Raspberry Pi. I keep the lid on the hard drive enclosure op…

[o1eal]

I believe I’ve upgraded to Lubuntu 21.04, but that was not super recent so it’s not directly responsible.

I have considered migrating this to a different Mac Mini with Ubuntu-Server installed, since obviously it doesn’t need a desktop environment, etc. Maybe now is the time to simply move it to a different system, if that would bypass the problem instead of solving it?

[deathlessdd]

Dont really know how stable a mac is running a node not alot of people using macs

[o1eal]

Mac is just the hardware, it’s a pretty straightforward Ubuntu-based OS. No Mac software involved. But, if I can find a guide I will move the whole setup to a different Mini this evening and see if that does the trick. It’s something I’ve been meaning to do for a while and I have the hardware.

[deathlessdd]

Thats the thing though how many people do you know running ubuntu on a mac that is from 2007? Its not like its hardware that is widely used in systems. I have a simlar aged system but its a dell and I its been running for over 3 years now with little to no issues other then my SMR drives.

But its not of the mobile chips

[o1eal]

I’m not sure if this part is worth pursuing? It’s a core2duo chip, similar to yours. Yes, some kind of hardware failure is possible (regardless of whether it’s Apple, Dell, etc.) A software issue relating to an update is possible too. I’m not convinced that the focus on the fact that the hardware is Apple will be useful in diagnosing that, though?

[Alexey]

@o1eal

structure needs cleaning

You need to stop and remove the storagenode container, check and fix errors on your drive, fix databases (at least bandwidth.db) and run the container back with all your parameters.

[o1eal]

Thanks Alexey! I have already stopped and removed the storj container. Not sure which drive (the OS drive or the one with Storj data) likely needs fixing, or what kind of fixing it would need. Any hints? (Do you think this has to do with the “How to fix a “database disk image is malformed”” issue I found?

[Alexey]

You need to fix the disk with storagenode data, your OS complaining about it.
To fix databases you should use “How to fix a “database disk image is malformed””.

[o1eal]

Update: I have been running fsck on the drive for about 12 hours, it continues to find problems. I will leave it running for a while. Maybe the storage drive has unrecoverable hardware problems, which would be a shame; however, I think I just barely managed to cover the cost of the drive after running it for about a year, so it wouldn’t be the end of the world if I have to start over B-)

Then again, I’m seeing forum posts elsewhere, in which people had fsck run for up to 45 days and were ultimately successful. I imagine my node would be permanently disqualified before that much time passes, but…maybe it’s worth letting it go for up to another week or so, to see what happens.