Node Thrashing Disk all the time

[Stubbsey]

Hi all,
I’ve got a node that has run fine for many months (in excess of a year maybe close to 2). in recent months the disk usage has gone from normal to 95% all the time. Node seems to run ok in all ways except dashboard can take a long while to load and there only seems to be one reoccurring error in logs.

2023-07-01T10:23:44.222+0100 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database is locked”, “errorVerbose”: “bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:855\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:769\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:783\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}

I believe drive is CMR WD Purple but disk speed is 5400 rpms.

Any clues? I’m relatively non technical and i’ve search forum but alot of what i’m reading is above me. Just to add its a windows node running on win 10.

Thanks

[Alexey]

Your disk subsystem cannot keep up. You have options to move your databases to SSD or add another node in the same /24 subnet of the public network.
Or restart it from time to time.
Make sure that the defragmentation job is enabled for this disk, since you use Windows.

[Stubbsey]

I’ve recently just added a node and that has made no difference. i do have spare ssd’s. is the database easy enough to move?

[Alexey]

I think so, see

Move databases on Windows storagenode

Hi guys. So, I troubleshooted a lot on one of my nodes, and disk activity is always at 100%. The HDD just can´t keep up with the demand (:P) it´s a WD Blue but it gets almost occupied writing in databases. Can I move the databases to my quicker system drive? If so, how?

[Stubbsey]

So within the storj folder move all .db files to a folder on ssd. change config path for database and restart.

[daki82]

Stop the node first.
Use ultradefrag like mentioned somewhere to defragment while in use.

[Stubbsey]

Yes done that. I’ve done it all and restarted node and all seems well. no errors up to yet. D Drive is still at 100% but am i right in saying that’s because I’ve just restarted node?

[daki82]

How full is the node? I try figuring out when i defrag mine the second time at best.

[Stubbsey]

its an 8tb disk. Was intially set up with a 7.7tb node that was 5.7tb full. when i started having these issues i set disk size to 5.7tb (so now effectivly full according to config setup) and started a new node on a 20tb disc. So stats are as follows for this drive 5.7 TB disc used 5.58TB free 0B trash 139.16GB overused 19.64GB.
My entire setup consists of 8.8TB full on one windows 10 pc and on second pc windows 10, 1st node is the 5.7TB node and the second node is 1TB node (on the 20TB disk) @ 588GB full. All behind 1 ip.

[daki82]

This is verry similar to what i do in 2 weeks!
Also the data amount is verry similar.
I would conclude considering defragmentation every 4-5TB.

[daki82]

@Stubbsey

100% but am i right in saying that’s because I’ve just restarted node?

Yes. exept you wrote the command in yaml file not to do so.

[Stubbsey]

I defragged the drive over night and the issue is still the same this morning. i’ve stopped the storage node service and the drive returns to 0. Any other ideas. The logs seem to show the node operating normally- these are just before i stopped node.

2023-07-02T10:24:23.776+0100 INFO piecestore downloaded {“Piece ID”: “XSKBHDJM2NREQXEFMWBZFWCGFKT53TUVALI5IO4B5DY3QM7INWJA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 11264, “Remote Address”: “167.235.19.43:36270”}
2023-07-02T10:24:25.325+0100 INFO piecestore download started {“Piece ID”: “DMFUOALTS5K2PWSNSDQZC2SMM7UM77KVAP73O3GA4KARSIZBAOAQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 1792, “Remote Address”: “5.161.149.40:23178”}
2023-07-02T10:24:25.440+0100 INFO piecestore download canceled {“Piece ID”: “DMFUOALTS5K2PWSNSDQZC2SMM7UM77KVAP73O3GA4KARSIZBAOAQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 0, “Remote Address”: “5.161.149.40:23178”}
2023-07-02T10:24:34.736+0100 INFO piecestore download started {“Piece ID”: “ODR2QJZTAKDDNLUFPYHALRZHB5E2CBH3TQL4XBL3M3WYXYY5VSYQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “Offset”: 723200, “Size”: 2560, “Remote Address”: “184.104.224.99:48888”}
2023-07-02T10:24:34.956+0100 INFO piecestore downloaded {“Piece ID”: “ODR2QJZTAKDDNLUFPYHALRZHB5E2CBH3TQL4XBL3M3WYXYY5VSYQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “Offset”: 723200, “Size”: 2560, “Remote Address”: “184.104.224.99:48888”}
2023-07-02T10:24:45.278+0100 INFO piecestore download started {“Piece ID”: “T4LKZVUEVUQC6RFEM75R5S57ONFVK7KWWATG6TKGQQ7PEMXDBCVQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 1973248, “Size”: 1280, “Remote Address”: “216.209.198.112:61143”}
2023-07-02T10:24:45.570+0100 INFO piecestore downloaded {“Piece ID”: “T4LKZVUEVUQC6RFEM75R5S57ONFVK7KWWATG6TKGQQ7PEMXDBCVQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 1973248, “Size”: 1280, “Remote Address”: “216.209.198.112:61143”}
2023-07-02T10:24:47.006+0100 INFO piecestore download started {“Piece ID”: “GF7PC7SGPYWTQEB77GG4X5OF5ZH5EAGNM3MJ3RDFD3KJYLBUV7UQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 1075200, “Size”: 2560, “Remote Address”: “216.66.40.82:44816”}
2023-07-02T10:24:47.240+0100 INFO piecestore downloaded {“Piece ID”: “GF7PC7SGPYWTQEB77GG4X5OF5ZH5EAGNM3MJ3RDFD3KJYLBUV7UQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 1075200, “Size”: 2560, “Remote Address”: “216.66.40.82:44816”}
2023-07-02T10:24:47.348+0100 INFO piecestore download started {“Piece ID”: “RAVJGPKQIO53CVO65VBC64DEYYCTYKMK7GBJ6CWUZ4YGQUJ2IYLA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 1583104, “Size”: 2560, “Remote Address”: “216.66.40.82:44816”}
2023-07-02T10:24:47.585+0100 INFO piecestore downloaded {“Piece ID”: “RAVJGPKQIO53CVO65VBC64DEYYCTYKMK7GBJ6CWUZ4YGQUJ2IYLA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 1583104, “Size”: 2560, “Remote Address”: “216.66.40.82:44816”}
2023-07-02T10:24:47.653+0100 INFO piecestore download started {“Piece ID”: “RAVJGPKQIO53CVO65VBC64DEYYCTYKMK7GBJ6CWUZ4YGQUJ2IYLA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 1585408, “Size”: 2560, “Remote Address”: “216.66.40.82:44816”}
2023-07-02T10:24:47.864+0100 INFO piecestore downloaded {“Piece ID”: “RAVJGPKQIO53CVO65VBC64DEYYCTYKMK7GBJ6CWUZ4YGQUJ2IYLA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 1585408, “Size”: 2560, “Remote Address”: “216.66.40.82:44816”}
2023-07-02T10:24:49.646+0100 INFO piecestore download started {“Piece ID”: “ZXJSO35UOCHIVGBFM3YWF7RJT4T6YU7765NDA2VQKQUCYSWKFBCA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”, “Offset”: 2202112, “Size”: 256, “Remote Address”: “34.148.62.121:34430”}
2023-07-02T10:24:49.947+0100 INFO piecestore downloaded {“Piece ID”: “ZXJSO35UOCHIVGBFM3YWF7RJT4T6YU7765NDA2VQKQUCYSWKFBCA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”, “Offset”: 2202112, “Size”: 256, “Remote Address”: “34.148.62.121:34430”}
2023-07-02T10:24:53.738+0100 INFO piecestore download started {“Piece ID”: “3B6OCNSWX3WSYQXKWFUJRLEB64H73TDIVZQFPONFV75I5VU45WDA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 75008, “Remote Address”: “5.161.50.62:52680”}
2023-07-02T10:24:54.234+0100 INFO piecestore downloaded {“Piece ID”: “3B6OCNSWX3WSYQXKWFUJRLEB64H73TDIVZQFPONFV75I5VU45WDA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 75008, “Remote Address”: “5.161.50.62:52680”}
2023-07-02T10:24:56.505+0100 INFO piecestore download started {“Piece ID”: “IPZ3AGPXCP2EB4C5S6UWJWZSME6PLIAM7R6JTNG5L3Y77HP3FUYQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “Offset”: 461568, “Size”: 4864, “Remote Address”: “50.7.22.66:57116”}
2023-07-02T10:24:56.695+0100 INFO piecestore downloaded {“Piece ID”: “IPZ3AGPXCP2EB4C5S6UWJWZSME6PLIAM7R6JTNG5L3Y77HP3FUYQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “Offset”: 461568, “Size”: 4864, “Remote Address”: “50.7.22.66:57116”}
2023-07-02T10:25:01.718+0100 INFO piecestore download started {“Piece ID”: “CHZOCU7LPVW7TYNZQAD46GHR23ZFMMMXWNVD2CQPBL6MEG7QAJRQ”, “Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “Action”: “GET_AUDIT”, “Offset”: 142848, “Size”: 256, “Remote Address”: “35.231.177.59:44126”}
2023-07-02T10:25:02.018+0100 INFO piecestore downloaded {“Piece ID”: “CHZOCU7LPVW7TYNZQAD46GHR23ZFMMMXWNVD2CQPBL6MEG7QAJRQ”, “Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “Action”: “GET_AUDIT”, “Offset”: 142848, “Size”: 256, “Remote Address”: “35.231.177.59:44126”}
2023-07-02T10:25:02.048+0100 INFO piecestore download started {“Piece ID”: “ER5NJRSNC32MWJLUMDPZ4J22IJGYM6RCJU3RBNRZH7VOMZ2N4MUA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “Offset”: 1297408, “Size”: 4864, “Remote Address”: “50.7.22.66:57116”}
2023-07-02T10:25:02.219+0100 INFO piecestore downloaded {“Piece ID”: “ER5NJRSNC32MWJLUMDPZ4J22IJGYM6RCJU3RBNRZH7VOMZ2N4MUA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “Offset”: 1297408, “Size”: 4864, “Remote Address”: “50.7.22.66:57116”}
2023-07-02T10:25:04.422+0100 INFO piecestore download started {“Piece ID”: “DN7R6KZS4K5NPLUPSTQC6HOJF46VWKOW2KQN3E42HPVALU5OU3XQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET_AUDIT”, “Offset”: 933376, “Size”: 256, “Remote Address”: “35.235.103.20:54246”}
2023-07-02T10:25:04.765+0100 INFO piecestore downloaded {“Piece ID”: “DN7R6KZS4K5NPLUPSTQC6HOJF46VWKOW2KQN3E42HPVALU5OU3XQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET_AUDIT”, “Offset”: 933376, “Size”: 256, “Remote Address”: “35.235.103.20:54246”}
2023-07-02T10:25:04.925+0100 INFO piecestore download started {“Piece ID”: “UWL2XLBZXPOZE7FAJ3YPIT7JZQGVR5B5VDR4TI476MLP6SCC4QJQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “Offset”: 1376512, “Size”: 2560, “Remote Address”: “184.104.224.99:48888”}
2023-07-02T10:25:05.170+0100 INFO piecestore downloaded {“Piece ID”: “UWL2XLBZXPOZE7FAJ3YPIT7JZQGVR5B5VDR4TI476MLP6SCC4QJQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “Offset”: 1376512, “Size”: 2560, “Remote Address”: “184.104.224.99:48888”}
2023-07-02T10:25:08.840+0100 INFO piecestore download started {“Piece ID”: “UVSR6KIXJ3MBT4C4YTNF3PATFNHBDXCRSO4HDPGYVB3DQOAIBECQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 16384, “Remote Address”: “184.104.224.99:53694”}
2023-07-02T10:25:09.200+0100 INFO piecestore downloaded {“Piece ID”: “UVSR6KIXJ3MBT4C4YTNF3PATFNHBDXCRSO4HDPGYVB3DQOAIBECQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 16384, “Remote Address”: “184.104.224.99:53694”}
2023-07-02T10:25:12.273+0100 INFO piecestore download started {“Piece ID”: “KRFGZ5Z3KDS77HIMJXZM44OQAZMA6E3XFWPGZLN7H6UHIGIO6ODA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 1817344, “Size”: 2560, “Remote Address”: “184.104.224.98:20328”}
2023-07-02T10:25:12.381+0100 INFO piecestore download started {“Piece ID”: “Q4RCD2IXARB4BSQCH3XTLH5XLYMVKPMXMCP7UKIXKJKHXNLBG4OA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 184064, “Size”: 2560, “Remote Address”: “216.66.40.82:9212”}
2023-07-02T10:25:12.539+0100 INFO piecestore download canceled {“Piece ID”: “KRFGZ5Z3KDS77HIMJXZM44OQAZMA6E3XFWPGZLN7H6UHIGIO6ODA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 1817344, “Size”: 0, “Remote Address”: “184.104.224.98:20328”}
2023-07-02T10:25:12.551+0100 INFO piecestore download started {“Piece ID”: “4HN4ARYWFDPS5PUCC7BU335YEKLNKQIQQJVHAYGQKBZ2M5DFUUDQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “Offset”: 1792768, “Size”: 2560, “Remote Address”: “184.104.224.98:50776”}
2023-07-02T10:25:12.704+0100 INFO piecestore downloaded {“Piece ID”: “Q4RCD2IXARB4BSQCH3XTLH5XLYMVKPMXMCP7UKIXKJKHXNLBG4OA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 184064, “Size”: 2560, “Remote Address”: “216.66.40.82:9212”}
2023-07-02T10:25:12.791+0100 INFO piecestore downloaded {“Piece ID”: “4HN4ARYWFDPS5PUCC7BU335YEKLNKQIQQJVHAYGQKBZ2M5DFUUDQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “Offset”: 1792768, “Size”: 2560, “Remote Address”: “184.104.224.98:50776”}
2023-07-02T10:25:12.824+0100 INFO piecestore download started {“Piece ID”: “Q4RCD2IXARB4BSQCH3XTLH5XLYMVKPMXMCP7UKIXKJKHXNLBG4OA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 186368, “Size”: 2560, “Remote Address”: “216.66.40.82:9212”}
2023-07-02T10:25:13.101+0100 INFO piecestore downloaded {“Piece ID”: “Q4RCD2IXARB4BSQCH3XTLH5XLYMVKPMXMCP7UKIXKJKHXNLBG4OA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 186368, “Size”: 2560, “Remote Address”: “216.66.40.82:9212”}
2023-07-02T10:25:15.888+0100 INFO piecestore download started {“Piece ID”: “BKVZ26AZDVC6JB3ASUXW6JIJAMCXIB7LBAAQ73YJQHCLTSC44PVA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 37376, “Size”: 1280, “Remote Address”: “216.209.198.112:63732”}
2023-07-02T10:25:16.217+0100 INFO piecestore downloaded {“Piece ID”: “BKVZ26AZDVC6JB3ASUXW6JIJAMCXIB7LBAAQ73YJQHCLTSC44PVA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 37376, “Size”: 1280, “Remote Address”: “216.209.198.112:63732”}
2023-07-02T10:25:19.673+0100 INFO piecestore download started {“Piece ID”: “WRKPO3BENHAO3STL66YX244HCM6C43XKOCTHZTJECRJKCERRN2MA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 181504, “Remote Address”: “5.161.217.169:58184”}
2023-07-02T10:25:19.693+0100 INFO piecestore download started {“Piece ID”: “JZPCTPMYM33V53VAYS2CLVM3GVSKNNV5BXFCYEHVZJWOFYBYAX4Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 181504, “Remote Address”: “5.161.44.25:55690”}
2023-07-02T10:25:20.369+0100 INFO piecestore downloaded {“Piece ID”: “JZPCTPMYM33V53VAYS2CLVM3GVSKNNV5BXFCYEHVZJWOFYBYAX4Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 181504, “Remote Address”: “5.161.44.25:55690”}
2023-07-02T10:25:20.472+0100 INFO piecestore downloaded {“Piece ID”: “WRKPO3BENHAO3STL66YX244HCM6C43XKOCTHZTJECRJKCERRN2MA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 181504, “Remote Address”: “5.161.217.169:58184”}
2023-07-02T10:25:24.381+0100 INFO piecestore download started {“Piece ID”: “5L3NDC7KLLRWR6Q565QX3X2AXUBHW53YX7PKPTA5KVROYK6TTISA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 1536, “Remote Address”: “184.104.224.98:59858”}
2023-07-02T10:25:24.483+0100 INFO piecestore download started {“Piece ID”: “A5K7NM3WHWNRPHO6C6KSHVE2WHIFA3Z2DAOIXHDXOBFTYEOPXCQQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 1760000, “Size”: 2560, “Remote Address”: “38.88.241.42:29602”}
2023-07-02T10:25:24.534+0100 INFO piecestore downloaded {“Piece ID”: “5L3NDC7KLLRWR6Q565QX3X2AXUBHW53YX7PKPTA5KVROYK6TTISA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 1536, “Remote Address”: “184.104.224.98:59858”}
2023-07-02T10:25:24.709+0100 INFO piecestore downloaded {“Piece ID”: “A5K7NM3WHWNRPHO6C6KSHVE2WHIFA3Z2DAOIXHDXOBFTYEOPXCQQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 1760000, “Size”: 2560, “Remote Address”: “38.88.241.42:29602”}
2023-07-02T10:25:29.115+0100 INFO Stop/Shutdown request received.
2023-07-02T10:25:29.122+0100 INFO piecestore downloaded {“Piece ID”: “XFAYJVNOPTJ2EP64HVQXZH5TIISHWQZSNG3IHF5L7WKEMQBFOMRA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 77312, “Size”: 608256, “Remote Address”: “72.52.83.203:61490”}
2023-07-02T10:25:29.152+0100 ERROR piecestore:cache error getting current used space: {“error”: “filewalker: context canceled; filewalker: context canceled”, “errorVerbose”: “group:\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:718\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75\n— filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:69\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:74\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:718\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}

Any other config i can alter?

just restarted Node. disk at 100% then a few of these errors before logs settle into normal operation

2023-07-02T10:38:34.598+0100 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database is locked”, “errorVerbose”: “bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:855\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func7:769\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:783\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:251\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35”}

[Alexey]

@Stubbsey

bandwidthdb: database is locked

yes, the disk become very slow and unresponsive. Perhaps you need to move databases to the SSD if possible.
If databases finally become unlocked, you may leave it as is. If you see these errors every time, then it’s time to move databases to SSD.

[Stubbsey]

i’ve already done this as you suggested earlier

[Alexey]

If you moved databases to SSD you should not see this error anymore. Please make sure that you saved the config file after the change and restarted the storagenode service.
Databases should be moved while the storagenode stopped, you need to add/uncomment parameter storage2.database-dir: with the path to a new databases location, save the config and restart the service.

[Stubbsey]

here are some screen shots of set up.

Yaml config

screen shot of folder with database in on ssd C drive. Note times when last modified, very recent and have been set up since yesterday.

C drive ssd

D drive where node is hosted 100%

Up to date log image.

I believe i have everything in place. yet still 100% disk usage

[Toyoo]

I do not use Windows, but I’ve seen some recommendations here on the forum to disable indexing services and antivirus on the drive you keep Storj data on. Have you tried that?

[daki82]

@Toyoo

disable indexing services and antivirus on the drive you keep Storj data on

and the database folder/drive.
btw. bandwidth.db seems a bit exaggerated @Alexey 555 MB? mine is 33mb at 7.6TB node.
maybe vacuum? or delete and recreate? (i have read this somewhere here despite no knowledge about this)

[Alexey]

Yes, the vacuum could help sometimes, but it contains a bandwidth history, see

"Action": "GET_AUDIT", "error": "usedserialsdb error: database is locked",

I’m running a docker version on Debian GNU/Linux. Here’s the procedure I use to vacuum and check the databases: Stop the node Vacuum the dbs integrity_check the dbs Restart the node Here’s a bash check which should do that for you. Please change the database directory to reflect where yours are on your system. docker stop -t 300 storagenode && dbs=$(ls /opt/storj/storage/*.db) c1="VACUUM;" c2="PRAGMA integrity_check;" for i in $dbs do sqlite3 $i "$c1" done for i in $dbs do sqlite3 $i "…

And read also

Is the vacuum necessary?

I cannot help but wonder if vaccuming the database is a good or a bad thing… people on here has been talking about that a lot lately, most sort of just go… oh look at this thing… and then i do this and it’s smaller after i’m done… and on harddrives smaller is better… so must be right… vacuuming is better… i don’t really know a lot about database management, i’ve worked with a few here and there…pretty simple stuff at first… you and like rows and columns of information, that you basically can se…

Yes, you may re-create it using the instruction https://support.storj.io/hc/en-us/articles/4403032417044-How-to-fix-database-file-is-not-a-database-error, but the usage history will be lost.

[Stubbsey]

Ok thanks. I’ll take a look when i get home from work.