Suspected disk corruption

[zeyacry]

Hi, I am running two nodes on one raspberry pi. One of the nodes suddenly stopped functioning in the middle of the night.

I tried to power cycle and it stuck at the bootup. I thought it was /etc/fstab issue, but it is not.

After checking the disk mounting, I tried to restart the docker service a few times. But the node dashboard is unable to display.

I guess it is some disk corruption. Am I correct? How do I resolve this issue?

[Pac]

Usually, issues related to databases can be solved by following posts like these ones:

More help about database disk image is malformed

You could try to fix it by following these instructions: If it doesn’t work, a node can still be recovered even if some databases were to be lost: here is how they can be recreated: In last resort if everything else fails, deleting all database files should work, but you would lose all statistics for current month, and possibly some of your earnings for current month also, but at least the node would be “saved”.

Preflight check error input/output

You can delete this DB, but node would not start. You need to do some extra steps: Delete corrupted DB Move all remained DBs to the other place Start the node, it should re-create databases Stop the node Move databases from backup and replace all databases Start the node.

However, what seems odd is that in your case it doesn’t find the database at all…

Maybe it’s time to run a full check on your disk. I faced some issues with a disk once and had to run some checks myself, maybe you’ll find some useful tips there:

Node crashed and restarted after some "error getting current used space", "lstat […] bad message" and "Unrecoverable error"

Hi there I noticed one of my node was suspiciously very active, and found out that was because it crashed at some point and restarted: the disk activity I’m seing is simply the filewalker rebrowsing the whole disk. Here are the few log lines around when it crashed and abruptluy restarted: [...] 2021-03-30T17:15:21.127Z INFO piecestore download started {"Piece ID": "BSBUCI6M2C3AKHRSJ3JHM4PFVND55RNJA7XUR4QEEVUZPYVS62DA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqk…

[Alexey]

Please:

1. Stop the storagenode
2. As @Pac said please, first run a fsck on this disk.
3. Move all *.db files to the backup folder
4. Start the storagenode
   it should create all databases, include missed ones
5. Stop the storagenode
6. Move backed up databases back
7. Start the storagenode
8. Check your logs

[zeyacry]

Thanks for the guidance.

It’s a lot of errors. I’ll let it run.

[zeyacry]

My node recovered after performing fsck. Thanks @Pac & @Alexey !

[zeyacry]

My node (8TB) stopped running for no reason again. This raspberry pi is running two nodes (8TB & 2TB). The other one is running normally. Hence, I omitted the possibility of a power outage or raspberry sd card issue.

This 8TB node is a 17 months old node and it is not fully filled.

fsck shows that there is no error this time.

The logs error messages are similar to my first post.

Unlike my initial situation, I am able to resume my node after verifying with fsck.

What else should I be checking?

[Alexey]

Please, copy the error message for that node.

[zeyacry]

Oops. I think I am unable to retrieve the error logs as I removed and restarted the container.

I’ll remember to keep the logs when it occurs again.

[Pac]

To prevent this from happening again in the future, you might want to redirect logs to a file:

How do I redirect my logs to a file?

[zeyacry]

Thanks. Configured it!

[zeyacry]

Uptimerobot detected the node is down at 2218hr.

Checked that *.db files in storage are ok.

There are five “download failed” error msg from 2210hr onwards.

2021-05-15T22:10:37.278Z	ERROR	piecestore	download failed	{Piece ID: GITFVDRXC4K5R6YFYZPX3J7NXWHY2OTUVB6XF77YMGGZKRGRAMKQ, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: GET_REPAIR, error: pieces error: filestore error: unable to open \config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/gi/tfvdrxc4k5r6yfyzpx3j7nxwhy2otuvb6xf77ymggzkrgramkq.sj1: open config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/gi/tfvdrxc4k5r6yfyzpx3j7nxwhy2otuvb6xf77ymggzkrgramkq.sj1: input/output error, errorVerbose: pieces error: filestore error: unable to open \config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/gi/tfvdrxc4k5r6yfyzpx3j7nxwhy2otuvb6xf77ymggzkrgramkq.sj1: open config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/gi/tfvdrxc4k5r6yfyzpx3j7nxwhy2otuvb6xf77ymggzkrgramkq.sj1: input/output error\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:71\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:504\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:217\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51}
2021-05-15T22:10:37.546Z	INFO	piecestore	download started	{Piece ID: HVMNTVRPXUW5VV63H2JBGBBTPC363Z7ZI6QXXT5SPXFVMZRTVYFA, Satellite ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6, Action: GET_REPAIR}
2021-05-15T22:10:37.978Z	ERROR	piecestore	download failed	{Piece ID: HVMNTVRPXUW5VV63H2JBGBBTPC363Z7ZI6QXXT5SPXFVMZRTVYFA, Satellite ID: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6, Action: GET_REPAIR, error: pieces error: filestore error: unable to open \config/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/hv/mntvrpxuw5vv63h2jbgbbtpc363z7zi6qxxt5spxfvmzrtvyfa.sj1: open config/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/hv/mntvrpxuw5vv63h2jbgbbtpc363z7zi6qxxt5spxfvmzrtvyfa.sj1: bad message, errorVerbose: pieces error: filestore error: unable to open \config/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/hv/mntvrpxuw5vv63h2jbgbbtpc363z7zi6qxxt5spxfvmzrtvyfa.sj1: open config/storage/blobs/qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa/hv/mntvrpxuw5vv63h2jbgbbtpc363z7zi6qxxt5spxfvmzrtvyfa.sj1: bad message\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:71\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:504\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:217\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51}
2021-05-15T22:10:38.010Z	INFO	piecestore	uploaded	{Piece ID: HHTLCQBDB7PXWITMD6UGSHTRN2MJWVBCDCRSS5Q2UPAEHLIFUSTQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 1792}
2021-05-15T22:10:38.717Z	INFO	piecestore	upload started	{Piece ID: P45PMVLUD6QESJEJUDGKKMCFZFPIEJ2BGGHRB3IFP2HBXXZFNQLA, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Available Space: 947189619456}
2021-05-15T22:10:38.862Z	INFO	piecestore	uploaded	{Piece ID: P45PMVLUD6QESJEJUDGKKMCFZFPIEJ2BGGHRB3IFP2HBXXZFNQLA, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Size: 12544}
2021-05-15T22:10:48.847Z	INFO	piecestore	upload started	{Piece ID: 7PE5DPCAJSHU6ENZ65LFCYDXT7Q5OFV7M6DFKFAWSIFKK32DTWFQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 947189606400}
2021-05-15T22:10:48.989Z	INFO	piecestore	uploaded	{Piece ID: 7PE5DPCAJSHU6ENZ65LFCYDXT7Q5OFV7M6DFKFAWSIFKK32DTWFQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 2816}
2021-05-15T22:10:54.504Z	INFO	piecestore	upload started	{Piece ID: ZISGVAWJFN6EWAUWCUUYCZB27ZKBF24ZFYZS77REZMMS7YGBNUCQ, Satellite ID: 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo, Action: PUT, Available Space: 947189603072}
2021-05-15T22:10:54.664Z	INFO	piecestore	uploaded	{Piece ID: ZISGVAWJFN6EWAUWCUUYCZB27ZKBF24ZFYZS77REZMMS7YGBNUCQ, Satellite ID: 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo, Action: PUT, Size: 3584}
2021-05-15T22:10:59.586Z	INFO	piecestore	upload started	{Piece ID: LKEZ3ET6RNAU56GRYIDIUG5RN52T4KVYXEA2SHZTTMMNNWKBUHWQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 947189598976}
2021-05-15T22:11:01.286Z	INFO	piecestore	upload started	{Piece ID: WFITWSPHIYTKVLTQT5DLJ5ISE4CKRMQ5YVBS6SL2UDL73SGHRPOQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Available Space: 947189597952}
2021-05-15T22:11:01.444Z	INFO	piecestore	uploaded	{Piece ID: WFITWSPHIYTKVLTQT5DLJ5ISE4CKRMQ5YVBS6SL2UDL73SGHRPOQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Size: 1536}
2021-05-15T22:11:02.259Z	INFO	piecestore	upload started	{Piece ID: OX3W4HHY5LUNQC64AV4UHK5SWBNSZDZN4H2EQ62NYHGHYQSGG5QQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Available Space: 947189595904}
2021-05-15T22:11:02.349Z	INFO	piecestore	uploaded	{Piece ID: OX3W4HHY5LUNQC64AV4UHK5SWBNSZDZN4H2EQ62NYHGHYQSGG5QQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Size: 1024}
2021-05-15T22:11:03.804Z	INFO	piecestore	upload started	{Piece ID: KZR6JZ3BFYUX47Z2N5DHQ5SV3IOIW4UQPHCEW4CNZOLYDSECGSYQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 947189594368}
2021-05-15T22:11:03.903Z	INFO	piecestore	uploaded	{Piece ID: KZR6JZ3BFYUX47Z2N5DHQ5SV3IOIW4UQPHCEW4CNZOLYDSECGSYQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 1792}
2021-05-15T22:11:04.436Z	INFO	piecestore	upload started	{Piece ID: UQMVZZST3QRI3Y2S2TKPC5D436PINGZ3ZAC5RU3YWWUOCUMSUOCQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Available Space: 947189592064}
2021-05-15T22:11:05.752Z	INFO	piecestore	upload started	{Piece ID: ROYUGPTUUH6E22TZHJXNZOX7WNUZQZM7QGDGAYYQ73FJCYNGA7TQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Available Space: 947189592064}
2021-05-15T22:11:06.005Z	INFO	piecestore	uploaded	{Piece ID: ROYUGPTUUH6E22TZHJXNZOX7WNUZQZM7QGDGAYYQ73FJCYNGA7TQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Size: 14080}
2021-05-15T22:11:06.169Z	INFO	piecestore	upload started	{Piece ID: E6MEQJOOFTA326EY4DFHLEZH6VGRCBRIA2MGVGDRLM3TVA6DTCOQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 947189577472}
2021-05-15T22:11:06.298Z	INFO	piecestore	uploaded	{Piece ID: E6MEQJOOFTA326EY4DFHLEZH6VGRCBRIA2MGVGDRLM3TVA6DTCOQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 768}
2021-05-15T22:11:07.919Z	INFO	piecestore	upload started	{Piece ID: 3T4JRKVMVY36V5JKUUBZ5VKFNIL633OM2MKFDVDK2CFP6N6SZ4VQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Available Space: 947189576192}
2021-05-15T22:11:08.264Z	INFO	piecestore	download started	{Piece ID: IZDIEV2YZQ7VDAVUG6RIXABQROO3JWE4J2QKZNPKVXU77OS7UAFA, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: GET}
2021-05-15T22:11:08.590Z	INFO	piecestore	upload started	{Piece ID: MH6IM3EZ2UNJHFXM2HNL7PZENLCOP54STCHHSH4XMVW7RYZYRQSQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 947189576192}
2021-05-15T22:11:09.965Z	INFO	piecestore	upload started	{Piece ID: B7V4V5XTVGA2SNV36KOTL22RK3NWK7TLRRF27UZE37SWZOP3RKDA, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 947189573888}
2021-05-15T22:11:13.395Z	INFO	piecestore	download started	{Piece ID: QXJKN63X3V4ER3X7I6NHSZ66PRIKMURSSYG7H4Y2WTOFAYCGORPA, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: GET}
2021-05-15T22:11:16.815Z	INFO	piecestore	upload started	{Piece ID: ZJ6IE33U4QDCSBPPGCGUINUSOAPOPORPFRPFSNF4AXUI4GFKZJHQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Available Space: 947189571840}
2021-05-15T22:11:17.481Z	INFO	piecestore	upload started	{Piece ID: 5Q4JET7UXR3A3UDXG3ENJ67VRUGJI36IQFAGAPE5F5LIKEDGF6GA, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Available Space: 947189571840}
2021-05-15T22:11:18.951Z	INFO	piecestore	download started	{Piece ID: VSI2BC7ODE3QLMSKHDPLRXRGGY6ACOY4UCIRABPAVRE6SIRQ2DVA, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: GET_REPAIR}
2021-05-15T22:11:19.201Z	INFO	piecestore	upload started	{Piece ID: 2A3YMZWGAJHVMSN46GB6U45KGNDWHFL4BJSE3G6FD7EZRWTNE2ZA, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Available Space: 947189571840}
2021-05-15T22:11:19.656Z	INFO	piecestore	upload canceled	{Piece ID: UQMVZZST3QRI3Y2S2TKPC5D436PINGZ3ZAC5RU3YWWUOCUMSUOCQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Size: 1843200}
2021-05-15T22:11:19.880Z	INFO	piecestore	uploaded	{Piece ID: LKEZ3ET6RNAU56GRYIDIUG5RN52T4KVYXEA2SHZTTMMNNWKBUHWQ, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Size: 512}
2021-05-15T22:11:20.813Z	INFO	piecestore	upload started	{Piece ID: SVZKZQKAKH6TPRODQB7ADDW64YBXLDYOCG6IYIMISCDGRWKTZU6Q, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Available Space: 947189545472}
2021-05-15T22:11:22.359Z	ERROR	piecestore	download failed	{Piece ID: VSI2BC7ODE3QLMSKHDPLRXRGGY6ACOY4UCIRABPAVRE6SIRQ2DVA, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: GET_REPAIR, error: pieces error: filestore error: unable to open \config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/vs/i2bc7ode3qlmskhdplrxrggy6acoy4ucirabpavre6sirq2dva.sj1: open config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/vs/i2bc7ode3qlmskhdplrxrggy6acoy4ucirabpavre6sirq2dva.sj1: input/output error, errorVerbose: pieces error: filestore error: unable to open \config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/vs/i2bc7ode3qlmskhdplrxrggy6acoy4ucirabpavre6sirq2dva.sj1: open config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/vs/i2bc7ode3qlmskhdplrxrggy6acoy4ucirabpavre6sirq2dva.sj1: input/output error\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:71\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:504\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:217\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51}
2021-05-15T22:11:22.359Z	ERROR	piecestore	download failed	{Piece ID: IZDIEV2YZQ7VDAVUG6RIXABQROO3JWE4J2QKZNPKVXU77OS7UAFA, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: GET, error: pieces error: filestore error: unable to open \config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/iz/diev2yzq7vdavug6rixabqroo3jwe4j2qkznpkvxu77os7uafa.sj1: open config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/iz/diev2yzq7vdavug6rixabqroo3jwe4j2qkznpkvxu77os7uafa.sj1: input/output error, errorVerbose: pieces error: filestore error: unable to open \config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/iz/diev2yzq7vdavug6rixabqroo3jwe4j2qkznpkvxu77os7uafa.sj1: open config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/iz/diev2yzq7vdavug6rixabqroo3jwe4j2qkznpkvxu77os7uafa.sj1: input/output error\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:71\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:504\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:217\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51}
2021-05-15T22:11:22.367Z	ERROR	piecestore	download failed	{Piece ID: QXJKN63X3V4ER3X7I6NHSZ66PRIKMURSSYG7H4Y2WTOFAYCGORPA, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: GET, error: pieces error: filestore error: unable to open \config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/qx/jkn63x3v4er3x7i6nhsz66prikmurssyg7h4y2wtofaycgorpa.sj1: open config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/qx/jkn63x3v4er3x7i6nhsz66prikmurssyg7h4y2wtofaycgorpa.sj1: input/output error, errorVerbose: pieces error: filestore error: unable to open \config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/qx/jkn63x3v4er3x7i6nhsz66prikmurssyg7h4y2wtofaycgorpa.sj1: open config/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/qx/jkn63x3v4er3x7i6nhsz66prikmurssyg7h4y2wtofaycgorpa.sj1: input/output error\n\tstorj.io/storj/storage/filestore.(*Dir).Open:279\n\tstorj.io/storj/storage/filestore.(*blobStore).Open:71\n\tstorj.io/storj/storagenode/pieces.(*Store).Reader:262\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:504\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:217\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51}
2021-05-15T22:11:22.566Z	INFO	piecestore	upload canceled	{Piece ID: 3T4JRKVMVY36V5JKUUBZ5VKFNIL633OM2MKFDVDK2CFP6N6SZ4VQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Size: 1843200}
2021-05-15T22:11:23.385Z	INFO	piecestore	download started	{Piece ID: H7OIP2NQLDXUV7SSMOIUML5VHAVANUSEFLJQ7XF4U4UUDYLXHJTA, Satellite ID: 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE, Action: GET}

The next earlier warning/error msg was around 2128hr. I guess it may not be related.

[deathlessdd]

Is that 8TB drive happen to be an SMR drive?

[zeyacry]

It is SATA HDD
Seagate IronWolf 8TB ST8000VN004
17 months old (New HDD when starting this node)

[deathlessdd]

Ahh ok then id say your drive is probably dying… I had a similar issue with mine but it was an SMR drive… Have you ran a smart on it to see what the health is? Smartmontools - Community Help Wiki

[zeyacry]

Here’s the result

[deathlessdd]

Looks to me that it is failing if im reading everything correctly…
Heres from mine you can compare to it Any time you see something with sectors its usually bad news, Also your drive is almost at the temp limit Id get some fans on it asap…

[zeyacry]

Am I able to reduce the used disk space from 6TB to below 4TB? This is for me to transfer the existing data onto another 4TB disk and not to start the node from scratch?.

[Alexey]

Yes you can: How do I change values like wallet address or storage capacity? - Node Operator
However, it will not free-up occupied space, it will only reduce an allocation.

[Pac]

@zeyacry

Am I able to reduce the used disk space from 6TB to below 4TB?

The only thing you can do is reduce the amount of space the node is authorized to use. But as @Alexey said, it won’t free up overused space.
As your node already uses 6.54TB (according to one of your previous screenshots), I’m afraid you cannot move your node to a 4TB disk as it won’t fit

So in short: in your case no, you can’t.

This is a features StorjLabs are aiming to develop eventually in the future, but it does’nt exist yet (currently known as “partial exit”).

[zeyacry]

This community is the very reason I still want to be an SNO.
Thanks for the guidance, advice and clarification.