My node got full and correctly stopped receiving data. Then some data got deleted and it started to get data again. However, it has been full 12 hours ago and it is still receiving new data. It currently has -2.12GB
What is the version of the storagenode?
Are you sure that it receives data and does not moving the garbage to the trash?
Version is 0.29.3.
I checked the logs and it is receiving new uploads:
2020-01-15T09:57:38.365Z INFO piecestore upload started {“Piece ID”: “I7Z5R7AMG4J3UGK7IYT4BYE4HIRII2YT3UGNXTRNLTQZ4JSKXMRA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”}
2020-01-15T09:57:38.693Z INFO piecestore uploaded {“Piece ID”: “I7Z5R7AMG4J3UGK7IYT4BYE4HIRII2YT3UGNXTRNLTQZ4JSKXMRA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”}
2020-01-15T09:57:44.122Z INFO piecestore deleted {“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Piece ID”: “AZOVVL5JRUAYSH5VT3B6NY32DMGPXR3Z46RFSH3CW2HUIMZ36GVA”}
2020-01-15T09:57:44.132Z INFO piecestore downloaded {“Piece ID”: “OTQFJLCHKQWQ3RHFG3ZRKPIH5R3A6O7XA5L2P3PFNPDJIVABH76Q”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET”}
2020-01-15T09:57:52.442Z INFO piecestore download started {“Piece ID”: “LQZAB5PMRBWSFGJVL5LOPCACUHZOWPB7F6OGNYCU2XQPWDOOSCSA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET”}
2020-01-15T09:57:53.295Z INFO piecestore download started {“Piece ID”: “HWMKJSTBK4J4DAN3634DGOZOIFDPCIKP4S2P36AEMLRHJRH3FC6A”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET”}
2020-01-15T09:57:56.895Z INFO piecestore downloaded {“Piece ID”: “LQZAB5PMRBWSFGJVL5LOPCACUHZOWPB7F6OGNYCU2XQPWDOOSCSA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET”}
2020-01-15T09:58:08.197Z INFO piecestore upload started {“Piece ID”: “ATGO5IOHXVEAAGKQM3T7L36CYEYJPIWAGGX3XBFKICMLGO4BSFEA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”}
2020-01-15T09:58:08.436Z INFO piecestore uploaded {“Piece ID”: “ATGO5IOHXVEAAGKQM3T7L36CYEYJPIWAGGX3XBFKICMLGO4BSFEA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”}
The trash folder does however take 3.6GB.
My node now has -6.2GB, the trash folder is 3.2GB.
So if it continues to grow, I’ll have to restart the node in the hopes of “fixing” this, the HDD still has 60GB of free space, so I should be good another day but need to watch it closely.
This is very likely a bug?
What max amount did you configure when you setup the node?
Not sure what that should have to do with it but it’s 900GB on a 1TB drive.
So it should have 100GB free space, but it now have 60G. Therefore it should not receive new data, I think that is relevant
no because your assumption is just wrong. There’s 66GB because a 1TB drive doesn’t have exactly 1TB but that doesn’t matter. It could have been a 2TB drive and I still only have 900GB configured for the storagenode. HDD size and free space doesn’t matter (as long as there is free space).
I believe the main problem is the satellites dont know its full fast enough so thats why its a good idea to always keep 10% extra hard drive space incase it happens. But you should account for even having less space available to your node so it stops sooner rather then later. So if your drive is 1TB reality is its only about 931gigs, I only allow for 800gigs for a 1TB hard drive I rather be safe then sorry.
of course but that is not the problem here. I had almost 100GB of free space but the satellites send data since more than 24 hours…
Is it possible that it was garbage collecting as well so it thought there was more free space?
dashboard shows -7.7GB, free tash is 3GB. so it should reject any new uploads. It all started after some delete operations or garbage collect6ion
Ive seen -400gigs once and I still was getting some data here and there, But that was a garbage collection at the same time. was also version 28.3. How much trash/garbage do you have right now?
The trash folder is 3GB and I get a constant stream of data like the node got never full.
That is odd I remember also after a few mins after the Node getting full I did see it stop getting data, Was an error saying Node is full and it kept rejecting the uploads. Do you see any errors at all for it?
I just reduced my allowed space and all data stopped havent gotten any data. Going to adjust it so its a bit higher and see when the data stops.
no there are no errors at all.
The node was once full and correctly stopped getting data, then there was a lot of deletes/garbage collection and it had maybe 1GB of free space and started to get data again. Didn’t stop since and didn’t reject anything.
It works fine for me this is the error I get
2020-01-16T11:20:06.424-0500 INFO piecestore upload failed {“Piece ID”: “2XKK3SLZ7P76J3HWGJAY52BHG63JS2XPOGAUZAJHYQT6OJOWY6TA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: out of space”, “errorVerbose”: “piecestore protocol: out of space\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:422\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:268\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
yes I did get those too when it got full, before the node started accepting new data again…
Did you try restarting the node yet?
No I haven’t because that is besides the point…
This thread was a bug report, not a troubleshooting.
I could have restarted the node a long time ago and it would probably resolve the issue but the bug would still be there.