WARNING: Full nodes receiving data

I raised that bug some time back (https://github.com/storj/storj/issues/3751) but with the current update it became important again.
After updating my nodes, the full nodes received data again!
A restart didn’t seem to help with that either.

Since my full nodes are still receiving upload requests, I have to assume that they continue to ignore that they are full. Therefore I have to stop my 2 full nodes until the error is fixed (i only have 60GB left on my HDD…).

Wanted to warn you to check up on your full nodes after the update. Don’t run out of space.

2 Likes

Is this space left from total allotted space for the node ?

The free allocated space for the nodes is at -20GB and -200GB for the 3rd node.
My available HDD disk space is only 60GB. Therefore I stopped the nodes because I don’t want to run out of harddrive space.

Our devs are aware of this issue and are working to address it as we speak.

Update: The dev team has identified the problem. While they are working on implementing the fix, we have temporarily halted uploads and downloads in order to prioritize network durability over availability. Thank you for your patience, our team will post an update when the issue has been fixed.

6 Likes

@kevink

3 Likes

I try to update the latest version, the the latest version not available on docker repository yet.

That sounds weird, my node has updated automatically awhile ago. I’m running it in docker with watchtower.

1 Like

I’ll report back once I get some uploads.

My nodes have updated to 31.10.
New node uploads put_repair without a problem but dont accept uploads. Download is no problem.

Old node same as new node but also alot of deletes

Update. Uploads “put” is uploaded to both my nodes but not many of them

Hi,
At most an hour ago I had version 0.31.9 and disk space remaining -4.51 GB. Now after upgrading to version 0.31.10 I have disk space remaining -4.65 GB. Surely the problem with the negative space was solved?

But i do have some of these.

But that have nothing to do with the update

2020-01-30T17:38:24.080Z ERROR piecestore delete failed {“error”: “pieces error: context canceled; v0pieceinfodb error: context canceled”, “errorVerbose”: “pieces error: context canceled; v0pieceinfodb error: context canceled\n\tstorj.io/storj/storagenode/pieces.(*Store).Delete:289\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).DeletePiece:225\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func4:1091\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Piece ID”: “OT4QRTGM47C2UBERY6WXHC2WRB2ZYWUKC5TS5HIYSCP5E7N5RUSA”}

It is not fully resolved yet. With v0.31.9 the storage nodes would accept pieces even if there are full. They will run out of space at some point. That is the difference you see in the frontend.

With v0.31.10 the storage node will stop accepting more pieces. There is still an issue with the used space calculation but at least the storage nodes will not run out of space.

We are still working on a second fix. As far as I know there is still a bug with garbage collection.

There’s currently not much upload traffic but it looks to me like it is resolved. Thanks for the fast bugfix!

If in version 0.31.10 no more data is accepted beyond the available space then it is okay for the moment.

yes that’s what this fix is about.

Still limited?
Thanks

we will give an update when there is any news. You can monitor the status at https://status.tardigrade.io

Update: the issue has been resolved, status is updated on https://status.tardigrade.io
Please be sure to update your nodes to the latest version, see Changelog v0.31.12

5 Likes

Updated both nods and still there is nearly no traffic.

1 Like

Just be patient until all nodes are updated, satellite is updated and tests resumed.

4 Likes