I have been running an ultra stable but small 1TB node for 16+ months. It has always been full until the recent large deletion. Since then I think I have had 0 ingress, perhaps all the way back to last update. Egress continues fine. Any ideas?
Can you try restarting your node?
what does the log file say?
Ive got the same problem with my node… I stoped getting data after the first week of september.
That probably means there wasn’t a big deletion like you thought, and for some reason the node was displaying the used amount incorrectly. Possibly it was building the used space again, and your node was actually full the entire time. I would recommend running fsck/chkdsk.
I agree, looks like it had to rebuild the used amount (reason for this unknown, only one I can think of is a missing DB?), which means there should have been high IO during that time because it went through all files. But should be fine from now on.
Is this a linux node? Can you give the results of df -H?
Filesystem Size Used Avail Use% Mounted on
/dev/md0 2.3G 1.1G 1.2G 47% /
none 1.9G 0 1.9G 0% /dev
/tmp 1.9G 1.7M 1.9G 1% /tmp
/run 1.9G 3.6M 1.9G 1% /run
/dev/shm 1.9G 4.0K 1.9G 1% /dev/shm
none 4.0K 0 4.0K 0% /sys/fs/cgroup
cgmfs 100K 0 100K 0% /run/cgmanager/fs
/dev/vg1000/lv 2.7T 2.0T 646G 76% /volume1
It’s a Synology NAS, with a single raid 1 volume.
Hmmm. I wonder if your issue is related to this:
I see a bunch more trash got dumped since your last post. Are you sure you typed df -H with a capital H? Because 646 GiB = 639 GB, which is the exact amount your node is reporting as free. Which is a “symptom” of that bug. I am leaning towards this as being the problem.
My bad. I did use a lowercase h, here is df -H
Filesystem Size Used Avail Use% Mounted on
/dev/md0 2.5G 1.1G 1.3G 47% /
none 2.1G 0 2.1G 0% /dev
/tmp 2.1G 1.8M 2.1G 1% /tmp
/run 2.1G 3.8M 2.1G 1% /run
/dev/shm 2.1G 4.1k 2.1G 1% /dev/shm
none 4.1k 0 4.1k 0% /sys/fs/cgroup
cgmfs 103k 0 103k 0% /run/cgmanager/fs
/dev/vg1000/lv 2.9T 2.2T 693G 76% /volume1
I think you are seeing your node go slightly above the allocated amount which is causing the free space calculation to be done incorrectly (well, correctly according to the code, but the calculation shouldn’t be done that way). If you take a read of that thread, you should see what I am talking about. Unfortunately that doesn’t look like it will be fixed until v1.14.x . For now, just keep an eye on it, but I believe your node is actually full (hence no ingress) and you are just seeing the results of that bug.
1 Like