Nodes Show Incorrect Stored Data After IP Change and Restart

Node Operators troubleshooting
1

Hi everyone,

I recently started 3 new nodes on a separate machine. My ISP changes the public IP every 3–4 weeks. I can’t use No-IP or similar dynamic DNS services because I constantly get attacked when I use them — somehow bots always find the hostname. So I’ve been using the raw public IP directly for over a year, and it works well for me, so I don’t want to change that.

I wrote a script that checks for IP changes every 3 minutes, and if the IP changes, it updates the configs and restarts the node. This method has been working perfectly on my main machine, and I improved and reused the same logic on the new one.

However, I’ve run into a strange issue. Early this morning the first IP change happened, and the script handled it correctly — the downtime was about 1.5 minutes. But after the nodes restarted, they only saw a fraction of the actual stored data. I stopped them, deleted all the database files, and restarted again. After that, the nodes correctly recognized the real amount of stored data.

I tested it again, and the same thing happened.

My question is: why does this happen?

These are small test nodes for now, each with around 25 GB of data. On my main node, which has 7–8 TB, I have never seen this issue.

Does anyone have an idea what could cause this?

Thanks!

2

Very similar setup to what I did a few years ago, before I got static IPs. Good solution to a bad problem imo.

It’s not advised to edit configuration files while the node is running

Instead of edit config → restart, you should do stop node → remove node → edit config → start node.

There was a lot of similar issues with users converting to hash store, where settings wouldn’t stick, if made while the node was online.

2 Likes
3

The used space donut graph will correct itself with the next restart roughly after a week. There is a prefix database, which prevents recalculation of used space too often to reduce IOPS pressure on the disk. So, if you restarting every day, it will prevent recalculation on each restart until some days will pass after the last recalculation.

You can delete just the prefix database and restart the node if it is so important to have accurate data in the dashboard at the cost of faster disk wear and reduced payouts due to losing races caused by slow disk performance during recalculation.

1 Like
4

Okay, thanks for the info. Next time I restart, I’ll just let it correct itself.

1 Like
5

I can’t get past the fact that hackers keep attacking you. Did you changed the host name? Did you changed the ports?

6

The hostname/port won’t matter: basically the entire IP range of the Internet is always being nmap’d by someone. I can open a random port… and in about 2-4 hours I can see bots banging on the door.

It’s not personal: nobody is out to get you: just another day on the public Internet…

7

It’s still not entirely clear what happened. Whatat is certain is that while I was using the DDNS service (I switched domain names three times, all completely random), after about 3–4 hours the router was getting around 15,000–16,000 attacks per second from the outside, and the ISP dropped the connection. Ever since I’ve been using it directly via IP, this problem hasn’t occurred