Windows node restarting almost every day since 1.16.1

Storage Node Operators troubleshooting
#1

Okay since 1.16.1 i see nodes restart almost daily.
its pointless from my point of view.
They are restarting, and every restart is HDD working like crazy for many hours.
it angers me, because there is no egress or ingress made in that time, but HDD is shortening its life by pointless work.

Please stop that somehow. Yes im sure there was never such behavior before.
Im using GUI on windows 10 pro. I have windows updates dissabled in hard way by policys and such, windows updates not a problem anymore.

Edit: Someone noticed the same?
Not every Node restarts daily, but the resratrs happens too often for sure.
mayby i will catch it and see if theres any errors in log acording to this particular problem yet. Seems like it just restarts and no problem, only that unpayed HDD work…

#2

Your node doesnt restart less something is failing there is no restart in normal use.

#3

I am not seeing this - stable as ever…

#4

Am seeing it now for example, in one node, but i cnat open the log to check, its too big my notepad ++ says, its 2,2GB, how to open it now? :smiley:

#5

In the PowerShell: https://documentation.storj.io/resources/faq/check-logs
Also, it’s worth to rename the current log while the storagenode stopped and archive it. When the storagenode starts it will create a new empty log.

#6

I believe egress and ingress should carry on even during the filewalker is rebrowsing files.

But you’re right and as others said, the node shouldn’t restart everyday and that’s gonna wear disks out.

#7

"Get-Content : A parameter cannot be found that matches parameter name “Tail”. i guess power shell is old, how to update it. (its ver 1.0, nevermind im bypasing it)
@Pac its ALMOST everyday, diffrent nodes difrent days, buts its most def more OFFTEn than befre 1.16.1 i basicly had 0 problems before that

#8
#9

Are all your nodes running on the same system? Theres no reason they should be restarting at all.

#11

is it possible that “filewalker” can be turned on without restart?
beacuse i checked the log from last case, and found no command to shutdown/restart the node has normal INFO log continously last “-Tail 2000”
but the HDD worked from certain point like filewalker was on just like restart begavior, and HDD on 100%, lol, will post more data when i catch next case

Edit: okay i got another case, and no sign of restarting in the logs, but HDD suddenly is working 100%, like after restart, and “storagenode program for Storj” in Processes has the highest % CPU usage, its 5,3-10% so low chances thats anything out of storagenode program, hmm whyy, the ver is 1.16.1

#12

The filewalker process will run every 1-2 day iirc. That’s normal and your hdd will be used 100% during that time. However, this shouldn’t have any impact on your ingress and egress.
How is your HDD connected?

#13

Its normal sata connection, to a motherboard. I didnt notice this problem before. only now a days such frequently. Before if hdd was working that hard it was always according to ndoe restart or startup.

Edit: okay this is anoying as heck. Same serwer just stopped that “filewalking”, CPU % felt to 0 for storagenode program. Some time passed, 15-30minutes. And now he started again! CPu % again 5-19% for storagenode program. dashboard says its 1.16.1 and online from 1h 40m, so no talk about any restart, its just filewalking without restart! whyyyy?

#14

Another behaviour that could hit the hard drive is when the satellites send the bloom filter to move files to the trash. If your node has been restarting often, it’s possible you have a lot of files that should have been deleted but weren’t. I would check the size of your trash folder.

#15

No, just like stated above.
it wasnt restarting at all i figured out.
“Filewalker” or whatever its called, was turned on without restarting!

#16

Do you see any errors related to storage_usage.db in the logs? The filewalker process shouldn’t run as often as you describe unless there is a problem with the node. If there is a problem with this database, it might be causing the recalculation of used space more often than it should.

The filewalker also shouldn’t stop uploads or downloads unless your node is I/O bottlenecked. Do you know the model of hard drive you are using? If it is an SMR drive, it could be causing a bottleneck.

#17

No, no SMR drive. And no errors in log i checked.
Its not that often but still, i catched one node with smoking gun! He did walking, then after while he did it again, as stated above.

#18

Is the I/O mostly read only? Or is it both reading and writing?

#19

Durring walking? its mostly reading, and writing almost none. Listen those nodes are jobless already, the egress or ingress is not existing almost and nodes got high repitation, 99%, sometimes 98% lately.

#20

How do you know its not an SMR drive, and how is the system connected to the drive, are you using the drive for anything else. Can you show a screen shot of what you actually talking about as well? Because your running on windows and it could be defragging in the background…

#21

because i know its not SMR. its 8TB ultrastar.no the drive is idle, only bussy when i do something mannualy, and dont do anything there lately. It was not defragmenting because as stated above i analized the CPU % usage, it showed me its storagenode program is doing this for sure. No i cannot provide any screens at this moment, i have enough evidence what happened, and i dont like it, i will observer more and update this topic