Only "Upload started" in the logs (very low ingress of a sudden)

Node Operators
1

Hey guys,

perhaps you can help me where to dig for the problem.
For almost 12 hours my node which was running fine for 2 months with a very high upload successs rate just got the worst piece of crap ever :slight_smile:

image
image1874Ă—899 200 KB

image
image1895Ă—903 244 KB

I already increased the log level to debug, but it’s not helping at all.
My storage is SAS connected, nothing changed, I have like 11 other nodes running with the same configuration and they work like a charm.

If I compare the stats to my other nodes, the egress traffic is quite the same, just the ingress is almost non existent.

image
image917Ă—343 20.2 KB

The node runs on an ESXI with a 250/50 VDSL connection (and it’s the only node in this /24 subnet).

It also takes like 3 minutes or so to stop the docker node… where my other nodes stop in a snap.
Do you think it is a problem with one of my dbs or something? They are relocated on an internal SSD.

The only good news though: I’m not failing any AUDITS it seems :slight_smile:

Thanks for any help guys

2

I could finally solve it myself… if anyone encounters the same problem:

-> my /storage/temp directory was clogged to the prim… with like 100s of temp data.
After I renamed it to “temp1” and restarted the node it worked again…
AND I recreated the orders.db/used_serial.db

Seems that the cleanup proccess had some problems. Perhaps storage latency or whatever.

A nice sunday to ya’ll.

1 Like
3

This was more likely a database related issue if you ask me. The temp files were there because those uploads couldn’t finish.

I hope by recreating you meant repairing them. Repairing the databases would likely have fixed this issue.