Please turn off Traffic Analyzer on ASUS routers with this feature to prevent being unable to change router settings in the future!
I’m posting this to prevent other SNOs from having to go through what I just did.
There seems to be a really nasty issue with Traffic Analyzer in certain ASUS routers that causes it to fill up the jffs storage of the router. When this happens a lot of settings changes can no longer be saved (including port forwarding changes) and the logs of the router will get filled with messages that garbage collection can’t be executed because there is no free space on the jffs partition.
kernel: jffs2: jffs2_reserve_space_gc of 196 bytes for garbage_collect_dnode failed: -28 kernel: jffs2: Error garbage collecting node at 01bd5ca0! kernel: jffs2: Argh. No free space left for GC. nr_erasing_blocks is 0. nr_free_blocks is 0. (erasableempty: yes, erasingempty: yes, erasependingempty: yes)
My guess is this was likely caused by the amount of traffic storagenodes generate on the network. So while this definitely isn’t caused by Storj, running a storagenode might increase the chances of running into this issue.
So, why can’t you just free up some space on the jffs partition? Well, because the jffs filesystem has a weird and very annoying quirk that prevents it from deleting files when it’s entirely full. Apparently a delete or even writing /dev/null to an empty file requires a metadata write first and there is no slack built in to allow for those writes to happen.
Even worse, this issue survives a factory reset of the router in many cases (it did in my case).
So, what should you do?
If your logs don’t show any errors yet regarding the full jffs partition and garbage collection. Just turn off Traffic Analyzer in your router settings. This will prevent its database from growing further and filling the entire partition. Optionally you can delete data with the trash icon on the Traffic Analyzer page (this option may only be available on some models). While this isn’t entirely necessary, it’s probably a good idea to clean up what’s already there.
Some screenshots of where to find this option in different router models:
If your router is already running into issues, there is unfortunately no simple fix. I went through the process outlined on this page:
In short it explains how to export the partition to a usb drive, then mount it on a different system with a little more space to do the removal, trim it back down to its original size and copy it back to the router. Unfortunately this requires having another linux system (I had to create a VM) to do these operations on. And if SSH wasn’t enabled on your router before this issue occurred, you can’t enable it afterwards, it simply won’t work. I had to temporarily enable telnet instead. If you have to, please disable it again right after for security reasons. And if you use SSH, only ever open it on the LAN side.
I sincerely hope nobody else runs into this issue, it kind of ruined my morning today.