I’m not sure what translates into “enough RAM” for storagenodes… with 2x22TB drives what is enough RAM? What about 4 drives?
I use the Badger Cache since 4th August. no Problems at all.
My Node is a StableBit Drivepool with over 60TB (5HDD) and duplication for the Storj folders only. There is a SSD WriteCache too. Every 1GB its copied to the HDDs
The Badger Cache Folder is on a SSD with a SymLink in Windows with the Native Windows Node. (With StableBit Drivepool it is possible to store individual folders on specific hard drives. It would also be possible to integrate an SSD and only direct the database of the Badger folder to it).
After few Reset and unexpected Bluescreens from a driver Issue absolutely no Problems.
The Speed is as well very good.
The size is now over 2,5GB
All Storj Database are on a separate SSD outside the Pool.
Its a old SATA SSD with live span 11% left still no issues.
So RAM usage seems to scale both with node size and also from “other stuff”… probably buffered write activity.
Anecdotally, I had one node with a 12TB drive, the system had only 1GB of ram + swap, and I had a limit of 900MB in docker, and sometimes the node would run out of memory and get killed. But 1GB is enough with only a 7TB node with relatively sedate activity (unsure how it would deal with a torrent of data like during testing).
other nodes with like 6-10TB of active data are taking anywhere between 1GB And 4.5GB of RAM (which is their docker limit), but they haven’t had restart problems.
that’s just the node software itself, not disk caching or anything.
If the node is starting to use RAM, then the disk has a latency big enough to force the node to buffering uploads to the RAM.
That means for uploaded, downloaded, expiry deletion, garbage collection and trash deletion? Anything else?
What is the data this cache is storing? Size and what else? Inode number?
For downloads it should be used too, uploads and deletions would likely only update the cache.
The badger cache is mostly helpful for reads, not writes (likely would increase used IOPS a little bit, not much, but still) or deletions (at least only partially, when we do the stat, but the deletion itself would not be speed up).
unlikely. It’s designed to be universal, not bind to the FS implementation. Only usual metadata, timestamps and size.
Quick question, anyone here running badger on 1.12? It runs fine on 1.11 but just crashes on 1.12 for me, filewalker error in logs.
Could you please post a error?
Use two lines with a three ``` to outline the code, like this:
```
here logs
```
2024-09-11T09:52:00+03:00 ERROR pieces used-space-filewalker failed {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Lazy File Walker": true, "error": "lazyfilewalker: exit status 1", "errorVerbose": "lazyfilewalker: exit status 1\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:133\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:722\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-11T09:52:00+03:00 ERROR pieces used-space-filewalker failed {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Lazy File Walker": false, "error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:731\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-11T09:52:00+03:00 ERROR piecestore:cache encountered error while computing space used by satellite {"error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:731\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "SatelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-09-11T09:52:00+03:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "context canceled"}
2024-09-11T09:52:00+03:00 ERROR pieces used-space-filewalker failed {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Lazy File Walker": true, "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:133\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:722\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-11T09:52:00+03:00 ERROR pieces used-space-filewalker failed {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Lazy File Walker": false, "error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:731\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-11T09:52:00+03:00 ERROR piecestore:cache encountered error while computing space used by satellite {"error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:731\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-09-11T09:52:00+03:00 ERROR piecestore:cache error getting current used space for trash: {"error": "filestore error: failed to walk trash namespace 7b2de9d72c2e935f1918c058caaf8ed00f0581639008707317ff1bd000000000: context canceled", "errorVerbose": "filestore error: failed to walk trash namespace 7b2de9d72c2e935f1918c058caaf8ed00f0581639008707317ff1bd000000000: context canceled\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
no fatal errors in logs
Please make sure that the lazy mode is off, please also check for FATAL and Unrecoverable errors before these errors. Because right now it looks like a consequence of stopping the node.
There is no fatal or unrecoverable errors I did ctrl + f and there is none
Please use a CLI instead: How do I check my logs? - Storj Docs.
Could you please post all messages since restart?
I doesnt give anylogs if I do
pieces.enable-lazy-filewalker: false
pieces.file-stat-cache: badger
I tried again 13:24 today when I stoped the node
and restarted few times
2024-09-11T13:24:04+03:00 INFO piecestore uploaded {"Piece ID": "WTZXYDYBFTMPWA2XV4DVZQ47EDJ2GB27FK3OR7ZXTO7DCQDNLP3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.100:43710", "Size": 14080}
2024-09-11T13:24:05+03:00 INFO piecestore download started {"Piece ID": "524IZ2ERPCEX6W5KPIYLN3FY7YBABB5N3SN7ALGL35OL72CRBQGA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 295424, "Remote Address": "79.127.226.98:43402"}
2024-09-11T13:24:06+03:00 INFO piecestore downloaded {"Piece ID": "524IZ2ERPCEX6W5KPIYLN3FY7YBABB5N3SN7ALGL35OL72CRBQGA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "Offset": 0, "Size": 295424, "Remote Address": "79.127.226.98:43402"}
2024-09-11T13:24:06+03:00 INFO piecestore upload started {"Piece ID": "72MX5RTZDKQNF2LIXHOLGP2UL2JTODFT7675OFR5JNPH2P2OW7AA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.97:58320", "Available Space": 3607322781970}
2024-09-11T13:24:06+03:00 INFO piecestore uploaded {"Piece ID": "72MX5RTZDKQNF2LIXHOLGP2UL2JTODFT7675OFR5JNPH2P2OW7AA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.226.97:58320", "Size": 295680}
2024-09-11T13:24:07+03:00 INFO piecestore upload started {"Piece ID": "F4Y3PIDV65Y5W6OL2LWDSNH25534RMJM7BKHJFC6QWNDWZCI4FTQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "169.150.225.81:54920", "Available Space": 3607322485778}
2024-09-11T13:24:07+03:00 INFO piecestore uploaded {"Piece ID": "F4Y3PIDV65Y5W6OL2LWDSNH25534RMJM7BKHJFC6QWNDWZCI4FTQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "169.150.225.81:54920", "Size": 15360}
2024-09-11T13:24:08+03:00 INFO piecestore download started {"Piece ID": "HDL5JNYIJSI4ONJLJZ5NNIJ7OAGP2CVP2Y2WPSZEMQIS3LL2N2YQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8704, "Remote Address": "109.61.92.73:47698"}
2024-09-11T13:24:08+03:00 INFO piecestore downloaded {"Piece ID": "HDL5JNYIJSI4ONJLJZ5NNIJ7OAGP2CVP2Y2WPSZEMQIS3LL2N2YQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 8704, "Remote Address": "109.61.92.73:47698"}
2024-09-11T13:24:09+03:00 INFO piecestore download started {"Piece ID": "F4Y3PIDV65Y5W6OL2LWDSNH25534RMJM7BKHJFC6QWNDWZCI4FTQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 15104, "Remote Address": "79.127.201.213:51656"}
2024-09-11T13:24:09+03:00 INFO piecestore downloaded {"Piece ID": "F4Y3PIDV65Y5W6OL2LWDSNH25534RMJM7BKHJFC6QWNDWZCI4FTQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 15104, "Remote Address": "79.127.201.213:51656"}
2024-09-11T13:24:09+03:00 INFO piecestore upload started {"Piece ID": "RSUIJ7G7LWHODCEAMGZJV4JQJZCTES4OU4SEQNS6HCCAWKAQ3F2A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.205.235:35562", "Available Space": 3607322469906}
2024-09-11T13:24:09+03:00 INFO piecestore uploaded {"Piece ID": "RSUIJ7G7LWHODCEAMGZJV4JQJZCTES4OU4SEQNS6HCCAWKAQ3F2A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Remote Address": "79.127.205.235:35562", "Size": 4352}
2024-09-11T13:24:11+03:00 INFO Stop/Shutdown request received.
2024-09-11T13:24:11+03:00 INFO piecestore upload canceled {"Piece ID": "LPNNCLD67XJL6HLAV2VRJXRFIRISCTQJYKCKXFEITN2MFPTHIHGQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Remote Address": "79.127.226.101:56252", "Size": 131072}
2024-09-11T13:24:11+03:00 INFO piecestore upload canceled {"Piece ID": "ZXBZ6J76EB3NVQHN6S4WYACFKFTCGGRRM7L4QDIRXARUCTJRBS7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Remote Address": "79.127.226.97:58310", "Size": 393216}
2024-09-11T13:24:19+03:00 INFO Configuration loaded {"Location": "C:\\Program Files\\Storj2\\Storage Node\\config.yaml"}
2024-09-11T13:24:20+03:00 INFO Anonymized tracing enabled
2024-09-11T13:24:20+03:00 INFO Operator email {"Address": "tadas.mauza@gmail.com"}
2024-09-11T13:24:20+03:00 INFO Operator wallet {"Address": "0xab55e21235b55952392c4b7201cc4eff9dd6d4b5"}
its 13:35 right now, no new logs
and yes this is from cli
and I just got this in logs on a diffrent node, again tried to enable badger
2024-09-11T16:27:30+03:00 ERROR piecestore:cache encountered error while computing space used by satellite {"error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:731\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "SatelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-09-11T16:27:30+03:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {"satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "context canceled"}
2024-09-11T16:27:30+03:00 ERROR pieces used-space-filewalker failed {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Lazy File Walker": true, "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:133\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:722\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-11T16:27:30+03:00 ERROR pieces used-space-filewalker failed {"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Lazy File Walker": false, "error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:731\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-11T16:27:30+03:00 ERROR piecestore:cache encountered error while computing space used by satellite {"error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:731\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "SatelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-09-11T16:27:30+03:00 ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {"satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "context canceled"}
2024-09-11T16:27:30+03:00 ERROR pieces used-space-filewalker failed {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Lazy File Walker": true, "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:133\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:722\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-11T16:27:30+03:00 ERROR pieces used-space-filewalker failed {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Lazy File Walker": false, "error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:731\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-11T16:27:30+03:00 ERROR piecestore:cache encountered error while computing space used by satellite {"error": "filewalker: context canceled", "errorVerbose": "filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkAndComputeSpaceUsedBySatellite:731\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:81\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-09-11T16:27:30+03:00 ERROR piecestore:cache error getting current used space for trash: {"error": "filestore error: failed to walk trash namespace 7b2de9d72c2e935f1918c058caaf8ed00f0581639008707317ff1bd000000000: context canceled", "errorVerbose": "filestore error: failed to walk trash namespace 7b2de9d72c2e935f1918c058caaf8ed00f0581639008707317ff1bd000000000: context canceled\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).SpaceUsedForTrash:273\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:100\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-11T16:27:30+03:00 ERROR pieces lazyfilewalker failed {"error": "lazyfilewalker: exit status 1", "errorVerbose": "lazyfilewalker: exit status 1\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkSatellitePiecesToTrash:163\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:575\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:380\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:265\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-11T16:27:30+03:00 ERROR filewalker failed to get progress from database
2024-09-11T16:27:30+03:00 ERROR retain retain pieces failed {"cachePath": "C:\\Program Files\\Storj\\Storage Node/retain", "error": "retain: filewalker: context canceled", "errorVerbose": "retain: filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:181\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:582\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:380\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:265\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-11T16:27:47+03:00 ERROR db.filestatcache Received err: Opening table: "D:\\Storj\\filestatcache\\004987.sst" error: failed to initialize table error: failed to read index. error: failed to verify checksum for table: D:\Storj\filestatcache\004987.sst error: actual: 4166837864, expected: 2629516371 error: checksum mismatch
github.com/dgraph-io/badger/v4/y.init
/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/y/checksum.go:29
runtime.doInit
/usr/local/go/src/runtime/proc.go:6527
runtime.doInit
/usr/local/go/src/runtime/proc.go:6504
runtime.doInit
/usr/local/go/src/runtime/proc.go:6504
runtime.doInit
/usr/local/go/src/runtime/proc.go:6504
runtime.doInit
/usr/local/go/src/runtime/proc.go:6504
runtime.doInit
/usr/local/go/src/runtime/proc.go:6504
runtime.main
/usr/local/go/src/runtime/proc.go:233
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1598. Cleaning up...
2024-09-11T16:27:47+03:00 ERROR failure during run {"error": "Error opening database on storagenode: Opening table: \"D:\\\\Storj\\\\filestatcache\\\\004987.sst\" error: failed to initialize table error: failed to read index. error: failed to verify checksum for table: D:\\Storj\\filestatcache\\004987.sst error: actual: 4166837864, expected: 2629516371 error: checksum mismatch\ngithub.com/dgraph-io/badger/v4/y.init\n\t/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/y/checksum.go:29\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6527\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\n\tstorj.io/storj/storagenode/storagenodedb.cachedBlobstore:231\n\tstorj.io/storj/storagenode/storagenodedb.OpenExisting:250\n\tmain.cmdRun:67\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "errorVerbose": "Error opening database on storagenode: Opening table: \"D:\\\\Storj\\\\filestatcache\\\\004987.sst\" error: failed to initialize table error: failed to read index. error: failed to verify checksum for table: D:\\Storj\\filestatcache\\004987.sst error: actual: 4166837864, expected: 2629516371 error: checksum mismatch\ngithub.com/dgraph-io/badger/v4/y.init\n\t/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/y/checksum.go:29\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6527\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\n\tstorj.io/storj/storagenode/storagenodedb.cachedBlobstore:231\n\tstorj.io/storj/storagenode/storagenodedb.OpenExisting:250\n\tmain.cmdRun:67\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n\tmain.cmdRun:69\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-09-11T16:27:47+03:00 FATAL Unrecoverable error {"error": "Error opening database on storagenode: Opening table: \"D:\\\\Storj\\\\filestatcache\\\\004987.sst\" error: failed to initialize table error: failed to read index. error: failed to verify checksum for table: D:\\Storj\\filestatcache\\004987.sst error: actual: 4166837864, expected: 2629516371 error: checksum mismatch\ngithub.com/dgraph-io/badger/v4/y.init\n\t/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/y/checksum.go:29\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6527\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\n\tstorj.io/storj/storagenode/storagenodedb.cachedBlobstore:231\n\tstorj.io/storj/storagenode/storagenodedb.OpenExisting:250\n\tmain.cmdRun:67\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "errorVerbose": "Error opening database on storagenode: Opening table: \"D:\\\\Storj\\\\filestatcache\\\\004987.sst\" error: failed to initialize table error: failed to read index. error: failed to verify checksum for table: D:\\Storj\\filestatcache\\004987.sst error: actual: 4166837864, expected: 2629516371 error: checksum mismatch\ngithub.com/dgraph-io/badger/v4/y.init\n\t/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/y/checksum.go:29\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6527\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6504\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\n\tstorj.io/storj/storagenode/storagenodedb.cachedBlobstore:231\n\tstorj.io/storj/storagenode/storagenodedb.OpenExisting:250\n\tmain.cmdRun:67\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n\tmain.cmdRun:69\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tstorj.io/common/process.ExecWithCustomConfig:72\n\tstorj.io/common/process.Exec:62\n\tmain.(*service).Execute.func1:107\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
could this be caused by electrical outage?
Looks like lazy filewalker is enabled, while it should be disabled when enabling the badger cache:
It seems this can happen when the node is suddenly stopped. Removing the cache will recreate it after restarting the node:
1 Like
Yes, the filestatcache folder.
Well after electrical outage every nodes badger files corrupted, thats a dumb problem I think. I got everything running now. Thank you everyone.
1 Like
Good to hear all is working again!
Badger cache is still in its early stages, so it’s not enabled by default just yet. SNOs like yourself are helping by testing it out, which is awesome. It’s totally normal to run into a few bumps at this point, but by reporting these issues, you’re directly contributing to making the feature better for everyone.
2 Likes