Failed to create storage node peer: hashstore: unable to flock: hashstore: resource temporarily unavailable

Hello.
Could someone pass some ideas on how to i deal with this error?
Node does not start. No other errors visible. Tried to restart all machine, did not helped.

Thank you

2025-01-31T19:22:34Z INFO server kernel support for server-side tcp fast open remains disabled. {“Process”: “storagenode”}
2025-01-31T19:22:34Z INFO server enable with: sysctl -w net.ipv4.tcp_fastopen=3 {“Process”: “storagenode”}
2025-01-31T19:22:34Z ERROR failure during run {“Process”: “storagenode”, “error”: “Failed to create storage node peer: hashstore: unable to flock: hashstore: resource temporarily unavailable\n\tstorj.io/storj/storagenode/hashstore.NewStore:105\n\tstorj.io/storj/storagenode/hashstore.New:85\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:271\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:105\n\tstorj.io/storj/storagenode.New:618\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272”, “errorVerbose”: “Failed to create storage node peer: hashstore: unable to flock: hashstore: resource temporarily unavailable\n\tstorj.io/storj/storagenode/hashstore.NewStore:105\n\tstorj.io/storj/storagenode/hashstore.New:85\n\tstorj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:271\n\tstorj.io/storj/storagenode/piecestore.NewHashStoreBackend:105\n\tstorj.io/storj/storagenode.New:618\n\tmain.cmdRun:84\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272\n\tmain.cmdRun:86\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272”}
Error: Failed to create storage node peer: hashstore: unable to flock: hashstore: resource temporarily unavailable
storj.io/storj/storagenode/hashstore.NewStore:105
storj.io/storj/storagenode/hashstore.New:85
storj.io/storj/storagenode/piecestore.(*HashStoreBackend).getDB:271
storj.io/storj/storagenode/piecestore.NewHashStoreBackend:105
storj.io/storj/storagenode.New:618
main.cmdRun:84
main.newRunCmd.func1:33
storj.io/common/process.cleanup.func1.4:392
storj.io/common/process.cleanup.func1:410
github.com/spf13/cobra.(*Command).execute:983
github.com/spf13/cobra.(*Command).ExecuteC:1115
github.com/spf13/cobra.(*Command).Execute:1039
storj.io/common/process.ExecWithCustomOptions:112
main.main:34
runtime.main:272
2025-01-31 19:22:34,343 WARN stopped: storagenode (exit status 1)
2025-01-31 19:22:34,344 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)

1 Like

How is your datastore mounted.

See the comments from Alexey in this thread.

1 Like

It is HDD (ext4) passed trough proxmox to VM.
This setup was working fine for years. But yesterday failed and i cannot repair it :expressionless:

This error seems not related to not supported flock on a network filesystems.
Could you please check this disk for issues and fix them?

That error indicates some other process already has a lock on the same hashstore. Is it possible you have multiple nodes configured to use the same storage directory?

2 Likes

It also happened to one of my nodes last night, which was working fine, after updating to 1.121.2, it gave the hashstore flock error. I haven’t activated the migration, I’m still on piecestore. I threw away the bin, piecestore, filestatcache folders, removed and recreated the node and it finally started

2 Likes

Did it say "unable to flock: hashstore: resource temporarily unavailable" or was it "unable to flock: ..." something else?

There should be no reason for “resource temporarily unavailable” to show unless something else has a flock on the file for some reason.

1 Like

Unfortunately I didn’t save the error, I was busy restarting the node, but I think that was it. In my opinion it has to do with the badger cache, but I’m not an expert, I’ll throw it out there