Error Codes: What they mean and Severity Level [READ FIRST]

Node Operators FAQ
69

6 posts were split to a new topic: Usedserialsdb error: disk I/O error: The device is not ready

72

Post your log between 3 backticks ```

73

2 posts were split to a new topic: Node offline, messed up with identity

74

A post was split to a new topic: Pieces error: marshaled piece header too big!

75

Hello,

I got this error today (It happened only once) :

2021-03-11T17:02:11.121Z	ERROR	piecestore	download failed	{"Piece ID": "WY64G3BUUNWGA7QD2ZVY33HXSIQ3ZLBWFFD7R2RCOSLTE35FXRCQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "file does not exist", "errorVerbose": "file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}

The disk shows no error. The node never hung-up, just the regular Storj container upgrades restarts.

Is there something I need to worry about ?

All the best !

76

You lost the piece that is why it says “file does not exist”. Did you have a sudden restart or power outage ?

77

Nope, never. the node is on an UPS

78

How old is your node ? You can search your log when the piece was uploaded. Your hardware could have restarted a month back and this only showed up now when the piece was getting downloaded.

79

Hello,

Thanks for your reply.

The node is up from early January, but I cleared the log after the upload.

I’ve searched more into the log.

  • A grep on the file returns this interesting entries :
2021-03-09T23:43:34.521Z INFO piecestore download started {“Piece ID”: “WY64G3BUUNWGA7QD2ZVY33HXSIQ3ZLBWFFD7R2RCOSLTE35FXRCQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2021-03-09T23:43:35.479Z INFO piecestore downloaded {“Piece ID”: “WY64G3BUUNWGA7QD2ZVY33HXSIQ3ZLBWFFD7R2RCOSLTE35FXRCQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2021-03-10T20:23:47.160Z INFO **collector delete expired** {“Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Piece ID”: “WY64G3BUUNWGA7QD2ZVY33HXSIQ3ZLBWFFD7R2RCOSLTE35FXRCQ”}
2021-03-10T21:42:54.825Z INFO piecestore download started {“Piece ID”: “WY64G3BUUNWGA7QD2ZVY33HXSIQ3ZLBWFFD7R2RCOSLTE35FXRCQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2021-03-10T21:42:54.873Z ERROR piecestore download failed {“Piece ID”: “WY64G3BUUNWGA7QD2ZVY33HXSIQ3ZLBWFFD7R2RCOSLTE35FXRCQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”, “error”: “file does not exist”, “errorVerbose”: “file does not exist\n\tstorj.io/common/rpc/rpcstatus.Wrap:74\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:506\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1033\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

What’s interesting is that happening after the " delete expired " collector message yesterday.

  • Then I greped on the “delete expired” string and I got many files from 4 to 11 March 2021, all from the EU West satellite.

So How a file which is supposed to be deleted is requested for download ? There is something that I don’t understand.

80

Hi, this is an expected behavior of the system.

There are a bunch of things that happen asynchronously, hence there’s some eventual consistency behaviors that might not be obvious at first sight.

Let’s take an example scenario that demonstrates it:

  1. At 11:15:00 Alice on computer A sends a request to download an object to the satellite, then satellite responds with the needed order limits for the specific storage nodes.
  2. At 11:15:01 Bob on computer B sends a delete request for that object, which ends up sending a delete request to the storage node. Storage node deletes the piece.
  3. At 11:15:03 Alice on computer A starts using the order limits (there was a latency spike on network, which caused it to take a bit longer)… however, now, none of the storage nodes have that data anymore.

This ends up in the logs as:

  1. object deleted
  2. download started
  3. download failed

With object expiration, the timeframe when this can happen is larger, because the satellite takes more time to delete expired objects/segments than the storage node.

6 Likes
81

Hello,

Thanks for the explanation. It’s clear now.

I was just worrying that something happens on my node potentially causing issue, especially for the GE process.

All the best !

82

7 posts were split to a new topic: Error: certificates peer error: authorization already claimed:

83

./identity.exe authorize storejnode email:*****@gmail.com:1XtU1ovrtStgSavWMrZoRfmuZwGHdm1P7nyKDm3nxuerx9pPQcMb1YiaaBT6YUDPMnFWz5aX5622PFv9Szaxz4ZAywokex
Error: certificates peer error: authorization not found:

84

Did you have that text in the command ?

It should be

/identity.exe authorize storejnode abaig60156@gmail.com:1XtU1ovrtStgSavWMrZoRfmuZwGHdm1P7nyKDm3nxuerx9pPQcMb1YiaaBT6YUDPMnFWz5aX5622PFv9Szaxz4ZAywokex
85

Hi All

having trouble getting a second node open with this error after creating a node with the windows GUI. and using GUI Toolbox to open the second node. I am studying how multiple nodes function so I can master the software. This is the error I am getting

2021-07-02T19:52:28.856+1000 FATAL Unrecoverable error {“error”: “Error starting master database on storagenode: group:\n— CreateFile D:\blobs: The system cannot find the file specified.\n— CreateFile D:\temp: The system cannot find the file specified.\n— CreateFile D:\garbage: The system cannot find the file specified.\n— CreateFile D:\trash: The system cannot find the file specified.”, “errorVerbose”: “Error starting master database on storagenode: group:\n— CreateFile D:\blobs: The system cannot find the file specified.\n— CreateFile D:\temp: The system cannot find the file specified.\n— CreateFile D:\garbage: The system cannot find the file specified.\n— CreateFile D:\trash: The system cannot find the file specified.\n\tmain.cmdRun:162\n\tstorj.io/private/process.cleanup.func1.4:363\n\tstorj.io/private/process.cleanup.func1:381\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.Exec:65\n\tmain.(*service).Execute.func1:64\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

and the node is not writing anything to the data disk. I think the error involves storage path and Database2 path as I had no idea what to put in Database2. Can someone help me with putting the correct files back in please?

86

This is often a permissions problem writing directly into the root of a drive. Reinstall the node or adjust the config.yaml with the StorJ directory set to D:\Node or D:\Storj instead of the root of the D:\ drive.

87

2021-07-02T20:06:44.999+1000 FATAL Unrecoverable error {“error”: “Error starting master database on storagenode: group:\n— CreateFile D:\Storagenode1\blobs: The system cannot find the file specified.\n— CreateFile D:\Storagenode1\temp: The system cannot find the file specified.\n— CreateFile D:\Storagenode1\garbage: The system cannot find the file specified.\n— CreateFile D:\Storagenode1\trash: The system cannot find the file specified.”, “errorVerbose”: “Error starting master database on storagenode: group:\n— CreateFile D:\Storagenode1\blobs: The system cannot find the file specified.\n— CreateFile D:\Storagenode1\temp: The system cannot find the file specified.\n— CreateFile D:\Storagenode1\garbage: The system cannot find the file specified.\n— CreateFile D:\Storagenode1\trash: The system cannot find the file specified.\n\tmain.cmdRun:162\n\tstorj.io/private/process.cleanup.func1.4:363\n\tstorj.io/private/process.cleanup.func1:381\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.Exec:65\n\tmain.(*service).Execute.func1:64\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

still doing it after the rename

not sure what is going on here

88

What user is running the ‘storagenode’ service? Normally it would be ‘system’ , ‘local system’ or ‘network service’. The account which is running the service needs permissions to the D:\ drive. Also just checking you’re not using an SMB share? That’s not supported. It must be a local or iSCSI device.

89

Full control was not switched on for D: which i changed I started the node and for a second it went green and then went offline again. the error log is showing nothing now when I start the node and the node is not showing on localhost, thanks for your patience with inexperience

90

At this point I would probably clear the D:\ drive, uninstall and put the correct permissions in place before re-installing.