FATAL Unrecoverable error {"error": "trust: malformed cache: invalid character '\\x00' looking for beginning of value"

thev · May 2, 2021, 3:40am

I got following errors on node startup:

|2021-05-02T05:13:17.264+0200|FATAL|Unrecoverable error|{error: trust: malformed cache: invalid character '\\x00' looking for beginning of value, errorVerbose: trust: malformed cache: invalid character '\\x00' looking for beginning of value\n\tstorj.io/storj/storagenode/trust.LoadCacheData:96\n\tstorj.io/storj/storagenode/trust.LoadCache:36\n\tstorj.io/storj/storagenode/trust.NewPool:86\n\tstorj.io/storj/storagenode.New:379\n\tmain.cmdRun:177\n\tstorj.io/private/process.cleanup.func1.4:363\n\tstorj.io/private/process.cleanup.func1:381\n\tgithub.com/spf13/cobra.(*Command).execute:842\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:950\n\tgithub.com/spf13/cobra.(*Command).Execute:887\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.ExecCustomDebug:70\n\tmain.main:385\n\truntime.main:204}|
|---|---|---|---|
|2021-05-02T05:17:16.249+0200|FATAL|Unrecoverable error|{error: trust: malformed cache: invalid character '\\x00' looking for beginning of value, errorVerbose: trust: malformed cache: invalid character '\\x00' looking for beginning of value\n\tstorj.io/storj/storagenode/trust.LoadCacheData:96\n\tstorj.io/storj/storagenode/trust.LoadCache:36\n\tstorj.io/storj/storagenode/trust.NewPool:86\n\tstorj.io/storj/storagenode.New:379\n\tmain.cmdRun:177\n\tstorj.io/private/process.cleanup.func1.4:363\n\tstorj.io/private/process.cleanup.func1:381\n\tgithub.com/spf13/cobra.(*Command).execute:842\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:950\n\tgithub.com/spf13/cobra.(*Command).Execute:887\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.ExecCustomDebug:70\n\tmain.main:385\n\truntime.main:204}|
|2021-05-02T05:17:16.403+0200|FATAL|Unrecoverable error|{error: trust: malformed cache: invalid character '\\x00' looking for beginning of value, errorVerbose: trust: malformed cache: invalid character '\\x00' looking for beginning of value\n\tstorj.io/storj/storagenode/trust.LoadCacheData:96\n\tstorj.io/storj/storagenode/trust.LoadCache:36\n\tstorj.io/storj/storagenode/trust.NewPool:86\n\tstorj.io/storj/storagenode.New:379\n\tmain.cmdRun:177\n\tstorj.io/private/process.cleanup.func1.4:363\n\tstorj.io/private/process.cleanup.func1:381\n\tgithub.com/spf13/cobra.(*Command).execute:842\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:950\n\tgithub.com/spf13/cobra.(*Command).Execute:887\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.ExecCustomDebug:70\n\tmain.main:385\n\truntime.main:204}|
|2021-05-02T05:31:52.699+0200|FATAL|Unrecoverable error|{error: trust: malformed cache: invalid character '\\x00' looking for beginning of value, errorVerbose: trust: malformed cache: invalid character '\\x00' looking for beginning of value\n\tstorj.io/storj/storagenode/trust.LoadCacheData:96\n\tstorj.io/storj/storagenode/trust.LoadCache:36\n\tstorj.io/storj/storagenode/trust.NewPool:86\n\tstorj.io/storj/storagenode.New:379\n\tmain.cmdRun:177\n\tstorj.io/private/process.cleanup.func1.4:363\n\tstorj.io/private/process.cleanup.func1:381\n\tgithub.com/spf13/cobra.(*Command).execute:842\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:950\n\tgithub.com/spf13/cobra.(*Command).Execute:887\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.ExecCustomDebug:70\n\tmain.main:385\n\truntime.main:204}|
|2021-05-02T05:37:33.141+0200|FATAL|Unrecoverable error|{error: trust: malformed cache: invalid character '\\x00' looking for beginning of value, errorVerbose: trust: malformed cache: invalid character '\\x00' looking for beginning of value\n\tstorj.io/storj/storagenode/trust.LoadCacheData:96\n\tstorj.io/storj/storagenode/trust.LoadCache:36\n\tstorj.io/storj/storagenode/trust.NewPool:86\n\tstorj.io/storj/storagenode.New:379\n\tmain.cmdRun:177\n\tstorj.io/private/process.cleanup.func1.4:363\n\tstorj.io/private/process.cleanup.func1:381\n\tgithub.com/spf13/cobra.(*Command).execute:842\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:950\n\tgithub.com/spf13/cobra.(*Command).Execute:887\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.ExecCustomDebug:70\n\tmain.main:385\n\truntime.main:204}|

Can you please help me?

peem · May 2, 2021, 12:13pm

If you expect an answer, please provide more details, e.g. how in which system do you run the node …

thev · May 2, 2021, 4:40pm

Yeah, sorry. I am running on Windows 10, command line. This started to happen after a bluescreen.

peem · May 2, 2021, 4:47pm

Sorry, I can’t help with the node in Windows (I am not using Windows), but does Windows support a node with CLI? I don’t know, there are Windows specialists here…

This is Windows without a GUI and Docker?

thev · May 2, 2021, 7:39pm

Yes, it is without GUI and Docker, I have manually configured my nodes and update them manually too. I use CLI to run it and was successfull from October 2020.

peem · May 2, 2021, 7:51pm

Unusual “construction” … sorry, I don’t have the knowledge to help

thev · May 2, 2021, 11:11pm

Ok, it seems my trust-cache.json has been corrupted. I removed it and new one has been downloaded. I am back online

peem · May 3, 2021, 5:28am

How did you discover it?

thev · May 3, 2021, 6:44pm

“Malformed cache” message in logs sounded like trust-cache.json. “Invalid character …” seemed there is something wrong with this file so I opened it in notepad and saw only this:

Now it was clear this file has been corrupted so I removed it and lunched node again. To be honest I was surprised storagenode.exe has downloaded this without any hussle and node became online.

thev · May 3, 2021, 6:46pm

Yes, it is unusual but this is the easiest and clearest solution for me to run multiple nodes on the same Windows machine. I have everything in control and don’t need to fight with docker or program files garbage

chesterz · June 10, 2021, 5:57am

Hey there,

got a small problem starting my storagenode since today. As soon as I am trying to start my node via service I am prompted with an unexpected error 1067.

The storagenode.log got the following entries:

|2021-06-10T07:42:22.229+0200|INFO|Configuration loaded|{Location: C:\\Program Files\\Storj\\Storage Node\\config.yaml}|
|---|---|---|---|
|2021-06-10T07:42:22.236+0200|INFO|Operator email|{Address: ---}|
|2021-06-10T07:42:22.236+0200|INFO|Operator wallet|{Address: ---}|
|2021-06-10T07:42:22.262+0200|FATAL|Unrecoverable error|{error: trust: malformed cache: invalid character '\\x00' looking for beginning of value, errorVerbose: trust: malformed cache: invalid character '\\x00' looking for beginning of value\n\tstorj.io/storj/storagenode/trust.LoadCacheData:96\n\tstorj.io/storj/storagenode/trust.LoadCache:36\n\tstorj.io/storj/storagenode/trust.NewPool:86\n\tstorj.io/storj/storagenode.New:378\n\tmain.cmdRun:177\n\tstorj.io/private/process.cleanup.func1.4:363\n\tstorj.io/private/process.cleanup.func1:381\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.Exec:65\n\tmain.(*service).Execute.func1:64\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}|
|2021-06-10T07:42:25.953+0200|INFO|Configuration loaded|{Location: C:\\Program Files\\Storj\\Storage Node\\config.yaml}|
|2021-06-10T07:42:25.961+0200|INFO|Operator email|{Address: ---}|
|2021-06-10T07:42:25.961+0200|INFO|Operator wallet|{Address: ---}|
|2021-06-10T07:42:25.986+0200|FATAL|Unrecoverable error|{error: trust: malformed cache: invalid character '\\x00' looking for beginning of value, errorVerbose: trust: malformed cache: invalid character '\\x00' looking for beginning of value\n\tstorj.io/storj/storagenode/trust.LoadCacheData:96\n\tstorj.io/storj/storagenode/trust.LoadCache:36\n\tstorj.io/storj/storagenode/trust.NewPool:86\n\tstorj.io/storj/storagenode.New:378\n\tmain.cmdRun:177\n\tstorj.io/private/process.cleanup.func1.4:363\n\tstorj.io/private/process.cleanup.func1:381\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.Exec:65\n\tmain.(*service).Execute.func1:64\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57}|

I can start the node via commandline without any errors. Is there anything I can do to fix the service-thingy?

Sincerely,

Stefan

PS: I am using the windows gui.

Alexey · June 10, 2021, 7:05am

Hello @chesterz ,
Welcome to the forum!

See

Mario · August 29, 2021, 9:40am

Hi there,

My node is not working, the service give error 1067

2021-08-29T11:37:12.359+0200 FATAL Unrecoverable error {“error”: “trust: malformed cache: invalid character ‘\x00’ looking for beginning of value”, “errorVerbose”: “trust: malformed cache: invalid character ‘\x00’ looking for beginning of value\n\tstorj.io/storj/storagenode/trust.LoadCacheData:96\n\tstorj.io/storj/storagenode/trust.LoadCache:36\n\tstorj.io/storj/storagenode/trust.NewPool:89\n\tstorj.io/storj/storagenode.New:378\n\tmain.cmdRun:177\n\tstorj.io/private/process.cleanup.func1.4:363\n\tstorj.io/private/process.cleanup.func1:381\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfig:88\n\tstorj.io/private/process.Exec:65\n\tmain.(*service).Execute.func1:64\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

any idea??? thank you

peem · August 29, 2021, 12:04pm

That’s what Alexey proposes:

To fix an issue - please:

Stop the storagenode service either from the Services applet or from the elevated PowerShell:

Stop-Service storagenode

Check disk for errors with correction.
Remove the trust-cache.json from the data location
Restart the storagenode service either from the Services applet or from the elevated PowerShell:

Restart-Service storagenode

Check your logs.

Lumicron · November 13, 2021, 6:35pm

hi, my node stoped working since 4 days, i checked everything, hdd has no errors, have an static ipv4, restard diddnt helped, identity is ok. but port is shown as closed and i cant figure out why. reseted my router and made a new rule, puplic ip hasnt changed and is correct in yams. in the log it looks like it startet with an clocksync error, fixed that, but now i get an fatal error since 4 days. can post log because new mewmember?

Alexey · November 13, 2021, 8:50pm

Hello @Lumicron ,
Welcome to the forum!

Lumicron · November 14, 2021, 5:45am

thx for the quick response. but where is trust-cache.json exactly located? i diddnt changed the install path.

Alexey · November 14, 2021, 7:27am

In the data location. The folder, which contains all customers data. Depending on type of your storagenode it’s either specified in the storage.path: parameter of your config.yaml or in the --mount type=bind,source=/here/your/path/to/the/data/location,destination=/app/config option of the docker run command.

Lumicron · November 18, 2021, 5:28am

found it…under programs/storj…in jams its just c:…searched for hours…

Lumicron · November 18, 2021, 5:49am

can i delete storagenode.log after i got my node back online? it is almost 4 GB big…and i need a special programm that needs more than 10 minutes to open it.