"error": "piecestore: serial number is already used: usedserialsdb error: file is not a database"

Node Operators troubleshooting
1

Hi all!!

I just changed the drive of my node and now I’m getting this error message. What must I do?? Seems the database is corrupted. Is there a good tutorial to repair it??

Thanks in advance.

2

Let it run for few minutes then it will go away. Check log if you not failing any audits.

3

Ok, I’ll wait.

Thanks.

4

Hmmm… Seem that the error keeps happening. How I can check the audits??

Thanks in advance.

5

How long is your node up?

Show output of docker logs --tail 20 storagenode

6

The node has been running for 4 hours now.
Here is the log:

2019-12-03T11:07:45.219Z ERROR server gRPC stream error response {“error”: “piecestore: serial number is already used: usedserialsdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:444\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”, “errorVerbose”: “piecestore: serial number is already used: usedserialsdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:444\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:445\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”}
2019-12-03T11:10:31.190Z INFO piecestore download started {“Piece ID”: “EBGBINR7IN6EDU42GIEPN2STVQXQRM23KYKH36ELQS5P24UDPKVQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_AUDIT”}
2019-12-03T11:11:47.156Z INFO piecestore download started {“Piece ID”: “UZW36NPWSR2Z7KROPFPGOAGWAIJ3ZEK6KVGP62U4GGTBOAXXHHOQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2019-12-03T11:12:06.144Z INFO piecestore download started {“Piece ID”: “V2LPRVHDECDM7Y3RQGDGDX7MRGL5VZOIRXMGDBKW3PQTB2QENCCQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET”}
2019-12-03T11:12:06.148Z ERROR server gRPC stream error response {“error”: “piecestore: serial number is already used: usedserialsdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:444\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”, “errorVerbose”: “piecestore: serial number is already used: usedserialsdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:444\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:445\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”}
2019-12-03T11:12:08.619Z INFO piecestore download started {“Piece ID”: “3HVLG5E4GQPHZZV5XBSN36K7LXBM6JRR7G2CPBJ6TAFJV3DPDGUA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2019-12-03T11:12:16.988Z INFO piecestore download started {“Piece ID”: “GFEPIX5XQFCRLH5L63ACZEQ2KTUB2RQ4UKTKCQETO4STV6S2OVDA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2019-12-03T11:12:16.992Z ERROR server gRPC stream error response {“error”: “piecestore: serial number is already used: usedserialsdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:444\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”, “errorVerbose”: “piecestore: serial number is already used: usedserialsdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:444\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:445\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”}
2019-12-03T11:12:45.666Z INFO piecestore download started {“Piece ID”: “3MTAPEDUCJYJ7FZ7LUEUISQOWTBVBFONQX2NUSRYHGJUJPK6ANFA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2019-12-03T11:12:45.669Z ERROR server gRPC stream error response {“error”: “piecestore: serial number is already used: usedserialsdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:444\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”, “errorVerbose”: “piecestore: serial number is already used: usedserialsdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:444\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:445\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”}
2019-12-03T11:14:49.347Z INFO piecestore download started {“Piece ID”: “C3MM56SAO7VDG5WNQASWH56KNMGIZCXCUHKTDTFG2MJKCHZK3QPA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”}
2019-12-03T11:16:19.501Z INFO piecestore download started {“Piece ID”: “37CLBQG2TFO4OW2M3Y2C6QTTPIVRLXRDGGDO2MUCTIJTBKZMN3LA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET_AUDIT”}
2019-12-03T11:16:46.745Z INFO version running on version v0.26.2
2019-12-03T11:16:47.153Z INFO piecestore download started {“Piece ID”: “UZW36NPWSR2Z7KROPFPGOAGWAIJ3ZEK6KVGP62U4GGTBOAXXHHOQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2019-12-03T11:17:08.636Z INFO piecestore download started {“Piece ID”: “3HVLG5E4GQPHZZV5XBSN36K7LXBM6JRR7G2CPBJ6TAFJV3DPDGUA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2019-12-03T11:17:17.072Z INFO piecestore download started {“Piece ID”: “GFEPIX5XQFCRLH5L63ACZEQ2KTUB2RQ4UKTKCQETO4STV6S2OVDA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2019-12-03T11:17:17.075Z ERROR server gRPC stream error response {“error”: “piecestore: serial number is already used: usedserialsdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:444\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”, “errorVerbose”: “piecestore: serial number is already used: usedserialsdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:444\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:445\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”}
2019-12-03T11:17:45.421Z INFO piecestore download started {“Piece ID”: “3MTAPEDUCJYJ7FZ7LUEUISQOWTBVBFONQX2NUSRYHGJUJPK6ANFA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2019-12-03T11:17:45.425Z ERROR server gRPC stream error response {“error”: “piecestore: serial number is already used: usedserialsdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:444\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”, “errorVerbose”: “piecestore: serial number is already used: usedserialsdb error: file is not a database\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:444\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:445\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”}
2019-12-03T11:21:47.103Z INFO piecestore download started {“Piece ID”: “UZW36NPWSR2Z7KROPFPGOAGWAIJ3ZEK6KVGP62U4GGTBOAXXHHOQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}

7

This is a problem. When you changed drive did you copy data and identity folders properly ?

8

Yes, I did copy all (robocopy, node stopped), but I fear I had some error with the drive (wasn’t the reason, it was full).

9

Time for expert & official help @Alexey

1 Like
10

Tried to repair the database but I got a “Error: file is not a database” :frowning:

I fear I’m going to lose the node.

11

You don’t have to lose the node. As a last resort you can always stop the node and rename the usedserials.db before starting it again. It’s not ideal, but I’m fairly certain this database isn’t critical and will be recreated if missing.

Last resort though, give @Alexey or someone else a chance to respond first with possible better solutions.

12

Your keyboard hates the word node

13

Stupid auto correct… lol. Well, I made it look like it never happened, haha. You’d think my phone would know the word node by now…

14

Yeah, I did so, but didn’t work. I’m going to do it again…

15

Done. Now is talking about “ordersdb”. Node stopped and database deleted. Now seems it’s working…
Lot of activity now, we will see tomorrow…

16

Nope, not working, I get this error:
"“error”: “piecestore: serial number is already used: usedserialsdb error: no such table: used_serial_\n\tstorj.io…”

17

Try the one I shared in this post

1 Like
18

Done. Now I get this
“019-12-05T18:11:01.337Z ERROR piecestore failed to add order {“error”: “ordersdb error: no such table: unsent_order”, “errorVerbose”: “ordersdb error: no such table: unsent_order\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:625\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:379\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}”

Thanks for your help!

19

Remember when I said last resort? Not a good idea to remove db’s willy-nilly. I’m not entirely sure what the impact would be from removing the orders.db as well.

Regardless, I created an empty version that contains the right tables that you can try, but you will have lost unsent orders which you likely won’t be paid for (they are sent every hour, so you won’t be missing much). This db also holds order history, but I don’t know exactly what that’s used for.

Try this empty version here
https://alpha.transfer.sh/5H3rL/orders.db

1 Like
20

Yeah, I know, but my node “died” last saturday, so…
I’m going to try your last version.

Edit: YES!!! It’s working!! Thank you very much!!! Very grateful!!! I was having trouble with the node being offline but after resetting Docker now works!!!

Thank you again!!

1 Like