Database disk image is malformed - Error coming from nowhere

I’ve had this problem before when stopping the Storj docker with to low timeout and getting the Database disk image is malformed error, but recently i got it from nowhere. Here’s some logs when it happened:

2019-09-27T21:54:03.339Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "L5WA75AKNLXV4EFN5EIV5HB4TWDE7MKPDRKAOQSIC5EZKSAXRZ6Q", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:31.519Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "NCRV45MCMYA5NMCF52ZDBP2L6EMLIW2A7ILTOWDV4CBT5YSUYXRQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:54:31.764Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:37.084Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "3QK24YF3HFUIIN5ETA3FI6ADF2I6JHTMSVYH7X75WNP6B7IJ57XA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:38.711Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:39.285Z	e[34mINFOe[0m	piecestore	uploaded	{"Piece ID": "NCRV45MCMYA5NMCF52ZDBP2L6EMLIW2A7ILTOWDV4CBT5YSUYXRQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:54:42.607Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "3QK24YF3HFUIIN5ETA3FI6ADF2I6JHTMSVYH7X75WNP6B7IJ57XA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:43.842Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "JVOXRUKXJIOXVZDRTBP3SI4CIQVCXA5UKLOULS5ZZTJXRRUFDAPQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:46.613Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "JVOXRUKXJIOXVZDRTBP3SI4CIQVCXA5UKLOULS5ZZTJXRRUFDAPQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:03.164Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:55:08.185Z	e[34mINFOe[0m	piecestore	uploaded	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:55:16.126Z	e[31mERRORe[0m	server	gRPC unary error response	{"error": "rpc error: code = PermissionDenied desc = untrusted peer 1QzDKGHDeyuRxbvZhcwHU3syxTYtU1jHy5duAKuPxja3XC8ttk"}
2019-09-27T21:55:24.181Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "JVOXRUKXJIOXVZDRTBP3SI4CIQVCXA5UKLOULS5ZZTJXRRUFDAPQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:29.670Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "JVOXRUKXJIOXVZDRTBP3SI4CIQVCXA5UKLOULS5ZZTJXRRUFDAPQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:33.951Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "O43WOPVU3TLDQHNAGAHRIFQIZTVS3TLXKQEBMJQ2JXULNLQUH5TA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:36.250Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "O43WOPVU3TLDQHNAGAHRIFQIZTVS3TLXKQEBMJQ2JXULNLQUH5TA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:37.927Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "NJE4ZQW76G773UOGMCXGPRIW6PKWIZEBP64KP5FQL2IYRQJNAVLQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:55:37.928Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:55:50.777Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "GNLUAAV4K2RTBP2Y7523SXUZQPW5PISRMQZ5ROMFGXIBDZ3UB5RQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:55:50.779Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:56:05.947Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:56:05.949Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:56:40.109Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:56:40.110Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:56:55.124Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "Q7ASDDELS7E3SSIHATPS33UKSGRI5M7FAC4EMQ3ICYAGISWQ6FUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:56:55.124Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:57:05.774Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:57:05.774Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:58:11.664Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "SRQEYFT2O6MGE2NA6RWECBNSTQXSKWGL2JZT3OBSHNJ3CZQA3J5Q", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:58:11.665Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:58:11.963Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:58:11.965Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:58:32.196Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:58:32.197Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:59:13.058Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:59:13.060Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:59:20.351Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "2XMSV355IV6ZOCHJOUYHCOSHYRQKJ2XCRMHSQPG2N3WGJCQ32PMA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:59:20.353Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:59:34.079Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:59:34.080Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T22:00:26.225Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T22:00:26.227Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T22:00:38.922Z	e[34mINFOe[0m	bandwidth	Performing bandwidth usage rollups
2019-09-27T22:00:39.483Z	e[34mINFOe[0m	version	running on version v0.21.3
2019-09-27T22:00:39.733Z	e[33mWARNe[0m	orders	DB contains invalid marshalled orders	{"error": "ordersdb error: proto: pb.OrderLimit: illegal tag 0 (wire type 0)", "errorVerbose": "ordersdb error: proto: pb.OrderLimit: illegal tag 0 (wire type 0)\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).ListUnsentBySatellite:162\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders:153\n\tstorj.io/storj/internal/sync2.(*Cycle).Run:147\n\tstorj.io/storj/internal/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-09-27T22:00:39.733Z	e[34mINFOe[0m	orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW	sending	{"count": 419}
2019-09-27T22:00:39.733Z	e[34mINFOe[0m	orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs	sending	{"count": 1}
2019-09-27T22:00:39.918Z	e[34mINFOe[0m	orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs	finished
2019-09-27T22:00:40.972Z	e[34mINFOe[0m	orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW	finished
2019-09-27T22:00:42.130Z	e[33mWARNe[0m	orders	some unsent order aren't in the DB	{"error": "order not found: order not found: satellite: 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW, serial number: F7VQJLZUEJFPLKFCBOPSGNM6RI; order not found: (22 000 more chars after this)
2019-09-27T22:00:51.261Z	e[31mERRORe[0m	server	gRPC unary error response	{"error": "rpc error: code = PermissionDenied desc = untrusted peer 1QzDKGHDeyuRxbvZhcwHU3syxTYtU1jHy5duAKuPxja3XC8ttk"}
2019-09-27T22:01:02.293Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "SNNVFYECHIP5YQAQZA2BKT7U5EXZRRJMK5GC2KB42RGMOBLM7X4A", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T22:01:02.295Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}

I also get a the warning:
DB contains invalid marshalled orders, which i cannot find any information about.

Would be greatful if someone could figure out where the errors come from so i don’t have to constantly monitor the logs in case this would happen again.

This error will popup if the container was a abruptly stopped for any reason - system update, reboot and so on.
The graceful stop is docker stop -t 300 storagenode, by default the daemon is uses a 10 seconds timeout, which is not enough to gracefully stop the storagenode.
You can take a look on this topic to configure your docker daemon:

To fix a malformed database you can use this guide:

Thank you for the quick reply, but as i wrote in the beginning, this issue came out of nowhere. If you check the logs and the timestamps, you can see that it goes from working to not working in less than a second.

I’ve fixed the database twice before because I forgot to set the timeout when shutting off the docker, but this time it had been running and working for over 20 hours. If the problem occured because of a faulty shutdown, wouldn’t the error appear immediately?

The error is revealed, when the storagenode tried to access the database. So, I would like to see how long it was working after the last restart of the container before the error is occur.

By the way how is you HDD connected to the host with storagenode?

Here is the full log from the restart: https://drive.google.com/open?id=16mI2XhJUzJ1DiRgkWTSGGDHdidoJyqFq

The HDD’s are configured in Unraid, with a share over 3 HDD’s where one is a parity. They are connected trough SATA.

Unraid is known as unreliable platform for the storagenode. It is abruptly killing containers during reboot. It mounts the filesystem after the container start, it’s corrupt an sqlite databases and so on.
I would recommend to migrate from this platform to something more reliable, if it possible and do not use the NFS as a protocol for the storage.

1 Like

Thank you, that explained everything. Will see if I can make Unraid stable enough for usage with Storj