Database disk image is malformed - Error coming from nowhere

[Coec]

I’ve had this problem before when stopping the Storj docker with to low timeout and getting the Database disk image is malformed error, but recently i got it from nowhere. Here’s some logs when it happened:

2019-09-27T21:54:03.339Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "L5WA75AKNLXV4EFN5EIV5HB4TWDE7MKPDRKAOQSIC5EZKSAXRZ6Q", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:31.519Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "NCRV45MCMYA5NMCF52ZDBP2L6EMLIW2A7ILTOWDV4CBT5YSUYXRQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:54:31.764Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:37.084Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "3QK24YF3HFUIIN5ETA3FI6ADF2I6JHTMSVYH7X75WNP6B7IJ57XA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:38.711Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:39.285Z	e[34mINFOe[0m	piecestore	uploaded	{"Piece ID": "NCRV45MCMYA5NMCF52ZDBP2L6EMLIW2A7ILTOWDV4CBT5YSUYXRQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:54:42.607Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "3QK24YF3HFUIIN5ETA3FI6ADF2I6JHTMSVYH7X75WNP6B7IJ57XA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:43.842Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "JVOXRUKXJIOXVZDRTBP3SI4CIQVCXA5UKLOULS5ZZTJXRRUFDAPQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:46.613Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "JVOXRUKXJIOXVZDRTBP3SI4CIQVCXA5UKLOULS5ZZTJXRRUFDAPQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:03.164Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:55:08.185Z	e[34mINFOe[0m	piecestore	uploaded	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:55:16.126Z	e[31mERRORe[0m	server	gRPC unary error response	{"error": "rpc error: code = PermissionDenied desc = untrusted peer 1QzDKGHDeyuRxbvZhcwHU3syxTYtU1jHy5duAKuPxja3XC8ttk"}
2019-09-27T21:55:24.181Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "JVOXRUKXJIOXVZDRTBP3SI4CIQVCXA5UKLOULS5ZZTJXRRUFDAPQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:29.670Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "JVOXRUKXJIOXVZDRTBP3SI4CIQVCXA5UKLOULS5ZZTJXRRUFDAPQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:33.951Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "O43WOPVU3TLDQHNAGAHRIFQIZTVS3TLXKQEBMJQ2JXULNLQUH5TA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:36.250Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "O43WOPVU3TLDQHNAGAHRIFQIZTVS3TLXKQEBMJQ2JXULNLQUH5TA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:37.927Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "NJE4ZQW76G773UOGMCXGPRIW6PKWIZEBP64KP5FQL2IYRQJNAVLQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:55:37.928Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:55:50.777Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "GNLUAAV4K2RTBP2Y7523SXUZQPW5PISRMQZ5ROMFGXIBDZ3UB5RQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:55:50.779Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:56:05.947Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:56:05.949Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:56:40.109Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:56:40.110Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:56:55.124Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "Q7ASDDELS7E3SSIHATPS33UKSGRI5M7FAC4EMQ3ICYAGISWQ6FUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:56:55.124Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:57:05.774Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:57:05.774Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:58:11.664Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "SRQEYFT2O6MGE2NA6RWECBNSTQXSKWGL2JZT3OBSHNJ3CZQA3J5Q", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:58:11.665Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:58:11.963Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:58:11.965Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:58:32.196Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:58:32.197Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:59:13.058Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:59:13.060Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:59:20.351Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "2XMSV355IV6ZOCHJOUYHCOSHYRQKJ2XCRMHSQPG2N3WGJCQ32PMA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:59:20.353Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:59:34.079Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:59:34.080Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T22:00:26.225Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T22:00:26.227Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T22:00:38.922Z	e[34mINFOe[0m	bandwidth	Performing bandwidth usage rollups
2019-09-27T22:00:39.483Z	e[34mINFOe[0m	version	running on version v0.21.3
2019-09-27T22:00:39.733Z	e[33mWARNe[0m	orders	DB contains invalid marshalled orders	{"error": "ordersdb error: proto: pb.OrderLimit: illegal tag 0 (wire type 0)", "errorVerbose": "ordersdb error: proto: pb.OrderLimit: illegal tag 0 (wire type 0)\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).ListUnsentBySatellite:162\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders:153\n\tstorj.io/storj/internal/sync2.(*Cycle).Run:147\n\tstorj.io/storj/internal/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-09-27T22:00:39.733Z	e[34mINFOe[0m	orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW	sending	{"count": 419}
2019-09-27T22:00:39.733Z	e[34mINFOe[0m	orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs	sending	{"count": 1}
2019-09-27T22:00:39.918Z	e[34mINFOe[0m	orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs	finished
2019-09-27T22:00:40.972Z	e[34mINFOe[0m	orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW	finished
2019-09-27T22:00:42.130Z	e[33mWARNe[0m	orders	some unsent order aren't in the DB	{"error": "order not found: order not found: satellite: 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW, serial number: F7VQJLZUEJFPLKFCBOPSGNM6RI; order not found: (22 000 more chars after this)
2019-09-27T22:00:51.261Z	e[31mERRORe[0m	server	gRPC unary error response	{"error": "rpc error: code = PermissionDenied desc = untrusted peer 1QzDKGHDeyuRxbvZhcwHU3syxTYtU1jHy5duAKuPxja3XC8ttk"}
2019-09-27T22:01:02.293Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "SNNVFYECHIP5YQAQZA2BKT7U5EXZRRJMK5GC2KB42RGMOBLM7X4A", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T22:01:02.295Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}

I also get a the warning:
DB contains invalid marshalled orders, which i cannot find any information about.

Would be greatful if someone could figure out where the errors come from so i don’t have to constantly monitor the logs in case this would happen again.

[Alexey]

This error will popup if the container was a abruptly stopped for any reason - system update, reboot and so on.
The graceful stop is docker stop -t 300 storagenode, by default the daemon is uses a 10 seconds timeout, which is not enough to gracefully stop the storagenode.
You can take a look on this topic to configure your docker daemon:

Docker post installation configuration

I would like suggest add to the documentation docker post installation steps: limit docker log files and add retention: "log-driver": "json-file", "log-opts": { "max-size": "100m", "max-file": "3" } and change default shutdown timeout parameter: "shutdown-timeout": 300 I tested both parameters on my node, and evrything is working fine. If you will reboot host machine, no need to stop container before with docker stop storagenode -t 300 command, now daemon will handle it easy…

To fix a malformed database you can use this guide:

How to fix a "database disk image is malformed"

[Coec]

Thank you for the quick reply, but as i wrote in the beginning, this issue came out of nowhere. If you check the logs and the timestamps, you can see that it goes from working to not working in less than a second.

I’ve fixed the database twice before because I forgot to set the timeout when shutting off the docker, but this time it had been running and working for over 20 hours. If the problem occured because of a faulty shutdown, wouldn’t the error appear immediately?

[Alexey]

The error is revealed, when the storagenode tried to access the database. So, I would like to see how long it was working after the last restart of the container before the error is occur.

[Alexey]

By the way how is you HDD connected to the host with storagenode?

[Coec]

Here is the full log from the restart: https://drive.google.com/open?id=16mI2XhJUzJ1DiRgkWTSGGDHdidoJyqFq

The HDD’s are configured in Unraid, with a share over 3 HDD’s where one is a parity. They are connected trough SATA.

[Alexey]

Unraid is known as unreliable platform for the storagenode. It is abruptly killing containers during reboot. It mounts the filesystem after the container start, it’s corrupt an sqlite databases and so on.
I would recommend to migrate from this platform to something more reliable, if it possible and do not use the NFS as a protocol for the storage.

[Coec]

Thank you, that explained everything. Will see if I can make Unraid stable enough for usage with Storj

[Coec]

Update: After downgrading to Unraid 6.6.7, the Storj docker has been running stable for over a month now. The sql-corruption problem is trying to be fixed and the progress can be tracked in this thread https://forums.unraid.net/bug-reports/prereleases/sqlite-data-corruption-testing-r664/. It seems like the problem has been fixed in 6.8.0-rc5 as seen in this comment. It’s also important to not stop the Storj docker from the ui, but instead type docker stop -t 300 storagenode in the terminal. If using an autoupdater, make sure to also set its timeout to 300 before stopping.

[miicar204]

x2 sites. Both Nodes (we will call Site A and Site B) on unRAID running with Docker; data in their own drive pools…no changes on either (besides normal reboots, server and container updates, as per-usual). They are still on V. 1.66.1 (according to logs). They are set up to always auto update. Sites are on static IP’s from ISP.

• Both sites started throwing this in logs; first Site A, then Site B:
  “ERROR piecestore failed to add bandwidth usage {“Process”: “storagenode”, “error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:171\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:113\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:731\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:670\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:694\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}” Constantly;

and occasionally, i get this at both sites:
“ERROR bandwidth Could not rollup bandwidth usage {“Process”: “storagenode”, “error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Rollup:324\n\tstorj.io/storj/storagenode/bandwidth.(*Service).Rollup:53\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/bandwidth.(*Service).Run:45\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}”

• Site A started having issues before Site B, and showed lower then 100% suspension scores. It seems to have resolved itself, after a 2 or 3 days (for the most part), and only rarely throws the “error peicestore” or “Could not rollup bandwidth usage” message in the log any longer; the suspension scores have returned to 100%. Site A never showed any QUIC errors, or offline during its trouble.

• Site B was running smoothly (during Site A’s issues), until a standard server restart was performed. Now Site B has the constant “error peicestore” and occasional “Could not rollup bandwidth usage” errors (although it seems to still be uploading/downloading from satellites, in the log, between errors, and constant read/write disk activity as normal), BUT the Node Dashboard also shows offline, QUIC misconfigured, no Node ID is shown in the Dashboard, and all the stats on the main page are wrong (although the payout info seems to still exist; unknown if its correct or not) (see screenshot). I am extremely worried about losing this node, as it is the one with the most space and activity (with plans on adding a large amount of more space soon)!

I have not touched ANYTHING on either STORj Node Sites since these errors started, other then server and network equipment reboots, confirming the ports are (still) open, and the drives are all in good health! Prior to this, they have been running trouble free for 6 and 9 months respectively!

Site A’s issues seems to be different but similar to Site B. Site A seems to have gone back to trouble free working order, except for the rare “error peicestore” error, and “Could not rollup bandwidth usage” if i restart it.

Please help! It seems that it has happened since the latest version rollout? Will there be any amnesty to our reputation scores, as it seems to be out of our control or anything we have done as operators?

UPDATE: in trying to resolve this ongoing issue, I have been comparing settings for both sites (matching the working A to B etc). I have noticed that, even tho Site A seems to be performing properly, at start up the log shows a number of “ERROR piecestore”, then one “ERROR bandwidth”, errors before, seemingly, settling into the normal download/upload logs (“GET” and “GET_REPAIR” info lines) that i am used to seeing. […but, curiously, no “PUT”; i suspect that is due to the allotted disk space being full till i can get bigger drives]

…just thought this might be of some use to a dev who is reading this. (Site B is still having major issues as described above)

[Stob]

@miicar204

“bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed

If both nodes are reporting this error then you need to run the fix on the bandwidth.db database.

How to fix a "database disk image is malformed"

FYI you should stop the node before restarting the server. This type of error can happen if a node isn’t cleanly shutdown.

[miicar204]

Thank you for the suggestion; and i plan on checking that once i resolve the offline issue. As i said…the Site A seems to no longer have that issue, and the nodes are set to shut off cleanly before a reboot, to prevent any write issues. This has not happened before, and nothing has changed!
That doesn’t explain the “OFFLINE/QUIC/Misconfigured” issue.

This seems to be an issue many are having recently, so i don’t know if its as it seems.

[Alexey]

Please fix databases first, when your nodes will start working and does not crash after start, you can fix the offline and QUIC issues (more likely the local IP of your UnRaid got changed and you need to update your port forwarding rules on your router).

[miicar204]

local IP was not changed; it is static both in in unraid and reserved on the router. its on a static (public) IP as well. It doesn’t crash after start.

I will try fixing the database and see if that helps (as i stated before, the other node site seemed to have fixed itself). but that still leaves the question on how this randomly happened after 6+ mths of no issues!

[Alexey]

Unfortunately SQLite databases are unable to fix itself, so they need your help there.

[miicar204]

So if there is only the bandwidth.db error, and my IP’s, ports etc have not changed (i have triple checked). why does the dashboard say offline, QUIC misconfigured, and node ID show as blank?

UPDATE: I followed the instructions here (Used_serial.db malformed - #4 by Alexey), on Site A. Everything seemed to proceed smoothly but the resulting bandwidth.db file was,
A. much smaller then the old one, and
B. after trying to start the container, it crashed storj, my server and even the logs (server and docker containers) would auto-close after a couple seconds (making it impossible to read/troubleshoot)
Thankfully…I had made a backup of the .DB, and after a server re-boot its back up and running with the occasional errors, as mentioned in my OP.

Without the ramdisk, i it would take days of being offline and disqualify me (for sure, if i am not already), but why was the output file so small? Can i copy the DB file somewhere else, to do the repair while leaving the node running to keep it online? Or is that going to cause confusion/disqualification in the node?

What happened? I am still trying to ascertain why the QUIC is saying misconfigured on Site B, when nothing has been changed, and all the IP, and port forwarding is clearly correct!?!?!?! Am i going to lose this node (siteB) and the reputation i was careful to maintain?

For future reference, can i use this backup again? Or, as i suspect, now that the node has been running again, the DB’s are changed and a backup must be made every time i shut down the node to repair the SQLite DB?

Sorry if i am long winded, i am trying to both understand and learn how this all works so i can be better!

[Alexey]

@miicar204

but why was the output file so small?

perhaps it has been corrupted too much. In that case you can recreate this database using this guide:

How to fix database: file is not a database error

@miicar204

For future reference, can i use this backup again? Or, as i suspect, now that the node has been running again, the DB’s are changed and a backup must be made every time i shut down the node to repair the SQLite DB?

you may do so, however it’s not necessary. If your UnRaid is updated to the latest version, it should not broke sqlite databases anymore.

After databases will be fixed, we can troubleshoot the offline issue too.

[miicar204]

@Alexey

perhaps it has been corrupted too much. In that case you can recreate this database using this guide:

Will this fix my “offline, QUIC misconfigured” issue?

@Alexey

you may do so, however it’s not necessary. If your UnRaid is updated to the latest version, it should not broke sqlite databases anymore.

I have always kept my servers up to date, and this corruption didn’t start till the last couple weeks…i check almost daily

[Alexey]

I cannot say until the obvious error will be fixed.

[miicar204]

@Alexey

https://support.storj.io/hc/en-us/articles/4403032417044-How-to-fix-database-file-is-not-a-database-error

ok, that seemed to fix Site B…its back online etc in the dashboard. Lost my “Bandwidth used this month” but it seems like its not having the issues as before.

The log is running clean; besides this new error appearing once when i first started the node:

“ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 3, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatelliteOnce:136\n\tstorj.io/storj/storagenode/contact.(*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/common/sync2.(*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}”

I will monitor it and report back any other issues.

I am assuming the last month earnings will be lost?

I am going to try to repair that bandwidth.db from the backup without using the ramdisk and see if that works…can i safely put that back in, to get this months earning back, if it repairs correctly? or is that just going to cause more issues?

Thank you for all your assistance.