# Database disk image is malformed - Error coming from nowhere

**URL:** <https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968>\
**Category:** troubleshooting\
**Tags:** database, unraid\
**Created:** [September 29, 2019, 10:21am UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968 "2019-09-29T10:21:08Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Coec](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/coec/32/67_2.png) [@Coec](https://forum.storj.io/u/Coec)\
**Post date:** [September 29, 2019, 10:21am UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/1 "2019-09-29T10:21:08Z")

</div>

I’ve had this problem before when stopping the Storj docker with to low timeout and getting the _Database disk image is malformed_ error, but recently i got it from nowhere. Here’s some logs when it happened:

```
2019-09-27T21:54:03.339Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "L5WA75AKNLXV4EFN5EIV5HB4TWDE7MKPDRKAOQSIC5EZKSAXRZ6Q", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:31.519Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "NCRV45MCMYA5NMCF52ZDBP2L6EMLIW2A7ILTOWDV4CBT5YSUYXRQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:54:31.764Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:37.084Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "3QK24YF3HFUIIN5ETA3FI6ADF2I6JHTMSVYH7X75WNP6B7IJ57XA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:38.711Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:39.285Z	e[34mINFOe[0m	piecestore	uploaded	{"Piece ID": "NCRV45MCMYA5NMCF52ZDBP2L6EMLIW2A7ILTOWDV4CBT5YSUYXRQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:54:42.607Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "3QK24YF3HFUIIN5ETA3FI6ADF2I6JHTMSVYH7X75WNP6B7IJ57XA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:43.842Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "JVOXRUKXJIOXVZDRTBP3SI4CIQVCXA5UKLOULS5ZZTJXRRUFDAPQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:54:46.613Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "JVOXRUKXJIOXVZDRTBP3SI4CIQVCXA5UKLOULS5ZZTJXRRUFDAPQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:03.164Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:55:08.185Z	e[34mINFOe[0m	piecestore	uploaded	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:55:16.126Z	e[31mERRORe[0m	server	gRPC unary error response	{"error": "rpc error: code = PermissionDenied desc = untrusted peer 1QzDKGHDeyuRxbvZhcwHU3syxTYtU1jHy5duAKuPxja3XC8ttk"}
2019-09-27T21:55:24.181Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "JVOXRUKXJIOXVZDRTBP3SI4CIQVCXA5UKLOULS5ZZTJXRRUFDAPQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:29.670Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "JVOXRUKXJIOXVZDRTBP3SI4CIQVCXA5UKLOULS5ZZTJXRRUFDAPQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:33.951Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "O43WOPVU3TLDQHNAGAHRIFQIZTVS3TLXKQEBMJQ2JXULNLQUH5TA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:36.250Z	e[34mINFOe[0m	piecestore	downloaded	{"Piece ID": "O43WOPVU3TLDQHNAGAHRIFQIZTVS3TLXKQEBMJQ2JXULNLQUH5TA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:55:37.927Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "NJE4ZQW76G773UOGMCXGPRIW6PKWIZEBP64KP5FQL2IYRQJNAVLQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:55:37.928Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:55:50.777Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "GNLUAAV4K2RTBP2Y7523SXUZQPW5PISRMQZ5ROMFGXIBDZ3UB5RQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:55:50.779Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:56:05.947Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:56:05.949Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:56:40.109Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:56:40.110Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:56:55.124Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "Q7ASDDELS7E3SSIHATPS33UKSGRI5M7FAC4EMQ3ICYAGISWQ6FUA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:56:55.124Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:57:05.774Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:57:05.774Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:58:11.664Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "SRQEYFT2O6MGE2NA6RWECBNSTQXSKWGL2JZT3OBSHNJ3CZQA3J5Q", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:58:11.665Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:58:11.963Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:58:11.965Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:58:32.196Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:58:32.197Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:59:13.058Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:59:13.060Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:59:20.351Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "2XMSV355IV6ZOCHJOUYHCOSHYRQKJ2XCRMHSQPG2N3WGJCQ32PMA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T21:59:20.353Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T21:59:34.079Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "KQJX3R57YNPMFMX3SHC3AVSF42D5NEYCDMM3GKJ3CM4EWHBPRQDA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T21:59:34.080Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T22:00:26.225Z	e[34mINFOe[0m	piecestore	download started	{"Piece ID": "IBS7VMM6L4SPPDZYDBIBWIS3AKEKBRT2LODRPMRTCY7MG26FOBNQ", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-09-27T22:00:26.227Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696", "errorVerbose": "piecestore: rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:407\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:408\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:356\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-27T22:00:38.922Z	e[34mINFOe[0m	bandwidth	Performing bandwidth usage rollups
2019-09-27T22:00:39.483Z	e[34mINFOe[0m	version	running on version v0.21.3
2019-09-27T22:00:39.733Z	e[33mWARNe[0m	orders	DB contains invalid marshalled orders	{"error": "ordersdb error: proto: pb.OrderLimit: illegal tag 0 (wire type 0)", "errorVerbose": "ordersdb error: proto: pb.OrderLimit: illegal tag 0 (wire type 0)\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).ListUnsentBySatellite:162\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders:153\n\tstorj.io/storj/internal/sync2.(*Cycle).Run:147\n\tstorj.io/storj/internal/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-09-27T22:00:39.733Z	e[34mINFOe[0m	orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW	sending	{"count": 419}
2019-09-27T22:00:39.733Z	e[34mINFOe[0m	orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs	sending	{"count": 1}
2019-09-27T22:00:39.918Z	e[34mINFOe[0m	orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs	finished
2019-09-27T22:00:40.972Z	e[34mINFOe[0m	orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW	finished
2019-09-27T22:00:42.130Z	e[33mWARNe[0m	orders	some unsent order aren't in the DB	{"error": "order not found: order not found: satellite: 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW, serial number: F7VQJLZUEJFPLKFCBOPSGNM6RI; order not found: (22 000 more chars after this)
2019-09-27T22:00:51.261Z	e[31mERRORe[0m	server	gRPC unary error response	{"error": "rpc error: code = PermissionDenied desc = untrusted peer 1QzDKGHDeyuRxbvZhcwHU3syxTYtU1jHy5duAKuPxja3XC8ttk"}
2019-09-27T22:01:02.293Z	e[34mINFOe[0m	piecestore	upload started	{"Piece ID": "SNNVFYECHIP5YQAQZA2BKT7U5EXZRRJMK5GC2KB42RGMOBLM7X4A", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-09-27T22:01:02.295Z	e[31mERRORe[0m	server	gRPC stream error response	{"error": "rpc error: code = Unauthenticated desc = serial number is already used: usedserialsdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:41\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:205\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}

```

I also get a the warning:  
_DB contains invalid marshalled orders_, which i cannot find any information about.

Would be greatful if someone could figure out where the errors come from so i don’t have to constantly monitor the logs in case this would happen again.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [September 29, 2019, 10:47am UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/2 "2019-09-29T10:47:54Z")

</div>

This error will popup if the container was a abruptly stopped for any reason - system update, reboot and so on.  
The graceful stop is `docker stop -t 300 storagenode`, by default the daemon is uses a 10 seconds timeout, which is not enough to gracefully stop the storagenode.  
You can take a look on this topic to configure your docker daemon:

> [@Docker post installation configuration](https://forum.storj.io/t/docker-post-installation-configuration/850):
>
> I would like suggest add to the [documentation](https://documentation.storj.io/setup/storage-node#running-the-storage-node) docker post installation steps: limit docker log files and add retention: "log-driver": "json-file", "log-opts": { "max-size": "100m", "max-file": "3" } and change default [shutdown timeout parameter](https://docs.docker.com/engine/reference/commandline/dockerd/): "shutdown-timeout": 300 I tested both parameters on my node, and evrything is working fine. If you will reboot host machine, no need to stop container before with docker stop storagenode -t 300 command, now daemon will handle it easy…

To fix a malformed database you can use this guide:

> **[How to fix a "database disk image is malformed"](https://support.storj.io/hc/en-us/articles/360029309111-How-to-fix-a-database-disk-image-is-malformed-)**
>
> Problem
> Sometimes a Storage Node Operator may encounter the "database disk image is malformed" error in their log. This could happen during unplanned shutdown or reboot. The error indicates that on...

---

<div class="post-metadata">

**Author:** ![Coec](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/coec/32/67_2.png) [@Coec](https://forum.storj.io/u/Coec)\
**Post date:** [September 29, 2019, 11:45am UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/3 "2019-09-29T11:45:45Z")

</div>

Thank you for the quick reply, but as i wrote in the beginning, this issue came out of nowhere. If you check the logs and the timestamps, you can see that it goes from working to not working in less than a second.

I’ve fixed the database twice before because I forgot to set the timeout when shutting off the docker, but this time it had been running and working for over 20 hours. If the problem occured because of a faulty shutdown, wouldn’t the error appear immediately?

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [September 29, 2019, 12:49pm UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/4 "2019-09-29T12:49:50Z")

</div>

The error is revealed, when the storagenode tried to access the database. So, I would like to see how long it was working after the last restart of the container before the error is occur.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [September 29, 2019, 12:52pm UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/5 "2019-09-29T12:52:16Z")

</div>

By the way how is you HDD connected to the host with storagenode?

---

<div class="post-metadata">

**Author:** ![Coec](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/coec/32/67_2.png) [@Coec](https://forum.storj.io/u/Coec)\
**Post date:** [September 29, 2019, 1:14pm UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/6 "2019-09-29T13:14:59Z")

</div>

Here is the full log from the restart: [https://drive.google.com/open?id=16mI2XhJUzJ1DiRgkWTSGGDHdidoJyqFq](https://drive.google.com/open?id=16mI2XhJUzJ1DiRgkWTSGGDHdidoJyqFq)

The HDD’s are configured in Unraid, with a share over 3 HDD’s where one is a parity. They are connected trough SATA.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [September 29, 2019, 10:04pm UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/7 "2019-09-29T22:04:42Z")

</div>

Unraid is known as unreliable platform for the storagenode. It is abruptly killing containers during reboot. It mounts the filesystem after the container start, it’s [corrupt an sqlite databases](https://forums.unraid.net/topic/80439-downgraded-back-to-667-due-to-sqlite-corruption/) and so on.  
I would recommend to migrate from this platform to something more reliable, if it possible and do not use the NFS as a protocol for the storage.

---

<div class="post-metadata">

**Author:** ![Coec](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/coec/32/67_2.png) [@Coec](https://forum.storj.io/u/Coec)\
**Post date:** [September 30, 2019, 3:21pm UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/8 "2019-09-30T15:21:55Z")

</div>

Thank you, that explained everything. Will see if I can make Unraid stable enough for usage with Storj

---

<div class="post-metadata">

**Author:** ![Coec](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/coec/32/67_2.png) [@Coec](https://forum.storj.io/u/Coec)\
**Post date:** [November 4, 2019, 9:08pm UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/9 "2019-11-04T21:08:43Z")

</div>

Update: After downgrading to Unraid 6.6.7, the Storj docker has been running stable for over a month now. The sql-corruption problem is trying to be fixed and the progress can be tracked in this thread [https://forums.unraid.net/bug-reports/prereleases/sqlite-data-corruption-testing-r664/](https://forums.unraid.net/bug-reports/prereleases/sqlite-data-corruption-testing-r664/). It seems like the problem has been fixed in 6.8.0-rc5 as seen in [this comment](https://forums.unraid.net/bug-reports/prereleases/sqlite-data-corruption-testing-r664/page/4/?tab=comments#comment-6650). It’s also important to not stop the Storj docker from the ui, but instead type `docker stop -t 300 storagenode` in the terminal. If using an autoupdater, make sure to also set its timeout to 300 before stopping.

---

<div class="post-metadata">

**Author:** ![miicar204](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/miicar204/32/1133_2.png) [@miicar204](https://forum.storj.io/u/miicar204)\
**Post date:** [November 23, 2022, 9:37pm UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/10 "2022-11-23T21:37:13Z")

</div>

x2 sites. Both Nodes (we will call Site A and Site B) on unRAID running with Docker; data in their own drive pools…no changes on either (besides normal reboots, server and container updates, as per-usual). They are still on V. 1.66.1 (according to logs). They are set up to always auto update. Sites are on static IP’s from ISP.

- Both sites started throwing this in logs; first Site A, then Site B:  
“ERROR piecestore failed to add bandwidth usage {“Process”: “storagenode”, “error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(\*bandwidthDB).getSummary:171\n\tstorj.io/storj/storagenode/storagenodedb.(\*bandwidthDB).Summary:113\n\tstorj.io/storj/storagenode/storagenodedb.(\*bandwidthDB).Add:52\n\tstorj.io/storj/storagenode/piecestore.(\*Endpoint).beginSaveOrder.func1:731\n\tstorj.io/storj/storagenode/piecestore.(\*Endpoint).Download.func6:670\n\tstorj.io/storj/storagenode/piecestore.(\*Endpoint).Download:694\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(\*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(\*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(\*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(\*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(\*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(\*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(\*Tracker).track:52”}” Constantly;

and occasionally, i get this at both sites:  
“ERROR bandwidth Could not rollup bandwidth usage {“Process”: “storagenode”, “error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(\*bandwidthDB).Rollup:324\n\tstorj.io/storj/storagenode/bandwidth.(\*Service).Rollup:53\n\tstorj.io/common/sync2.(\*Cycle).Run:99\n\tstorj.io/storj/storagenode/bandwidth.(\*Service).Run:45\n\tstorj.io/storj/private/lifecycle.(\*Group).Run.func2.1:87\n\truntime/pprof.Do:40\n\tstorj.io/storj/private/lifecycle.(\*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(\*Group).Go.func1:75”}”

- Site A started having issues before Site B, and showed lower then 100% suspension scores. It seems to have resolved itself, after a 2 or 3 days (for the most part), and only rarely throws the “error peicestore” or “Could not rollup bandwidth usage” message in the log any longer; the suspension scores have returned to 100%. Site A never showed any QUIC errors, or offline during its trouble.

- Site B was running smoothly (during Site A’s issues), until a standard server restart was performed. Now Site B has the constant “error peicestore” and occasional “Could not rollup bandwidth usage” errors (although it seems to still be uploading/downloading from satellites, in the log, between errors, and constant read/write disk activity as normal), BUT the Node Dashboard also shows offline, QUIC misconfigured, no Node ID is shown in the Dashboard, and all the stats on the main page are wrong (although the payout info seems to still exist; unknown if its correct or not) (see screenshot). I am extremely worried about losing this node, as it is the one with the most space and activity (with plans on adding a large amount of more space soon)!

I have not touched ANYTHING on either STORj Node Sites since these errors started, other then server and network equipment reboots, confirming the ports are (still) open, and the drives are all in good health! Prior to this, they have been running trouble free for 6 and 9 months respectively!

Site A’s issues seems to be different but similar to Site B. Site A seems to have gone back to trouble free working order, except for the rare “error peicestore” error, and “Could not rollup bandwidth usage” if i restart it.

Please help! It seems that it has happened since the latest version rollout? Will there be any amnesty to our reputation scores, as it seems to be out of our control or anything we have done as operators?

 ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/c/c1ac5a69255e6b3ee44e98d275bb8c540a63b1b1.png)

 ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/9/9ae260a0fed9df9bd280f626343e250941dea2aa.png)

UPDATE: in trying to resolve this ongoing issue, I have been comparing settings for both sites (matching the working A to B etc). I have noticed that, even tho Site A seems to be performing properly, at start up the log shows a number of “ERROR piecestore”, then one “ERROR bandwidth”, errors before, seemingly, settling into the normal download/upload logs (“GET” and “GET\_REPAIR” info lines) that i am used to seeing. […but, curiously, no “PUT”; i suspect that is due to the allotted disk space being full till i can get bigger drives]

…just thought this might be of some use to a dev who is reading this. (Site B is still having major issues as described above)

---

<div class="post-metadata">

**Author:** ![Stob](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/stob/32/1137_2.png) [@Stob](https://forum.storj.io/u/Stob)\
**Post date:** [November 24, 2022, 12:50am UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/11 "2022-11-24T00:50:22Z")

</div>

> [@miicar204](#):
>
> “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed

If both nodes are reporting this error then you need to run the fix on the bandwidth.db database.

> **[How to fix a "database disk image is malformed"](https://support.storj.io/hc/en-us/articles/360029309111)**
>
> Problem
> Sometimes a Storage Node Operator may encounter the "database disk image is malformed" error in their log. This could happen during unplanned shutdown or reboot. The error indicates that on...

FYI you should stop the node before restarting the server. This type of error can happen if a node isn’t cleanly shutdown.

---

<div class="post-metadata">

**Author:** ![miicar204](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/miicar204/32/1133_2.png) [@miicar204](https://forum.storj.io/u/miicar204)\
**Post date:** [November 24, 2022, 1:03am UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/12 "2022-11-24T01:03:39Z")

</div>

Thank you for the suggestion; and i plan on checking that once i resolve the offline issue. As i said…the Site A seems to no longer have that issue, and the nodes are set to shut off cleanly before a reboot, to prevent any write issues. This has not happened before, and nothing has changed!  
That doesn’t explain the “OFFLINE/QUIC/Misconfigured” issue.

This seems to be an issue many are having recently, so i don’t know if its as it seems.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [November 24, 2022, 5:36am UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/13 "2022-11-24T05:36:14Z")

</div>

Please fix databases first, when your nodes will start working and does not crash after start, you can fix the offline and QUIC issues (more likely the local IP of your UnRaid got changed and you need to update your port forwarding rules on your router).

---

<div class="post-metadata">

**Author:** ![miicar204](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/miicar204/32/1133_2.png) [@miicar204](https://forum.storj.io/u/miicar204)\
**Post date:** [November 24, 2022, 6:22am UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/14 "2022-11-24T06:22:58Z")

</div>

local IP was not changed; it is static both in in unraid and reserved on the router. its on a static (public) IP as well. It doesn’t crash after start.

I will try fixing the database and see if that helps (as i stated before, the other node site seemed to have fixed itself). but that still leaves the question on how this randomly happened after 6+ mths of no issues!

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [November 24, 2022, 6:47am UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/15 "2022-11-24T06:47:09Z")

</div>

Unfortunately SQLite databases are unable to fix itself, so they need your help there.

---

<div class="post-metadata">

**Author:** ![miicar204](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/miicar204/32/1133_2.png) [@miicar204](https://forum.storj.io/u/miicar204)\
**Post date:** [November 24, 2022, 10:06am UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/16 "2022-11-24T10:06:51Z")

</div>

So if there is only the bandwidth.db error, and my IP’s, ports etc have not changed (i have triple checked). why does the dashboard say offline, QUIC misconfigured, and node ID show as blank?

UPDATE: I followed the instructions here ([Used\_serial.db malformed - #4 by Alexey](https://forum.storj.io/t/used-serial-db-malformed/7512/4?u)), on Site A. Everything seemed to proceed smoothly but the resulting bandwidth.db file was,  
A. much smaller then the old one, and  
B. after trying to start the container, it crashed storj, my server and even the logs (server and docker containers) would auto-close after a couple seconds (making it impossible to read/troubleshoot)  
Thankfully…I had made a backup of the .DB, and after a server re-boot its back up and running with the occasional errors, as mentioned in my OP.

Without the ramdisk, i it would take days of being offline and disqualify me (for sure, if i am not already), but why was the output file so small? Can i copy the DB file somewhere else, to do the repair while leaving the node running to keep it online? Or is that going to cause confusion/disqualification in the node?

What happened? I am still trying to ascertain why the QUIC is saying misconfigured on Site B, when nothing has been changed, and all the IP, and port forwarding is clearly correct!?!?!?! Am i going to lose this node (siteB) and the reputation i was careful to maintain?

For future reference, can i use this backup again? Or, as i suspect, now that the node has been running again, the DB’s are changed and a backup must be made every time i shut down the node to repair the SQLite DB?

Sorry if i am long winded, i am trying to both understand and learn how this all works so i can be better!

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [November 25, 2022, 8:11am UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/17 "2022-11-25T08:11:21Z")

</div>

> [@miicar204](#):
>
> but why was the output file so small?

perhaps it has been corrupted too much. In that case you can recreate this database using this guide:

> **[How to fix database: file is not a database error](https://support.storj.io/hc/en-us/articles/4403032417044-How-to-fix-database-file-is-not-a-database-error)**
>
> If your node was abruptly terminated due to power failure, the database file could be irreversible corrupted, you can see an error Error starting master database on storagenode: database: file is n...

> [@miicar204](#):
>
> For future reference, can i use this backup again? Or, as i suspect, now that the node has been running again, the DB’s are changed and a backup must be made every time i shut down the node to repair the SQLite DB?

you may do so, however it’s not necessary. If your UnRaid is updated to the latest version, it should not broke sqlite databases anymore.

After databases will be fixed, we can troubleshoot the offline issue too.

---

<div class="post-metadata">

**Author:** ![miicar204](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/miicar204/32/1133_2.png) [@miicar204](https://forum.storj.io/u/miicar204)\
**Post date:** [November 25, 2022, 8:13am UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/18 "2022-11-25T08:13:21Z")

</div>

> [@Alexey](#):
>
> perhaps it has been corrupted too much. In that case you can recreate this database using this guide:

Will this fix my “offline, QUIC misconfigured” issue?

> [@Alexey](#):
>
> you may do so, however it’s not necessary. If your UnRaid is updated to the latest version, it should not broke sqlite databases anymore.

I have always kept my servers up to date, and this corruption didn’t start till the last couple weeks…i check almost daily

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [November 25, 2022, 8:17am UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/19 "2022-11-25T08:17:54Z")

</div>

I cannot say until the obvious error will be fixed.

---

<div class="post-metadata">

**Author:** ![miicar204](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/miicar204/32/1133_2.png) [@miicar204](https://forum.storj.io/u/miicar204)\
**Post date:** [November 25, 2022, 8:36am UTC](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968/20 "2022-11-25T08:36:49Z")

</div>

> [@Alexey](#):
>
> [https://support.storj.io/hc/en-us/articles/4403032417044-How-to-fix-database-file-is-not-a-database-error](https://support.storj.io/hc/en-us/articles/4403032417044-How-to-fix-database-file-is-not-a-database-error)

ok, that seemed to fix Site B…its back online etc in the dashboard. Lost my “Bandwidth used this month” but it seems like its not having the issues as before.

The log is running clean; besides this new error appearing once when i first started the node:

“ERROR contact:service ping satellite failed {“Process”: “storagenode”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “attempts”: 3, “error”: “ping satellite: check-in ratelimit: node rate limited by id”, “errorVerbose”: “ping satellite: check-in ratelimit: node rate limited by id\n\tstorj.io/storj/storagenode/contact.(\*Service).pingSatelliteOnce:136\n\tstorj.io/storj/storagenode/contact.(\*Service).pingSatellite:98\n\tstorj.io/storj/storagenode/contact.(\*Chore).updateCycles.func1:87\n\tstorj.io/common/sync2.(\*Cycle).Run:99\n\tstorj.io/common/sync2.(\*Cycle).Start.func1:77\n\tgolang.org/x/sync/errgroup.(\*Group).Go.func1:75”}”

I will monitor it and report back any other issues.

I am assuming the last month earnings will be lost?

I am going to try to repair that bandwidth.db from the backup without using the ramdisk and see if that works…can i safely put that back in, to get this months earning back, if it repairs correctly? or is that just going to cause more issues?

Thank you for all your assistance.

[Next page](https://forum.storj.io/t/database-disk-image-is-malformed-error-coming-from-nowhere/1968.md?page=2)