I upgraded a storage node from v0.13.5 to v0.15.2. It was offline for a month.
I noticed a lot of “infodb: database is locked” errors after the upgrade:
019-07-17T03:18:43.309-0400 INFO db.migration Latest Version {"version": 3}
2019-07-17T03:18:43.309-0400 INFO db.migration.4 Add index on pieceinfo expireation
2019-07-17T03:18:54.236-0400 INFO db.migration.5 Partial Network Wipe - Tardigrade Satellites
2019-07-17T03:18:55.101-0400 INFO db.migration.6 Add creation date.
2019-07-17T03:18:55.107-0400 INFO db.migration.7 Drop certificate table.
2019-07-17T03:31:02.810-0400 INFO db.migration.8 Drop old used serials and remove pieceinfo_deletion_failed index.
2019-07-17T03:34:01.931-0400 INFO db.migration.9 Add order limit table.
2019-07-17T03:34:01.932-0400 INFO db.migration.10 Optimize index usage.
2019-07-17T03:34:25.877-0400 INFO db.migration.11 Create bandwidth_usage_rollup table.
2019-07-17T03:34:25.878-0400 INFO db.migration.12 Clear Tables from Alpha data
2019-07-17T03:36:34.940-0400 INFO db.migration.13 Free Storagenodes from trash data
2019-07-17T03:57:03.567-0400 INFO vouchers Checking vouchers
2019-07-17T03:57:03.586-0400 INFO Node 15Y7KzGXxnjfo3wQDzLBxdkuYeyDc9AMiXpdHxNmsmsuLfyePb started
2019-07-17T03:57:03.586-0400 INFO Public server started on [::]:28967
2019-07-17T03:57:03.586-0400 INFO Private server started on 127.0.0.1:7778
2019-07-17T03:57:03.617-0400 INFO vouchers Requesting voucher {"satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2019-07-17T03:57:03.620-0400 INFO vouchers Requesting voucher {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2019-07-17T03:57:03.623-0400 INFO vouchers Requesting voucher {"satellite": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW"}
2019-07-17T03:57:03.634-0400 INFO vouchers Requesting voucher {"satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2019-07-17T03:57:04.445-0400 INFO running on version v0.15.2
2019-07-17T03:57:10.915-0400 INFO piecestore:orderssender.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {"count": 6}
2019-07-17T03:57:10.915-0400 INFO piecestore:orderssender.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {"count": 7}
2019-07-17T03:57:10.915-0400 INFO piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW sending {"count": 7062}
2019-07-17T03:57:38.023-0400 INFO piecestore:monitor Remaining Bandwidth {"bytes": 10995116277760}
2019-07-17T03:57:38.023-0400 WARN piecestore:monitor Disk space is less than requested. Allocating space {"bytes": 1729649541120}
2019-07-17T03:57:38.118-0400 ERROR vouchers Error requesting voucher {"satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "database is locked"}
2019-07-17T03:57:40.521-0400 ERROR vouchers Error requesting voucher {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "database is locked"}
2019-07-17T03:57:41.336-0400 ERROR piecestore:orderssender.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs failed to archive order as rejected: serial: SUDULPEAJNCIXMXB7LELT5VVHA, infodb: database is locked
2019-07-17T03:57:41.770-0400 ERROR piecestore:orderssender.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S failed to archive order as accepted: serial: BJSNW3UDLVGGPED7XOFGU3Y2R4, infodb: database is locked
2019-07-17T03:57:44.438-0400 ERROR piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW failed to archive order as rejected: serial: 5D3O66MRSNDCPEMOANWGB7S2OY, infodb: database is locked
2019-07-17T03:57:51.494-0400 ERROR piecestore:orderssender.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs failed to archive order as rejected: serial: 7ANQNP6TXRFX5A3I3ZTFKZZE6I, infodb: database is locked
2019-07-17T03:57:51.938-0400 ERROR piecestore:orderssender.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S failed to archive order as accepted: serial: I65G3H2HGBAMFEBWBBK7GDGOC4, infodb: database is locked
2019-07-17T03:57:54.539-0400 ERROR piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW failed to archive order as rejected: serial: Y453CWFA7FBJHEH756WIUSC2LE, infodb: database is locked
2019-07-17T03:58:00.603-0400 ERROR vouchers Error requesting voucher {"satellite": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "error": "database is locked"}
2019-07-17T03:58:01.595-0400 ERROR piecestore:orderssender.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs failed to archive order as rejected: serial: AGHVZ4UK2FGOVFKIO54SVKXKZM, infodb: database is locked
2019-07-17T03:58:02.026-0400 ERROR piecestore:orderssender.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S failed to archive order as accepted: serial: 7VY5YKCTVJGCLBNRY5YBLIKSFE, infodb: database is locked
2019-07-17T03:58:04.622-0400 ERROR piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW failed to archive order as rejected: serial: ZEFXCMM3GNEPDB2ZHOQ7KXSQXQ, infodb: database is locked
2019-07-17T03:58:04.722-0400 ERROR vouchers Error requesting voucher {"satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "database is locked"}
2019-07-17T03:58:11.680-0400 ERROR piecestore:orderssender.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs failed to archive order as accepted: serial: 2WUFXNBITJGGRHXTBJF6ATUVSY, infodb: database is locked
2019-07-17T03:58:12.110-0400 ERROR piecestore:orderssender.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S failed to archive order as accepted: serial: HBPVHNPHG5GQFIQYY2OPQ4BA3Y, infodb: database is locked
2019-07-17T03:58:14.704-0400 ERROR piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW failed to archive order as rejected: serial: F4JW3HSGWBCJTFZFB6O3RA7JQI, infodb: database is locked
2019-07-17T03:58:21.767-0400 ERROR piecestore:orderssender.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs failed to archive order as accepted: serial: S54XRLKICNDQXJWUO6K5CXCUKU, infodb: database is locked
2019-07-17T03:58:22.196-0400 ERROR piecestore:orderssender.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S failed to archive order as accepted: serial: X7WOTHZ3INBPBBPW53LOLWLCYI, infodb: database is locked
2019-07-17T03:58:24.788-0400 ERROR piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW failed to archive order as rejected: serial: WOGDS4LJGJAOJOKEJG64K4L4IU, infodb: database is locked
2019-07-17T03:58:31.849-0400 ERROR piecestore:orderssender.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs failed to archive order as accepted: serial: MV2ANHEPENBIPFEB6SZ43OTZD4, infodb: database is locked
2019-07-17T03:58:32.283-0400 ERROR piecestore:orderssender.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S failed to archive order as accepted: serial: CNFIEVH7DZAHJFK5W42J4EDJQY, infodb: database is locked
2019-07-17T03:58:32.283-0400 INFO piecestore:orderssender.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished
2019-07-17T03:58:32.284-0400 ERROR piecestore:orderssender.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S failed to settle orders {"error": "order: failed to archive order as accepted: serial: BJSNW3UDLVGGPED7XOFGU3Y2R4, infodb: database is locked; order: failed to archive order as accepted: serial: I65G3H2HGBAMFEBWBBK7GDGOC4, infodb: database is locked; order: failed to archive order as accepted: serial: 7VY5YKCTVJGCLBNRY5YBLIKSFE, infodb: database is locked; order: failed to archive order as accepted: serial: HBPVHNPHG5GQFIQYY2OPQ4BA3Y, infodb: database is locked; order: failed to archive order as accepted: serial: X7WOTHZ3INBPBBPW53LOLWLCYI, infodb: database is locked; order: failed to archive order as accepted: serial: CNFIEVH7DZAHJFK5W42J4EDJQY, infodb: database is locked", "errorVerbose": "group:\n--- order: failed to archive order as accepted: serial: BJSNW3UDLVGGPED7XOFGU3Y2R4, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:208\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as accepted: serial: I65G3H2HGBAMFEBWBBK7GDGOC4, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:208\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as accepted: serial: 7VY5YKCTVJGCLBNRY5YBLIKSFE, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:208\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as accepted: serial: HBPVHNPHG5GQFIQYY2OPQ4BA3Y, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:208\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as accepted: serial: X7WOTHZ3INBPBBPW53LOLWLCYI, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:208\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as accepted: serial: CNFIEVH7DZAHJFK5W42J4EDJQY, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:208\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-07-17T03:58:34.871-0400 ERROR piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW failed to archive order as rejected: serial: 323EAZP2F5H3RDDZUIOK5LB26Q, infodb: database is locked
2019-07-17T03:58:41.961-0400 ERROR piecestore:orderssender.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs failed to archive order as accepted: serial: G6QP6VCCXNEFLFJQHJYMQE3TPM, infodb: database is locked
2019-07-17T03:58:41.961-0400 INFO piecestore:orderssender.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished
2019-07-17T03:58:41.962-0400 ERROR piecestore:orderssender.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs failed to settle orders {"error": "order: failed to archive order as rejected: serial: SUDULPEAJNCIXMXB7LELT5VVHA, infodb: database is locked; order: failed to archive order as rejected: serial: 7ANQNP6TXRFX5A3I3ZTFKZZE6I, infodb: database is locked; order: failed to archive order as rejected: serial: AGHVZ4UK2FGOVFKIO54SVKXKZM, infodb: database is locked; order: failed to archive order as accepted: serial: 2WUFXNBITJGGRHXTBJF6ATUVSY, infodb: database is locked; order: failed to archive order as accepted: serial: S54XRLKICNDQXJWUO6K5CXCUKU, infodb: database is locked; order: failed to archive order as accepted: serial: MV2ANHEPENBIPFEB6SZ43OTZD4, infodb: database is locked; order: failed to archive order as accepted: serial: G6QP6VCCXNEFLFJQHJYMQE3TPM, infodb: database is locked", "errorVerbose": "group:\n--- order: failed to archive order as rejected: serial: SUDULPEAJNCIXMXB7LELT5VVHA, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:213\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as rejected: serial: 7ANQNP6TXRFX5A3I3ZTFKZZE6I, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:213\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as rejected: serial: AGHVZ4UK2FGOVFKIO54SVKXKZM, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:213\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as accepted: serial: 2WUFXNBITJGGRHXTBJF6ATUVSY, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:208\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as accepted: serial: S54XRLKICNDQXJWUO6K5CXCUKU, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:208\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as accepted: serial: MV2ANHEPENBIPFEB6SZ43OTZD4, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:208\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as accepted: serial: G6QP6VCCXNEFLFJQHJYMQE3TPM, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:208\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-07-17T03:58:44.957-0400 ERROR piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW failed to archive order as rejected: serial: K47AKAB53RF6VMFBS6YPK6EARE, infodb: database is locked
2019-07-17T03:58:55.033-0400 ERROR piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW failed to archive order as rejected: serial: QGJZGF5GLNFSTPMAGOCHIV4FXI, infodb: database is locked
2019-07-17T03:59:05.174-0400 ERROR piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW failed to archive order as rejected: serial: 5LIDMBKXOFGDFAI425CYGCRSM4, infodb: database is locked
2019-07-17T03:59:49.124-0400 ERROR piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW failed to receive response: rpc error: code = Unavailable desc = transport is closing
2019-07-17T03:59:49.124-0400 ERROR piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW sending agreements returned an error: EOF
2019-07-17T03:59:49.124-0400 INFO piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW finished
2019-07-17T03:59:49.124-0400 ERROR piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW failed to settle orders {"error": "order: failed to archive order as rejected: serial: 5D3O66MRSNDCPEMOANWGB7S2OY, infodb: database is locked; order: failed to archive order as rejected: serial: Y453CWFA7FBJHEH756WIUSC2LE, infodb: database is locked; order: failed to archive order as rejected: serial: ZEFXCMM3GNEPDB2ZHOQ7KXSQXQ, infodb: database is locked; order: failed to archive order as rejected: serial: F4JW3HSGWBCJTFZFB6O3RA7JQI, infodb: database is locked; order: failed to archive order as rejected: serial: WOGDS4LJGJAOJOKEJG64K4L4IU, infodb: database is locked; order: failed to archive order as rejected: serial: 323EAZP2F5H3RDDZUIOK5LB26Q, infodb: database is locked; order: failed to archive order as rejected: serial: K47AKAB53RF6VMFBS6YPK6EARE, infodb: database is locked; order: failed to archive order as rejected: serial: QGJZGF5GLNFSTPMAGOCHIV4FXI, infodb: database is locked; order: failed to archive order as rejected: serial: 5LIDMBKXOFGDFAI425CYGCRSM4, infodb: database is locked; order: failed to receive response: rpc error: code = Unavailable desc = transport is closing; order: sending agreements returned an error: EOF", "errorVerbose": "group:\n--- order: failed to archive order as rejected: serial: 5D3O66MRSNDCPEMOANWGB7S2OY, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:213\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as rejected: serial: Y453CWFA7FBJHEH756WIUSC2LE, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:213\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as rejected: serial: ZEFXCMM3GNEPDB2ZHOQ7KXSQXQ, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:213\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as rejected: serial: F4JW3HSGWBCJTFZFB6O3RA7JQI, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:213\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as rejected: serial: WOGDS4LJGJAOJOKEJG64K4L4IU, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:213\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as rejected: serial: 323EAZP2F5H3RDDZUIOK5LB26Q, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:213\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as rejected: serial: K47AKAB53RF6VMFBS6YPK6EARE, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:213\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as rejected: serial: QGJZGF5GLNFSTPMAGOCHIV4FXI, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:213\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to archive order as rejected: serial: 5LIDMBKXOFGDFAI425CYGCRSM4, infodb: database is locked\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:213\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: failed to receive response: rpc error: code = Unavailable desc = transport is closing\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:200\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n--- order: sending agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Sender).settle.func3:191\n\tstorj.io/storj/storagenode/orders.(*Sender).settle:221\n\tstorj.io/storj/storagenode/orders.(*Sender).Settle:142\n\tstorj.io/storj/storagenode/orders.(*Sender).runOnce.func1:127\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-07-17T04:12:04.203-0400 INFO running on version v0.15.2
2019-07-17T04:27:04.179-0400 INFO running on version v0.15.2
2019-07-17T04:42:04.163-0400 INFO running on version v0.15.2
After restarting the storage node everything seems to be fine:
2019-07-17T04:50:30.167-0400 INFO running on version v0.15.2
2019-07-17T04:50:30.190-0400 INFO db.migration Latest Version {"version": 13}
2019-07-17T04:50:30.192-0400 INFO vouchers Checking vouchers
2019-07-17T04:50:30.194-0400 INFO Node 15Y7KzGXxnjfo3wQDzLBxdkuYeyDc9AMiXpdHxNmsmsuLfyePb started
2019-07-17T04:50:30.207-0400 INFO Public server started on [::]:28967
2019-07-17T04:50:30.207-0400 INFO Private server started on 127.0.0.1:7778
2019-07-17T04:50:30.218-0400 INFO vouchers Requesting voucher {"satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2019-07-17T04:50:30.221-0400 INFO vouchers Requesting voucher {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2019-07-17T04:50:30.221-0400 INFO vouchers Requesting voucher {"satellite": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW"}
2019-07-17T04:50:30.223-0400 INFO vouchers Requesting voucher {"satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2019-07-17T04:50:30.233-0400 INFO piecestore:monitor Remaining Bandwidth {"bytes": 10995116277760}
2019-07-17T04:50:30.233-0400 WARN piecestore:monitor Disk space is less than requested. Allocating space {"bytes": 1729663594496}
2019-07-17T04:50:30.327-0400 INFO running on version v0.15.2
2019-07-17T04:50:30.446-0400 INFO piecestore:orderssender.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {"count": 6}
2019-07-17T04:50:30.460-0400 INFO piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW sending {"count": 4450}
2019-07-17T04:50:30.463-0400 INFO piecestore:orderssender.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {"count": 7}
2019-07-17T04:50:36.296-0400 INFO piecestore:orderssender.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished
2019-07-17T04:50:38.369-0400 INFO piecestore:orderssender.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished
2019-07-17T04:50:40.252-0400 INFO vouchers Voucher received {"satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2019-07-17T04:50:43.829-0400 INFO vouchers Voucher received {"satellite": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2019-07-17T04:51:09.991-0400 INFO piecestore:orderssender.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW finished
2019-07-17T04:51:17.456-0400 INFO vouchers Voucher received {"satellite": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW"}
2019-07-17T04:51:20.258-0400 INFO vouchers Voucher received {"satellite": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2019-07-17T05:05:30.769-0400 INFO running on version v0.15.2