I am having the same issue with one of my nodes.
2020-04-02T22:05:23.926Z ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb error: disk I/O error: input/output error”
I can run earnings.py with feb and april but not march.
Another strange thing is, It is the only node I have connected to a USB drive, but not sure it is related.
I keep getting successfull PUT/GET.
Meaning that your disk have problems.
What is your OS?
Version of storagenode (Docker, binary, Windows GUI)?
How is your HDD connected to the PC with storagenode?
Debian 10, storage node is in v1.0.1 and the disk is connected via USB.
Yeah my first thought was that the disk is dying
I think it’s still should be taken into consideration.
Please, check your disk for errors.
Also, the USB disk should have an additional power supply to work under load.
It is better to use the internal disk instead if possible
The setup is a combo of a XU4 board and an external USB3 drive with its own power supply.
I can’t access it physically until confinement is finished.
Can you check it with fsck?
Sorry for the late reply.
fsck is returning clean status.
root@XXX:~# fsck -y /dev/sda1
fsck from util-linux 2.33.1
e2fsck 1.44.5 (15-Dec-2018)
WD: clean, 1237633/183148544 files, 578516532/732565760 blocks
The full error message:
2020-04-07T14:29:52.537Z ERROR piecestore failed to add bandwidth usage {"error": "bandwidthdb error: disk I/O error: input/output error", "errorVerbose": "bandwidthdb error: disk I/O error: input/output error\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:170\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:112\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:51\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:721\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:988\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:199\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:173\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:124\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:161\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}/dev/sda is not a USB drive …
unless you only have one drive on your computer
Its a USB drive, system is running on SD card.
root@XXX:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 2.7T 0 disk
└─sda1 8:1 0 2.7T 0 part /data
mmcblk1 179:0 0 14.9G 0 disk
└─mmcblk1p1 179:1 0 14.7G 0 part /Please, show permissions:
ls -l /data/storage
Are you able to manually create files on that USB drive?
I had one of my USB drives fail with a bad superblock. I could still mount it correctly and read my data but storagenode would crash and I was not able to create any file anymore.
root@XXX:/data/storj-storage# ls -l
total 42396
-rw------- 1 root root 7012 Feb 4 16:39 config.yaml
-rw-r--r-- 1 root root 39364261 Apr 8 13:40 node.log
-rw-r--r-- 1 root root 34386 Apr 3 00:00 node.log-20200403.gz
-rw-r--r-- 1 root root 861698 Apr 4 00:00 node.log-20200404.gz
-rw-r--r-- 1 root root 958509 Apr 5 00:00 node.log-20200405.gz
-rw-r--r-- 1 root root 2136985 Apr 8 00:00 node.log-20200408.gz
-rw------- 1 root root 32768 Apr 7 16:29 revocations.db
drwx------ 6 root root 4096 Apr 7 16:29 storage
-rw------- 1 root root 1202 Apr 8 06:47 trust-cache.json
Yes I can create files on that USB drive.
Container can create files as well
2020-04-08T11:41:55.479Z INFO piecestore uploaded {"Piece ID": "MUWFY52KQ35PDF6ZDTISC5327EGFQB4NAPGSFQMS7E2HU76OMTQA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"}
Please, try to check the integrity of that database:
Also, could you show the result of the command:
root@XXX:/data/storj-storage# ls -l storage/Hello, I have the same problem.
Since I updated to 1.0.1 and yesterday to 1.1.1, the problem persists.
Somehow it is wired that when I start my node I can access my dashboard via port 14002 and with Docker exec -it storagenode /app/dashboard.sh.
If I wait 10 minutes and access my dashboard again, I get the same error
2020-04-10T21:12:55.925Z ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb error: disk I/O error: input/output error”
If I just look at the logs, I see that there are a lot of uploads and everything else is working fine.
root@XXX:/data/storj-storage/storage# ll
total 512392
-rw-r--r-- 1 root root 28635136 Apr 11 19:25 bandwidth.db
drwx------ 7 root root 4096 Feb 13 17:12 blobs
drwx------ 2 root root 4096 Apr 11 18:31 garbage
-rw-r--r-- 1 root root 32768 Apr 9 10:21 heldamount.db
-rw-r--r-- 1 root root 32768 Apr 9 10:21 heldamount.db-shm
-rw-r--r-- 1 root root 32992 Apr 9 10:21 heldamount.db-wal
-rw-r--r-- 1 root root 16384 Apr 9 10:21 info.db
-rw-r--r-- 1 root root 32768 Apr 9 10:21 info.db-shm
-rw-r--r-- 1 root root 32992 Apr 9 10:21 info.db-wal
-rw-r--r-- 1 root root 24576 Apr 9 10:21 notifications.db
-rw-r--r-- 1 root root 32768 Apr 9 10:21 notifications.db-shm
-rw-r--r-- 1 root root 32992 Apr 9 10:21 notifications.db-wal
-rw-r--r-- 1 root root 303411200 Apr 11 19:19 orders.db
-rw-r--r-- 1 root root 98304 Apr 11 19:19 orders.db-shm
-rw-r--r-- 1 root root 44479552 Apr 11 19:19 orders.db-wal
-rw-r--r-- 1 root root 36864 Apr 9 10:21 piece_expiration.db
-rw-r--r-- 1 root root 32768 Apr 9 10:21 piece_expiration.db-shm
-rw-r--r-- 1 root root 32992 Apr 9 10:21 piece_expiration.db-wal
-rw-r--r-- 1 root root 24576 Apr 9 10:21 pieceinfo.db
-rw-r--r-- 1 root root 32768 Apr 9 10:21 pieceinfo.db-shm
-rw-r--r-- 1 root root 32992 Apr 9 10:21 pieceinfo.db-wal
-rw-r--r-- 1 root root 24576 Apr 11 19:25 piece_spaced_used.db
-rw-r--r-- 1 root root 20480 Apr 11 19:25 reputation.db
-rw-r--r-- 1 root root 32768 Apr 9 10:21 satellites.db
-rw-r--r-- 1 root root 32768 Apr 9 10:21 satellites.db-shm
-rw-r--r-- 1 root root 32992 Apr 9 10:21 satellites.db-wal
-rw-r--r-- 1 root root 86016 Apr 11 19:25 storage_usage.db
drwx------ 2 root root 110592 Apr 11 19:23 temp
drwx------ 7 root root 4096 Feb 17 10:58 trash
-rw-r--r-- 1 root root 43438080 Apr 11 19:18 used_serial.db
-rw-r--r-- 1 root root 229376 Apr 11 19:18 used_serial.db-shm
-rw-r--r-- 1 root root 103523272 Apr 11 19:18 used_serial.db-wal
root@XXX:/data/storj-storage/storage
I followed the procedure and fixed the integrity of bandwidth.db and used_serial.db
I restarted the node and it seems OK now.
The system hasnt been restarted for months, maybe it got corrupted during an update.
Thanks !