Piecemigrate:chore couldn't migrate

[donald.m.motsinger]

I migrated a storagenode to hashtable and it finished apart from 1 piece. There seems to be a bad sector on this disk as I get an input/output error when I try to copy this piece somewhere else.

Now the node keeps trying to migrate this piece again and again every 10 minutes.

2025-02-19T10:11:13Z    INFO    piecemigrate:chore      couldn't migrate        {"Process": "storagenode", "error": "while copying the piece: pieces error: read config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/cu/7l36frf6lc37dbvozudjyqpzxxrwgwkylt5kaea3fbfzk6a4jq.sj1: input/output error", "errorVerbose": "while copying the piece: pieces error: read config/storage/blobs/ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa/cu/7l36frf6lc37dbvozudjyqpzxxrwgwkylt5kaea3fbfzk6a4jq.sj1: input/output error\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).copyPiece:379\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).migrateOne:342\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).processQueue:260\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).Run.func2:167\n\tstorj.io/common/errs2.(*Group).Go.func1:23", "sat": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "id": "CU7L36FRF6LC37DBVOZUDJYQPZXXRWGWKYLT5KAEA3FBFZK6A4JQ"}

Should I try to write zeroes to this sector to stop the migration attempts? The pieces seems to be lost anyway.

[Roxor]

I’d ‘fsck -y’ the disk first and see what it can take care of.

[ItsHass]

@donald.m.motsinger

I migrated a storagenode to hashtable

hm… is this something we’re supposed to be doing?

[donald.m.motsinger]

@Roxor

I’d ‘fsck -y’ the disk first and see what it can take care of.

I forced a fsck but it doesn’t fix this piece.

@ItsHass

hm… is this something we’re supposed to be doing?

Not yet. You can read about it here.

[Tech Preview] Hashstore backend for storage nodes

Do you remember that one big customer we simulated on SLC? If not here is a short recap. That customer uploads a high amount of small and medium size files with a short TTL. It turns out storage nodes can handle the initial uploads but things gets expensive when the TTL kicks in. The storage nodes simply needs too much time deleting such an high amount of pieces. The solution for that problem is the new hashstore backend. Instead of storing all pieces as a file on disk the hashstore backend com…

[arrogantrabbit]

@donald.m.motsinger

Should I try to write zeroes to this sector

Just delete the file.