Piecemigrate:chore couldn’t migrate {“error”: "duplicate pieces differ: headers don’t match

Node Operators troubleshooting
,
1

Hello, I’m seeing a lot of these messages in the log:

The blobs are no longer deleted after the 10-minute migration cycle.

Is this something to be worried about?

piecemigrate:chore couldn’t migrate {“error”: "duplicate pieces differ: headers don’t match; new=creation_time:<seconds:-62135596800 > order_limit:<serial_number:"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000" satellite_id:"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000" uplink_public_key:"" storage_node_id:"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000" piece_id:"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000" piece_expiration:<seconds:-62135596800 > order_expiration:<seconds:-62135596800 > order_creation:<seconds:-62135596800 > > != old=format_version:FORMAT_V1FORMAT_V1 hash:"J\340\305\307\004\230M\225\331\316x[\347\344\341V\215\357\345TomW\361\tm\222\2441\202\262)" hash_algorithm:BLAKE3BLAKE3 creation_time:<seconds:1738512128 nanos:741401965 > signature:"\370\\\353\254\000\343\030#\235\023\255\372\023\\017\\260\\223Qa\\207\\265\\241Gh\\240\\374\\273\\232\\003!}\\353&;\\231\\321?E\\001\\231\\225\\007\\357\\241RvY\\301\\202odM:\\317\\304j\\266yU\\312~\\254\\027\\005\" order_limit:<serial_number:\"\\000\\000\\000\\000g\\240\\350\\200\\036\\315n\\301\\010\\027\\262>\" satellite_id:\"\\257,B\\000>\\374\\202j\\2646\\037s\\371\\330\\220\\224!F\\376\\016\\276\\200g\\206\\370\\347\\031\\010\\000\\000\\000\\000\" uplink_public_key:\"F\\272\\376\\225\\002&\\354\\373\\225b\\364\\203\\017MW\\374\\256[\\333\\322\\022\\241i~X$eX\\252\\270\\022\\252\" storage_node_id:\"\\235\\266w\\315\\240\\n\\374\\007\\272\\326\\261\\007\\204\\037n\\317$<\\001\\024\\232\\210\\306\\301'Z\\234\000\000\000\000" piece_id:"\237\201\263o\317\352Mt\337\0269\211\357\354[W\223\331\033\373\266\250\326\311q\327\201\376\257\341\365\330" limit:2319360 action:PUTPUT piece_expiration:<seconds:-62135596800 > order_expiration:<seconds:1738598528 nanos:741401965 > order_creation:<seconds:1738512128 nanos:741401965 > encrypted_metadata_key_id:"l\320\375\325h\235\264\314" encrypted_metadata:"3<\"1\344\223\307\016\005B\313\234p\340.\230\264tN\256\330\274\254\310;\370b\275;\360ZG\255_\030y\360\230S\301\026_6\361\254\013\244{\000!\316\t\247\310\3551\266\036ku" satellite_signature:"0D\002 @\226\026\375\267C\037\263>\232\254HOi[\210\022\300\311!\004R\326$f\325\337\272?\376j\340\002 \037\016d%\2622\232o\026\376R\212\260K\3777\027\254\177\207\324P\306\014\027\357p\247\363\301#\246" > ", “errorVerbose”: “duplicate pieces differ: headers don’t match; new=creation_time:<seconds:-62135596800 > order_limit:<serial_number:"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000" satellite_id:"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000" uplink_public_key:"" storage_node_id:"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000" piece_id:"\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000" piece_expiration:<seconds:-62135596800 > order_expiration:<seconds:-62135596800 > order_creation:<seconds:-62135596800 > > != old=format_version:FORMAT_V1FORMAT_V1 hash:"J\340\305\307\004\230M\225\331\316x[\347\344\341V\215\357\345TomW\361\tm\222\2441\202\262)" hash_algorithm:BLAKE3BLAKE3 creation_time:<seconds:1738512128 nanos:741401965 > signature:"\370\\\353\254\000\343\030#\235\023\255\372\023\\017\\260\\223Qa\\207\\265\\241Gh\\240\\374\\273\\232\\003!}\\353&;\\231\\321?E\\001\\231\\225\\007\\357\\241RvY\\301\\202odM:\\317\\304j\\266yU\\312~\\254\\027\\005\" order_limit:<serial_number:\"\\000\\000\\000\\000g\\240\\350\\200\\036\\315n\\301\\010\\027\\262>\" satellite_id:\"\\257,B\\000>\\374\\202j\\2646\\037s\\371\\330\\220\\224!F\\376\\016\\276\\200g\\206\\370\\347\\031\\010\\000\\000\\000\\000\" uplink_public_key:\"F\\272\\376\\225\\002&\\354\\373\\225b\\364\\203\\017MW\\374\\256[\\333\\322\\022\\241i~X$eX\\252\\270\\022\\252\" storage_node_id:\"\\235\\266w\\315\\240\\n\\374\\007\\272\\326\\261\\007\\204\\037n\\317$<\\001\\024\\232\\210\\306\\301'Z\\234\000\000\000\000" piece_id:"\237\201\263o\317\352Mt\337\0269\211\357\354[W\223\331\033\373\266\250\326\311q\327\201\376\257\341\365\330" limit:2319360 action:PUTPUT piece_expiration:<seconds:-62135596800 > order_expiration:<seconds:1738598528 nanos:741401965 > order_creation:<seconds:1738512128 nanos:741401965 > encrypted_metadata_key_id:"l\320\375\325h\235\264\314" encrypted_metadata:"3<\"1\344\223\307\016\005B\313\234p\340.\230\264tN\256\330\274\254\310;\370b\275;\360ZG\255_\030y\360\230S\301\026_6\361\254\013\244{\000!\316\t\247\310\3551\266\036ku" satellite_signature:"0D\002 @\226\026\375\267C\037\263>\232\254HOi[\210\022\300\311!\004R\326$f\325\337\272?\376j\340\002 \037\016d%\2622\232o\026\376R\212\260K\3777\027\254\177\207\324P\306\014\027\357p\247\363\301#\246" > \n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).copyPiece:395\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).migrateOne:342\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).processQueue:260\n\tstorj.io/storj/storagenode/piecemigrate.(*Chore).Run.func2:167\n\tstorj.io/common/errs2.(*Group).Go.func1:23”, “sat”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “id”: “T6A3G36P5JGXJXYWHGE673C3K6J5SG73W2UNNSLR26A75L7B6XMA”}

2

Asked the team to take a look, but for me it looks like a corrupted piece.

3

After half a day i have over 1GB of them unable to delete out of the blobfolder after migration.

4

can you upload the hashtbl files for the satellite 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs somewhere? they should be named something like hashstore/<satellite>/s0/meta/hashtbl-<number> and hashstore/<satellite>/s1/meta/hashtbl-<number>. this would help me get a lead on what is going on. hopefully they aren’t too big and they should compress to about half size. let me know if you run into any challenges.