WARN hashstore mismatched log tail

Node Operators troubleshooting
,
1

One more question, what this mean and how to fix?

2026-03-05T00:05:15+03:00 WARN hashstore mismatched log tail {"process": "storagenode", "satellite": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "store": "s1", "id": 2433, "path": "config/storage/hashstore/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs/s1/81/log-0000000000000981-00005058", "table": [{"Key":"WLWJCVYFAOQNB24SPXY5R2NXDR7Y74SP5IRDLKXQ3DMY7FTV47FQ","Offset":6959808,"Log":2433,"Length":2319872,"Created":20476,"Expires":41136},{"Key":"H7SZP3CQXMYW6N7KQKKJA5KF3NWN4ELNZJ5BPN7XM6PBE62MKD2A","Offset":2319936,"Log":2433,"Length":2319872,"Created":20476,"Expires":41136},{"Key":"LQVHN26REMAPH5DJHHQHQCF5X64NABEVC5ZEFZI52JG53A2SMHHQ","Offset":0,"Log":2433,"Length":2319872,"Created":20476,"Expires":41136},{"Key":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA","Offset":0,"Log":0,"Length":0,"Created":0,"Expires":0},{"Key":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA","Offset":0,"Log":0,"Length":0,"Created":0,"Expires":0}], "log": [{"Key":"WLWJCVYFAOQNB24SPXY5R2NXDR7Y74SP5IRDLKXQ3DMY7FTV47FQ","Offset":6959808,"Log":2433,"Length":2319872,"Created":20476,"Expires":41136},{"Key":"2PYBIOF4HSRAAEQKFQ6UVQGPYM7AS4P4BTGM23KV6D2TSRCQ4LWQ","Offset":4639872,"Log":2433,"Length":2319872,"Created":20476,"Expires":41136},{"Key":"H7SZP3CQXMYW6N7KQKKJA5KF3NWN4ELNZJ5BPN7XM6PBE62MKD2A","Offset":2319936,"Log":2433,"Length":2319872,"Created":20476,"Expires":41136},{"Key":"LQVHN26REMAPH5DJHHQHQCF5X64NABEVC5ZEFZI52JG53A2SMHHQ","Offset":0,"Log":2433,"Length":2319872,"Created":20476,"Expires":41136},{"Key":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA","Offset":0,"Log":0,"Length":0,"Created":0,"Expires":0}]}

PS: Will answer on my own question: warnings gone after second restart, clean start in the logs. So i think second restart not needed, it (if any) was fixed on first start after update.

2

Yes, the restart was not needed, this issue should be fixed on start (there is fsck for hashstore, so it checked, printed and fixed this minor issue).

3

Same here.
After the “Mismatched log tail” error and accordingly with “storagenode.exe” on Windows, it stops the service, restarts but the error appears again and it´s looping!
After the second service restart, again…and there´s no third as the third Windows service action is “Take no action”.

4

This is a warning, not error. It shouldn’t stop the service.
Could you please search for the FATAL or Unrecoverable error in your logs?

5

For some reason it´s working now :face_with_raised_eyebrow:

6

Of course it should. There is a hashstore fsck, which is started on restart, it should check and fix issues.

7

I’m having the same issue right now with one of my nodes.

Here is my docker compose file:

services:

  storj4:
    image: "storjlabs/storagenode:${IMAGE_TAG}"
    container_name: storj4
    restart: unless-stopped
    stop_grace_period: 300s
    user: "${UID}:${GID}"
    ports:
      - "28970:28967/tcp"
      - "28970:28967/udp"
      - "14004:14002"
    environment:
      WALLET: "${WALLET}"
      EMAIL: "${EMAIL}"
      ADDRESS: "${ADDRESS_HOST}:28970"
      STORAGE: "3.3TiB"
      TZ: "America/Boston"
    volumes:
      - /mnt/storj4/storj4_identity:/app/identity
      - /mnt/storj4/storj4_data:/app/config

It was working well. Then, I stopped all the containers (sudo docker compose down), made a few changes on another container in the docker compose file, then started everything back again (sudo docker compose up -d). After that, the container 4 keeps displaying this in the log, and I can’t see any transaction going through the node, neither I can access the dashboard:

Logs

2026-07-07T23:54:40-04:00 WARN hashstore mismatched log tail {“process”: “storagenode”, “satellite”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “store”: “s0”, “id”: 2732, “path”: “config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/ac/log-0000000000000aac-000050dd”, “table”: [{“Key”:“4SSLKMCQZNALFG4HE7WCI2LUVHPBE4QXDFNXQKP42NBTKLJCHENQ”,“Offset”:0,“Log”:2732,“Length”:327168,“Created”:20610,“Expires”:41402},{“Key”:“AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA”,“Offset”:0,“Log”:0,“Length”:0,“Created”:0,“Expires”:0},{“Key”:“AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA”,“Offset”:0,“Log”:0,“Length”:0,“Created”:0,“Expires”:0},{“Key”:“AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA”,“Offset”:0,“Log”:0,“Length”:0,“Created”:0,“Expires”:0},{“Key”:“AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA”,“Offset”:0,“Log”:0,“Length”:0,“Created”:0,“Expires”:0}], “log”: [{“Key”:“JZRN6GOUJUAYHCJSS35X6LDV4Z4QWGHT4HG5CLHQSRUZMMQZZVGQ”,“Offset”:327232,“Log”:2732,“Length”:2560,“Created”:20610,“Expires”:41402},{“Key”:“4SSLKMCQZNALFG4HE7WCI2LUVHPBE4QXDFNXQKP42NBTKLJCHENQ”,“Offset”:0,“Log”:2732,“Length”:327168,“Created”:20610,“Expires”:41402},{“Key”:“AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA”,“Offset”:0,“Log”:0,“Length”:0,“Created”:0,“Expires”:0},{“Key”:“AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA”,“Offset”:0,“Log”:0,“Length”:0,“Created”:0,“Expires”:0},{“Key”:“AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA”,“Offset”:0,“Log”:0,“Length”:0,“Created”:0,“Expires”:0}]}

8

Hello @smokemodel,
Welcome back!