Recreating node after disqualification

So my node was disqualified (due to my own fault). I tried getting new token, got it, but node still doesn’t run. What should I do? Or is it permanent disqualification per IP?

You need to delete old identity and data if you want to start from scratch. Your identity is tied to your data so you can’t use new identity for old data.

No

Also make sure you don’t repeat said mistake for your new node.

1 Like

I DID delete the old one. Restarted everything from docker to Linux. Nothing helps…

Just make sure you deleted the whole folder with blobs and database files in them. What does your log show ?

I completely formatted HDD, unmounted, mounted again. I’ll check tomorrow and reply here.

So it starts and runs now, just earnings.py shows DQ for all sattelites. I know it’s 3rd party tool but still… Should I wait? Is there any other way to check if node is correctly operational? I have no way to use GUI dashboard, installed node runs on Linux without GUI.

Check web dashboard by visiting 127.0.0.1:14002 in Chrome or firefox

I can’t :slight_smile: as I mentioned, I have no possibility to use GUI. Linux is only with CLI.

You can open web dashboard remotely
https://documentation.storj.io/resources/faq/how-to-remote-access-the-web-dashboard#install-a-ssh-client-on-your-device

Thanks for the tip :slight_smile: This is what I see:

Show me result your docker run command and remove any personal info from it.

Just restarted the node:

[root@localhost ~]# docker logs storagenode 2>&1 | grep -v Operator
2020-05-06T12:25:32.869Z        INFO    Configuration loaded    {"Location": "/app/config/config.yaml"}
2020-05-06T12:25:32.918Z        INFO    tracing disabled
2020-05-06T12:25:38.301Z        INFO    db.migration    Database Version        {"version": 36}
2020-05-06T12:25:39.063Z        INFO    preflight:localtime     start checking local system clock with trusted satellites' system clock.
2020-05-06T12:25:40.125Z        INFO    preflight:localtime     local system clock is in sync with trusted satellites' system clock.
2020-05-06T12:25:40.128Z        INFO    bandwidth       Performing bandwidth usage rollups
2020-05-06T12:25:40.139Z        INFO    Node 15hQPfQjpvjXhWVuEcJBmcAEtA6df3SsgW1Uba15AFyWZxf6EL started
2020-05-06T12:25:40.139Z        INFO    Public server started on [::]:28967
2020-05-06T12:25:40.139Z        INFO    Private server started on 127.0.0.1:7778
2020-05-06T12:25:40.166Z        INFO    trust   Scheduling next refresh {"after": "9h1m4.39887855s"}
2020-05-06T12:25:44.879Z        INFO    piecestore      upload started  {"Piece ID": "MM5SGBQ6S5W34FSKZZZKT4W3EBUQ5ALACY2DA4IYYXEQ42W2NXSQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 798826333946}
2020-05-06T12:25:47.197Z        INFO    piecestore      upload canceled {"Piece ID": "MM5SGBQ6S5W34FSKZZZKT4W3EBUQ5ALACY2DA4IYYXEQ42W2NXSQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-06T12:25:47.901Z        INFO    piecestore      upload started  {"Piece ID": "XTZQBMH2NIKMITWZES7TRDVSNPI3IWJX7F4UTQJW5V4GIA7ORTVA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 798824014074}
2020-05-06T12:25:51.486Z        INFO    piecestore      upload canceled {"Piece ID": "XTZQBMH2NIKMITWZES7TRDVSNPI3IWJX7F4UTQJW5V4GIA7ORTVA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-06T12:26:38.716Z        INFO    piecestore      upload started  {"Piece ID": "BYTCENTLG4FUYU5CKPW7ST6AZGFJ4PHYLUGDU5FCF74HN66HT33Q", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 798821694202}
2020-05-06T12:26:40.814Z        INFO    piecestore      upload canceled {"Piece ID": "BYTCENTLG4FUYU5CKPW7ST6AZGFJ4PHYLUGDU5FCF74HN66HT33Q", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-06T12:26:48.184Z        INFO    piecestore      upload started  {"Piece ID": "Z6BA5EBP4OZFQNT2MFSG7H7FNBHKYVIPLMWL6D6BLRBJKBQER3CQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 798819374330}
2020-05-06T12:26:51.059Z        INFO    piecestore      upload started  {"Piece ID": "A5E6LLJNSSDTGDVSJTZEEB5L7DXPMK56TOHUZJXQYXSAN4US6XCA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 798819374330}
2020-05-06T12:26:51.556Z        INFO    piecestore      upload canceled {"Piece ID": "Z6BA5EBP4OZFQNT2MFSG7H7FNBHKYVIPLMWL6D6BLRBJKBQER3CQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-06T12:26:52.124Z        INFO    piecestore      upload canceled {"Piece ID": "A5E6LLJNSSDTGDVSJTZEEB5L7DXPMK56TOHUZJXQYXSAN4US6XCA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-06T12:27:44.436Z        INFO    orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       sending {"count": 5}
2020-05-06T12:27:44.437Z        INFO    orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB      sending {"count": 96}
2020-05-06T12:27:44.669Z        INFO    orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       finished
2020-05-06T12:27:44.763Z        INFO    orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB      finished
2020-05-06T12:28:19.956Z        INFO    piecestore      upload started  {"Piece ID": "IQKU7PQ4LOZ3U3UWVAFPJCA46NOJWSE7WA6UXZM5MUSQGBUUTMTA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 798814734586}
2020-05-06T12:28:21.722Z        INFO    piecestore      upload canceled {"Piece ID": "IQKU7PQ4LOZ3U3UWVAFPJCA46NOJWSE7WA6UXZM5MUSQGBUUTMTA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-06T12:28:49.084Z        INFO    piecestore      upload started  {"Piece ID": "J6PJSWMEHPPLAQKLBNWRPTLLVZ7WQWJK4DNCXZSVQK2F2IT32Y4Q", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 798812414714}
2020-05-06T12:28:49.968Z        INFO    piecestore      upload started  {"Piece ID": "INUBPWYAYPJOMSWAL53E67EN6OQAZWTRBSIV2HIKLBXU5D2BLNKA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 798812414714}
2020-05-06T12:28:51.882Z        INFO    piecestore      upload canceled {"Piece ID": "J6PJSWMEHPPLAQKLBNWRPTLLVZ7WQWJK4DNCXZSVQK2F2IT32Y4Q", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-06T12:28:52.574Z        INFO    piecestore      upload canceled {"Piece ID": "INUBPWYAYPJOMSWAL53E67EN6OQAZWTRBSIV2HIKLBXU5D2BLNKA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-06T12:30:01.668Z        INFO    piecestore      upload started  {"Piece ID": "X56CY66GCNWII7ZXKSAV4LP6B4LULYS3FD4BGZVRLWNZP6MZ7ZKQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 798807774970}
2020-05-06T12:30:03.687Z        INFO    piecestore      upload started  {"Piece ID": "JKMLN4JZGCAP2OJBBEPKR5LAX36GGFACLCGNLPGJ2P3ZDEQYWAFQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 798807774970}
2020-05-06T12:30:04.137Z        INFO    piecestore      upload canceled {"Piece ID": "X56CY66GCNWII7ZXKSAV4LP6B4LULYS3FD4BGZVRLWNZP6MZ7ZKQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-06T12:30:10.673Z        INFO    piecestore      upload canceled {"Piece ID": "JKMLN4JZGCAP2OJBBEPKR5LAX36GGFACLCGNLPGJ2P3ZDEQYWAFQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-06T12:30:26.488Z        INFO    piecestore      upload started  {"Piece ID": "ZLMYRQJAOXLLOIIRG373P2K3OJPHGNCNY6MVWDECUV2F2E4FHY7Q", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 798803135226}
2020-05-06T12:30:30.007Z        INFO    piecestore      upload canceled {"Piece ID": "ZLMYRQJAOXLLOIIRG373P2K3OJPHGNCNY6MVWDECUV2F2E4FHY7Q", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-06T12:30:50.067Z        INFO    piecestore      download started        {"Piece ID": "X56CY66GCNWII7ZXKSAV4LP6B4LULYS3FD4BGZVRLWNZP6MZ7ZKQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET"}
2020-05-06T12:30:54.694Z        INFO    piecestore      downloaded      {"Piece ID": "X56CY66GCNWII7ZXKSAV4LP6B4LULYS3FD4BGZVRLWNZP6MZ7ZKQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET"}
2020-05-06T12:30:58.074Z        INFO    piecestore      upload started  {"Piece ID": "BVLDI3VXC2WONARCBPWXC34QEKX7ZWFK7U2S3RNAFOQWFXNQWXHQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 798800815354}
2020-05-06T12:31:04.792Z        INFO    piecestore      upload canceled {"Piece ID": "BVLDI3VXC2WONARCBPWXC34QEKX7ZWFK7U2S3RNAFOQWFXNQWXHQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
[root@localhost ~]#

Your node is DQed on 4 satellites. Check your log for audit failures. Search for download failed and GET_AUDIT

PS: You won’t find anything if your log isn’t redirected to a file. Every time you restart it deletes the log

But general idea is - why? That is fresh auth token :slight_smile:

Show your docker run command and remove any personal info.

Also show result of

df -h

[root@localhost ~]# egrep -v "WALL|EM|ADDR" storj.sh
docker run -d --restart unless-stopped -p 28967:28967 \
    -p 127.0.0.1:14002:14002 \
    -e STORAGE="845GB" \
    -e BANDWIDTH="60TB" \
    --mount type=bind,source="/root/.local/share/storj/identity/storagenode/",destination=/app/identity \
    --mount type=bind,source="/storj",destination=/app/config \
    --name storagenode storjlabs/storagenode:beta
[root@localhost ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 479M     0  479M   0% /dev
tmpfs                    491M     0  491M   0% /dev/shm
tmpfs                    491M  7.1M  484M   2% /run
tmpfs                    491M     0  491M   0% /sys/fs/cgroup
/dev/mapper/centos-root   50G  3.8G   47G   8% /
/dev/mapper/centos-home  180G   34M  180G   1% /home
/dev/sdb1                932G  1.4G  930G   1% /storj
/dev/sda1               1014M  199M  816M  20% /boot
tmpfs                     99M     0   99M   0% /run/user/0
overlay                   50G  3.8G   47G   8% /var/lib/docker/overlay2/d1e160930199acfe5c4ec5889b270e5ff839fe3e55cda1ddcdbaecab7644bd4d/merged
overlay                   50G  3.8G   47G   8% /var/lib/docker/overlay2/14351afad8c078a5c26fb8d5e1ba70f10f223796e3801e348a570c72d88546e2/merged
[root@localhost ~]#

I had to recreate a sno after a partition got damaged after only 20 days of activity. I requested a new token and redid the correct procedure with a new identity and cleaned everything. Now I risk that I disqualify this new node in your opinion?

Did you follow static mount point ? This is shown here

Did you figure out why your partition got damaged ?

We all learn from our mistakes but the important part here is learning from it. If you make sure your previous mistake won’t be repeated then you are good.

You have to make sure you aren’t failing audits. Currently DQ due to low uptime is not implemented so your node probably lost data during partition loss and therefore failed audits.

Use monitoring options like uptimerobot.com to get email notification when your node is offline. Check logs for download failed and 'GET_AUDIT` to make sure you aren’t failing audits.

The disk was very old. Now i bought a new HP Proliant DL380E G8 with 14 bay with 14 hgst 3TB SAS for new SNO and P822 Controller with 2GB Cache. Now the errors are egual to zero or i hope. One question: I requested the token with the same email as the previous one. Maybe here I made the mistake. Quite right? Do I have to request it with a new email? Because it’s been 24 hours that it’s online and it’s not downloading anything. He basically downloaded 5GB in total between all the satellites. And from Saltlake = 0.