Want the results now that it’s going or after it quits again?
The web dashboard and dashboard CLI are NOT working when it stops.
The results are too big to post. Is linking to a pastebin-type site ok?
Then it’s perhaps the problem
docker container prune -f
In the bash or PowerShell:
docker rm --force $(docker ps -qf ancestor=storjlabs/watchtower)
Then run the watchtower by guide: https://documentation.storj.io/setup/cli/software-updates#automatic-updates
After that, please, show the output of
docker ps -a
storjlabs/storagenode:latest “/entrypoint” 46 hours ago Up 13 hours 0.0.0.0:14002->14002/tcp, 0.0.0.0:28967->28967/tcp storj
If you were talking to me @Alexey…
Mine’s using docker-compose so I don’t use the watchtower.
My current version is 1.14.7
Please see here for the debug while it’s NOT working:
Just curious - are you able to see the data at /api/sno or at /api/sno/satellite/<satellite ID>?
In the Pastebin output you provided, I do not see evidence of many failed attempts at loading the dashboard - did you attempt to load the dashboard unsuccessfully between starting the node and copying the output of /mon/funcs? I ask because the most relevant function names for the dashboards are:
GUI:
GetDashboardData or StorageNode
CLI:
Dashboard or getDashboardData
I do not see any function calls for the CLI in your output - I am assuming you have issues with the GUI. However, for the GUI function (GetDashboardData), I see
[8230671656014584756] storj.io/storj/storagenode/console.(*Service).GetDashboardData
parents: 5576563438954870221
current: 0, highwater: 1, success: 2, errors: 0, panics: 0
...
[5576563438954870221] storj.io/storj/storagenode/console/consoleapi.(*StorageNode).StorageNode
parents: entry
current: 0, highwater: 1, success: 2, errors: 0, panics: 0
In other words, no failures. Could you clarify if you did attempt and fail to load the dashboard at least one time between restarting the node and producing the output you provided?
Thank you. I really appreciate the time you have put into helping us get to the bottom of this.
duh on my part. I’ll try to load it up here when it fails and dump you a new pastebin.
Thanks for taking a look at this odd problem!
1 Like
Thank you - I intend to look at this in the next couple hours, or tomorrow if I can’t get to it today.
1 Like
/api/sno appeared. JSON I believe?
I tried this (not sure it’s right) and it just hangs:
/api/sno/satellite/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE saltlake
JSON is right. Were you able to load the dashboard when you got the JSON?
if I understood correctly you tried to load this in the browser
http://<host>:<port>/api/sno/satellite/1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE'
and saltlake was not part of the URL but you mostly wanted to indicate that the node ID corresponds to Saltlake sattellite.
correct?
could try with us-central-1 this one http://<host>:<port>/api/sno/satellite/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S?
And also try to open the browser dev-console and load the web dashboard and check if you see any error in the console pane. In such a case, please paste them or a screenshot here.
Thanks for your collaboration.
That’s correct, saltlake was just one I was trying to connect to. Both of them just spin and spin when the dashboard fails to load.
The console in dev shows nothing when it’s trying to load… almost like the site doesn’t exist.
Network is showing this:
http://192.168.1.15:14002/static/dist/js/app_de1d649d449345eb0742.js is the “pending” satellites url
I would say that web server is hanging for some unknown reason, considering that you said
I too am getting this. Activity is going in the logs. Restarting the docker image fixes it for about 6 hours then quits again.
And now you showed that.
Can you try to do the request with cURL? You can easily get the cURL syntax with right click on the satellites endpoint, or at least you can if it doesn’t hang, not sure if you will be able when it hangs, but it’s worth to try it.
In case that you don’t know how to do it, see below my screenshot
Once you have it, can you execute it in your terminal?
If you don’t have cURL installed in your machine you can use this docker image Docker
On the other hand, can you share the docker container logs when you are getting that issue?
curl 'http://192.168.1.15:14002/api/sno/satellites' \
> -H 'Referer: http://192.168.1.15:14002/' \
> -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36' \
> -H 'DNT: 1' \
> --compressed
Then blank… no response. I’ll leave it running to see if curl shows an error.
Log files are showing activity:
| 2020-10-16T14:41:31.789Z | INFO | piecestore | download started | {“Piece ID”: “5MCGETOQEEIOFZU32OGO2TNC2MNYCDMLEANZWYQ4SWNJJ5HENEMA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “GET”} |
|---|---|---|---|---|
| 2020-10-16T14:41:31.843Z | INFO | piecestore | downloaded | {“Piece ID”: “5MCGETOQEEIOFZU32OGO2TNC2MNYCDMLEANZWYQ4SWNJJ5HENEMA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “GET”} |
| 2020-10-16T14:41:32.132Z | INFO | piecestore | upload started | {“Piece ID”: “EYKV6OY5OICM5U4VJDSLOL3RQMJXILRBLRJ5MHRLNGBC4YHBM7UA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “PUT”, “Available Space”: 3512794947541} |
| 2020-10-16T14:41:32.922Z | INFO | piecestore | upload started | {“Piece ID”: “ZER7PNKSFSJSLEQODMHRUYANT2S2RIBXJDZXHEKQ4AFNLAORYMWA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”, “Available Space”: 3512794946261} |
| 2020-10-16T14:41:32.974Z | INFO | piecestore | upload started | {“Piece ID”: “YPIKUWLWCOPNWMBNNDQATCSEEVV2KW7DRUMZKP54DKBAGZ3WMD6A”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”, “Available Space”: 3512794946261} |
| 2020-10-16T14:41:33.756Z | INFO | piecestore | download started | {“Piece ID”: “EYKV6OY5OICM5U4VJDSLOL3RQMJXILRBLRJ5MHRLNGBC4YHBM7UA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “GET”} |
| 2020-10-16T14:41:33.956Z | INFO | piecestore | upload started | {“Piece ID”: “5NTG7WSAGT4JFKZGAWQW2KZNVIKLYMQKXECTQIQY5TFJU5ERMDNQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 3512794946261} |
| 2020-10-16T14:41:34.696Z | INFO | piecestore | download started | {“Piece ID”: “EYKV6OY5OICM5U4VJDSLOL3RQMJXILRBLRJ5MHRLNGBC4YHBM7UA”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “GET”} |
| 2020-10-16T14:41:35.637Z | INFO | piecestore | upload started | {“Piece ID”: “RST5Q4QDGFVT3LTYLIJ3UZ6PANTHPCNHHUHGANUOD6OSIRZJ3XKQ”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “PUT”, “Available Space”: 3512790943701} |
| 2020-10-16T14:41:36.801Z | INFO | piecestore | download started | {“Piece ID”: “RST5Q4QDGFVT3LTYLIJ3UZ6PANTHPCNHHUHGANUOD6OSIRZJ3XKQ”, “Satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “Action”: “GET”} |
System removed my post for spam so hopefully it’ll show back up here. 
curl response: curl: (52) Empty reply from server
The same thing has been happening to me for the last few days. It seemed a little better after a restart but soon goes unresponsive. My bandwidth also dropped to nothing, and I now have missed audits. I manually updated the node and I’ll closely monitor for a few days.
Hello @nagolmas,
Welcome to the forum!
What the version was before update? Is it a binary or docker? What is your OS?
What is dashboard - CLI or web? If web - is your service running?
v1.14.7 before and after removing and pulling the latest image. Running on docker on ubuntu 18.04.5. Web dashboard, and the service is running. It took forever to load, like a few minutes if it ever did load.