Support for problematic node

[support26]

@Alexey

This is weird, the command is valid. >> usually mean that the shell expecting something. Seems you may pressed the Shift-Enter?
Please try to press Enter in the >> prompt

Just responds with prompts

>>
>>
>>
>>
>>

@Alexey

You may copy only the last few lines and then you should be able to post them.

From which command?

[nerdatwork]

Could you make a video of executing the command and getting the output ? It would help understand the issue better.

[pangolin]

@support26

I was directed to by the support forum. Perhaps incorrectly.

I told you the options, stay with 1.05 until auto update happens or do a manual update. It was your decision not to wait.

[support26]

@pangolin

I told you the options, stay with 1.05 until auto update happens or do a manual update. It was your decision not to wait.

Not really helpful in getting to aresolution of the issue(unfortunately, seems a common occurrence here), however, I was referring to the databases deletion.

[pangolin]

SNOs are not customers. So any support is voluntarily…

[support26]

What does that have to do with this forum presenting to be a supportive community? (I am not sure if you were responding to me).

[Julio]

haha, miraculous set of circumstances here.

Stuck key? sxs corruption? dism + sfc should fix that. ummm …

Try running Powershell as administrator, and then re-trying that command Support22/23/24/25/26. You may have set that disk up with restricted user permissions not knowing it.

also there is a bug in windows which can do that (ie: command prompt input useless)- can’t recall if restart of explorer will fix it or re-start of DWM service will do the trick… but a re-boot certainly will, until it resurfaces - generally it’s prolly by bad overtaxed windows mem management.

2 cents

[Julio]

Maybe you should run a simple stress test on that drive. Waaaay back in this thread, didn’t I see a 1 minute timeout? Crystal disk mark that drive or mount point, see if you have less than .67 megs/s on the default bottom 4k random read/write, or what iops there really are… let us see those results.

[Alexey]

@support26

From which command?

This one:

cat "$env:ProgramFiles\Storj\Storage Node\storagenode.log" | sls "database|filewalker" | select -last 10

Please use a button on the line above, seems you tried to copy with a mouse and didn’t copy all commands. The button is appearing when you move your mouse over the line of code.

[support26]

@Julio

Try running Powershell as administrator, and then re-trying that command Support22/23/24/25/26. You may have set that disk up with restricted user permissions not knowing it.

Powershell is always run as administrator (also, this has never come up before this issue arose)

@Julio

also there is a bug in windows which can do that (ie: command prompt input useless)- can’t recall if restart of explorer will fix it or re-start of DWM service will do the trick… but a re-boot certainly will, until it resurfaces - generally it’s prolly by bad overtaxed windows mem management.

Does not seem to be of that regard - as whenever the dashboard access is restricetd - I have to restart the system anyhow. Nothing that indicates overtaxed mem - i ihave only ever seen the memory useage at a max of 8GB (currently at 6). So, unlikely, I suspect.

[support26]

@Julio

Maybe you should run a simple stress test on that drive. Waaaay back in this thread, didn’t I see a 1 minute timeout? Crystal disk mark that drive or mount point, see if you have less than .67 megs/s on the default bottom 4k random read/write, or what iops there really are… let us see those results.

I don’t know if there is/was such a timeout. I can easily enough do that - I’ve got HDTune at the moment and the results That(Crystal disk mark) should be run with the node service stopped or started?

[support26]

[quote=“Alexey, post:29, topic:27179, full:true”]

@support26

From which command?

This one:

cat "$env:ProgramFiles\Storj\Storage Node\storagenode.log" | sls "database|filewalker" | select -last 10

2024-07-25T09:34:16-04:00       INFO    db.migration    Database Version        {"version": 57}
2024-07-25T09:48:48-04:00       ERROR   piecestore:cache        error getting current used space:       {"error": "filewalker: context canceled; filewalker: context canceled; filewalker: context canceled; filewalker: context canceled; filewalker: context canceled; filewalker: context canceled",
"errorVerbose": "group:\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatel
lite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n--- filewalker: context cance
led\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheS
ervice).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker)
.WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*G
roup).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/
pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.i
o/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellit
e:79\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgol
ang.org/x/sync/errgroup.(*Group).Go.func1:78\n--- filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkAndComputeSpaceUsedBySatellite:79\n\tstorj.io/storj/storagenode/pieces.(*Store).Spa
ceUsedTotalAndBySatellite:716\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-07-25T09:48:48-04:00       ERROR   retain  retain pieces failed    {"cachePath": "C:\\Program Files\\Storj\\Storage Node/retain", "error": "retain: filewalker: context canceled", "errorVerbose": "retain: filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatelliteP
ieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:181\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:568\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:373\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:
259\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-07-25T09:48:49-04:00       ERROR   retain  retain pieces failed    {"cachePath": "C:\\Program Files\\Storj\\Storage Node/retain", "error": "retain: filewalker: context canceled", "errorVerbose": "retain: filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatelliteP
ieces:74\n\tstorj.io/storj/storagenode/pieces.(*FileWalker).WalkSatellitePiecesToTrash:181\n\tstorj.io/storj/storagenode/pieces.(*Store).WalkSatellitePiecesToTrash:568\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:373\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:
259\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-07-25T09:59:41-04:00       INFO    db.migration    Database Version        {"version": 60}
2024-07-25T09:59:42-04:00       INFO    pieces  used-space-filewalker started   {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"}
2024-07-25T11:07:14-04:00       INFO    pieces  used-space-filewalker completed {"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Lazy File Walker": false, "Total Pieces Size": 198760435712, "Total Pieces Content Size": 198674775552}
2024-07-25T11:07:14-04:00       INFO    pieces  used-space-filewalker started   {"Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"}
2024-07-25T11:14:48-04:00       INFO    pieces  used-space-filewalker completed {"Satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "Lazy File Walker": false, "Total Pieces Size": 6462498048, "Total Pieces Content Size": 6454004480}
2024-07-25T11:14:48-04:00       INFO    pieces  used-space-filewalker started   {"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}

[Alexey]

Now need to wait until the folders of these two last satellites will be scanned.
The error on start is not good - the disk is struggling to respond, but at least the node is not crashed and your filewalkers still running.

@support26

I don’t know if there is/was such a timeout.

sls "writab|readab" "$env:ProgramFiles\Storj\Storage Node\storagenode.log" | select -last 10

[Julio]

@support26

I don’t know if there is/was such a timeout. I can easily enough do that - I’ve got HDTune at the moment and the results That(Crystal disk mark) should be run with the node service stopped or started?

Don’t matter which, if there’s anything awry it’ll show in the numbers, just note if disk is already 100% active so we could compensate. May not be an issue… HDD Tune sound good too, whatever.

[Julio]

@support26

Powershell is always run as administrator (also, this has never come up before this issue arose)

Not accurate. OS will provide shortcuts to either, you’ll only know if you have ULAC set, and it forces you to confirm running it as administrator.

Nevertheless, even a read only attribute set by SYSTEM process wouldn’t let an admin access shit, if it created the directory. You’d have to add permission, or take ownership.

.75 cents

[Alexey]

I just wonder… Do you really spent cents when answering?

[Julio]

The cents I display at the closing of a message, indicate how much of a sh#t I cared to answer the question. Or it’s relative weight of importance to me or the reader. For example, if I ever say ‘10 cents’ at the end of a message: then I really actually thought through everything I said in the reply, and the answer holds a lot of technical merit/expertise, etc., or I would not have bothered. “2 cents” represents the average common colloquial phrasing meaning somewhat unstudied and informal commentary. So, should I ever give more than 2 cents, I’m probably rather intrigued at the topic or question. think of it as scale of difficulty or level of ignorance.

It also works as a time management tool for me, to add up the cents I’ve given to this board & it’s members for any given day. That it doesn’t become an unpaid, unappreciated time vampire burdon to participate daily here.

2 1/2 cents

[support26]

@Alexey

Now need to wait until the folders of these two last satellites will be scanned.
The error on start is not good - the disk is struggling to respond, but at least the node is not crashed and your filewalkers still running.

I do not know what errors you are talking about. It did crash - as I could not access the dashboard(is that not a crash?) and I had to restart it. Happens often.

[support26]

@Julio

@support26

Powershell is always run as administrator (also, this has never come up before this issue arose)

Not accurate. OS will provide shortcuts to either, you’ll only know if you have ULAC set, and it forces you to confirm running it as administrator.

I stated that to mean I always ruin powershell as administrator.

[support26]

@Julio

Maybe you should run a simple stress test on that drive. Waaaay back in this thread, didn’t I see a 1 minute timeout? Crystal disk mark that drive or mount point, see if you have less than .67 megs/s on the default bottom 4k random read/write, or what iops there really are… let us see those results.

RND4K
Q1T1

Read: 261.96
Write 1023.19