Bandwidth 0GB, but node is working correctly

kosti11 · February 28, 2021, 10:35am

Hello.
I have problem with my node dashboard. When i’m looking on Bandwidth usage it is 0GB for some days.
Last time i migrated my node from Raspberry Pi to Synology. Only what i do i changed chown from root to one user on Synology because it given me no permissions. After it node started. Disk is connected through USB so i don’t made RAID by him.
Second problem is no “sudo docker logs storagenode” respond by terminal.
Synology is only storage for this node. Node is working on VPS with running Storj docker container through SSHFS connected to my Synology because of NATing by my LTE ISP.
Can someone help me with fixing these problems?

Krystof · February 28, 2021, 10:49am

Hello,
What is your version? Please update to the latest version…

Btw its too complicated connect with more potential issue but if the node is online i see only a version issue

kosti11 · February 28, 2021, 10:52am

Node version is v1.22.2. Docker run is with :latest and watchtower runs correctly.
Yeah i know it’s complicated but it worked on Raspberry PI with 200-400h uptime (from update to update) without any problems.

andrew2.hart · February 28, 2021, 11:03am

Does it say online?
Have you got any other nodes running?
Have you checked the logs?
Are you suspended for downtime? The large amount of trash might suggest downtime…

kosti11 · February 28, 2021, 11:36am

Yeah. All the time it’s online. I have Telegram notification bot turned on too. When i refreshing it after some time it gives me 0 min ago last contact (normal affect before migration).
Yes i have smaller node running too on my Synology but mounted on second VPS (using now +30GB with SHR - 4TB with 3 disks) but last time after my mistake i damaged database. It’s still working for my luck. I need to make another topic when sq3lite help guide on Storj forum will not fix my problem.
Logs on second node works correctly.
I don’t know i’m suspended. I got some payouts.
I’ll tell you a story. Node 1 (8TB from pics) worked on FTTH in my friends house from 03.2020. After some months my node was offline for 3 months because he needed to change location. After found solution for my LTE NATing problem I turned on my node and it started working to this time (5 months).

Alexey · February 28, 2021, 11:47am

Any remote storage is suspect to problems with sqlite databases. They do not like any remote storage.
Please, check your databases:

kosti11 · February 28, 2021, 11:55am

Yeah. I talked about this topic. I need more time to look at this and do this step by step
But first i need to stand up my first node
What too much is not healthy

andrew2.hart · February 28, 2021, 12:47pm

Sorry, I didn’t see before. You have no logs at all from docker?
Did you redirect the logs to a file?

kosti11 · March 3, 2021, 4:42pm

Yeah i saw they’re magically transferred to file (10 GB :D). Weird but after changing it command “sudo docker logs storagenode” works properly now. Przechwytywanie

nerdatwork · March 3, 2021, 4:47pm

You have issue with adding orders to file. Orders older than 2 days are discarded and you will lose payment. Fix that issue asap.

kosti11 · March 3, 2021, 4:52pm

How can i fix it? Is it permission issue?

nerdatwork · March 3, 2021, 5:12pm

Yes. The error message does say “operation not permitted”.

kosti11 · March 4, 2021, 4:53pm

I had wrong permissions. I fixed it by “chown -R ‘user’ ‘location’” and “chmod -R 777 ‘location’’” and problem still exists. Any ideas? 5fe807d8-7cac-4779-b236-357587841865

Alexey · March 4, 2021, 6:11pm

How logs looks now?
Could you please, copy a text instead of screenshot?
You can put it between two new lines with three backticks ``` each. The last 20 lines is enough.

The remote connected filesystem can do not work as expected. The storagenode designed to use local drives.

kosti11 · March 4, 2021, 6:28pm

Log looks the same now.
Yes, of course.
I know it too that remote connected filesystem can do not work as expected but it worked over half a year correctly. I’m looking for better ISP but my location is hard to make this.

2021-03-04T18:26:41.766Z        ERROR   piecestore      upload failed   {"Piece ID": "2YD53MMQLRTJFAZBR4AMRQSLQ7NWK7GWXJAGH73BFYKL22NR4JVQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "pieces error: filestore error: open config/storage/temp/blob-534474434.partial: operation not permitted", "errorVerbose": "pieces error: filestore error: open config/storage/temp/blob-534474434.partial: operation not permitted\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:166\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:212\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:291\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51", "Size": 0}
2021-03-04T18:26:42.042Z        INFO    piecestore      download started        {"Piece ID": "PO7HY3I7L7YED5BAXXMKT4MKPLNYRTDXT5JEDHHFJSQ7RI6NS65A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-03-04T18:26:42.895Z        INFO    piecestore      upload started  {"Piece ID": "XQDCQX43P2UEL3GGM3DP4BUKMVVZBVL3CHFXXB2PVKBWDAWM5VQQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 1168998336256}
2021-03-04T18:26:43.457Z        INFO    piecestore      upload started  {"Piece ID": "23SRRODDKYTCNPRKCMXYIYZNL62KZOJ6XE3SXGHMKUC4IPK6ZJFQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 1168998336256}
2021-03-04T18:26:44.815Z        ERROR   orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6      failed to archive orders        {"error": "order: ordersfile: rename config/orders/unsent/unsent-orders-121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6-1613890800000000000.v1 config/orders/archive/archived-orders-121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6-1613890800000000000-1614882374574819495-REJECTED.v1: operation not permitted", "errorVerbose": "order: ordersfile: rename config/orders/unsent/unsent-orders-121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6-1613890800000000000.v1 config/orders/archive/archived-orders-121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6-1613890800000000000-1614882374574819495-REJECTED.v1: operation not permitted\n\tstorj.io/storj/storagenode/orders/ordersfile.MoveUnsent:143\n\tstorj.io/storj/storagenode/orders.(*FileStore).Archive:278\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func1:215\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2021-03-04T18:26:44.815Z        INFO    piecestore      downloaded      {"Piece ID": "RBP25FSXDISSM62CDPFU5ISIC2QCXUEBUUFMPDFIRY6KC7PAGUNA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET"}
2021-03-04T18:26:44.816Z        INFO    piecestore      downloaded      {"Piece ID": "M6GXMARHDRACED3Y4LJNNYBRS2CFZH5DGTYO6C6T4EPV7WOPDWIQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET"}
2021-03-04T18:26:44.816Z        INFO    piecestore      downloaded      {"Piece ID": "M6GXMARHDRACED3Y4LJNNYBRS2CFZH5DGTYO6C6T4EPV7WOPDWIQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET"}
2021-03-04T18:26:44.816Z        INFO    piecestore      downloaded      {"Piece ID": "PVHTLBWRSAZ2FOHZU6GBVIXN6COJB3WHAAINF33S6PSSQVFJCLNA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-03-04T18:26:44.816Z        INFO    piecestore      downloaded      {"Piece ID": "2IUAGAJONNIV3SKN7WHIVQUHLM67IPLWFYSF6E2KR5PBO66BXOPQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-04T18:26:44.817Z        INFO    piecestore      downloaded      {"Piece ID": "4YDOTXYIIAF3BF3J6BOH2GR7QIP6NSDQIMJTR5IF7CK53LEYLVOQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-03-04T18:26:44.819Z        ERROR   piecestore      upload failed   {"Piece ID": "XQDCQX43P2UEL3GGM3DP4BUKMVVZBVL3CHFXXB2PVKBWDAWM5VQQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "pieces error: filestore error: open config/storage/temp/blob-264867129.partial: operation not permitted", "errorVerbose": "pieces error: filestore error: open config/storage/temp/blob-264867129.partial: operation not permitted\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:166\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:212\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:291\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51", "Size": 0}
2021-03-04T18:26:49.367Z        ERROR   piecestore      upload failed   {"Piece ID": "23SRRODDKYTCNPRKCMXYIYZNL62KZOJ6XE3SXGHMKUC4IPK6ZJFQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "error": "pieces error: filestore error: open config/storage/temp/blob-247869764.partial: operation not permitted", "errorVerbose": "pieces error: filestore error: open config/storage/temp/blob-247869764.partial: operation not permitted\n\tstorj.io/storj/storage/filestore.(*blobStore).Create:166\n\tstorj.io/storj/storagenode/pieces.(*Store).Writer:212\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:291\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51", "Size": 0}
2021-03-04T18:26:50.967Z        ERROR   piecestore      failed to add order     {"error": "order: ordersfile: open config/orders/unsent/unsent-orders-12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S-1614880800000000000.v1: operation not permitted", "errorVerbose": "order: ordersfile: open config/orders/unsent/unsent-orders-12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S-1614880800000000000.v1: operation not permitted\n\tstorj.io/storj/storagenode/orders/ordersfile.OpenWritableV1:49\n\tstorj.io/storj/storagenode/orders/ordersfile.OpenWritableUnsent:80\n\tstorj.io/storj/storagenode/orders.(*FileStore).BeginEnqueue.func1:113\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:678\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:622\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:646\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:29\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2021-03-04T18:26:50.968Z        INFO    piecestore      downloaded      {"Piece ID": "5QEC7I5CXPOLCYVVH5VX575HZHZ3ZPAKHXCWSJGWTF7XPSHXEIBA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET"}
2021-03-04T18:26:50.968Z        INFO    piecestore      downloaded      {"Piece ID": "5TJUSWRWXWJVMFGZDBVIJ6PDIG5G5SX2SGPWEAWKKC4C6R54B2KQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR"}
2021-03-04T18:26:50.968Z        INFO    piecestore      downloaded      {"Piece ID": "FEB434JBYGVCF7W5JXTVACKY3RJPEEP7QNV2NCG3RUWHGUPW3APQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET"}
2021-03-04T18:26:50.968Z        INFO    piecestore      downloaded      {"Piece ID": "FEB434JBYGVCF7W5JXTVACKY3RJPEEP7QNV2NCG3RUWHGUPW3APQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET"}
2021-03-04T18:26:50.968Z        INFO    piecestore      downloaded      {"Piece ID": "A3FTCI6DNEBP4NVNNSZPNLC3TGJYGVWQ3H5ROIHYH5CNKBG5N4TA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-03-04T18:26:50.969Z        INFO    piecestore      downloaded      {"Piece ID": "BYVO7DDQNOQJ5DIGCSQN7NHS5TSTHMBU672L6MPZVE6X2S6CTFLA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET"}```

Alexey · March 4, 2021, 7:16pm

Maybe it’s better to configure VPN instead of sshfs. You also can use a ssh as a VPN and forward port.

kosti11 · March 10, 2021, 5:11pm

Hello everyone! Node1 was repaired by me. It was permissions problem.
I don’t know why is Synology doing that problem on USB drive but when i inserted it directly on Synology SATA pocket and typed these commands again it worked. My main node works pretty well now.