Fatal Error on my Node / Timeout after 1 min

Node Operators troubleshooting
, ,
510

image
image626×933 91.2 KB

Check disk seems happy?

1 Like
511

Yes,
also clustersize is fine.
chose next what you want to do, i send the link here already.

I recommend timeouts up (see link how to) and
set the node to full (just for the time of the defrag that follows)

How many free ram and ssd space do you have? do you have an UPS?
how much space for nodes overall?
maybe primocache is worth a shot.

512

Force of habit? I don’t know why people recommend it. I recommend against it.

  1. Majority of files that storagenode stores are very small under 4k, and hence, fragmentation is not a factor. Moving them around the disk makes zero sense. Moreover, they are likely stored within MFT to begin with (resident files)
  2. Modern filesystems are very good at preventing fragmentation in the first place (that’ why the recommendation to keep 10% space free at least)
  3. One can make a case for MTF fragmentation on NFTS – but over time MTF ends up completed cached in memory anyway.
  4. Node reads data non-sequentially. There is no difference if it randomly reads files from a neatly arranged pile versus scattered all over the place.

Essentially you are doing a lot of work to benefit small amount of non-resident files.

Hence, it’s a waste of time.

Next, once you move to better filesystems, such as ZFS and BTRFS, defragmenting actually hurts, so the sooner people drop the notion of needing babysitting the filesystem – the better.

This contradicts my understanding of how NTFS operates. Can you elaborate why would that be the case?

Defragmentation makes sense in only one usecase – when you have a very few gigantic files on disk, that you need to stream in realtime at max speed, and cannot afford latency in the middle during transfer. This is a very niche usecase. Exact opposite one from what storagenode does.

1 Like
513

Great. In this case it’s likely the case of IO pressure occasionally increasing, going over what HDD can service, causing long delays and time-outs.

Start logging disk IO stats with timestamps, and next time the issue happens, reconcile with the log/graph. The easiest would be to start logging with Windows Performance Monitor.

514

But MFT is distributed all over the disk (for real, check it up even ultradefrag shows it)

and the other half of files bigger than 4k? they could be fragmented. and they are for sure mixed up all together. its a result of the constant writes and deletes of node data. realy logical.
after a certain point, wich many nodes reached (mine to) it s a big mess of clusters.

NTFS is neighter modern, or good at preventing fragmentation.

NO, god NO, windows is to stupid to do that, only primocache combined with defrag does that.
IF enough ram/ssd is present.
(living example, my first node, MFT=16GB RAM=8GB, see its not possilble)

yes, the customer upload is random, but can still go over fragmented files.
how about filewalks? wich sequence does the node? eh?

How do you think it operates?
even i don’t know that fully.

Its the other way around, this is the only usecase where defragmenters say: fuck it, im not moving this 2gb to its other two 1gb parts. because the 2 repositionings of the heads don’t matter for the stream of this data. the buffers will even that out.

why? the solutions are all where i posted them, the 1 min timeout is introduced in june, and many nodes got them instant, resolved over MFT defrag or timeout increase. (if the drives were ok)

So, still no prove for your not recomending? arrogantrabbit.
it reduces seek time while filewalking, that is the whole thing for defragmentation, and it prevents further defrag for a while. also helps preventing the triggering of the 1min timeout.

its called the filewalks, and garbage collection.

1 Like
515

I’ll keep everyone posted, but I got sketched out and run chkdsk /r which will prob keep my node offline for a bit, current ETA is 54 hours so I’ll let that finish just for peace of mind. I set the timeout to 1:30 using storage2.monitor.verify-dir-writable-timeout: 1m30s so we’ll see if that has an impact once I turn it back on.

516

I have 32gb DDR3 ram the ssd is 50% full

image
Those 3 are my node HDD, the 8TB is the one being chkdsk right now. The 3TB is a bit full I know, but it’s not accepting anymore and been happy like this for over a year now

517

do you have a good guide for it? I haven’t used that myself - I’d call my self a bit tech savvy but there’s so many bad guides I don’t want to fall for some dumb pitfall

518

Unfortunately, no, I don’t have access to any windows machine anymore.

Don’t read random tutorials, go straight to the source, like this one: Windows Performance Monitor Overview - Microsoft Community Hub

Or better yet, just play with the app. It’s pretty self-explanatory, add a bunch of counters and see how they work. It all UI.

519

Unfortunately you have a misunderstanding or did not have an experience with node’s data on Windows.
The defragmentation is needed for NTFS, include MFT. Windows is much worse in caching than Linux or freeBSD, it’s not very effective by default.
So for NTFS disks you must not disable an automatic defragmentation, this would keep it with low fragmentation level and good average latency. If you disable it, after some time and when the disk would be almost full it will work much more slower than initially, and having even 3GB RAM per TB wouldn’t help much - it will just not use it and will still crawling across fragments.

So, please, do not suggest to Windows users with NTFS to disable the defragmentation or do not perform it at all - it’s bad advice.
Running defragmentation explicitly is recommended only when they clearly have problems with latency (writeable/readable timeouts), because an automatic defragmentation uses an idle usage policy by default, so it may not happen frequently enough, thus requires to execute it manually.

1 Like
520

please edit the corresponding timeout interval to the same number.
do not exeed 5min(4m55s maximum)

521

You may exceed it actually. But you need to change both - the timeout and the interval (for writeability checks the interval is 5m0s by default), for readability checks you need to increase both - the interval and the timeout (because they are equal to 1m0s by default).

Just note, if it’s exceeds the 5m0s timeout - you likely have a huge issue with your storage and need to check disks and/or the storage configuration - it’s likely not effective for the node or broken.

1 Like
522

Like this?

storage2.monitor.verify-dir-writable-timeout: 1m30s
storage2.monitor.verify-dir-writable-interval: 1m30s
1 Like
523

Yes, but writeable interval is 5m0s by default, so you do not need to modify it until your writeable timeout would be greater than 5m0s.
However, for a readability timeout you need to increase both - the readable timeout and the readable check interval, because they both are 1m0s by default.

1 Like
524

Is this correct to have the readable set as well?

storage2.monitor.verify-dir-writable-timeout: 1m30s
#storage2.monitor.verify-dir-writable-interval: 1m30s default is 5m, no need to change

# how frequently to verify the location and readability of the storage directory
storage2.monitor.verify-dir-readable-interval: 1m30s

# how long to wait for a storage directory readability verification to complete
storage2.monitor.verify-dir-readable-timeout: 1m30s
525

Yes, now it’s correct. However, I did not see a readable timeout error in your logs, do you have them? If not - you may comment out them as well.

1 Like
526

Hi, I wonder if someone can check my logs for me as i cant get to bottom of why my node keeps stopping and the restarting.

2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “N3HHGWWTETUPZXAWN22X6SHELLDJTVNMIZSQWWZ4YP6FSPZZTRVQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708115998724, “Remote Address”: “79.127.226.97:54480”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “6Q5EPQUDLUJHF2PKUH2XMPPSTKVVFMJFS57R5ERZLIQ4CYFVKINQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT_REPAIR”, “Available Space”: 708115998724, “Remote Address”: “94.130.111.189:35952”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “JY6FUWFRTQYJ2QDCI3TLAQLHEIT7UDFIUEO57PRLOMNUE3NOV3NA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708115998724, “Remote Address”: “5.161.143.41:15084”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “YMQBFXZVBWBZ5A4EI4NORYD52MGRDP2OIBSTRRLBWPROYGLIRWWQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708115998724, “Remote Address”: “79.127.220.99:57108”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “AZC7VXA7IRQ6KHDP35FCFE2MIMJMXPFOC5K4BL25CPGKJRJMEGMQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708115998724, “Remote Address”: “147.189.193.86:42426”}
2024-01-31T14:08:49Z ERROR piecestore error sending hash and order limit {“error”: “context canceled”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “SWYRS5UJ43VEXU5YZRUBLRKTEQE4MYKJ37MCJWGXZTZPMHPLY2XQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT_REPAIR”, “Available Space”: 708115998724, “Remote Address”: “5.161.123.157:56402”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “DTAG7CPRVB7XMAZSTYDJFIIBATNW6IZBAJVABVVUM5UP6VUVTEHQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708115998724, “Remote Address”: “79.127.220.99:57122”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “GYCK6XHZVDBVBW2O4PNCTZW5NASJDODRQAG7PUOULMCF5QSSOTLQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.205.226:53496”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “K67GRQRLTOTJ2E4IZGKXUQEWINOHKHK4DO2LHTDL65XYMDWQ3CGQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.205.226:52576”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “OPDGOO2N25SMFY5RXYKXMR2LNJWF4DTILUDLLNRPX7KLEKGDL4OQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.219.34:39636”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “UDW4TFXJVIQYZ4UIVSEYMATN6GJA2MQF6SYDZDS4AANFQJB3JYEA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.219.33:46228”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “U2YQNCUZ3EDZN47IV7QJENUKXRV5ERG36ULLA7Y46HH6ZJZ7D4BQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.220.97:36268”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “6IXEF7QYTNLJ6UWFFLGJGLOSZAAJG4SN27QQ5ZUZLVJDP6MGHFGQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “5.161.176.200:27674”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “E3MVNTXKUW3AQ55DY6PNRA532PFJ3JJU2A2WZM5DE75LJF7QHYQQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.220.98:36736”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “MQ4GBVUIZN3VQN2XWR2QBCU46IXB67KIQJ3GRBGW6DC7J6LOU34A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.226.100:45784”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “R6PHYUB6XTM4VHX3ELHZGVV5O5O7ZKDBIJ6B63KQSW7G7YDJ5UUQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708115998724, “Remote Address”: “5.161.176.200:45460”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “MCCGVTF323L24VHGGI5ETWMK2VGRFWRXKMIE7RWHCDBDABDSKFLQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “5.161.143.41:16018”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “RAU72Q5LMGUZSYUXLBP44PRGXTP4DBZB47GK3AGGGYDLQ2SMAL2A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.219.33:37958”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “TIPD3RYTQMXQIKXVHLNBX67FBDCS7RBZNSSFOBXRTJZCWBAEVXPA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708115998724, “Remote Address”: “79.127.219.33:46212”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “44FIH2KN3PXWQFVD66WHHHR3BKPMGTLCF7PHWCJYQ6XDU2GT7HSQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708115998724, “Remote Address”: “147.189.193.14:59608”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “2D4IWXLUSLCEDYNP6CBBV7VRUPKATCM2E4SUTSXYSQGKBQBWHMYA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.205.225:35104”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “DDOAM2JTNCWJ7GYSQAR7JFM367ICEAJWYTHIWWTMXOMIR5Z2MEJQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “5.161.192.57:52310”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “NAZUQBOAK6WPJ5LVPDDRGITVEAK5TUJETQUNQV6XT6OIPGH4WC6Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.219.34:34900”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “3NCGCKIY7IQ7L4GKTZKZWLEJPCDB7Z7V7W3HWFKUILPZ2FBMPBMA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “5.161.176.200:27050”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “CCI6472JXYGVTZBXRN6FRJZK3CS2CYWYCFRIH4WNEKBIJC76OQTQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.201.209:48154”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “3KGZ4X4DRPLLVGH5L6IS3J3RHC4MAZYBEVG4V6JWYBM5EBYVKV6A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.220.98:51772”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “PFQJVZ4CBKUPJSBXURMKSWSYPFWAKXUVG7WV4SQBSAMJQFYLHF3Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708115998724, “Remote Address”: “79.127.205.225:35068”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “LMPSWW32K477CZTUHRYGGYX6XWNWO43JA5ZXGDZR4ESEEAC32QCQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “72.83.198.17:59146”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “POD7CAXEZLRMLUQIIOSNAL5XCKUBULAW7QZBEMMCKW5JWPXTRTSQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Available Space”: 708115998724, “Remote Address”: “79.127.226.98:51350”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “JXCVWSMJPJZ6JT4WXBCLSMSBJL73FKOL3R3XRLZ6WYZFPQ44KTDQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708115998724, “Remote Address”: “147.189.195.37:51686”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “JULAMACXXUOAW6LWTPOUSKGT7YMYRKJRTWVI35DYEGSPXNGHLROQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.226.100:40366”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “L37Q7OZAG3XA4PIFB3FURMIVCZDW3NYQMUWGZOUQDHADYH2KZCCQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708115998724, “Remote Address”: “79.127.205.226:52592”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “6EADGKQMWFR72TNCC4UAKR43BBLXOSWFH5AOF5B4CT6JW7ML7JFA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.220.98:33014”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “ADA7KRYEVK7KK7DKAVLVUUU34LUGJBT5I4IQZZ7ZAF2PH6PH3FBQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.205.225:44832”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “W2ZOMTDINTFRLPYK3ZFO7NRDSC6TJSEYBQUR3FEIKL5PZKSZLA7A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “79.127.213.33:59344”}
2024-01-31T14:08:49Z INFO piecestore download canceled {“Piece ID”: “OF7I6ZSAZXKQ6JXE632RCBEP5OORQ5Q4HPNJBAOTBMYI4EJMUWMA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_REPAIR”, “Offset”: 0, “Size”: 0, “Remote Address”: “5.161.85.184:59896”}
2024-01-31T14:08:49Z INFO piecestore download canceled {“Piece ID”: “3TS53TVQ2OQKC5TTR6R273HDLGMXA6S6OZPQG7Y2I7TILHSIEUTA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 0, “Remote Address”: “79.127.201.209:38534”}
2024-01-31T14:08:49Z INFO piecestore downloaded {“Piece ID”: “DS2QENEJITAEEYDHGGQYGSTPB55FR26MQ6MT4WC4EHEYDMUURPRQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 112384, “Remote Address”: “79.127.226.99:36112”}
2024-01-31T14:08:49Z INFO piecestore download canceled {“Piece ID”: “GDS3B3K625GZ6FR74ZHT7ZDT3IV4MIYMWWESM4WCTTZV2TUGMKMQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 0, “Remote Address”: “79.127.213.33:60528”}
2024-01-31T14:08:49Z INFO piecestore upload started {“Piece ID”: “D2U7M74UQDULFIJ25OR4A23DVN24AFO7HNXF3UL65Y6D5YJPKOQA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 708116548356, “Remote Address”: “5.161.77.10:64112”}
2024-01-31T14:08:49Z INFO piecestore download canceled {“Piece ID”: “FTROI73C3XDSJYVSMWDWHZUSOQYDZHHHBYPFKGUEYR2BMXJRS3VQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 0, “Remote Address”: “79.127.220.99:58864”}
2024-01-31T14:08:49Z INFO piecestore download canceled {“Piece ID”: “ITZBG5VO6IORRGSEMIUKXI6FBNVR63QGIKKDC2RM3RPWTAVTQBIA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 0, “Remote Address”: “79.127.205.226:45224”}
2024-01-31T14:08:49Z INFO piecestore download canceled {“Piece ID”: “B2B4EN3ANMO2RTCBIMW3DZPP2NC5427F2TPEGLKJPZ5QMVN2MZIA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 0, “Remote Address”: “79.127.220.98:50024”}
2024-01-31T14:08:49Z INFO piecestore download canceled {“Piece ID”: “MVO6B6DMLHQHK3CF24ZZFYC3RXNEE3TEANOKXYEX5L5F2VCWWMBQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 0, “Remote Address”: “79.127.220.98:43148”}
2024-01-31T14:08:50Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “4IAUEDB2KX5XDAUDE456QZRQXZL6MWFZ5LO2S5CHZMOQ32BMCAJA”}
2024-01-31T14:08:50Z INFO piecestore download canceled {“Piece ID”: “KVUBJB3WCCILNQD2V7XCATRXIASKFMTBWLWTDWDYDFRVKUICY35Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 0, “Remote Address”: “79.127.226.97:58060”}
2024-01-31T14:08:50Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “QGXV6X4WY3LCOL2CWUSVQOXX47UPDV467BFI4JFQUCEP7K4BGUFA”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “DDOAM2JTNCWJ7GYSQAR7JFM367ICEAJWYTHIWWTMXOMIR5Z2MEJQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “5.161.192.57:52310”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “LMPSWW32K477CZTUHRYGGYX6XWNWO43JA5ZXGDZR4ESEEAC32QCQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote Address”: “72.83.198.17:59146”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “FU5KWLY53GZPSOTN2MVSDJY5WCH2L3X6GGRRT2ERVK73NB2Y62EA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT_REPAIR”, “Size”: 0, “Remote Address”: “5.161.206.9:43634”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “4YX6GXHMVLZV7NZGU4OFGLMPLZGQ5DI3UFJHPZNQUNBIUXT3QAXA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT_REPAIR”, “Size”: 0, “Remote Address”: “157.90.17.108:53440”}
2024-01-31T14:08:50Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “Q6ZP7KDIYCY5MCFB7X3PHXGW3NILTU4Y2YZIOVYEEFOJUVTMZXNQ”}
2024-01-31T14:08:50Z INFO piecestore download canceled {“Piece ID”: “VWZW6NL3FE4J7MZM5E7T4KGZPW3NTNQ4VA3EHLHG2BH6WFZ6RSDA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 0, “Remote Address”: “79.127.201.209:53378”}
2024-01-31T14:08:50Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “A5OIM2VG2NLGRO5AMWO73UJXU7YTWIZEFAEFQ5YCHJ2CSOOQIRUQ”}
2024-01-31T14:08:50Z INFO piecestore download canceled {“Piece ID”: “3AXSQZJBWPJ76MQAL2L7WJC66RWIMUJP3VTP7WXTCWOMR3T5PKDA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”, “Offset”: 0, “Size”: 0, “Remote Address”: “79.127.220.99:57144”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “DJ25VY35V7IQ6HC3B3F755TTTCDF64WVFPY5S3DHFSVDVR6ZKLQQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 0, “Remote Address”: “72.83.198.17:47214”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “JTWKIFDVX4KQDPTBCUXRFUSZGJTTPCAT3O7F6LLP3Q277ZAEOHIQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT_REPAIR”, “Size”: 0, “Remote Address”: “116.203.196.91:44052”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “JLRLIFJPRUBWBSSJBOJGDKL64I77EKHQFPCYLIRR6JRX7LDUVNOA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.220.97:35046”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “6Q5EPQUDLUJHF2PKUH2XMPPSTKVVFMJFS57R5ERZLIQ4CYFVKINQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT_REPAIR”, “Size”: 0, “Remote Address”: “94.130.111.189:35952”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “QAAUUAVMEOKSLSG7RJQTTP5EQCOULAVDHD3RGZGZDD26OU6L6B4A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.219.34:46646”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “KNILD2DDDPOYAX7WZ5V7O2K7VMM5CXVF4C4VPNAGWW5AWV2WPSBA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “147.189.192.224:37236”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “K67GRQRLTOTJ2E4IZGKXUQEWINOHKHK4DO2LHTDL65XYMDWQ3CGQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.205.226:52576”}
2024-01-31T14:08:50Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “4TSIGWXSKTDOXSAERGFH37G6ZMTJL7FLVPVWFRCIUTXBXOSPF4PA”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “XTO4CWZDVIUDNV3AN2UZ3LL3KDWQ57FCK7Y7ZXB3XFLWZ3MV3SXQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.226.97:35116”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “GYCK6XHZVDBVBW2O4PNCTZW5NASJDODRQAG7PUOULMCF5QSSOTLQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.205.226:53496”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “R6PHYUB6XTM4VHX3ELHZGVV5O5O7ZKDBIJ6B63KQSW7G7YDJ5UUQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “5.161.176.200:45460”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “PFQJVZ4CBKUPJSBXURMKSWSYPFWAKXUVG7WV4SQBSAMJQFYLHF3Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.205.225:35068”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “SWYRS5UJ43VEXU5YZRUBLRKTEQE4MYKJ37MCJWGXZTZPMHPLY2XQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT_REPAIR”, “Size”: 0, “Remote Address”: “5.161.123.157:56402”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “UPRQTF2G6CMAISIIAKM2BZ3LFZ66LXO2KK6KQYJ7N5BHQOY6HWAA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT_REPAIR”, “Size”: 0, “Remote Address”: “5.161.105.123:37784”}
2024-01-31T14:08:50Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “4EQOFVT2RZRQX45HHX2JXRSISDTBLYSTTOC6PMSIIWTKIEUKFQCQ”}
2024-01-31T14:08:50Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “JULAMACXXUOAW6LWTPOUSKGT7YMYRKJRTWVI35DYEGSPXNGHLROQ”}
2024-01-31T14:08:50Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “J3WOCYAAMUY5PVIQGP475V7SBTB27XBTVYHYMG7G5RLWN4WX6TIQ”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “LIIMEYBVIAXJGPRH4OEWM6ITT255GXMPWGWOHEUQAX3TDUJ3G24Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.219.34:59712”}
2024-01-31T14:08:50Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “G4RKITBQUVSVXJGA7QORE3XDBSSSSI4MUZDDYUTIYQOMBHCUCKIA”}
2024-01-31T14:08:50Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “MQUCZBDXACNWGVZPI6VZ5XM25RHLY4IY2G5DOLU5MCY66WKKFLWQ”}
2024-01-31T14:08:50Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “3Z5PGRWET7FOFMWWWFTIIOIXWK33UNG4C7SG634XZ2MYKBXOJ22A”}
2024-01-31T14:08:50Z INFO piecestore upload canceled {“Piece ID”: “AG5H2Q3O3A7OF5CLNF3ADVKQJ3MAURVSOSO2QGQPJ5HAIW54WYOQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “5.161.77.10:64200”}
2024-01-31T14:08:50Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “MQ4GBVUIZN3VQN2XWR2QBCU46IXB67KIQJ3GRBGW6DC7J6LOU34A”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “WMTHB2KYWDTGMFKFDTWLJSAOUT4R4LYJZF7F3ANTNW2QTKKD5NQA”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “ZMWOV4ANA2VMNHYDB5XKCGW5JPA2HBA7PYGIYNPONJULE64LKTHQ”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “AZC7VXA7IRQ6KHDP35FCFE2MIMJMXPFOC5K4BL25CPGKJRJMEGMQ”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “TGKFPPM6MJRIJTZERCPCWG2477H5I5R3EGUUJDF437RUCPSHKHSQ”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “X4EVEVZSNC6XISPN34SOWW3327O4QL5VD4XOBCTKAHYD3OFUOIOA”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “4AWRV5XLNRJC5Q4ADV727VRROCRDVGWQSNKTI3UM72QFBPS56TWA”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “YMQBFXZVBWBZ5A4EI4NORYD52MGRDP2OIBSTRRLBWPROYGLIRWWQ”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “RNC3ZHWKU7BYJ5WXWC2DEUBBYUICU52JQTQHXAFCSTUD2JDFHVKA”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “2YK6XHXGADZ4LN7LOYMK5JYYPV3MPN5OK2I3U42BGULPNQXARCBQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT_REPAIR”, “Size”: 0, “Remote Address”: “5.161.226.176:58618”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “L37Q7OZAG3XA4PIFB3FURMIVCZDW3NYQMUWGZOUQDHADYH2KZCCQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.205.226:52592”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “ADA7KRYEVK7KK7DKAVLVUUU34LUGJBT5I4IQZZ7ZAF2PH6PH3FBQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.205.225:44832”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “DJ6EMJVKHWP2U7Z5744UCLKOY2XFFSMRTZUADLFAWAM3JDYXQZQQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT_REPAIR”, “Size”: 0, “Remote Address”: “5.161.74.111:39646”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “CEZVHL4GODSIRFEGSVQFKEBK4M3OHQV2MOF636EVCLD7H3UTOL3Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.220.97:35066”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “NDDEYOV6EN6JM6LNXG6BI3EUGXLPUXMIIH4W4XCWGC6IFFQO6JJA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.226.97:51792”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “GYNIBGJLVBPBGX55FMHPC3E7GT35VO3PNH5HTYS26I3QBMFMCQHQ”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “5Y5CXPW7GGUUV22WNMXXPOIOQPBLLXWZQLRE5Y6DUIK6HVRDYDCQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.220.99:59330”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “6IXEF7QYTNLJ6UWFFLGJGLOSZAAJG4SN27QQ5ZUZLVJDP6MGHFGQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “5.161.176.200:27674”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “QF23254H3MEPBIJT52L2KD4UITQO7PZGS2CXBKZTMUTNVMKG5LTQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT_REPAIR”, “Size”: 0, “Remote Address”: “162.55.215.41:49232”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “OIGSTFJF6TPKY3ZQHJWPUOHQNUAJ7U5IPNR4NMQAQT3JLSGUJEBQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “189.191.152.246:35718”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “JUEKVA655WWTGY5RFVJPJ2UV7SYCKLLUCEESCFNOTWNVZBWPBTHA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT_REPAIR”, “Size”: 0, “Remote Address”: “5.161.72.113:45122”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “2D4IWXLUSLCEDYNP6CBBV7VRUPKATCM2E4SUTSXYSQGKBQBWHMYA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.205.225:35104”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “JY6FUWFRTQYJ2QDCI3TLAQLHEIT7UDFIUEO57PRLOMNUE3NOV3NA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “5.161.143.41:15084”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “6EADGKQMWFR72TNCC4UAKR43BBLXOSWFH5AOF5B4CT6JW7ML7JFA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.220.98:33014”}
2024-01-31T14:08:51Z INFO piecestore upload canceled {“Piece ID”: “POD7CAXEZLRMLUQIIOSNAL5XCKUBULAW7QZBEMMCKW5JWPXTRTSQ”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “Size”: 65536, “Remote Address”: “79.127.226.98:51350”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “DBYJYECJSUP35NFZBAAIVT3XLQPRWDIDL2G2UCSJM5RYBCRHVUDQ”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “E3MVNTXKUW3AQ55DY6PNRA532PFJ3JJU2A2WZM5DE75LJF7QHYQQ”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “BVYZ3KHZXYM7US7XMNYJUYHU2YZQUSKFKZD6GUOJCI7HXJWPUH2Q”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “44FIH2KN3PXWQFVD66WHHHR3BKPMGTLCF7PHWCJYQ6XDU2GT7HSQ”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “BUYFUCDHPLBJYWTYYIP24BAGPVSW6I6NZHW67OLSRQDQOXQFZLDA”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “F3KLOAMPFRZZDOINHXBC4QWNVSXWUJZBPXCM5KDJMOYA2SSAYV7Q”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “UDW4TFXJVIQYZ4UIVSEYMATN6GJA2MQF6SYDZDS4AANFQJB3JYEA”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “OPDGOO2N25SMFY5RXYKXMR2LNJWF4DTILUDLLNRPX7KLEKGDL4OQ”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “ZTKI5TY4FZTVSVDFENAIWYW55XGFED3I65XVXDQWSF5VTR46D7UA”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “TIPD3RYTQMXQIKXVHLNBX67FBDCS7RBZNSSFOBXRTJZCWBAEVXPA”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “RAU72Q5LMGUZSYUXLBP44PRGXTP4DBZB47GK3AGGGYDLQ2SMAL2A”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “U2YQNCUZ3EDZN47IV7QJENUKXRV5ERG36ULLA7Y46HH6ZJZ7D4BQ”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “L7INLMAEJ35V6OYF7H5HGXHNVOGDNECTED4H3IL66LSOGZVGUJBQ”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “OHG7KEGGFVVPPMHVGL5SXUKNR7YIPDQ2KPYEMGKBJV5CWNHDKTIA”}
2024-01-31T14:08:51Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “N3HHGWWTETUPZXAWN22X6SHELLDJTVNMIZSQWWZ4YP6FSPZZTRVQ”}
2024-01-31T14:08:52Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “CCI6472JXYGVTZBXRN6FRJZK3CS2CYWYCFRIH4WNEKBIJC76OQTQ”}
2024-01-31T14:08:52Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “MCCGVTF323L24VHGGI5ETWMK2VGRFWRXKMIE7RWHCDBDABDSKFLQ”}
2024-01-31T14:08:52Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “DTAG7CPRVB7XMAZSTYDJFIIBATNW6IZBAJVABVVUM5UP6VUVTEHQ”}
2024-01-31T14:08:52Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “W2ZOMTDINTFRLPYK3ZFO7NRDSC6TJSEYBQUR3FEIKL5PZKSZLA7A”}
2024-01-31T14:08:52Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “3KGZ4X4DRPLLVGH5L6IS3J3RHC4MAZYBEVG4V6JWYBM5EBYVKV6A”}
2024-01-31T14:08:52Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “QALUBHI4M6U66KFZMRJMNRBXO7YUPAKP7IHEK3IDLWUN5Q3Q37GA”}
2024-01-31T14:08:52Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “D2U7M74UQDULFIJ25OR4A23DVN24AFO7HNXF3UL65Y6D5YJPKOQA”}
2024-01-31T14:08:52Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “JXCVWSMJPJZ6JT4WXBCLSMSBJL73FKOL3R3XRLZ6WYZFPQ44KTDQ”}
2024-01-31T14:08:52Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “NAZUQBOAK6WPJ5LVPDDRGITVEAK5TUJETQUNQV6XT6OIPGH4WC6Q”}
2024-01-31T14:08:52Z INFO piecestore upload canceled (race lost or node shutdown) {“Piece ID”: “3NCGCKIY7IQ7L4GKTZKZWLEJPCDB7Z7V7W3HWFKUILPZ2FBMPBMA”}
2024-01-31T14:08:53Z FATAL Unrecoverable error {“error”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory”, “errorVerbose”: “piecestore monitor: timed out after 1m0s while verifying writability of storage directory\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2.1:176\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tstorj.io/storj/storagenode/monitor.(*Service).Run.func2:165\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}

Its a 6month old 20tb disk. everything else appears fine with pc and can access the disk normally using windows.
any help would be great.

527

Sound like the filesystem is going read-only due to errors?

528

I wonder how to solve this issue

529

Here, maybe describe your setup more.

As usual, something causes the system to bottlenec for more than one min.
Is it all 6-12h? thenits maybe the filewalker or something ressource intensive task like indexing, or a slow connection to the drive. (USB2?)

1 Like