Disk utilization not increasing

Hi,

About 9 months ago, I migrated two nodes from using smaller disks to larger disks, from roughly 3-5TB to 12TB (set to 10TB), but I have not seen basically any growth in disk utilization since then. Is this to be expected? They’re both still sitting at around 3.5TB each, no other issues, near 100% uptime. The only errors I can find in the redonculous log, otherwise filled with successful download logs, pertains to satellite connection timeouts and a few download/delete fails. What am I missing?

Disks are statically mounted via /etc/fstab in Ubuntu server 20.04 running in a Proxmox VM with physical disk passthrough.

System disks in question, via Ubuntu:

$ df -H

Filesystem   Size   Used   Avail   Use%  Mounted on
/dev/sdc1    12T    3.6T    7.8T    32%    /mnt/storj01
/dev/sdb1    12T    3.5T    7.9T    31%    /mnt/storj02

And flag in docker run command:

-e STORAGE="10TB" \

Thanks,
alfananuq

Hi @alfananuq
If there are no errors in the logs (other than the odd download or delete fail) and the uptime is near 100% then there’s very little else you can do.

On the dashboard of each node does it show available space?

Yeah, it all looks good on the dashboards, here’s one:

Total: 10TB
Used: 3.57TB
Free: 6.43TB
Trash: 187.74MB
Overused: 0B

Other than the fact that there’s nothing that can be done, can anyone offer an explanation as to why I’m seeing this behavior? Has anyone else seen this after an increase in node storage capacity? I’m wondering if there’s anything to gain by starting a new node instead. The nodes grew to max size pretty quickly when I first set them up, so seeing zero growth is unexpected to me.

Is your node up to date?

1 Like

I’m on v1.36.1 and running watchtower.

Once your version is to old (3 Releases?), storj stop sending Data to your nodes. I think you have to update your nodes to get Ingress again.

2 Likes

I guess my misplaced trust in Watchtower did me under, haha.

Interestingly, I stopped, deleted, pulled latest and started the nodes, but they installed different versions… The first one installed v1.54.2 and the second one is now on v1.53.1. I think docker is just trolling me at this point… I’ll see if I can straighten this out. Thanks @Arkina !

Yeah, I have no idea why my nodes are running different versions since they both use latest image. I updated watchtower as well, we’ll see if it catches it in the next couple of days.

And hopefully, I’ll start seeing some ingress again, I’ll report back!

Is normal, v1.54.2 is on rollout. Some node get update some others don’t.

2 Likes

the docker image switched to something new recently…
it will not change much anymore, the storagenode version isn’t controlled by the docker image anymore… so most likely there was some sort of issue during the switch over to the new solution.

this is why you can have multiple storagenode versions.

Having different versions during rollout is a normal state. The rollout is happening by waves, what node will update first and what next depends on theirs NodeID.

1 Like

yeah its pretty awesome to see that one cannot force change the node versions anymore.
had a weird issue, but i think it was unrelated a docker of mine broke, no clue why…
so i did a ton of stuff until i figured out the storage driver was wrong.

so fixed that and got everything back online, but then later noticed that my node versions had gone back to 1.49, i doubt that is really related to storj, but it did keel my ingress…

i just think maybe i somehow rolled my docker images back to an older version, that can happen when one uses different storage drivers.

did docker rm and ran the full run command, and they all snapped back to the version i’m sure they was before i wrecked my docker… i guess that is what i get for running docker on ext4 lol and randomly rebooting a bit to often.

anyways i feel it was a bit lucky that it wasn’t a completely antique storagenode version, as that could maybe have gotten the node DQ?

Might be time to get rid of old images with a docker system prune. Though please be aware that will clean up everything that isn’t linked to a running container. This includes paused containers and related images and volumes and stuff.

3 Likes

Good call … lol

$docker system prune
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all dangling images
  - all dangling build cache

Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: storjlabs/storagenode@sha256:fec0362afdbe1ee37f799265ff5fd5f73ca93d266d0f8a824eb33fe7d23f570e
deleted: sha256:3894d69ba88cdd4eede26b648a9af79c58c668932fb5c1b7102943285937feed
deleted: sha256:b9a335d7d6b676ba6e0aa12d844313451b5fd08431b3c6f6ce44359273b08cbc
deleted: sha256:b0a968e1c25e91f574a9b15409033fca796d0eddbff78680b7dc18c562ca03dd
deleted: sha256:b5255d2ca5241b19a05ccddd63d3c095a4f3d314dcde0b57d940e0ddd51724bb
deleted: sha256:0a9f10eab020c64e537b02b0167fa3888990c4879b78d815717811ad64607c23
deleted: sha256:7d3d080af79e8bf9b47b5659cb4243b6249e4b540b2327d40987524a31adf07a
deleted: sha256:1a058d5342cc722ad5439cacae4b2b4eedde51d8fe8800fcf28444302355c16d
untagged: storjlabs/storagenode@sha256:d7f0ffa2357d8153fdb9d578f7a9c48cd64397d8fc857099d44ffa8fbf1a895e
deleted: sha256:ba7191145d7f023e14201cd1af68ce42a4ca09c6b9a427c49ba647dda3608293
deleted: sha256:b43d22064dfacab95fd0be5fe76b4b2f10bfa333b619436ebdab139251477266
deleted: sha256:9bfa41552413283021ac340fed0103f8d5159d7b4f5b2740dda29b17bec6d1e2
deleted: sha256:e49f04b7313db3016d647eff7c9664cf2c91ebd28d140ece3e38c7842c5387d4
deleted: sha256:bee4ae0cf5357dbf525671e98685f55c76eb43ce391f7b46897e1ed4e9742c21
deleted: sha256:021109298b2570993ed0235a324e365941eac52e514e0058178197246d6feda6
untagged: storjlabs/storagenode@sha256:4453c04c31a4d1f9d2cde1def7c060167839e40ad47c3b85773cf0feea447d97
deleted: sha256:f1a22a725dd5c8a237543ce9bd182532b437924cb87d8e8a4933a376e1722f82
deleted: sha256:5c66bb9c3bf96ed0f49187e3078f75cb73dec67bbc0f063b4acc6137323e6b75
deleted: sha256:81770f0a5dcffa59a931f5088d81d9565c58cbe2dedf596adb9ac99b9a5e3c23
deleted: sha256:aae48e8d15848492d04a0a66957a8e13a28b8e313c8e8f3f32368f11de026941
deleted: sha256:4ac696a4f7d63b0a388e794f1989a6197097fd39309770257e1b416154f246e4
deleted: sha256:6b013687564f91243a84a11722b5c66b31179d6d1cc0ec6628517e27cb2f86bb
untagged: storjlabs/storagenode@sha256:7d2cbab3824683fe81e2632a9a938888e969bb0365fb45a92bce4c8a18087908
deleted: sha256:4a21cc7abd9a35bcb2e653673c2f33b38ab7ce865faab4d9e2b11b06535dc673
deleted: sha256:f830989acbef8b03a94408fedd9f57db334d41107cf5fa2922aeb2e68b29056e
deleted: sha256:0b40e743a64a8691f4c1d57354d3129d4185978575bf0129492528f56c5fa497
deleted: sha256:2a8b4306fa7751e85b6deb04224da5c657fe49d0f3a068d0a09922bfbdb10246
deleted: sha256:598f57716413cdb5deedc6b186ba8cb72ca2ce5abcf8132c322bcc82c8ffa236
deleted: sha256:07160d1b3559845eb3cbd88ad64223ec39e9de2d339c395f2e2289c8d67bda4e
untagged: storjlabs/storagenode@sha256:00b719a31761c28a25ad0d12fd9305080d835ba5a82eb38c16a84716d3d41939
deleted: sha256:a6a681a94b02dee95a0c47e30951683f9ccfd443fc44ab3e0e1698f2434342fb
deleted: sha256:1b4743d61b0c44f9c0ce7590394f57f7dfba190fe00b27be2ca67fa73aa1b90b
deleted: sha256:be025fdb79a166dda55ca5b0f3e8ae63e1b2c722a17842018a545039be9151af
deleted: sha256:31d1483b15f4d1c807a977faa24d85380876de0799ac09752e70b09571ebd1e4
deleted: sha256:9546ae8567152567670713701d8b3e2fb70002f752a24343d3818c505b3ee015
deleted: sha256:8a1e0b4cd27811e729874b5f4406fcd286ffa13605ea36338b20677c53046d79
untagged: storjlabs/storagenode@sha256:056fae20996180069812c42c561ac9ac0437c0f14806264422c1c78b81148523
deleted: sha256:4ff6c157798bc1bbc82b52033de33395698c8f4ffc451c249afd7d0444debb36
deleted: sha256:3f9a0ec331cf1531fc1706c588f868b402fa5f95234ebe6434ecdfa56e08af92
deleted: sha256:9fba2a5a5b6369722387a8f4c94e0f0c98b2f8399d0a5dfc94eab04767116648
deleted: sha256:e07a03e1bf9f1d3ed54eb0a395eee05dd4e752b038073979f14854dcc6ed2af5
deleted: sha256:b67fdf7cda8ad0de0c429b7aab0d117bbbbe57a0d49512a1c088ef3b5aa1420b
deleted: sha256:b958c29d878828690012139d58551dd4a93541b43528a40298ec521c9ea6ca35
untagged: storjlabs/storagenode@sha256:0f0f8b96edfa649c438a6f36f245da73ee5af9838a36b5213a50459b41f16fda
deleted: sha256:4b848bea4b49f687d594f8d358a2f4ff2a76396db55c3c06bbeec6b0e8a89a30
deleted: sha256:3bc5463795cd696893ab7f822dc3c2556a6aab5897236e13c47a553d18193973
deleted: sha256:cc6b46ae6853737805030b7d2ef413ae776c36dca2df76cc41adfe7ab2a060df
deleted: sha256:c7d29c258007813d416e0003360b5a0f013c4e2d1b955af9004a73ead43a0242
deleted: sha256:16e6d0a3e70d0235de4904612512e2a207faaf41cf198c0afaacbacdc30d3a65
deleted: sha256:9b63675a36a3d3c95ee9887fa9d60819d05824aae30f7527e87a37cbe019ecb6
untagged: storjlabs/storagenode@sha256:6361b586050af90b3ef78533d80cb21328a2c06a162d3fb043b8245ef258d547
deleted: sha256:75ac40115d3039022fada3c42c340c2862db3c260c59f877ad650d212165b0f2
deleted: sha256:e4be347f03ecf5922123cdd26e299797942f168447a14a1af28cd32593d065bf
deleted: sha256:43750c6e3d0a31e77c007c5852be2e58a818001d6ebc3323408daa3f39303587
deleted: sha256:a69f6b4b50a62eaaeb91b24da292dee0e8227f630bdbbfe4b661c3aa362006c3
deleted: sha256:99b97668fb1d811eb6315045940249a844764778b0fe09673fe791beb1c74c07
deleted: sha256:577410cf155161ee95ce8b76b7408be8c0040e03d806e9436be6b12447825077
untagged: storjlabs/storagenode@sha256:5d1ae18419206503cf57711eb1850b8c6cd5056984473ca2efffb8d2f8b48829
deleted: sha256:e6fd5a8bc6e0b0d3d1697dcc872dac689db50e8d12e5041f12d9d5667b764009
deleted: sha256:fda39ad8fe5fb238a9f5a59618a4b2f89c9b43c3c1ef048cb7035e304fa1512b
deleted: sha256:1baec77feb48ea658bdf3db65d24d969dc4a1fbabdc0c71fec8400bcad0577dc
deleted: sha256:d4d1a4b00b7c0f5dfd2c85dde6c3209af5f51d048ddcbd1fd0da1dea66c2f7f7
deleted: sha256:ec4e124c7cde4cadae05a4b34a180242e273b865d6b89ce65fe653d258fe0248
deleted: sha256:7a77bf6a729046c0af74011c7adf8cfabd2183d19b05c968d52a766f1269fcda
untagged: storjlabs/storagenode@sha256:db3cc24d0906f1c070e875dfca2d5ae5af7a3e6b95150343b60db5bfb330be21
deleted: sha256:fe847c069ef134d41a9e2b0b521afe092c72a6c57b26d1a76066c75563776394
deleted: sha256:7c0bc98d9ac4b10dae6a615c765e0d53b4ada028b6f9418dd0f45d7b9ad5282c
deleted: sha256:7de0157e42a5ac2a4314889f4b79ae6c88f4e8deae7b2db6413038ff0a34f49d
deleted: sha256:80ad8df21cac845ea700d2045a583e916791b9e8c7bbcb6681cb05cb010a0ee0
deleted: sha256:9a8cad9a5e28178ec53a5bceec0d9296ae069ff5991a7077e375572150d0d467
deleted: sha256:a2b60c6dfc684ffe5c8193f5aa55d6a8bbebcf9b32a1c34addfe025ce4c98582
deleted: sha256:8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759
untagged: 

storjlabs/storagenode@sha256:6286a3ec196149cd5a5377318eacaabf11232af59e1c017d0b7327bd971681ff
deleted: sha256:17500c570cc0d7972461c097e79912eb8b1840fb5448c62e8e94fef79cc66ffd
deleted: sha256:6d7702334a7106cab2b517e3abc996e2ea8e427feb982ea4fd6fcde3cf7a2ce2
deleted: sha256:b05165ba03b1fd8fae01fa777f414fcfb5216425ef692ee83b2ee0d8f5ad68ad
deleted: sha256:aa57b8261095f8401cdef43a80a17a7fc2c24c6dcc0561b5085bf1655f34f223
deleted: sha256:15d231b23b15a3fe22e63b133e91ef31dd5b5ef864a5390947215b49c9cac635
deleted: sha256:58db02b1430388206120fec690f031b4ed3a2ff11f9193467c609ba99be6ad73
deleted: sha256:e2eb06d8af8218cfec8210147357a68b7e13f7c485b991c288c2d01dc228bb68

Total reclaimed space: 381.3MB
1 Like