No space left on device

robion · April 18, 2020, 5:48pm

hey there, im new to Storj since 1 month. Since the last update i have this error in the log:

no space left on device ?,

Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
udev 1,9G 0 1,9G 0% /dev
tmpfs 387M 6,6M 381M 2% /run
/dev/sda2 454G 118G 314G 28% /
tmpfs 1,9G 0 1,9G 0% /dev/shm
tmpfs 5,0M 0 5,0M 0% /run/lock
tmpfs 1,9G 0 1,9G 0% /sys/fs/cgroup
/dev/sda1 511M 7,3M 504M 2% /boot/efi
/dev/sdb1 1,9T 1,6T 294G 85% /mnt/storj
tmpfs 387M 0 387M 0% /run/user/1000

since then my Disk space used this month and Bandwith used this month is going to the bottom.
So my disk space remaining is also freezed.

Alexey · April 18, 2020, 6:01pm

Hello @robion,
Welcome to the forum!

Please, give me result of the command:

docker inspect storagenode -f "{{ .Config.Env }}" | grep -Po "STORAGE=.*? " 
docker inspect storagenode -f "{{.Mounts}}"

robion · April 18, 2020, 9:54pm

Thanks for help

STORAGE=1.7TB

[{volume 421cabcf739e7a89d37187d1b00d6f540ef8ad5da336bc3d470f86e47c65cf35 /var/lib/docker/volumes/421cabcf739e7a89d37187d1b00d6f540ef8ad5da336bc3d470f86e47c65cf35/_data /root/.local/share/storj/storagenode local true } {bind /home/server/.local/share/storj/identity/storagenode /app/identity true rprivate} {bind /mnt/storj /app/config true rprivate}]

littleskunk · April 19, 2020, 9:39am

That error message is coming from your file system and not from the storage node. The storage node still thinks there is space available.

Alexey · April 19, 2020, 11:32am

There is also a difference in the measure units. Your OS calculates space in binary measure units - TiB, the Storj software usually uses a decimal measure units - TB.
So, the used space of 1.6 TiB it’s a roughly 1.76 TB.

The storagenode still thinks there 156GB is free in your allocation, though.
But seems it’s already used more space, than allocated.
Could you please, calculate the used space (it will print out in the decimal measure units)

du -s --si /mnt/storj/storage

robion · April 19, 2020, 12:19pm

1,7T /mnt/storj/storage
Unbenannt1

Alexey · April 19, 2020, 12:33pm

Please show permissions for your storage:

ls -l /mnt/storj/storage/temp

Also,

docker exec -it storagenode df -h

Your OS return an error when it tried to create a file.
Please, stop the storagenode and check the filesystem on the drive with fsck.

robion · April 19, 2020, 12:35pm

-rwxr-xr-x 1 root root 991232 Apr 15 05:59 blob-981308571.partial
-rwxr-xr-x 1 root root 4034560 Apr 16 03:09 blob-981320129.partial
-rwxr-xr-x 1 root root 1122304 Apr 15 05:59 blob-981320863.partial
-rwxr-xr-x 1 root root 987136 Apr 16 03:05 blob-981345034.partial
-rwxr-xr-x 1 root root 2924544 Apr 15 05:59 blob-981396714.partial
-rwxr-xr-x 1 root root 790528 Apr 16 03:09 blob-981438415.partial
-rwxr-xr-x 1 root root 167936 Apr 16 03:08 blob-981498115.partial
-rwxr-xr-x 1 root root 1052672 Apr 16 03:00 blob-981535569.partial
-rwxr-xr-x 1 root root 1482752 Apr 15 05:59 blob-981601166.partial
-rwxr-xr-x 1 root root 3149824 Apr 16 02:57 blob-981627751.partial
-rwxr-xr-x 1 root root 2134016 Apr 16 03:08 blob-981713243.partial
-rwxr-xr-x 1 root root 1970176 Apr 16 03:01 blob-981715387.partial
-rwxr-xr-x 1 root root 2568192 Apr 15 05:59 blob-981774070.partial
-rwxr-xr-x 1 root root 1609728 Apr 16 03:06 blob-981777305.partial
-rwxr-xr-x 1 root root 954368 Apr 16 03:06 blob-981851571.partial
-rwxr-xr-x 1 root root 1122304 Apr 15 05:59 blob-981924344.partial
-rwxr-xr-x 1 root root 2666496 Apr 15 05:59 blob-981943846.partial
-rwxr-xr-x 1 root root 823296 Apr 16 03:06 blob-982052598.partial
-rwxr-xr-x 1 root root 266240 Apr 16 03:01 blob-982100026.partial
-rwxr-xr-x 1 root root 2887680 Apr 15 06:05 blob-982156064.partial
-rwxr-xr-x 1 root root 897024 Apr 15 05:59 blob-982225150.partial
-rwxr-xr-x 1 root root 3649536 Apr 15 05:59 blob-982409010.partial
-rwxr-xr-x 1 root root 659456 Apr 16 03:11 blob-982442544.partial
-rwxr-xr-x 1 root root 1282048 Apr 15 06:05 blob-982501919.partial
-rwxr-xr-x 1 root root 958464 Apr 15 05:59 blob-982558100.partial
-rwxr-xr-x 1 root root 1544192 Apr 16 03:00 blob-982581547.partial
-rwxr-xr-x 1 root root 1122304 Apr 15 05:59 blob-982692272.partial
-rwxr-xr-x 1 root root 757760 Apr 16 03:03 blob-982789005.partial
-rwxr-xr-x 1 root root 1544192 Apr 16 03:01 blob-982803388.partial
-rwxr-xr-x 1 root root 1413120 Apr 16 02:50 blob-982857143.partial
-rwxr-xr-x 1 root root 2887680 Apr 16 03:11 blob-983031260.partial
-rwxr-xr-x 1 root root 1413120 Apr 16 03:09 blob-983033487.partial
-rwxr-xr-x 1 root root 1445888 Apr 13 16:32 blob-983121117.partial
-rwxr-xr-x 1 root root 1839104 Apr 16 03:08 blob-983129454.partial
-rwxr-xr-x 1 root root 1314816 Apr 15 06:05 blob-983179920.partial
-rwxr-xr-x 1 root root 204800 Apr 15 05:59 blob-983274148.partial
-rwxr-xr-x 1 root root 823296 Apr 13 16:32 blob-983283423.partial
-rwxr-xr-x 1 root root 3477504 Apr 16 03:11 blob-983291665.partial
-rwxr-xr-x 1 root root 2535424 Apr 15 05:59 blob-983370011.partial
-rwxr-xr-x 1 root root 1576960 Apr 16 03:09 blob-983407626.partial
-rwxr-xr-x 1 root root 593920 Apr 13 16:32 blob-983430831.partial
-rwxr-xr-x 1 root root 1544192 Apr 13 16:31 blob-983469055.partial
-rwxr-xr-x 1 root root 4034560 Apr 13 16:31 blob-983473909.partial
-rwxr-xr-x 1 root root 397312 Apr 16 03:01 blob-983512180.partial
-rwxr-xr-x 1 root root 323584 Apr 15 05:59 blob-983530944.partial
-rwxr-xr-x 1 root root 2854912 Apr 16 03:09 blob-983536914.partial
-rwxr-xr-x 1 root root 1576960 Apr 13 16:32 blob-983632308.partial
-rwxr-xr-x 1 root root 1413120 Apr 16 02:57 blob-983654229.partial
-rwxr-xr-x 1 root root 1519616 Apr 15 05:59 blob-983670013.partial
-rwxr-xr-x 1 root root 2789376 Apr 18 11:49 blob-983682010.partial
-rwxr-xr-x 1 root root 430080 Apr 16 03:00 blob-983699234.partial
-rwxr-xr-x 1 root root 172032 Apr 15 05:59 blob-983714083.partial
-rwxr-xr-x 1 root root 921600 Apr 16 03:01 blob-983784710.partial
-rwxr-xr-x 1 root root 1413120 Apr 16 03:00 blob-983798108.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:13 blob-983822070.partial
-rwxr-xr-x 1 root root 3805184 Apr 13 16:32 blob-983937576.partial
-rwxr-xr-x 1 root root 761856 Apr 15 05:59 blob-983992275.partial
-rwxr-xr-x 1 root root 1839104 Apr 16 03:01 blob-984021070.partial
-rwxr-xr-x 1 root root 1748992 Apr 15 05:59 blob-984095519.partial
-rwxr-xr-x 1 root root 1249280 Apr 13 16:32 blob-984181332.partial
-rwxr-xr-x 1 root root 135168 Apr 13 16:32 blob-984189972.partial
-rwxr-xr-x 1 root root 1970176 Apr 16 02:59 blob-984190132.partial
-rwxr-xr-x 1 root root 2166784 Apr 13 16:32 blob-984226397.partial
-rwxr-xr-x 1 root root 1257472 Apr 15 05:59 blob-984227798.partial
-rwxr-xr-x 1 root root 12288 Apr 15 05:59 blob-984243599.partial
-rwxr-xr-x 1 root root 2396160 Apr 16 03:11 blob-984376258.partial
-rwxr-xr-x 1 root root 307200 Apr 15 05:59 blob-984377671.partial
-rwxr-xr-x 1 root root 4165632 Apr 13 09:49 blob-984392702.partial
-rwxr-xr-x 1 root root 2887680 Apr 16 03:12 blob-984430168.partial
-rwxr-xr-x 1 root root 3256320 Apr 15 05:59 blob-984508483.partial
-rwxr-xr-x 1 root root 397312 Apr 16 03:01 blob-984552043.partial
-rwxr-xr-x 1 root root 1839104 Apr 16 02:55 blob-984588836.partial
-rwxr-xr-x 1 root root 3543040 Apr 13 16:32 blob-984602492.partial
-rwxr-xr-x 1 root root 1970176 Apr 13 16:31 blob-984633464.partial
-rwxr-xr-x 1 root root 1806336 Apr 13 16:32 blob-984766783.partial
-rwxr-xr-x 1 root root 3575808 Apr 16 03:05 blob-984803958.partial
-rwxr-xr-x 1 root root 471040 Apr 15 05:59 blob-984904417.partial
-rwxr-xr-x 1 root root 1642496 Apr 16 03:09 blob-984988773.partial
-rwxr-xr-x 1 root root 1937408 Apr 16 03:07 blob-985038156.partial
-rwxr-xr-x 1 root root 2764800 Apr 15 05:59 blob-985064350.partial
-rwxr-xr-x 1 root root 102400 Apr 16 03:01 blob-985088048.partial
-rwxr-xr-x 1 root root 1318912 Apr 15 05:59 blob-985172845.partial
-rwxr-xr-x 1 root root 4042752 Apr 15 05:59 blob-985240821.partial
-rwxr-xr-x 1 root root 761856 Apr 15 05:59 blob-985301868.partial
-rwxr-xr-x 1 root root 1642496 Apr 16 03:08 blob-985305312.partial
-rwxr-xr-x 1 root root 593920 Apr 16 03:05 blob-985343669.partial
-rwxr-xr-x 1 root root 790528 Apr 16 03:00 blob-985508627.partial
-rwxr-xr-x 1 root root 1806336 Apr 16 03:11 blob-985593498.partial
-rwxr-xr-x 1 root root 1028096 Apr 15 05:59 blob-985709028.partial
-rwxr-xr-x 1 root root 3182592 Apr 13 16:32 blob-985725085.partial
-rwxr-xr-x 1 root root 1675264 Apr 16 03:01 blob-985731631.partial
-rwxr-xr-x 1 root root 4165632 Apr 15 06:05 blob-985888584.partial
-rwxr-xr-x 1 root root 3387392 Apr 15 05:59 blob-985892017.partial
-rwxr-xr-x 1 root root 4169728 Apr 15 05:59 blob-985985203.partial
-rwxr-xr-x 1 root root 1576960 Apr 16 03:11 blob-985993866.partial
-rwxr-xr-x 1 root root 1249280 Apr 16 03:00 blob-986128255.partial
-rwxr-xr-x 1 root root 659456 Apr 15 06:05 blob-986154901.partial
-rwxr-xr-x 1 root root 3870720 Apr 16 02:54 blob-986206650.partial
-rwxr-xr-x 1 root root 1347584 Apr 16 03:06 blob-986212655.partial
-rwxr-xr-x 1 root root 561152 Apr 16 02:58 blob-986270749.partial
-rwxr-xr-x 1 root root 995328 Apr 15 05:59 blob-986411725.partial
-rwxr-xr-x 1 root root 2232320 Apr 16 03:06 blob-986418366.partial
-rwxr-xr-x 1 root root 139264 Apr 15 05:59 blob-986446862.partial
-rwxr-xr-x 1 root root 1904640 Apr 16 03:08 blob-986549159.partial
-rwxr-xr-x 1 root root 733184 Apr 15 05:59 blob-986563805.partial
-rwxr-xr-x 1 root root 2461696 Apr 16 03:06 blob-986600840.partial
-rwxr-xr-x 1 root root 233472 Apr 16 03:00 blob-986755493.partial
-rwxr-xr-x 1 root root 241664 Apr 15 05:59 blob-986774702.partial
-rwxr-xr-x 1 root root 3313664 Apr 16 03:09 blob-986827478.partial
-rwxr-xr-x 1 root root 1253376 Apr 15 05:59 blob-986850952.partial
-rwxr-xr-x 1 root root 143360 Apr 15 05:59 blob-986856178.partial
-rwxr-xr-x 1 root root 397312 Apr 13 09:49 blob-986905978.partial
-rwxr-xr-x 1 root root 864256 Apr 15 05:59 blob-986999424.partial
-rwxr-xr-x 1 root root 2490368 Apr 15 05:59 blob-987077790.partial
-rwxr-xr-x 1 root root 2035712 Apr 13 16:32 blob-987092371.partial
-rwxr-xr-x 1 root root 1380352 Apr 13 16:32 blob-987116890.partial
-rwxr-xr-x 1 root root 700416 Apr 15 05:59 blob-987147185.partial
-rwxr-xr-x 1 root root 630784 Apr 15 05:59 blob-987271717.partial
-rwxr-xr-x 1 root root 1191936 Apr 15 05:59 blob-987281646.partial
-rwxr-xr-x 1 root root 2789376 Apr 16 03:03 blob-987340893.partial
-rwxr-xr-x 1 root root 2101248 Apr 16 03:10 blob-987364795.partial
-rwxr-xr-x 1 root root 1773568 Apr 13 09:49 blob-987718295.partial
-rwxr-xr-x 1 root root 4165632 Apr 13 16:31 blob-987778222.partial
-rwxr-xr-x 1 root root 4001792 Apr 16 03:13 blob-987811146.partial
-rwxr-xr-x 1 root root 102400 Apr 16 02:59 blob-987838700.partial
-rwxr-xr-x 1 root root 3158016 Apr 15 05:59 blob-987965144.partial
-rwxr-xr-x 1 root root 102400 Apr 16 03:00 blob-988059190.partial
-rwxr-xr-x 1 root root 1970176 Apr 16 03:07 blob-988091859.partial
-rwxr-xr-x 1 root root 1589248 Apr 15 05:59 blob-988215651.partial
-rwxr-xr-x 1 root root 1355776 Apr 15 05:59 blob-988223682.partial
-rwxr-xr-x 1 root root 3248128 Apr 16 03:07 blob-988308291.partial
-rwxr-xr-x 1 root root 1478656 Apr 13 09:49 blob-988329217.partial
-rwxr-xr-x 1 root root 2199552 Apr 16 03:11 blob-988338678.partial
-rwxr-xr-x 1 root root 3575808 Apr 15 06:05 blob-988339661.partial
-rwxr-xr-x 1 root root 2756608 Apr 13 16:32 blob-988431676.partial
-rwxr-xr-x 1 root root 135168 Apr 16 03:11 blob-988442522.partial
-rwxr-xr-x 1 root root 1052672 Apr 16 03:01 blob-988514730.partial
-rwxr-xr-x 1 root root 2166784 Apr 16 03:07 blob-988534569.partial
-rwxr-xr-x 1 root root 3346432 Apr 13 16:32 blob-988575539.partial
-rwxr-xr-x 1 root root 2560000 Apr 16 03:09 blob-988576697.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:11 blob-988584736.partial
-rwxr-xr-x 1 root root 860160 Apr 15 05:59 blob-988589215.partial
-rwxr-xr-x 1 root root 3444736 Apr 16 03:05 blob-988749271.partial
-rwxr-xr-x 1 root root 364544 Apr 16 03:00 blob-988818133.partial
-rwxr-xr-x 1 root root 700416 Apr 15 05:59 blob-988820086.partial
-rwxr-xr-x 1 root root 3739648 Apr 13 16:31 blob-988834458.partial
-rwxr-xr-x 1 root root 1650688 Apr 15 05:59 blob-988840441.partial
-rwxr-xr-x 1 root root 3411968 Apr 13 16:32 blob-988939749.partial
-rwxr-xr-x 1 root root 831488 Apr 15 05:59 blob-988978102.partial
-rwxr-xr-x 1 root root 3379200 Apr 13 16:32 blob-989030615.partial
-rwxr-xr-x 1 root root 233472 Apr 15 06:05 blob-989053125.partial
-rwxr-xr-x 1 root root 1552384 Apr 15 05:59 blob-989064762.partial
-rwxr-xr-x 1 root root 3411968 Apr 16 03:05 blob-989144002.partial
-rwxr-xr-x 1 root root 3485696 Apr 15 05:59 blob-989154158.partial
-rwxr-xr-x 1 root root 2625536 Apr 16 02:54 blob-989190956.partial
-rwxr-xr-x 1 root root 1675264 Apr 16 03:06 blob-989197307.partial
-rwxr-xr-x 1 root root 3248128 Apr 13 16:32 blob-989207817.partial
-rwxr-xr-x 1 root root 495616 Apr 16 03:00 blob-989247420.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:11 blob-989270110.partial
-rwxr-xr-x 1 root root 1150976 Apr 16 03:08 blob-989276343.partial
-rwxr-xr-x 1 root root 3248128 Apr 13 16:32 blob-989342679.partial
-rwxr-xr-x 1 root root 3608576 Apr 16 03:08 blob-989442210.partial
-rwxr-xr-x 1 root root 3739648 Apr 16 03:09 blob-989477308.partial
-rwxr-xr-x 1 root root 45056 Apr 15 05:59 blob-989496231.partial
-rwxr-xr-x 1 root root 1454080 Apr 15 05:59 blob-989508048.partial
-rwxr-xr-x 1 root root 430080 Apr 16 03:00 blob-989509437.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:11 blob-989610462.partial
-rwxr-xr-x 1 root root 532480 Apr 15 05:59 blob-989642753.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:11 blob-989660744.partial
-rwxr-xr-x 1 root root 692224 Apr 16 02:57 blob-989690988.partial
-rwxr-xr-x 1 root root 2199552 Apr 13 16:31 blob-989735346.partial
-rwxr-xr-x 1 root root 1183744 Apr 13 09:49 blob-989741120.partial
-rwxr-xr-x 1 root root 397312 Apr 15 06:05 blob-989750960.partial
-rwxr-xr-x 1 root root 102400 Apr 16 03:00 blob-989757074.partial
-rwxr-xr-x 1 root root 2461696 Apr 13 16:32 blob-989800400.partial
-rwxr-xr-x 1 root root 4165632 Apr 16 03:13 blob-989838296.partial
-rwxr-xr-x 1 root root 2957312 Apr 15 05:59 blob-989941126.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:09 blob-989970794.partial
-rwxr-xr-x 1 root root 172032 Apr 15 05:59 blob-990027166.partial
-rwxr-xr-x 1 root root 2068480 Apr 16 03:00 blob-990090071.partial
-rwxr-xr-x 1 root root 987136 Apr 16 03:11 blob-990182529.partial
-rwxr-xr-x 1 root root 921600 Apr 16 02:59 blob-990268651.partial
-rwxr-xr-x 1 root root 3936256 Apr 16 03:08 blob-990283653.partial
-rwxr-xr-x 1 root root 684032 Apr 15 05:59 blob-990309992.partial
-rwxr-xr-x 1 root root 4108288 Apr 15 05:59 blob-990438766.partial
-rwxr-xr-x 1 root root 397312 Apr 16 02:59 blob-990520011.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:11 blob-990529960.partial
-rwxr-xr-x 1 root root 270336 Apr 15 05:59 blob-990532450.partial
-rwxr-xr-x 1 root root 2895872 Apr 15 05:59 blob-990737048.partial
-rwxr-xr-x 1 root root 667648 Apr 15 05:59 blob-990837223.partial
-rwxr-xr-x 1 root root 1576960 Apr 13 16:32 blob-990884752.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:11 blob-991004870.partial
-rwxr-xr-x 1 root root 3870720 Apr 13 16:32 blob-991097628.partial
-rwxr-xr-x 1 root root 2011136 Apr 15 05:59 blob-991127323.partial
-rwxr-xr-x 1 root root 2596864 Apr 15 05:59 blob-991197371.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:11 blob-991223376.partial
-rwxr-xr-x 1 root root 1085440 Apr 13 16:32 blob-991298221.partial
-rwxr-xr-x 1 root root 430080 Apr 15 06:05 blob-991322650.partial
-rwxr-xr-x 1 root root 12288 Apr 15 05:59 blob-991426662.partial
-rwxr-xr-x 1 root root 897024 Apr 15 05:59 blob-991430839.partial
-rwxr-xr-x 1 root root 1347584 Apr 13 09:49 blob-991476721.partial
-rwxr-xr-x 1 root root 929792 Apr 15 05:59 blob-991494540.partial
-rwxr-xr-x 1 root root 3051520 Apr 16 03:06 blob-991497271.partial
-rwxr-xr-x 1 root root 2625536 Apr 16 03:06 blob-991507520.partial
-rwxr-xr-x 1 root root 2560000 Apr 16 03:11 blob-991559052.partial
-rwxr-xr-x 1 root root 2461696 Apr 16 03:08 blob-991734274.partial
-rwxr-xr-x 1 root root 1118208 Apr 16 03:01 blob-991915590.partial
-rwxr-xr-x 1 root root 1712128 Apr 15 05:59 blob-991974751.partial
-rwxr-xr-x 1 root root 1282048 Apr 16 03:08 blob-992077284.partial
-rwxr-xr-x 1 root root 1740800 Apr 16 03:11 blob-992118753.partial
-rwxr-xr-x 1 root root 102400 Apr 16 03:01 blob-992258012.partial
-rwxr-xr-x 1 root root 3575808 Apr 16 02:49 blob-992323467.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:12 blob-992326764.partial
-rwxr-xr-x 1 root root 1126400 Apr 15 05:59 blob-992331595.partial
-rwxr-xr-x 1 root root 2199552 Apr 16 03:08 blob-992399564.partial
-rwxr-xr-x 1 root root 2363392 Apr 16 03:08 blob-992489037.partial
-rwxr-xr-x 1 root root 3117056 Apr 13 16:32 blob-992614619.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:13 blob-992624203.partial
-rwxr-xr-x 1 root root 790528 Apr 16 03:06 blob-992688378.partial
-rwxr-xr-x 1 root root 1122304 Apr 15 05:59 blob-992696241.partial
-rwxr-xr-x 1 root root 1740800 Apr 13 16:31 blob-992895932.partial
-rwxr-xr-x 1 root root 3313664 Apr 16 03:03 blob-992940646.partial
-rwxr-xr-x 1 root root 499712 Apr 15 05:59 blob-993107885.partial
-rwxr-xr-x 1 root root 602112 Apr 15 05:59 blob-993127787.partial
-rwxr-xr-x 1 root root 1417216 Apr 15 05:59 blob-993169599.partial
-rwxr-xr-x 1 root root 856064 Apr 16 03:11 blob-993212424.partial
-rwxr-xr-x 1 root root 139264 Apr 15 05:59 blob-993247317.partial
-rwxr-xr-x 1 root root 3313664 Apr 13 16:32 blob-993371579.partial
-rwxr-xr-x 1 root root 3616768 Apr 15 05:59 blob-993388116.partial
-rwxr-xr-x 1 root root 2789376 Apr 13 09:49 blob-993541956.partial
-rwxr-xr-x 1 root root 593920 Apr 16 03:13 blob-993553114.partial
-rwxr-xr-x 1 root root 692224 Apr 16 03:01 blob-993567775.partial
-rwxr-xr-x 1 root root 1351680 Apr 15 05:59 blob-993607687.partial
-rwxr-xr-x 1 root root 135168 Apr 13 16:32 blob-993618322.partial
-rwxr-xr-x 1 root root 3018752 Apr 13 16:32 blob-993677036.partial
-rwxr-xr-x 1 root root 2240512 Apr 15 05:59 blob-993684960.partial
-rwxr-xr-x 1 root root 69632 Apr 13 09:49 blob-993695615.partial
-rwxr-xr-x 1 root root 2007040 Apr 15 05:59 blob-993742092.partial
-rwxr-xr-x 1 root root 3280896 Apr 13 09:49 blob-993743478.partial
-rwxr-xr-x 1 root root 3444736 Apr 16 03:06 blob-993822464.partial
-rwxr-xr-x 1 root root 102400 Apr 16 03:00 blob-993854454.partial
-rwxr-xr-x 1 root root 856064 Apr 16 03:09 blob-993863795.partial
-rwxr-xr-x 1 root root 2134016 Apr 16 03:11 blob-993926418.partial
-rwxr-xr-x 1 root root 1216512 Apr 16 03:01 blob-993936402.partial
-rwxr-xr-x 1 root root 2269184 Apr 15 05:59 blob-993946027.partial
-rwxr-xr-x 1 root root 2764800 Apr 15 05:59 blob-993998509.partial
-rwxr-xr-x 1 root root 1159168 Apr 15 05:59 blob-994051387.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:11 blob-994055514.partial
-rwxr-xr-x 1 root root 724992 Apr 15 06:05 blob-994128029.partial
-rwxr-xr-x 1 root root 3743744 Apr 15 05:59 blob-994194702.partial
-rwxr-xr-x 1 root root 2297856 Apr 16 03:09 blob-994208033.partial
-rwxr-xr-x 1 root root 1708032 Apr 16 03:09 blob-994254809.partial
-rwxr-xr-x 1 root root 692224 Apr 16 03:00 blob-994384777.partial
-rwxr-xr-x 1 root root 2596864 Apr 15 05:59 blob-994389112.partial
-rwxr-xr-x 1 root root 110592 Apr 15 05:59 blob-994598627.partial
-rwxr-xr-x 1 root root 40960 Apr 15 05:59 blob-994649505.partial
-rwxr-xr-x 1 root root 2465792 Apr 15 05:59 blob-994714983.partial
-rwxr-xr-x 1 root root 1515520 Apr 15 05:59 blob-994730919.partial
-rwxr-xr-x 1 root root 3411968 Apr 16 03:06 blob-994764014.partial
-rwxr-xr-x 1 root root 3018752 Apr 16 03:11 blob-994825126.partial
-rwxr-xr-x 1 root root 1282048 Apr 13 16:31 blob-994984490.partial
-rwxr-xr-x 1 root root 831488 Apr 15 05:59 blob-995038376.partial
-rwxr-xr-x 1 root root 1118208 Apr 16 03:09 blob-995171326.partial
-rwxr-xr-x 1 root root 856064 Apr 13 09:49 blob-995261794.partial
-rwxr-xr-x 1 root root 2543616 Apr 15 05:59 blob-995302769.partial
-rwxr-xr-x 1 root root 12288 Apr 15 05:59 blob-995339682.partial
-rwxr-xr-x 1 root root 3313664 Apr 13 16:32 blob-995381268.partial
-rwxr-xr-x 1 root root 741376 Apr 15 05:59 blob-995432869.partial
-rwxr-xr-x 1 root root 102400 Apr 16 02:59 blob-995527028.partial
-rwxr-xr-x 1 root root 266240 Apr 15 06:05 blob-995570938.partial
-rwxr-xr-x 1 root root 761856 Apr 15 05:59 blob-995586121.partial
-rwxr-xr-x 1 root root 3706880 Apr 16 03:09 blob-995609778.partial
-rwxr-xr-x 1 root root 3936256 Apr 16 03:06 blob-995679370.partial
-rwxr-xr-x 1 root root 2953216 Apr 16 03:11 blob-995693690.partial
-rwxr-xr-x 1 root root 3117056 Apr 13 09:49 blob-995812746.partial
-rwxr-xr-x 1 root root 3608576 Apr 16 03:09 blob-995894233.partial
-rwxr-xr-x 1 root root 724992 Apr 16 03:05 blob-995950538.partial
-rwxr-xr-x 1 root root 2691072 Apr 16 03:09 blob-995959488.partial
-rwxr-xr-x 1 root root 3444736 Apr 16 03:10 blob-995972476.partial
-rwxr-xr-x 1 root root 4173824 Apr 15 05:59 blob-995981173.partial
-rwxr-xr-x 1 root root 3215360 Apr 13 16:32 blob-996001363.partial
-rwxr-xr-x 1 root root 4001792 Apr 16 03:09 blob-996001745.partial
-rwxr-xr-x 1 root root 3842048 Apr 15 05:59 blob-996030898.partial
-rwxr-xr-x 1 root root 1839104 Apr 16 02:56 blob-996111589.partial
-rwxr-xr-x 1 root root 3313664 Apr 16 03:12 blob-996168242.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:10 blob-996173143.partial
-rwxr-xr-x 1 root root 2699264 Apr 15 05:59 blob-996199392.partial
-rwxr-xr-x 1 root root 3641344 Apr 16 02:55 blob-996270339.partial
-rwxr-xr-x 1 root root 331776 Apr 16 03:01 blob-996301226.partial
-rwxr-xr-x 1 root root 921600 Apr 13 16:31 blob-996475604.partial
-rwxr-xr-x 1 root root 2793472 Apr 15 05:59 blob-996524667.partial
-rwxr-xr-x 1 root root 172032 Apr 15 05:59 blob-996538111.partial
-rwxr-xr-x 1 root root 1773568 Apr 13 16:32 blob-996547722.partial
-rwxr-xr-x 1 root root 2236416 Apr 15 05:59 blob-996593724.partial
-rwxr-xr-x 1 root root 3248128 Apr 13 16:32 blob-996593984.partial
-rwxr-xr-x 1 root root 1552384 Apr 15 05:59 blob-996624970.partial
-rwxr-xr-x 1 root root 733184 Apr 15 05:59 blob-996719154.partial
-rwxr-xr-x 1 root root 3805184 Apr 13 09:49 blob-996926308.partial
-rwxr-xr-x 1 root root 462848 Apr 16 03:01 blob-996949029.partial
-rwxr-xr-x 1 root root 790528 Apr 16 03:11 blob-996964727.partial
-rwxr-xr-x 1 root root 3153920 Apr 15 05:59 blob-996966452.partial
-rwxr-xr-x 1 root root 692224 Apr 16 03:11 blob-997074865.partial
-rwxr-xr-x 1 root root 2134016 Apr 13 09:49 blob-997115652.partial
-rwxr-xr-x 1 root root 2592768 Apr 16 03:11 blob-997157949.partial
-rwxr-xr-x 1 root root 3969024 Apr 13 09:49 blob-997207891.partial
-rwxr-xr-x 1 root root 724992 Apr 16 02:54 blob-997262117.partial
-rwxr-xr-x 1 root root 1257472 Apr 15 05:59 blob-997286853.partial
-rwxr-xr-x 1 root root 3411968 Apr 16 03:09 blob-997441442.partial
-rwxr-xr-x 1 root root 2789376 Apr 16 03:00 blob-997446389.partial
-rwxr-xr-x 1 root root 2273280 Apr 15 05:59 blob-997471725.partial
-rwxr-xr-x 1 root root 2723840 Apr 16 03:11 blob-997605042.partial
-rwxr-xr-x 1 root root 3772416 Apr 13 16:32 blob-997682270.partial
-rwxr-xr-x 1 root root 1683456 Apr 15 05:59 blob-997705891.partial
-rwxr-xr-x 1 root root 1708032 Apr 16 03:06 blob-997751116.partial
-rwxr-xr-x 1 root root 4071424 Apr 15 05:59 blob-997774462.partial
-rwxr-xr-x 1 root root 204800 Apr 15 05:59 blob-997798789.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:13 blob-997804528.partial
-rwxr-xr-x 1 root root 12288 Apr 15 05:59 blob-997821227.partial
-rwxr-xr-x 1 root root 73728 Apr 15 05:59 blob-997832885.partial
-rwxr-xr-x 1 root root 565248 Apr 15 05:59 blob-997887504.partial
-rwxr-xr-x 1 root root 2592768 Apr 16 03:08 blob-997908183.partial
-rwxr-xr-x 1 root root 77824 Apr 15 05:59 blob-997919256.partial
-rwxr-xr-x 1 root root 3969024 Apr 16 03:08 blob-997949610.partial
-rwxr-xr-x 1 root root 1261568 Apr 15 05:59 blob-998059294.partial
-rwxr-xr-x 1 root root 2629632 Apr 15 05:59 blob-998073446.partial
-rwxr-xr-x 1 root root 724992 Apr 16 03:05 blob-998166892.partial
-rwxr-xr-x 1 root root 3969024 Apr 16 03:08 blob-998200312.partial
-rwxr-xr-x 1 root root 954368 Apr 16 03:01 blob-998204451.partial
-rwxr-xr-x 1 root root 1839104 Apr 16 03:06 blob-998205030.partial
-rwxr-xr-x 1 root root 954368 Apr 13 16:31 blob-998226877.partial
-rwxr-xr-x 1 root root 372736 Apr 15 05:59 blob-998244425.partial
-rwxr-xr-x 1 root root 532480 Apr 15 05:59 blob-998274142.partial
-rwxr-xr-x 1 root root 1478656 Apr 15 06:06 blob-998291177.partial
-rwxr-xr-x 1 root root 2363392 Apr 13 09:49 blob-998350574.partial
-rwxr-xr-x 1 root root 3084288 Apr 16 03:09 blob-998382798.partial
-rwxr-xr-x 1 root root 561152 Apr 16 03:00 blob-998383190.partial
-rwxr-xr-x 1 root root 2695168 Apr 15 05:59 blob-998383543.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:06 blob-998439296.partial
-rwxr-xr-x 1 root root 1904640 Apr 16 03:01 blob-998478702.partial
-rwxr-xr-x 1 root root 1028096 Apr 15 05:59 blob-998491529.partial
-rwxr-xr-x 1 root root 1417216 Apr 15 05:59 blob-998502005.partial
-rwxr-xr-x 1 root root 2760704 Apr 15 05:59 blob-998527116.partial
-rwxr-xr-x 1 root root 3411968 Apr 16 03:09 blob-998599766.partial
-rwxr-xr-x 1 root root 1318912 Apr 15 05:59 blob-998614068.partial
-rwxr-xr-x 1 root root 200704 Apr 16 03:01 blob-998663382.partial
-rwxr-xr-x 1 root root 1388544 Apr 15 05:59 blob-998711609.partial
-rwxr-xr-x 1 root root 761856 Apr 15 05:59 blob-998750647.partial
-rwxr-xr-x 1 root root 200704 Apr 16 03:00 blob-998880146.partial
-rwxr-xr-x 1 root root 2957312 Apr 15 05:59 blob-999029855.partial
-rwxr-xr-x 1 root root 831488 Apr 15 05:59 blob-999057738.partial
-rwxr-xr-x 1 root root 1445888 Apr 15 06:05 blob-999068344.partial
-rwxr-xr-x 1 root root 3551232 Apr 15 05:59 blob-999139557.partial
-rwxr-xr-x 1 root root 888832 Apr 16 03:06 blob-999234617.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:13 blob-999265321.partial
-rwxr-xr-x 1 root root 1486848 Apr 15 05:59 blob-999352284.partial
-rwxr-xr-x 1 root root 4100096 Apr 16 03:11 blob-999428729.partial
-rwxr-xr-x 1 root root 3248128 Apr 16 03:06 blob-999472362.partial
-rwxr-xr-x 1 root root 1159168 Apr 15 05:59 blob-999516969.partial
-rwxr-xr-x 1 root root 1552384 Apr 15 05:59 blob-999549930.partial
-rwxr-xr-x 1 root root 2658304 Apr 16 03:06 blob-999550677.partial
-rwxr-xr-x 1 root root 4096 Apr 16 03:13 blob-999630472.partial
-rwxr-xr-x 1 root root 4136960 Apr 15 05:59 blob-999727969.partial
-rwxr-xr-x 1 root root 2199552 Apr 16 02:41 blob-999763468.partial
-rwxr-xr-x 1 root root 1314816 Apr 16 03:08 blob-999778434.partial
-rwxr-xr-x 1 root root 3280896 Apr 13 16:32 blob-999779660.partial
-rwxr-xr-x 1 root root 2363392 Apr 16 03:02 blob-999857135.partial
-rwxr-xr-x 1 root root 1875968 Apr 15 05:59 blob-999978186.partial

Filesystem Size Used Available Use% Mounted on
overlay 454.0G 117.5G 313.4G 27% /
tmpfs 64.0M 0 64.0M 0% /dev
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
shm 64.0M 0 64.0M 0% /dev/shm
/dev/sda2 454.0G 117.5G 313.4G 27% /app/identity
/dev/sdb1 1.8T 1.5T 293.3G 84% /app/config
/dev/sda2 454.0G 117.5G 313.4G 27% /etc/resolv.conf
/dev/sda2 454.0G 117.5G 313.4G 27% /etc/hostname
/dev/sda2 454.0G 117.5G 313.4G 27% /etc/hosts
/dev/sda2 454.0G 117.5G 313.4G 27% /root/.local/share/storj/storagenode
tmpfs 1.9G 0 1.9G 0% /proc/asound
tmpfs 1.9G 0 1.9G 0% /proc/acpi
tmpfs 64.0M 0 64.0M 0% /proc/kcore
tmpfs 64.0M 0 64.0M 0% /proc/keys
tmpfs 64.0M 0 64.0M 0% /proc/timer_list
tmpfs 64.0M 0 64.0M 0% /proc/timer_stats
tmpfs 64.0M 0 64.0M 0% /proc/sched_debug
tmpfs 1.9G 0 1.9G 0% /proc/scsi
tmpfs 1.9G 0 1.9G 0% /sys/firmware

Alexey · April 19, 2020, 2:36pm

Please, do it. It should fix possible issues

robion · April 19, 2020, 3:02pm

/dev/sda2 is mounted. e2fsck cannot continue aborting

robion · April 19, 2020, 3:11pm

umount /dev/sda2

umount: /: target is busy
(In some cases useful info about processes that
use the device is found by lsof(8) or fuser(1).)

Alexey · April 19, 2020, 3:57pm

Did you stop the storagenode before continue?

robion · April 19, 2020, 4:06pm

yes Storagenode is stopped

Alexey · April 19, 2020, 4:18pm

sudo umount /mnt/storj
sudo fsck /dev/sda2

robion · April 19, 2020, 8:24pm

server@serv:~ sudo umount /mnt/storj server@serv:~ sudo fsck /dev/sda2
fsck from util-linux 2.27.1
e2fsck 1.42.13 (17-May-2015)
/dev/sda2 is mounted.
e2fsck: Cannot continue, aborting.

donald.m.motsinger · April 19, 2020, 9:00pm

sudo lsof /mnt/storj

Alexey · April 20, 2020, 7:52pm

2 posts were split to a new topic: Would like to add another hard drive

robion · April 21, 2020, 7:04pm

lsof returns nothing, but the drive is only used for Storj. I dont understand this, but the problem consist still

robion · April 23, 2020, 6:16pm

No more answers ? xD

Alexey · April 23, 2020, 10:42pm

You need to check that drive.
Please, edit your /etc/fstab for that drive and replace the last 0 to 2, save the file and reboot the system.
You can read more there: https://linuxconfig.org/how-to-force-fsck-to-check-filesystem-after-system-reboot-on-linux