UPLOAD failed rate still at 100% since a few days

philippe456 · August 31, 2022, 7:32am

hello,
since a few days i have INGRESS at 0.

& UPLOAD failed rate at 100%

I did remove & restart again my docker storjnode but noway.

any idea ?

thanks
best regards

peem · August 31, 2022, 8:00am

What version of SN do you have?
version 1.62.3 ???

philippe456 · August 31, 2022, 8:16am

hello @peem

Stob · August 31, 2022, 8:22am

You should check the node log file, which will show the reason for failure.

philippe456 · August 31, 2022, 8:51am

hello @Stob & @peem
here is my very recent log :

2022-08-31T08:49:15.600Z	INFO	piecestore	upload started	{Process: storagenode, Piece ID: V5ME2E75FDKHY73JH2LY362JBMSZCNO5BIGVFQQVNFQFJMEKQ57A, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Available Space: 436958913024}
2022-08-31T08:49:15.614Z	ERROR	piecestore	upload failed	{Process: storagenode, Piece ID: V5ME2E75FDKHY73JH2LY362JBMSZCNO5BIGVFQQVNFQFJMEKQ57A, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, error: pieces error: filestore error: open config/storage/temp/blob-2211903295.partial: no space left on device, errorVerbose: pieces error: filestore error: open config/storage/temp/blob-2211903295.partial: no space left on device\n\tstorj.io/storj/storage/filestore.(blobStore).Create:170\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:213\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).Upload:303\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(Handler).HandleRPC:61\n\tstorj.io/drpc/drpcserver.(Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52, Size: 0}
2022-08-31T08:49:16.280Z	INFO	piecestore	download started	{Process: storagenode, Piece ID: O4HN7FLUK2NLNDLARKUORG3HCFRT32ZYG5SIIZUVAYO7CWDOQH4A, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET}
2022-08-31T08:49:16.453Z	INFO	piecestore	download started	{Process: storagenode, Piece ID: 3HCAZ2QCAPEF5ETCTSPIRBBU3F3WKGGHPFCLW2QOY7GFBUCGQWGQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET}
2022-08-31T08:49:17.110Z	INFO	piecestore	downloaded	{Process: storagenode, Piece ID: O4HN7FLUK2NLNDLARKUORG3HCFRT32ZYG5SIIZUVAYO7CWDOQH4A, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET}
2022-08-31T08:49:17.357Z	INFO	piecestore	downloaded	{Process: storagenode, Piece ID: 3HCAZ2QCAPEF5ETCTSPIRBBU3F3WKGGHPFCLW2QOY7GFBUCGQWGQ, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: GET}
2022-08-31T08:49:18.309Z	INFO	piecestore	upload started	{Process: storagenode, Piece ID: 6THENZJSUTSWTGQDRSLHJDSNBLX3KUCKLRYXHJVFNMXQWIPL2Q3A, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, Available Space: 436958913024}
2022-08-31T08:49:18.323Z	ERROR	piecestore	upload failed	{Process: storagenode, Piece ID: 6THENZJSUTSWTGQDRSLHJDSNBLX3KUCKLRYXHJVFNMXQWIPL2Q3A, Satellite ID: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S, Action: PUT, error: pieces error: filestore error: open config/storage/temp/blob-398689262.partial: no space left on device, errorVerbose: pieces error: filestore error: open config/storage/temp/blob-398689262.partial: no space left on device\n\tstorj.io/storj/storage/filestore.(blobStore).Create:170\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:213\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).Upload:303\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(Handler).HandleRPC:61\n\tstorj.io/drpc/drpcserver.(Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52, Size: 0}
2022-08-31T08:49:19.679Z	INFO	piecestore	upload started	{Process: storagenode, Piece ID: PVSWNDU6WP6JQLQ5AVU6MAKULDVFMUG4YCTOG7RLFX7IDXICQLIA, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, Available Space: 436958913024}
2022-08-31T08:49:19.699Z	ERROR	piecestore	upload failed	{Process: storagenode, Piece ID: PVSWNDU6WP6JQLQ5AVU6MAKULDVFMUG4YCTOG7RLFX7IDXICQLIA, Satellite ID: 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs, Action: PUT, error: pieces error: filestore error: open config/storage/temp/blob-1615615273.partial: no space left on device, errorVerbose: pieces error: filestore error: open config/storage/temp/blob-1615615273.partial: no space left on device\n\tstorj.io/storj/storage/filestore.(blobStore).Create:170\n\tstorj.io/storj/storagenode/pieces.(Store).Writer:213\n\tstorj.io/storj/storagenode/piecestore.(Endpoint).Upload:303\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(Handler).HandleRPC:61\n\tstorj.io/drpc/drpcserver.(Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52, Size: 0}

it seems to say no disk space left but it’s empty à 50%

any idea ?

thx

peem · August 31, 2022, 3:22pm

What will the command show:

df -Hk

jammerdan · August 31, 2022, 3:30pm

If it’s on Linux, check if you have enough Inodes left:

philippe456 · August 31, 2022, 4:13pm

@peem :
on my NAS synology DS720+ my node is on volumeUSB1/usbshare/storj1/data

df -Hk →

philippe@nas:~$ df -Hk
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/md0 2385528 1873216 393528 83% /
devtmpfs 2920460 0 2920460 0% /dev
tmpfs 2957472 180 2957292 1% /dev/shm
tmpfs 2957472 30692 2926780 2% /run
tmpfs 2957472 0 2957472 0% /sys/fs/cgroup
tmpfs 2957472 916 2956556 1% /tmp
/dev/mapper/cachedev_0 3746098056 2053660276 1692437780 55% /volume1
tmpfs 591496 0 591496 0% /run/user/196791
/dev/usb2p1 976640640 515688192 460952448 53% /volumeUSB1/usbshare

peem · August 31, 2022, 5:01pm

ls -la /volumeUSB1/usbshare/storj1/data/storage/temp/

philippe456 · August 31, 2022, 5:21pm

@peem
ls -la /volumeUSB1/usbshare/storj1/data/storage/temp/ :
a lot of .partial files . date : 20 & 21 august
can’t print all of them

-rwxrwxrwx 1 admin users 3055616 Aug 21 07:45 blob-992625113.partial
-rwxrwxrwx 1 admin users 3293184 Aug 21 07:45 blob-99285839.partial
-rwxrwxrwx 1 admin users 913408 Aug 20 16:36 blob-992911833.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 11:29 blob-99296476.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 11:32 blob-993061337.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 23:50 blob-993123610.partial
-rwxrwxrwx 1 admin users 4194304 Aug 21 01:21 blob-993123649.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 20:24 blob-993299585.partial
-rwxrwxrwx 1 admin users 2580480 Aug 21 07:45 blob-99373631.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 19:03 blob-99471671.partial
-rwxrwxrwx 1 admin users 999424 Aug 21 07:45 blob-995197970.partial
-rwxrwxrwx 1 admin users 2650112 Aug 20 16:36 blob-995243630.partial
-rwxrwxrwx 1 admin users 2236416 Aug 20 16:36 blob-995475792.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 18:38 blob-995557607.partial
-rwxrwxrwx 1 admin users 3686400 Aug 21 07:45 blob-995575978.partial
-rwxrwxrwx 1 admin users 1110016 Aug 20 16:36 blob-996039034.partial
-rwxrwxrwx 1 admin users 3211264 Aug 20 16:36 blob-996270379.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 19:28 blob-996451715.partial
-rwxrwxrwx 1 admin users 1748992 Aug 20 16:36 blob-996543059.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 18:20 blob-996623552.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 18:22 blob-996752128.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 20:22 blob-99724645.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 19:05 blob-997391864.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 18:33 blob-997778697.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 18:59 blob-997819194.partial
-rwxrwxrwx 1 admin users 765952 Aug 20 16:36 blob-997977797.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 18:53 blob-998026660.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 11:27 blob-99839921.partial
-rwxrwxrwx 1 admin users 1540096 Aug 20 16:36 blob-998589683.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 18:21 blob-998640239.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 19:05 blob-998844441.partial
-rwxrwxrwx 1 admin users 3973120 Aug 20 16:36 blob-998931345.partial
-rwxrwxrwx 1 admin users 4194304 Aug 20 18:27 blob-999095603.partial
-rwxrwxrwx 1 admin users 2338816 Aug 21 07:45 blob-999270989.partial

peem · August 31, 2022, 6:32pm

df -i /volumeUSB1/usbshare/

philippe456 · August 31, 2022, 6:36pm

@peem

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/usb2p1 0 0 0 - /volumeUSB1/usbshare

peem · August 31, 2022, 6:45pm

This is a drive formatted as fat32 or ntfs?

philippe456 · August 31, 2022, 6:48pm

@peem
FAT32

peem · August 31, 2022, 7:06pm

You need to check if the disk has no errors after connecting to Windows.
Stop the container, remove the container, shut down Synology, unplug the disk, connect to your Windows computer and check it …

philippe456 · August 31, 2022, 7:13pm

ok i’ll do that & post the result after
thank you @peem for your help

Alexey · August 31, 2022, 7:54pm

I would recommend to migrate from FAT32 to ext4. You need to backup your data, format disk to ext4 and restore data from backup.

philippe456 · August 31, 2022, 7:58pm

windows has found some erros & repair them.

restart my node but same :

i am going to migrate for ext4

thnk you @peem & @Alexey

peem · August 31, 2022, 8:06pm

Or maybe you mount “temp” in the section:
–mount type=bind,source=

philippe456 · August 31, 2022, 8:47pm

could you tell me how ?

my config when i start my node :

so i should add :
–mount type=bind,source=xxxxxxxxxxxxxxx ??