Question about upload data size

Node Operators FAQ
1

Hi,

When I see line like this:

2020-08-07T12:06:11.760Z        INFO    piecestore      uploaded        {"Piece ID": "HFPOLTEWNRBQC4DX2V5OXWN4KGK5LTB5SZYY7DNJT76MZLUS2DGA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}

what is size of data uploaded? Is it always in pieces of same size (like 2MB) or it can be any size (please give me average then)

Thanks in advance

2

You can deduce that from 2 consecutive available space lines in the log.

image
image1497×44 8.55 KB

2 Likes
3

Oh, thanks. I checked it like you suggest before, but results are somewthat strange.
Most pieces uploaded have size like few kB.
But it solves my other question - how my node can get 100k+ uploads in half a day, but ingress bandwidth is only like 20GB :wink:

Thanks!

2 Likes
4

Not if there were deletes. I actually wish for each upload/download to state the amount in the logs, would be nice to put on graphs.

5

If you’re decent with sqlite, this data is available in the node db’s.

6

I don’t think the database data is so granular?

7

Why assume when you can verify ? You do know @BrightSilence has posted earnings script which uses databases so when he says its there, it should be there. He never posts without valid claim.

image
image968×332 17.2 KB

As you can see in the pieceinfo.db there is literally a field for piece_size.

8

Again, not for deleted pieces, then.

9

The title is stating Question about upload data size, not deleted :man_shrugging:

However, it has this data too, include pieces in trash. You can use the join with table piece_expiration.db to figure out, which pieces are currently in trash.

Also, why do not use your filesystem to check that out?

find /mnt/storj/storagenode/storage/trash/ -type f -printf "%s %f\n"
10

Every deleted piece has been uploaded at some point. Its size is then lost if the piece if deleted and garbage-collected. This is the information I wish I had in the logs.

11

For post analysis it’s highly recommended to use some kind of log collection service and monitoring system in place.
The storagenode will remove the information regarding deleted pieces when they will be deleted from the disk, you can collect this information only while it is in the trash. You have only 7 days for that.

12

How about in the dashboard ?

13

For that to be implemented the node would have to keep more information on deletes in its db’s. I’m guessing this will be a new action in the existing tables in bandwidth.db. however, that data gets aggregated (rolled up) quickly to prevent the db from getting very large and slow. So you won’t have per piece information. But totals would be there. This is however still speculation on how that may be implemented.

14

Again. It would be enough for the uploads to report their sizes.

15

Again, they do, look inside the bandwidth.db for recent uploads (since last roll up) and pieceinfo.db for sizes of all pieces.

16

I’ve just noticed that somewhere between 1.49 and 1.53 this feature has been implemented… well, at least for uploads. Thank you!

1 Like