Hi, I have two nodes with two different VPN (I must use it on CGNAT network). One of this node seems very slow during downloading. Speedtest seems ok but I think is a delay or dns problem to slow down everything.
My question is: Can I benchmark my nodes? Can I simulate storj gateway test on my nodes?
personally i might try to switch the vpn’s around on the nodes…
that would indicate if one vpn is performing worse than another.
if that is not the case then it has to be the one node, that could be on a SMR HDD or something.
latency would usually be an indicator of a bad connection or stressed cpu, hdd or such things…
vpn’s use encryption and not all cpu’s can hardware offload such tasks.
if you are on linux i would recommend netdata it is very good for finding issues.
its an amazing tool and have helped me many times, it can be a bit resource heavy to run on smaller / older systems… so might want to turn it off again when you are done.
i’m sure there are similar good tools for windows, but nothing comes to mind right now.
I dont think I can emulate storj clients with netdata but simply my connection. I need to simulate clients requests understanding if I’m serving files with a decent “reactivity” or not
You could try capturing some incoming packets with wireshark, then replaying them with hping. Your node will obviously fail the request, as crypto will not match, but at least you can benchmark the establishment of the TCP connection/routing UDP and the first reply.
I think this will be very good featere from storj, if it will be posible to Benchmark nodes from different satelites. then node operators can think what they can do better, or if there is some bottleneck. This will be very healthy for the whole storj system. As node operators will start to understand why their node perform good or bad.
4 Likes
Data is transferred directly from the customers, not satellites. Measurement from the satellites’ location will be useless.
yes netdata is monitoring, but it can usually tell you if there is something misbehaving with the hardware or other odd behavior.
alternatively you could try to do read write and io tests on the HDD’s and see how they perform.
in most cases of nodes performing worse than one another, its the HDD’s that play the major factor even tho other things can impact ingress and downloads.
using storj as a benchmark, tho it can be done using the storj network private version, i don’t think anyone ever really tried that… and one would basically have to build a mini storj network to do that… so its just not really viable.
besides, tho it would be cool with an official benchmark tool from storj, thus far we haven’t really had much practical use for one, as most stuff can be benchmarked just fine using 3rd party software solutions.
but maybe you are right, its about time we got an official storj software suit recommendation of which 3rd party softwares to use for benchmarking node performance.
maybe a bit of a how to manual for the individual tests.
Then it cold be possibilities to send data to specific nodes for the test prepose. and read them.
It will help owners to improve nodes. It’s difficult to understand if you are going to “serve” your files fast or not during the whole process from client back to client.
It’s time to improve our performance to compete with s3 
with programming it could ofc be possible to have like temporary benchmark pieces, and then a node could request a benchmark which other nodes performed on it.
ofc this sort of feature would need some limitations and possible protections against exploits.
and not sure such a feature would be high on storjlabs to do list…
also there would be the question of difficulty, as it might involve rather deep changes to code / options the node software provides.
No, it could not. Again, data is transferred from the customers, not satellites. If we run own uplinks for that purpose, you can only measure something for that one location. This benchmark will be useless, It would be like testing your internet speed with Speedtest - many ISP optimizes their routes to get fastest result with Speedtest, this would be the same. You would optimize your nodes for this one location, but will have worse results with the real customers from other locations.
3 Likes
Ipv6 works but then you will serve only ipv6 clients, if I understand well.
My thoughts about performance testing is to start from simple and then trying to testing peering conditions. We can just start to test a file request from a specific location (maybe one of the gateways). This can help to understand if you have bottleneck somewhere in your configuration (vpn, smr, usb channel busy from other devices that slowdown everything… etc).
I think many nodes operators are not system administrators. We need simply tools for a better storj speed tuning. If speed matters…
I think there was an idea posted on the forum some time ago to build a benchmark into storage node code. As in, each node would periodically test other nodes and share results, either to a satellite, or directly to the peer node. As there is now ~15k nodes all over the world, each node operator would get a lot of data.
It would be quite a lot of coding though, as well as adding some sort of social contract (how much do you trust other nodes to give true results?).
up/dl?
who will pay for this?
1 Like
New social contract.
Yeah. It would still more useful than a synthetic benchmark from a single location.
There’s a saying, you never have the right data, so you need to leverage data you have (paraphrased).
And… Who need it? If you have 1 node with 2-5tb, you dont need another benchmark. You can benchmark network and disk with standard utils with google. If you have 10+ nodes and 50+tb, you dont need benchmark, becouse you can (and know how) benchmark your hardware by standard utils.
Network benchmark-torrents.
Hdd benchmark - torrents with low seeders and high leechers. And little script for create, read, delete thousands small files
I don’t know who needs it. I know I’d gladly trade off some of my bandwidth in exchange for some other nodes provide me additional insight into how well my nodes function.
How? I have a node in my home. I buy a VPS and I want to simulate e file request to my home node and check speed of the entire operation.
You’re going to test from that city to your pc. Yes, it’s easy. I’ve asked something different.
I need to test storj node too.
If you have nodes installed in VM even changing vm software can improve your latency, or just change some settings. There are multiple layers of latencies in some environment.