# Upcoming storage node improvements including benchmark tool **URL:** \ **Category:** STORJLINGS\ **Tags:** storagenode\ **Created:** [May 5, 2024, 3:28pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096 "2024-05-05T15:28:39Z")\ **Posts on this page:** 20\ **Page:** 6
**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/5_5575768a8748004e209b776fc1b2916d.png) [@Roxor](https://forum.storj.io/u/Roxor)\ **Post date:** [May 9, 2024, 6:38pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/109 "2024-05-09T18:38:31Z")
> [@Ruskiem](#): > > ppl could get some feedback, and chance to improve, or get interested and improve! We already have [a script](https://forum.storj.io/t/success-rate-script-now-updated-for-new-delete-terminology-since-v1-29-3/5114) any SNO can run, whenever they want, that shows if they’re winning most upload/download races… or have areas to improve. And it’s based on the success of client requests: the actions that directly influence payouts. I can understand Storj needing internal rankings though: that could help their dev and sales teams. ---
**Author:** ![Ruskiem](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/ruskiem/32/64_2.png) [@Ruskiem](https://forum.storj.io/u/Ruskiem)\ **Post date:** [May 9, 2024, 6:59pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/110 "2024-05-09T18:59:03Z")
yea but noone wants to run it, i tried and got some Powershell errors and give up. i rememebr back in days it worked, but then stopped and im currently unable to run any test, so will welcome with open hands any statistic if Storj can run it for me ---
**Author:** ![thelastspark](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/thelastspark/32/1132_2.png) [@thelastspark](https://forum.storj.io/u/thelastspark)\ **Post date:** [May 9, 2024, 7:00pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/111 "2024-05-09T19:00:38Z")
I have go, I have gcc but what do I enter into cmd to run the test? Additionally for the sqlite3 step: ![image](https://storj-s3.bcdn.literatehosting.net/original/3X/1/2/1242aa18385821a175336198b56717aeebe3fdbe.png) ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [May 9, 2024, 7:04pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/112 "2024-05-09T19:04:09Z")
> [@Ruskiem](#): > > if Storj can run it for me We don’t have access to your node and are unable to run it for you. Running a storage node is easy. Optimizing it requires additional knowledge and there is no shortcut. We can provide help for setting up grafana or the new benchmark. ---
**Author:** ![Mitsos](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mitsos/32/1125_2.png) [@Mitsos](https://forum.storj.io/u/Mitsos)\ **Post date:** [May 9, 2024, 9:56pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/113 "2024-05-09T21:56:43Z")
Installed 1.104.0-rc and skipped lazy used space because I saw the comment above about trash being fixed later today. Installed 1.104.1 and enabled lazy used. All I can say is it flies compared to before. ~1h for ~5TB (dbs still on disk). Well done. ---
**Author:** ![agente](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/agente/32/1137_2.png) [@agente](https://forum.storj.io/u/agente)\ **Post date:** [May 9, 2024, 10:49pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/115 "2024-05-09T22:49:01Z")
> [@IsThisOn](#): > > So from **532 MiB/s** down to **30MiB/s** Without fsync patch? Maybe less than 5mib… ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [May 10, 2024, 12:55am UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/117 "2024-05-10T00:55:39Z")
> [@Mitsos](#): > > Installed 1.104.1 and enabled lazy used. All I can say is it flies compared to before. ~1h for ~5TB (dbs still on disk). Well done. You mean the lazy file walker is faster now? How long was it before? This is just a side effect. We didn’t touch the lazy file walker itself. We made the uploads cheaper so that the lazy file walker has more IOPs available to run a little faster. ---
**Author:** ![Mitsos](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mitsos/32/1125_2.png) [@Mitsos](https://forum.storj.io/u/Mitsos)\ **Post date:** [May 10, 2024, 1:08am UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/118 "2024-05-10T01:08:26Z")
Yea it’s a lot faster since it doesn’t have to pause all the time waiting for IO. I don’t remember exact figures, but it was \> 3 hours for sure on that node (node runs on array that is used for other things as well). ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [May 10, 2024, 1:09am UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/119 "2024-05-10T01:09:46Z")
Awesome. Thats a nice side effect. ---
**Author:** ![Ambifacient](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/ambifacient/32/7815_2.png) [@Ambifacient](https://forum.storj.io/u/Ambifacient)\ **Post date:** [May 10, 2024, 4:53am UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/120 "2024-05-10T04:53:14Z")
I’ve also manually upgraded one of my nodes on XFS to v1.104.1, and you can clearly observe the difference (upgrade was taken at around 00:43): ![Screenshot 2024-05-10 at 12.50.22 AM](https://storj-s3.bcdn.literatehosting.net/original/3X/a/a/aa8675cf87fe3a59c295a2b9b12277f0466031a6.jpeg) ![Screenshot 2024-05-10 at 12.50.28 AM](https://storj-s3.bcdn.literatehosting.net/original/3X/a/b/abd2e665ff67ee5465f381a6d96341c5f41c8d78.png) And this is with **both** garbage collection and trash emptying taking place, and garbage collection was occurring before the upgrade. The read operations are getting a big boost. The intermittency in the reads seems related to when the data is being flushed from memory, not 100% sure though. Unfortunately I think a side effect of the bandwidth DB change is that the existing prometheus exporter no longer gives nice bandwidth charts in Grafana. ---
**Author:** ![agente](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/agente/32/1137_2.png) [@agente](https://forum.storj.io/u/agente)\ **Post date:** [May 10, 2024, 6:40am UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/121 "2024-05-10T06:40:31Z")
> [@Ambifacient](#): > > I’ve also manually upgraded one of my nodes on XFS to v1.104.1, and you can clearly observe the difference (upgrade was taken at around 00:43): This graph is better than any tests to explain the improvement PS: How long it takes usually to go live ver 1.104? ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [May 10, 2024, 10:00am UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/122 "2024-05-10T10:00:00Z")
> [@Ambifacient](#): > > Unfortunately I think a side effect of the bandwidth DB change is that the existing prometheus exporter no longer gives nice bandwidth charts in Grafana. I don’t see that on my Grafana or storage node dashboard. To me it looks like it should continue to work just fine. ---
**Author:** ![Ambifacient](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/ambifacient/32/7815_2.png) [@Ambifacient](https://forum.storj.io/u/Ambifacient)\ **Post date:** [May 10, 2024, 1:45pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/123 "2024-05-10T13:45:15Z")
For reference I am using this exporter: [GitHub - anclrii/Storj-Exporter: Prometheus exporter for monitoring Storj storage nodes](https://github.com/anclrii/Storj-Exporter) My graphs look like this: ![Screenshot 2024-05-10 at 9.44.02 AM](https://storj-s3.bcdn.literatehosting.net/original/3X/5/8/58eb25d21a80261c5dee4852afd8f4a37214f30c.png) It’s only picking up the periodic flushes. Are you using the same exporter? ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [May 10, 2024, 1:50pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/124 "2024-05-10T13:50:15Z")
> [@Ambifacient](#): > > Are you using the same exporter? No. I don’t like the idea of running some third party tool that basically gets full access to my storage node. Instead I use the build in metrics endpoint that works without any additional log scraping tool. ---
**Author:** ![Ambifacient](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/ambifacient/32/7815_2.png) [@Ambifacient](https://forum.storj.io/u/Ambifacient)\ **Post date:** [May 10, 2024, 1:52pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/125 "2024-05-10T13:52:01Z")
Ah gotcha. I will have to check that out. In the meantime I’ll see if I can patch this. ---
**Author:** ![zip](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/zip/32/7813_2.png) [@zip](https://forum.storj.io/u/zip)\ **Post date:** [May 10, 2024, 4:24pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/126 "2024-05-10T16:24:48Z")
Any hints on how to graph JSON data in Grafana? The exporter mentioned here is reading Storagenode API and exposing the data for Prometheus to scrape, but what would be a best way to do the same directly with Storagenode API? ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [May 10, 2024, 4:26pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/127 "2024-05-10T16:26:16Z")
> [@zip](#): > > what would be a best way to do the same directly with Storagenode API? Not storagenode API. Just the metrics endpoint with prometheus. ---
**Author:** ![zip](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/zip/32/7813_2.png) [@zip](https://forum.storj.io/u/zip)\ **Post date:** [May 10, 2024, 4:45pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/128 "2024-05-10T16:45:45Z")
Thank you, didn’t know about that. For anyone else wondering, it is at /metrics at the debug.addr port. ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [May 10, 2024, 4:47pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/129 "2024-05-10T16:47:00Z")
> [@\[Tech Preview\] Email alerts with Grafana and Prometheus](https://forum.storj.io/t/tech-preview-email-alerts-with-grafana-and-prometheus/16156): > > I would like to setup email alerts for my storage nodes. It turns out the storage node already has a Prometheus endpoint. It is running on the debug endpoint /metrics. If you haven’t set the debug endpoint please visit [Guide to debug my storage node, uplink, s3 gateway, satellite](https://forum.storj.io/t/guide-to-debug-my-storage-node-uplink-s3-gateway-satellite/1372) I have already a Prometheus instance running and Grafana inclusive email alerts. Here are my configs: docker-compose.yaml version: "3.3" services: prometheus: restart: unless-stopped user: 993:991 ports… ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [May 10, 2024, 5:28pm UTC](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096/130 "2024-05-10T17:28:51Z")
try this suggestion: > [@Node goes down/restarts every 10-15 minutes. thread allocation error in logs](https://forum.storj.io/t/node-goes-down-restarts-every-10-15-minutes-thread-allocation-error-in-logs/26010/108): > > docker run -m “1024m” --memory-reservation=“768m” [Previous page](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096.md?page=5) [Next page](https://forum.storj.io/t/upcoming-storage-node-improvements-including-benchmark-tool/26096.md?page=7)