ERROR piecestore upload rejected, too many requests

Vadim · December 29, 2019, 4:37pm

this is Windows GUI installation cmd not support this
I can send you log file to support last day

littleskunk · December 29, 2019, 4:40pm

For Windows I recommend notepad++. That will work as well.

Vadim · December 29, 2019, 5:08pm

installed it and
there is 12900+ upload rejected
It started to apear at 4.32 morning
and more and more, then 100% of Ingress is rejected untill restart.
Only some Egress loads.
I can send you today log file, that you can understand better.

BrightSilence · December 29, 2019, 7:55pm

While I did see this issue 5 days ago, I have since commented out the option in the config.yaml and didn’t see it repeat this time. Make sure you comment out the option and don’t forget to restart your node afterwards.

Alexey · December 30, 2019, 9:46am

Please, give me result of this command:

sls "concurrent" "$env:ProgramFiles\Storj\Storage Node\config.yaml"

littleskunk · December 30, 2019, 10:08am

github.com

storj/storj/blob/e99bdac9443f8dc452e6702e863e52d1b38ba295/storagenode/piecestore/endpoint.go#L61


      
          	Path                   string         `help:"path to store data in" default:"$CONFDIR/storage"`
          	WhitelistedSatellites  storj.NodeURLs `help:"a comma-separated list of approved satellite node urls (unused)" devDefault:"" releaseDefault:""`
          	AllocatedDiskSpace     memory.Size    `user:"true" help:"total allocated disk space in bytes" default:"1TB"`
          	AllocatedBandwidth     memory.Size    `user:"true" help:"total allocated bandwidth in bytes" default:"2TB"`
          	KBucketRefreshInterval time.Duration  `help:"how frequently Kademlia bucket should be refreshed with node stats" default:"1h0m0s"`
          }
          
          // Config defines parameters for piecestore endpoint.
          type Config struct {
          	ExpirationGracePeriod time.Duration `help:"how soon before expiration date should things be considered expired" default:"48h0m0s"`
          	MaxConcurrentRequests int           `help:"how many concurrent requests are allowed, before uploads are rejected. 0 represents unlimited." default:"0"`
          	OrderLimitGracePeriod time.Duration `help:"how long after OrderLimit creation date are OrderLimits no longer accepted" default:"24h0m0s"`
          	CacheSyncInterval     time.Duration `help:"how often the space used cache is synced to persistent storage" releaseDefault:"1h0m0s" devDefault:"0h1m0s"`
          
          	RetainTimeBuffer time.Duration `help:"allows for small differences in the satellite and storagenode clocks" default:"48h0m0s"`
          
          	Trust trust.Config
          
          	Monitor monitor.Config
          	Orders  orders.Config
          }

In the config file that would be max-concurrent-requests

Alexey · December 30, 2019, 10:10am

Thanks!
Edited the previous message.

Vadim · December 30, 2019, 10:33am

Ok i found that all the time i whached wrong thing #retain.concurrency: 100
but should whatch storage2.max-concurrent-requests: 40
Thank you @Alexey
But there looks like problem connections hang, bacase during time connections shold free, but so long time they not gone free, during 3-5h was no upload(ingress) connections all was rejected. today i had 40 max, but if there no max could be 1000 connections hanged this is big memory leack?

littleskunk · December 30, 2019, 11:06am

I won my bet

If you get too many downloads or more precisely if you don’t have the bandwith to finish them fast you will also end up with a lot of open connections. Take a look at this 3-5h time window. Not a single successful or failed upload/download? I would bet a second time that this is not the case.

That was the hole point of moving away from GRPC. With GRPC you get a memory leak even with 10 connections because GRPC is doing some magic that we don’t need. DRPC doesn’t has this problem. Just give it a try and watch.

Vadim · December 30, 2019, 11:23am

what conserning bandwidth i have 100/100Mbit and never seen over 40Mbit, and average not over 20Mbit
this ccant be botleneck