I have 1 node that is 5TB, full of data and it strangly consumting 127MB memory while other take only 25-40 MB
It looks like work fine, but strange taking big memory.
The RAM starts to using when you have troubles either with disk or with databases.
Please, check both.
Checkdisk is ok, Integrity check is ok, only Order DB was 2,5GB after compacting it it is 2GB not normaly big may be this is the reason?
also the memory usage can remain high for days after you have had an increase in disk latency…
from my VM going crazy yesterday… i’m still at 96mb of memory used and dropping slowly…
i forget what the norm is, but i think 30-50mb
the amount of memory can change depending on your host machine activity and hdd usage… but yeah like alexey says it can be an indicator of other issues… but 127-300mb for a storagenode memory i wouldn’t worry about…
i’ve seen mine be 300mb just because the we had like 5mb/s ingress and 500k - 1mb egress
it’s not uncommon for the number to change a bit if your node is kinda busy, just keep an eye on that it settles back or atleast keeps steady and not starts climbing without stopping…
doesn’t really matter if its 30mb or 127mb… so long as it’s stable… and stays stable…
if memory serve the what was it called… ffs version pervious version to this one… v1.6.3 or v1.8.2
whatever… the previous one… used more memory i think… then when we got 1.9.2 it dropped sharply to like 30-50mb…
maybe you got the new version and the new programming changes uses more ram, i think v1.10 is out
best guess i got actually xD
I have there 5 nodes, all other is about 20-40MB and this node is 330MB today.
Checkdisk show it is OK, Orded DBs integrity check is OK.
Node is Full, so no ingress, only some egress.
It can be so big as it is only one node where order DB is 2 GB, all others are only 200-600MB.
Before i vacumed it was 2,5GB. Node is like all other 1.10 newst version.
so far as i have seen the memory usage of the storagenode is always related to disk latency
the disk may have high latency because of various reasons, that will not show up in either disk integrity checks nor in common smart data…
i have one disk that is slower than it’s family members, seems to be due to the one disk being older and having 300k power load cycles or whatever its called , while the others have like 1400
if i was to run a storagenode on such a disk alone with higher latency it’s possible that it might create more latency and thus may create a larger cache…
however if you have multiple nodes then the load is split between the drives and thus you shouldn’t have any IO limitations like that and even if they did exist they should be near non existing unless if the disk is in really bad condition…
it may be a sign of an impending disk failure
does node cashing some part of order.db to the RAM?
you will have many caches … in the cpu there are like 3 levels atleast… then the hdd with have a level of caching, then the os will do caching, and maybe programs will also add caching…
basically the idea is to keep the information that is used most often close to where it’s needed and thus reduce latency of finding the data… or working on the data…
so stuff like databases will most likely live in multiple caches… ofc depending on the system and what it’s doing some datasets might be to big for some caches and then its cached in a slower system… often each cache level is a factor of 10 slower than the previous cache layer…
i know it doesn’t really answer your question, but i can tell you will almost certainty that any data often used will be cached in multiple places depending on the size of the data…
caching will also depend on filesystem, and OS in most cases, so it’s a difficult question to answer exactly as windows might act differently that windows…
just assume it will cache anything that’s often used, even if the storagenode doesn’t then something else most likely will…
This is an astronomical advancement from v2 where the node tried to fit 4 or 5 4 GB pieces into RAM, which meant you had to have close to 100 GB of RAM if you had 5 nodes running.
1 Like
i don’t remember it being that bad… tho i only ran v2 for a few months before the v3 development was announced… but that may just have been due to my node being small at the time… i do remember it having some ram requirements… nothing crazy tho… ofc may have been a fixed issue before i joined.
and only had 1 node running 
It was never fixed because development was dropped in favor of v3. If I understood correctly, v2 tried to keep everything that was being uploaded in RAM. Statistically it was bound to hit a snag eventually and get several large upload pieces in parallel, which would stay there for hours or days with bad upload speeds, eventually crashing the node.
V2 was mostly idle by then. You didn’t see the high activity times. RAM usage was massive at times.
lol that sounds horrible… also storing data in non ecc memory… lol good luck
i really cannot say only ran the node for 3 months… so mostly vetting and very limited actual exposure to the network… seemed to run fine… but i ofc was running on my current server just with 24 gb ecc ram back then.
so if it worked for normal pc’s i most likely wouldn’t have noticed any issues.
i have found this errors in logd on this node, when it starts
2020-08-18T21:50:07.236+0300 INFO orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW sending {“count”: 18968}
2020-08-18T21:50:07.236+0300 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {“count”: 634230}
2020-08-18T21:50:07.236+0300 INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE sending {“count”: 996566}
2020-08-18T21:50:07.236+0300 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 sending {“count”: 1141997}
2020-08-18T21:50:07.236+0300 INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB sending {“count”: 369}
2020-08-18T21:50:07.236+0300 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {“count”: 984424}
2020-08-18T21:50:07.394+0300 ERROR orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs rpc client error when receiving new order settlements {“error”: “order: failed to receive settlement response: only specified storage node can settle order”, “errorVerbose”: “order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:330\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:221\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:200\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-08-18T21:50:07.394+0300 ERROR orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs rpc client when sending new orders settlements {“error”: “order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:302\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”, “request”: {“limit”:{“serial_number”:“Q4N4YPJSD5GMJDNU3EGVKGJ6PU”,“satellite_id”:“12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”,“uplink_public_key”:{},“storage_node_id”:“12J6RTUfoxeavsPBPV93wJfKVt35b86ytnoVJ1E16cqiZqhK1TB”,“piece_id”:“JUZUV4RYPDDWN34ZR2R7GZMXOS642AMYLXPXWJ7ERKND7NSNEZAA”,“limit”:2319360,“action”:1,“piece_expiration”:“0001-01-01T00:00:00Z”,“order_expiration”:“2020-03-20T21:50:06.543099208Z”,“order_creation”:“2020-03-18T21:50:06.543409467Z”,“satellite_signature”:“MEYCIQDS1tWPm+n/SsFo1F/IgjLx7r8eBBfHZNr7d6N+oB5eqQIhAKNVOs9fFcjASgxlRWZSy4WZymlR6YJaqG9UpbgslxIc”,“satellite_address”:{}},“order”:{“serial_number”:“Q4N4YPJSD5GMJDNU3EGVKGJ6PU”,“amount”:2319360,“uplink_signature”:“QyiX7iB1v+ph2uf/Ot7l5jXs5cl1ZSwfuKrINsaY/uhIv1sQNOxJsfMin3/GRraKCG5KoDNw28mQJ1CZKcDfCw==”}}}
2020-08-18T21:50:07.394+0300 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished
2020-08-18T21:50:07.395+0300 ERROR orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs failed to settle orders {“error”: “order: failed to receive settlement response: only specified storage node can settle order; order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “group:\n— order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:330\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:221\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:200\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:302\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-08-18T21:50:07.417+0300 ERROR orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW rpc client error when receiving new order settlements {“error”: “order: failed to receive settlement response: only specified storage node can settle order”, “errorVerbose”: “order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:330\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:221\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:200\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-08-18T21:50:07.417+0300 ERROR orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW rpc client when sending new orders settlements {“error”: “order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:302\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”, “request”: {“limit”:{“serial_number”:“BXBMLDHRKNALVMJEC2LVP2VKN4”,“satellite_id”:“118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”,“uplink_public_key”:{},“storage_node_id”:“12J6RTUfoxeavsPBPV93wJfKVt35b86ytnoVJ1E16cqiZqhK1TB”,“piece_id”:“DOACZ5H3GK3XWH5JFHWQ67BZ7AN5IZGGX7765Q55P2TH2MOFS5QA”,“limit”:2317056,“action”:5,“piece_expiration”:“0001-01-01T00:00:00Z”,“order_expiration”:“2020-03-23T23:17:35.055434736Z”,“order_creation”:“2020-03-21T23:17:35.055677922Z”,“satellite_signature”:“MEUCIDt8/DtLJ9eOR1CfFzPnLMVMY4xinedy5OJtmafZdi24AiEAto/tJXHtNiPZ+KDskNNMbpY+kE/HReFS4q4Ooy3zc7Q=”,“satellite_address”:{“address”:“satellite.stefan-benten.de:7777”}},“order”:{“serial_number”:“BXBMLDHRKNALVMJEC2LVP2VKN4”,“amount”:2317056,“uplink_signature”:"+nFGD+EH8bo8QS3jLmsV7wcG/0knHnEP/JvTUoJrklW/p8C0sNpVCkmnRRCLBCW4VaInFAp4AtID476/UcXvDA=="}}}
2020-08-18T21:50:07.418+0300 INFO orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW finished
2020-08-18T21:50:07.418+0300 ERROR orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW failed to settle orders {“error”: “order: failed to receive settlement response: only specified storage node can settle order; order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “group:\n— order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:330\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:221\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:200\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:302\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-08-18T21:50:07.508+0300 INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB finished
2020-08-18T21:50:07.678+0300 ERROR orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S rpc client error when receiving new order settlements {“error”: “order: failed to receive settlement response: only specified storage node can settle order”, “errorVerbose”: “order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:330\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:221\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:200\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-08-18T21:50:07.807+0300 ERROR orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE rpc client error when receiving new order settlements {“error”: “order: failed to receive settlement response: only specified storage node can settle order”, “errorVerbose”: “order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:330\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:221\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:200\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-08-18T21:50:07.807+0300 ERROR orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE rpc client when sending new orders settlements {“error”: “order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:302\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”, “request”: {“limit”:{“serial_number”:“VMF5VQPMZFDNDBBLYHPKIIQQLQ”,“satellite_id”:“1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”,“uplink_public_key”:{},“storage_node_id”:“12J6RTUfoxeavsPBPV93wJfKVt35b86ytnoVJ1E16cqiZqhK1TB”,“piece_id”:“PIKPLTM42KFX5WEP7LXZJM5KEKN3NZFJDMQH5MDJMQTRTPQ4F2OA”,“limit”:2319360,“action”:1,“piece_expiration”:“0001-01-01T00:00:00Z”,“order_expiration”:“2020-03-20T21:08:10.589854907Z”,“order_creation”:“2020-03-18T21:08:10.59652051Z”,“satellite_signature”:“MEYCIQClAwQDDsNFGpPow7kFoek/paYi+FC13uJjGYyT/syz3AIhAJ/w0HWcJAChHH/3ZNQtZmO4Zp7F4hZh0R38LyQ/0cv/”,“satellite_address”:{}},“order”:{“serial_number”:“VMF5VQPMZFDNDBBLYHPKIIQQLQ”,“amount”:2319360,“uplink_signature”:“0tf13sHTOfn9t1Y0Dr3UU3hjOrg2XLUft5cLVDUdNKTB6eBhh/hL5kgA3rFM+ABOwzKYojKBTdU1cPHhlD9KAw==”}}}
2020-08-18T21:50:07.808+0300 INFO orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE finished
2020-08-18T21:50:07.808+0300 ERROR orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE failed to settle orders {“error”: “order: failed to receive settlement response: only specified storage node can settle order; order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “group:\n— order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:330\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:221\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:200\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:302\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-08-18T21:50:07.814+0300 ERROR orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S rpc client when sending new orders settlements {“error”: “order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:302\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”, “request”: {“limit”:{“serial_number”:“L6RZX7M6TRDUTBKBIBFLSVRDVM”,“satellite_id”:“12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”,“uplink_public_key”:{},“storage_node_id”:“12J6RTUfoxeavsPBPV93wJfKVt35b86ytnoVJ1E16cqiZqhK1TB”,“piece_id”:“R6NXE4KXNQYWU6TKU2SZKSOUC5JWBMW5J5O7CTGGIR7HJ4QD6JCQ”,“limit”:2317056,“action”:1,“piece_expiration”:“0001-01-01T00:00:00Z”,“order_expiration”:“2020-03-20T22:14:27.668431044Z”,“order_creation”:“2020-03-18T22:14:27.668849833Z”,“satellite_signature”:“MEQCIHvonzNNREt1vmRtJhxK9rGRSWaUfzsbPU/czWh6k37zAiA9hW985PCCBbwL+Rldinp6e8xO8O5O1QoUp0L9yKOhnw==”,“satellite_address”:{}},“order”:{“serial_number”:“L6RZX7M6TRDUTBKBIBFLSVRDVM”,“amount”:2317056,“uplink_signature”:“V9JuC5TRFfLoaN+rAgU3ywsfgvUJ3p9pfhDptwkeItKQ7qJau/awo0C9lrC9at1bdO8TisjJ0EYjh2sbxfM5BA==”}}}
2020-08-18T21:50:07.815+0300 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished
2020-08-18T21:50:07.815+0300 ERROR orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S failed to settle orders {“error”: “order: failed to receive settlement response: only specified storage node can settle order; order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “group:\n— order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:330\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:221\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:200\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:302\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-08-18T21:50:08.133+0300 ERROR orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 rpc client error when receiving new order settlements {“error”: “order: failed to receive settlement response: only specified storage node can settle order”, “errorVerbose”: “order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:330\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:221\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:200\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-08-18T21:50:08.174+0300 ERROR orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 rpc client when sending new orders settlements {“error”: “order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:302\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”, “request”: {“limit”:{“serial_number”:“BWSCSPBZ6RG7ZH7DUD7THAS7RU”,“satellite_id”:“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”,“uplink_public_key”:{},“storage_node_id”:“12J6RTUfoxeavsPBPV93wJfKVt35b86ytnoVJ1E16cqiZqhK1TB”,“piece_id”:“UYFOG2JTMPWHAZQLKOIR6NCN3VUS3FPQUBQ7ROOZY7SSQSG7JTZQ”,“limit”:2317056,“action”:1,“piece_expiration”:“0001-01-01T00:00:00Z”,“order_expiration”:“2020-03-20T23:04:29.281343291Z”,“order_creation”:“2020-03-18T23:04:29.281641989Z”,“satellite_signature”:“MEYCIQCtfUMG14OEvAPQ387TpKrWAmyO7W2sc6n3ieB5brAcKgIhALaYbUYMEFjcyauT+zUHlrM2RZp1EQcYrEk5tWlNn18r”,“satellite_address”:{}},“order”:{“serial_number”:“BWSCSPBZ6RG7ZH7DUD7THAS7RU”,“amount”:44288,“uplink_signature”:“TzZQnYRYQs0985AbETYIADA0TubzOc8436z/0VEp14/g8hEzGUXxDiM7EaryZhTHqrwyyiDhYU8VMZGopePoAA==”}}}
2020-08-18T21:50:08.174+0300 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 finished
2020-08-18T21:50:08.175+0300 ERROR orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 failed to settle orders {“error”: “order: failed to receive settlement response: only specified storage node can settle order; order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “group:\n— order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:330\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:221\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:200\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:302\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-08-18T21:50:10.389+0300 INFO piecestore download started
@Alexey do you know what this errors mean? I added RAM to this PC, and it started to eat up to 4GB RAM
no other sign of problems. dashboard looks like very fine.
Also i have 3776917 rown in unsended orders litlebit too many or it shold be like that?
Integrity check show it is OK
It could be if you messed up the identity and data. Seems this node has unsettled orders from other identity
how can i clean this messs?
You can drop all records not older than 7 days from the orders.db database. However, you will drop the valid orders too. Or just wait a week when they should expire.
how can i do that in proper way?