Uploads limited to 24 hours?

Developer Discussions DCS
1

Yesterday I started to upload 30 x 1TB files, and today (exactly 24 hours later) all uploads has been terminated due to "stream ID expired"

I could however split the files into smaller pieces, but it’s not uncommon for us to have file size like this, when working with eg. veeam backups.

Would it be possible to change that “limit” to like 48 Hours?

690.00 GiB / 931.32 GiB [------------------------->________] 74.09% 5.13 MiB p/s2020-03-22T16:35:52.020+0100    WARN     Failed deleting object  {"path": "test2/2758638435265215252650729261", "error": "object not found: metainfo error: object not found: key not found: \"09725ae4-4260-437e-a04c-2eaa561cd9dd/l/test2/\\x02y\\x91\\x9a\\xd7\\xde\\xf8JG\\xd2\\xfa\\x1fm\\xafI\\x06\\x10\\x90\\x19d\\xd1npUKNQV\\xc0  w\\x93\\xcb¤K\\xbd\\x8e'\\xfe\\x02\\xf6p\\xf5?`j\\xfa?#\\x8b\\xf1\\xa4q\\x9c\\xbfF\"", "errorVerbose": "object not found: metainfo error: object not found: key not found: \"09725ae4-4260-437e-a04c-2eaa561cd9dd/l/test2/\\x02y\\x91\\x9a\\xd7\\xde\\xf8JG\\xd2\\xfa\\x1fm\\xafI\\x06\\x10\\x90\\x19d\\xd1npUKNQV\\xc0  w\\x93\\xcb¤K\\xbd\\x8e'\\xfe\\x02\\xf6p\\xf5?`j\\xfa?#\\x8b\\xf1\\xa4q\\x9c\\xbfF\"\n\tstorj.io/uplink/private/metainfo.(*Client).BeginDeleteObject:662\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Delete:427\n\tstorj.io/uplink/private/storage/streams.(*streamStore).cancelHandler:531\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Put.func1:106\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Put:209\n\tstorj.io/uplink/private/storage/streams.(*shimStore).Put:54\n\tstorj.io/uplink/private/stream.NewUpload.func1:43\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
Error: metainfo error: stream ID expired

680.25 GiB / 931.32 GiB [------------------------>_________] 73.04% 6.15 MiB p/s2020-03-22T16:35:51.091+0100    WARN     Failed deleting object  {"path": "test2/8977197861375610972517030471", "error": "object not found: metainfo error: object not found: key not found: \"09725ae4-4260-437e-a04c-2eaa561cd9dd/l/test2/\\x02\\x05w\\xed!䚥\\x1f\\xcd\\x15\\xa2\\xeb\\xe76\\xaeG\\a\\xcf}\\xaf\\x87\\xb3\\t+\\x1db\\x9c7\\xd8\\xd1z\\x90\\ve\\xd6 \\xbal9\\xea{\\xfe\\x02\\xf2\\x1e\\xa6+jjR5cB\\xd6V.\\x028\"", "errorVerbose": "object not found: metainfo error: object not found: key not found: \"09725ae4-4260-437e-a04c-2eaa561cd9dd/l/test2/\\x02\\x05w\\xed!䚥\\x1f\\xcd\\x15\\xa2\\xeb\\xe76\\xaeG\\a\\xcf}\\xaf\\x87\\xb3\\t+\\x1db\\x9c7\\xd8\\xd1z\\x90\\ve\\xd6 \\xbal9\\xea{\\xfe\\x02\\xf2\\x1e\\xa6+jjR5cB\\xd6V.\\x028\"\n\tstorj.io/uplink/private/metainfo.(*Client).BeginDeleteObject:662\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Delete:427\n\tstorj.io/uplink/private/storage/streams.(*streamStore).cancelHandler:531\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Put.func1:106\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Put:209\n\tstorj.io/uplink/private/storage/streams.(*shimStore).Put:54\n\tstorj.io/uplink/private/stream.NewUpload.func1:43\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
Error: metainfo error: stream ID expired

688.13 GiB / 931.32 GiB [------------------------->________] 73.89% 5.44 MiB p/s2020-03-22T16:35:48.832+0100    WARN     Failed deleting object  {"path": "test2/25966214753076931798256119454", "error": "object not found: metainfo error: object not found: key not found: \"09725ae4-4260-437e-a04c-2eaa561cd9dd/l/test2/\\x02\\xf77ځ \\xb1 \\xd1\\xe4\\x18\\x94^\\x16\\xba\\x8e\\xe3\\x14\\xadÑ\\xc4ͧYÑ\\xb2\\xc8\\xf9\\xbd\\x92\\xfe\\x01\\x936\\x14UE\\xd0\\xf4\\xf5\\xa2\\x19\\xee\\x10\\n.\\x02\\xad\\xa3$1\\x95\\x8ct\\x16'ތ\"", "errorVerbose": "object not found: metainfo error: object not found: key not found: \"09725ae4-4260-437e-a04c-2eaa561cd9dd/l/test2/\\x02\\xf77ځ \\xb1 \\xd1\\xe4\\x18\\x94^\\x16\\xba\\x8e\\xe3\\x14\\xadÑ\\xc4ͧYÑ\\xb2\\xc8\\xf9\\xbd\\x92\\xfe\\x01\\x936\\x14UE\\xd0\\xf4\\xf5\\xa2\\x19\\xee\\x10\\n.\\x02\\xad\\xa3$1\\x95\\x8ct\\x16'ތ\"\n\tstorj.io/uplink/private/metainfo.(*Client).BeginDeleteObject:662\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Delete:427\n\tstorj.io/uplink/private/storage/streams.(*streamStore).cancelHandler:531\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Put.func1:106\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Put:209\n\tstorj.io/uplink/private/storage/streams.(*shimStore).Put:54\n\tstorj.io/uplink/private/stream.NewUpload.func1:43\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
Error: metainfo error: stream ID expired

....snip.....

684.31 GiB / 931.32 GiB [------------------------>_________] 73.48% 5.24 MiB p/s2020-03-22T16:35:02.182+0100    WARN     Failed deleting object  {"path": "test2/210842192773964874452527901", "error": "object not found: metainfo error: object not found: key not found: \"09725ae4-4260-437e-a04c-2eaa561cd9dd/l/test2/\\x02^\\x8a\\xd8\\xc0\\x19\\x91\\xdd`\\x9dp\\xa6C\\xb4|︰\\x1d\\x19\\xa4\\x88\\xd1\\xed\\x87\\x01\\x02\\x89\\x01\\x02\\x11\\x06\\x1c\\xbc\\xe6߯\\xa0\\xdf\\xd69\\xfdL\\xad%\\x84\\x9e\\xc0\\x9d\\x1f@ \\\\\\x82\\xe1T\\x84}\"", "errorVerbose": "object not found: metainfo error: object not found: key not found: \"09725ae4-4260-437e-a04c-2eaa561cd9dd/l/test2/\\x02^\\x8a\\xd8\\xc0\\x19\\x91\\xdd`\\x9dp\\xa6C\\xb4|︰\\x1d\\x19\\xa4\\x88\\xd1\\xed\\x87\\x01\\x02\\x89\\x01\\x02\\x11\\x06\\x1c\\xbc\\xe6߯\\xa0\\xdf\\xd69\\xfdL\\xad%\\x84\\x9e\\xc0\\x9d\\x1f@ \\\\\\x82\\xe1T\\x84}\"\n\tstorj.io/uplink/private/metainfo.(*Client).BeginDeleteObject:662\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Delete:427\n\tstorj.io/uplink/private/storage/streams.(*streamStore).cancelHandler:531\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Put.func1:106\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Put:209\n\tstorj.io/uplink/private/storage/streams.(*shimStore).Put:54\n\tstorj.io/uplink/private/stream.NewUpload.func1:43\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
Error: metainfo error: stream ID expired

682.88 GiB / 931.32 GiB [------------------------>_________] 73.32% 4.58 MiB p/s2020-03-22T16:34:54.833+0100    WARN     Failed deleting object  {"path": "test2/67324318327185104561793", "error": "object not found: metainfo error: object not found: key not found: \"09725ae4-4260-437e-a04c-2eaa561cd9dd/l/test2/\\x02\\x86\\\"M\\x8c\\xeai\\xacF\\x8a\\xc3\\xea\\x1d\\x19\\x12\\x19\\\"]\\xd3[\\xc3 \\xa1\\xbe\\x82\\x0ex~\\x94\\xc0\\xb1-\\x199d\\x1f\\x1e~6\\xa7\\x03\\xc5`\\xac=S\\xf57\\xeb\\x01\\x02\\xe9Y\"", "errorVerbose": "object not found: metainfo error: object not found: key not found: \"09725ae4-4260-437e-a04c-2eaa561cd9dd/l/test2/\\x02\\x86\\\"M\\x8c\\xeai\\xacF\\x8a\\xc3\\xea\\x1d\\x19\\x12\\x19\\\"]\\xd3[\\xc3 \\xa1\\xbe\\x82\\x0ex~\\x94\\xc0\\xb1-\\x199d\\x1f\\x1e~6\\xa7\\x03\\xc5`\\xac=S\\xf57\\xeb\\x01\\x02\\xe9Y\"\n\tstorj.io/uplink/private/metainfo.(*Client).BeginDeleteObject:662\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Delete:427\n\tstorj.io/uplink/private/storage/streams.(*streamStore).cancelHandler:531\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Put.func1:106\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Put:209\n\tstorj.io/uplink/private/storage/streams.(*shimStore).Put:54\n\tstorj.io/uplink/private/stream.NewUpload.func1:43\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
Error: metainfo error: stream ID expired

677.75 GiB / 931.32 GiB [------------------------>_________] 72.77% 5.56 MiB p/s2020-03-22T16:34:53.623+0100    WARN     Failed deleting object  {"path": "test2/8291546726828834195652808", "error": "object not found: metainfo error: object not found: key not found: \"09725ae4-4260-437e-a04c-2eaa561cd9dd/l/test2/\\x02Ӓ\\xe3\\x0e\\xfe\\x01\\xac\\xb2D\\xc7\\xe4\\xf3_\\x8e\\x86\\x97\\x9dJ\\t{C\\x10U\\xbf\\x8d\\xfe\\x02\\xb3\\x91\\x16k\\x84bWv\\xcfK\\xe0\\x02\\x84\\xbfe\\u007f\\x13aw<\\xccn\\xa3e)\\xfc\\xd9\\xe5\"", "errorVerbose": "object not found: metainfo error: object not found: key not found: \"09725ae4-4260-437e-a04c-2eaa561cd9dd/l/test2/\\x02Ӓ\\xe3\\x0e\\xfe\\x01\\xac\\xb2D\\xc7\\xe4\\xf3_\\x8e\\x86\\x97\\x9dJ\\t{C\\x10U\\xbf\\x8d\\xfe\\x02\\xb3\\x91\\x16k\\x84bWv\\xcfK\\xe0\\x02\\x84\\xbfe\\u007f\\x13aw<\\xccn\\xa3e)\\xfc\\xd9\\xe5\"\n\tstorj.io/uplink/private/metainfo.(*Client).BeginDeleteObject:662\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Delete:427\n\tstorj.io/uplink/private/storage/streams.(*streamStore).cancelHandler:531\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Put.func1:106\n\tstorj.io/uplink/private/storage/streams.(*streamStore).Put:209\n\tstorj.io/uplink/private/storage/streams.(*shimStore).Put:54\n\tstorj.io/uplink/private/stream.NewUpload.func1:43\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
Error: metainfo error: stream ID expired
7 Likes
2

Thanks for that suggestion, I have made a request to the dev team to get the limit increased to 48 hours.

3

My suggestion would be to have uplink get a new stream ID once in a while after uploading x segments of the file. I’m pretty sure that there would be some circumstances where someone needs to download or upload a large file using a slow connection.

3 Likes
4

This change has been implemented and should be available with next week’s satellite release.

1 Like
5

Which solution was implemented in the end?

I think @Pentium100 made a good point :slightly_smiling_face:

2 Likes
6

Limit changed to 48 hours

7

Transfers should never fail because of “taking too long”, you already limited them by upload speed (some sanity-check limit should be in place like a week or something). Transfers should be resumable, if they’re not already. Most people’s connections reset every day.

1 Like