Can't open bucket if store a lot of objects

kaichaosun · November 24, 2022, 1:42pm

I have a bucket stored 6399 objects, now I’m getting error to retrieve existing objects and create new objects. Both the web and filezilla failed,

error show in web console,

GET https://gateway.storjshare.io/flux?delimiter=%2F&prefix=

app_chunk-vendors_61be00bd6947233c.js:336 InternalError: We encountered an internal error, please try again.: cause(uplink: metainfo: invalid direction 0)
    at constructor.extractError (app_chunk-vendors_61be00bd6947233c.js:1:222622)
    at constructor.callListeners (app_chunk-vendors_61be00bd6947233c.js:1:198352)
    at constructor.emit (app_chunk-vendors_61be00bd6947233c.js:1:198062)
    at constructor.emitEvent (app_chunk-vendors_61be00bd6947233c.js:1:183872)
    at constructor.e (app_chunk-vendors_61be00bd6947233c.js:1:179429)
    at t.runTo (app_chunk-vendors_61be00bd6947233c.js:1:246787)
    at app_chunk-vendors_61be00bd6947233c.js:1:246993
    at constructor.<anonymous> (app_chunk-vendors_61be00bd6947233c.js:1:179640)
    at constructor.<anonymous> (app_chunk-vendors_61be00bd6947233c.js:1:183928)
    at constructor.callListeners (app_chunk-vendors_61be00bd6947233c.js:1:198458)

Error showed in filezilla,

   	Connection timed out after 20 seconds of inactivity
Error:        	Failed to retrieve directory listing
Status:      	Connecting to us1.storj.io...
Status:      	Retrieving directory listing of "/flux"...
Command:	list "/flux"
Error:        	Connection timed out after 20 seconds of inactivity
Error:        	Failed to retrieve directory listing
Status:      	Disconnected from server
Status:      	Connecting to us1.storj.io...
Status:      	Retrieving directory listing of

kaichaosun · November 24, 2022, 2:08pm

With FileZilla, I set the timeout to be 60 seconds, and now it can retrieve the files, but web still failed, seems bug here.

ifraixedes · November 24, 2022, 2:30pm

Hi @kaichaosun

Thank you for reporting it.
We’ll investigate it. Please, be mindful that our offices are closed until next Monday due to Thanksgiving festivity.

elek · December 15, 2022, 3:03pm

I believe that this issue can happen only if you either use wrong encryption key, or one bucket contains files which are encrypted with multiple different encryption keys.

Should be fixed anyway, the deep dive started here:

github.com/storj/gateway-st

metainfo: invalid direction 0

opened 10:59AM - 15 Dec 22 UTC

elek

question Needs Investigation edge

We found this error in the object browser of the nightly deployment: ``` <Er…ror><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metainfo: invalid direction 0)</Message><BucketName>bucket1</BucketName><Resource>/bucket1</Resource><RequestId>17276D17ED0D2F14</RequestId><HostId></HostId></Error> ``` But the same problem is reported on the forum, too: ``` https://forum.storj.io/t/cant-open-bucket-if-store-a-lot-of-objects/20440 ``` After debugging it with a local gateway-mt, the problem is restricted to this code fragment in `gateway.go`: ``` func (layer *gatewayLayer) listObjectsFast( ctx context.Context, project *uplink.Project, bucket, prefix, continuationToken, delimiter string, maxKeys int, startAfter string, ) ( prefixes []string, objects []minio.ObjectInfo, nextContinuationToken string, err error, ) { defer mon.Task()(&ctx)(&err) ... for limit > 0 && list.Next() { item := list.Item() limit-- if item.IsPrefix { prefixes = append(prefixes, item.Key) } else { objects = append(objects, minioObjectInfo(bucket, "", item)) } nextContinuationToken = item.Key } if list.Err() != nil { return nil, nil, "", list.Err() } more := list.Next() if list.Err() != nil { return nil, nil, "", list.Err() } ... } ``` Here we have two `listNext()` call after each other. I don't know exactly the reason, why. In case of *the encryption key is bad*, the first `list.Next` returns with `more=false`, and doesn't expect any more calls. Doesn't expect means that the options are reset to a zero object in `types.go` project of `storj/uplink`: ```` // NextPage returns options for listing the next page. func (opts ListOptions) NextPage(list ObjectList) ListOptions { if !list.More || len(list.Items) == 0 { return ListOptions{} } return ListOptions{ Prefix: opts.Prefix, Cursor: list.Items[len(list.Items)-1].Path, Delimiter: opts.Delimiter, Recursive: opts.Recursive, IncludeSystemMetadata: opts.IncludeSystemMetadata, IncludeCustomMetadata: opts.IncludeCustomMetadata, Direction: After, Limit: opts.Limit, } } ``` The problem is in the `return ListOptions{}`, which resets the direction (assuming nobody else will call the next, again. Not sure if it's an `uplink` or `gateway` error, maybe we can fix it on bot sides.