Presigned url does not expire - DDOS / Rate limiting questions - Cloudflare Worker using aws4fetch S3 with storj

Developer Discussions troubleshooting
1

Hey, I am trying to build a cloudflare worker using aws4fetch especially with the AwsV4Signer. Since I am not able to directly use the aws-sdk in the cf worker.

I want to be able to just send the url back to the frontend so that the user can do a second request directly to storj and display this object.

The setup is as follows:

import { AwsV4Signer } from "aws4fetch";

const url = new URL("https://gateway.storjshare.io/demo-bucket/my-object");
url.searchParams.set("X-Amz-Expires", "30"); // 30 seconds

const signer = new AwsV4Signer({
    url, 
    accessKeyId, 
    secretAccessKey, 
    method: "GET", 
    headers: {
      "content-type": "application/json",
    }, 
    signQuery: true,
    service: "s3",
  });

const { url } = await signer.sign();

return new Response(url.href, { status: 200 });

This works fine, however upon trying after 30+ seconds and more, the link still works. This begs the question what would happen if some attacker tried to DDOS this link, then potentially my storj service will go down and users wouldn’t be able see any contents on the site during the attack as I imagine because of rate limiting. If it’s a huge attack then bandwidth cost might also be a consideration.

Is there anything I can do to mitigate this or is there any other way to set the expire time?

2

Ah, of course right after posting this I realized that this is the link that will expire after certain amount of time before you can do the initial request. However, after doing the initial request I want to still make sure that the link will expire after the initial request to mitigate rate limitation and ddos attacks.

Another question: Will the rate limiter kick in even if an attacker would request before the initial expire and after the link itself has expired?

3

If I understand correctly, you’re looking to ensure the link expires after a single request has happened (initial request)?

There isn’t a way enforce 1 time use with presigned urls and it would need to be done either on your server or a proxy that purposes denies access to a link after the initial request.

As for the expire time, it seems like you’re seeing the links expire, which is good, but not within the 30 seconds? My guess is that there may be a minimum time it can be set to or it might have other limitations that keep it from being exact.

what would happen if some attacker tried to DDOS this link

To protect against DDOS, you’d either have to proxy traffic to your storj service with nginx or a similar service that has rate limiting enabled.

Another option is to use the Storj hosted s3 gateway and then you wouldn’t need to worry about that.

4 Likes
4

Thanks for the reply! :slight_smile:

If I understand correctly, you’re looking to ensure the link expires after a single request has happened (initial request)?

Yes, I’m only worried about the link expiry after the initial request due to rate limiting and DDOS attacks. An attacker might DDOS a link with more than 100/req a second, then a regular user won’t be able to browse any other presigned url because of this attack if I am not mistaken?

but not within the 30 seconds?

Well yes, it does expire after 30 seconds if I haven’t already queried it before 30 seconds, but I don’t know for how long the link will last after activation. Is it like 7 days?

Another option is to use the Storj hosted s3 gateway and then you wouldn’t need to worry about that.

This might be a dumb question, but am I not using the storj hosted s3 gateway when signing with this link https://gateway.storjshare.io ?

Based on your reply and if https://gateway.storjshare.io is the storj hosted s3 gateway, to make sure I’m not getting DDOS’d and potentially avoid bandwidth cost (if the attack is huge) or rate limited by storj, I need to fetch from storj on my rate limited backend then return it the user, I cannot just serve the presigned url directly?

5

The link doesn’t disappear, just after the expire time, it returns an error (example). The content is available for the duration you specify when creating the link.

This might be a dumb question, but am I not using the storj hosted s3 gateway when signing with this link https://gateway.storjshare.io ?

I misunderstood, I thought you were self-hosting the gateway. Yes, that is the Storj hosted s3 gateway. Yes, you can serve the presigned url directly.

An attacker might DDOS a link with more than 100/req a second, then a regular user won’t be able to browse any other presigned url because of this attack if I am not mistaken?

Yes if you hit the rate limit, an error is returned.

If you want additional protection against DDOS beyond the built-in rate limit, you’d have to add that with some other service.

1 Like
6

I feel very dumb right now, I initially thought since the link still worked after the expiry time when I’ve visited it at least once before the expiration, that my setup was incorrect somehow. However, upon further investigation it was due to… browser cache. In the network tab I can see the link is loaded through the cache and by visiting the same url in incognito, I can see that the request have expired.

Thank you again Dan for the clarifications!

My thinking to avoid any rate limiting/DDOS on storj, I obviously need to do some backend preparation, but at least setting the expire to a small amount of time will help mitigate the presigned url attacks.

Then, I have only one question left:

Does the rate limiting only count for successful request, or does it also count when i.e an attacker tries to DDOS an expired url?

7

I’m not sure, my guess is the rate limits would kick in for every request. In the case of the expired url, there isn’t any object data involved, so you wouldn’t be charged for egress.

1 Like
8

Oh man, I really hope that’s not the case, then everyone who’s using presigned urls are subject to a rate limit attack regardless if the signed url have expired, or maybe I’m misunderstanding. Is it not the case that the rate limit is for the whole account, or is the rate limit for each IP addresses?

If 1 guy query the same signed url 100 times a second, then other people will have trouble i.e fetching, right?

9

I believe it’s per IP, but would like to hear a confirmation.

1 Like
10

I believe it’s per IP, but would like to hear a confirmation.

:eyes:

What can I do to get a confirmation?

11

It’s was more for me :slight_smile:
I got a response, that rate limit on the satellite is per project, and we do not offer a DDoS protection service, you need to use something specific for that, if you need it.
The expired presigned requests will not hit the satellite rate limits, gateway rejects the expired requests when it’s validating the signature before talking to a satellite.

1 Like
12

This has been a tremendous help for me, thank you @Alexey & @danw ! :slight_smile:

1 Like