Steven Moon•6mo ago

Supabase S3 storage

I'm trying to connect a local windmill instance (docker compose) to a local Supabase instance (started with Supabase CLI). However, the S3 type in Windmill always fails with this error. I've tried all sorts of endpoint variations assuming it's a docker networking issue, but no luck so far. Is there some sort of validation that forces a AWS only S3 providers?

12 Replies

Steven Moon•6mo ago

I discovered the issue - the s3-lite-client windmill has chosen prevents slashes in both the endpoint string AND the bucket string. 😥 so Supabase's S3 endpoint looks like this: http://127.0.0.1:54321/storage/v1/s3 Since the endpoint MUST contain slashes it's not possible without changing the validators in s3-lite-client I guess the only way is to use a different S3 client library directly in my scripts?

Steven Moon•6mo ago

https://github.com/bradenmacdonald/s3-lite-client/blob/main/helpers.ts

// bucket should begin with alphabet/number and end with alphabet/number,
  // with alphabet/number/.- in the middle.
  if (bucket.match(/^[a-z0-9][a-z0-9.-]+[a-z0-9]$/)) {
    return true;
  }

// bucket should begin with alphabet/number and end with alphabet/number,
  // with alphabet/number/.- in the middle.
  if (bucket.match(/^[a-z0-9][a-z0-9.-]+[a-z0-9]$/)) {
    return true;
  }

No slashes for you!

GitHub

s3-lite-client/helpers.ts at main · bradenmacdonald/s3-lite-client

A lightweight but powerful JavaScript S3 client. Contribute to bradenmacdonald/s3-lite-client development by creating an account on GitHub.

Steven Moon•6mo ago

// Validate input params.
    if (
      typeof params.endPoint !== "string" || params.endPoint.length === 0 ||
      params.endPoint.indexOf("/") !== -1
    ) {
      throw new errors.InvalidEndpointError(
        `Invalid endPoint : ${params.endPoint}`,
      );
    }

// Validate input params.
    if (
      typeof params.endPoint !== "string" || params.endPoint.length === 0 ||
      params.endPoint.indexOf("/") !== -1
    ) {
      throw new errors.InvalidEndpointError(
        `Invalid endPoint : ${params.endPoint}`,
      );
    }

No slash zone in client.ts

rubenf•6mo ago

We don't use the S3 lite client in the workers themselves, that's just used for validation. So it might still work

Steven Moon•6mo ago

I couldn't get it to work - gave up We are also having issues accessing Azure storage via their S3 compatible API

rubenf•6mo ago

any reason to not just azure object storage / s3 directly ?

Steven Moon•6mo ago

We are trying to develop a pure local offering for certain customer requirements - Supabase now supports the S3 protocol for their storage. My hope was to just use the wmill S3Object and then our scripts will be good for customers who want local (supabase) and for those who want it in their cloud (AWS, Azure, etc.).

rubenf•6mo ago

we can investigate trying to make it work with supabase s3 on our side

Steven Moon•6mo ago

Regarding the issue with supabase S3 endpoints is that they contain slashes (S3 Storage URL: http://127.0.0.1:54321/storage/v1/s3). Since the wmill S3Object is instantiated from a form that validates against s3-lite-client, it means that I can't even create an S3Object to test it out in the actual runtime. The form validation of both the enpoint url and the bucket name is the problem. I expect that, since you are not using that s3 lib in the runtime env, it would work if we could just enter them and pass form validation.

Steven Moon•6mo ago

https://supabase.com/docs/guides/storage/s3/compatibility

Supabase

S3 Compatibility | Supabase Docs

Compatibility spec

Steven Moon•6mo ago

https://supabase.com/blog/s3-compatible-storage

Supabase

Supabase Storage: now supports the S3 protocol

Supabase Storage is now officially an S3-Compatible Storage Provider.

Steven Moon•6mo ago

By S3Object, I meant the S3 Storage resource in workspace settings