Steven Moon
Steven Moon•7mo ago

Supabase S3 storage

I'm trying to connect a local windmill instance (docker compose) to a local Supabase instance (started with Supabase CLI). However, the S3 type in Windmill always fails with this error. I've tried all sorts of endpoint variations assuming it's a docker networking issue, but no luck so far. Is there some sort of validation that forces a AWS only S3 providers?
No description
12 Replies
Steven Moon
Steven MoonOP•7mo ago
I discovered the issue - the s3-lite-client windmill has chosen prevents slashes in both the endpoint string AND the bucket string. 😥 so Supabase's S3 endpoint looks like this: http://127.0.0.1:54321/storage/v1/s3 Since the endpoint MUST contain slashes it's not possible without changing the validators in s3-lite-client I guess the only way is to use a different S3 client library directly in my scripts?
Steven Moon
Steven MoonOP•7mo ago
https://github.com/bradenmacdonald/s3-lite-client/blob/main/helpers.ts
// bucket should begin with alphabet/number and end with alphabet/number,
// with alphabet/number/.- in the middle.
if (bucket.match(/^[a-z0-9][a-z0-9.-]+[a-z0-9]$/)) {
return true;
}
// bucket should begin with alphabet/number and end with alphabet/number,
// with alphabet/number/.- in the middle.
if (bucket.match(/^[a-z0-9][a-z0-9.-]+[a-z0-9]$/)) {
return true;
}
No slashes for you!
GitHub
s3-lite-client/helpers.ts at main · bradenmacdonald/s3-lite-client
A lightweight but powerful JavaScript S3 client. Contribute to bradenmacdonald/s3-lite-client development by creating an account on GitHub.
Steven Moon
Steven MoonOP•7mo ago
// Validate input params.
if (
typeof params.endPoint !== "string" || params.endPoint.length === 0 ||
params.endPoint.indexOf("/") !== -1
) {
throw new errors.InvalidEndpointError(
`Invalid endPoint : ${params.endPoint}`,
);
}
// Validate input params.
if (
typeof params.endPoint !== "string" || params.endPoint.length === 0 ||
params.endPoint.indexOf("/") !== -1
) {
throw new errors.InvalidEndpointError(
`Invalid endPoint : ${params.endPoint}`,
);
}
No slash zone in client.ts
rubenf
rubenf•7mo ago
We don't use the S3 lite client in the workers themselves, that's just used for validation. So it might still work
Steven Moon
Steven MoonOP•7mo ago
I couldn't get it to work - gave up We are also having issues accessing Azure storage via their S3 compatible API
rubenf
rubenf•7mo ago
any reason to not just azure object storage / s3 directly ?
Steven Moon
Steven MoonOP•7mo ago
We are trying to develop a pure local offering for certain customer requirements - Supabase now supports the S3 protocol for their storage. My hope was to just use the wmill S3Object and then our scripts will be good for customers who want local (supabase) and for those who want it in their cloud (AWS, Azure, etc.).
rubenf
rubenf•7mo ago
we can investigate trying to make it work with supabase s3 on our side
Steven Moon
Steven MoonOP•7mo ago
Regarding the issue with supabase S3 endpoints is that they contain slashes (S3 Storage URL: http://127.0.0.1:54321/storage/v1/s3). Since the wmill S3Object is instantiated from a form that validates against s3-lite-client, it means that I can't even create an S3Object to test it out in the actual runtime. The form validation of both the enpoint url and the bucket name is the problem. I expect that, since you are not using that s3 lib in the runtime env, it would work if we could just enter them and pass form validation.
Steven Moon
Steven MoonOP•7mo ago
Supabase
Supabase Storage: now supports the S3 protocol
Supabase Storage is now officially an S3-Compatible Storage Provider.
Steven Moon
Steven MoonOP•7mo ago
By S3Object, I meant the S3 Storage resource in workspace settings