How to build a media pipeline around a S3 bucket and Docker
Hi, we're evaluating windmill.dev as an option to build a trigger based media pipeline. We have a bunch of locally maintaned docker containers doing different task (such as taking FBX files and doing format conversions for vendors) or transcribing media contents.
I would like to achieve the following:
1. trigger a process with an input (which is a remote URL)
2. download the remote URLs contents into a folder
3. download some contents from an S3 bucket (possibly multiple GB in size sometimes)
4. bind a docker container and run in
5. return the results with pre-signed S3 URLs
I'm currently struggeling to transition from downloading large binary files and binding them into a docker container, because I get to either run
An alternative may be to register specific works that are built for individual tasks, however, I'd like to avoid that currently.
I would like to achieve the following:
1. trigger a process with an input (which is a remote URL)
2. download the remote URLs contents into a folder
3. download some contents from an S3 bucket (possibly multiple GB in size sometimes)
4. bind a docker container and run in
5. return the results with pre-signed S3 URLs
I'm currently struggeling to transition from downloading large binary files and binding them into a docker container, because I get to either run
bunbashAn alternative may be to register specific works that are built for individual tasks, however, I'd like to avoid that currently.
