How to build a media pipeline around a S3 bucket and Docker
Hi, we're evaluating windmill.dev as an option to build a trigger based media pipeline. We have a bunch of locally maintaned docker containers doing different task (such as taking FBX files and doing format conversions for vendors) or transcribing media contents.
I would like to achieve the following:
1. trigger a process with an input (which is a remote URL)
2. download the remote URLs contents into a folder
3. download some contents from an S3 bucket (possibly multiple GB in size sometimes)
4. bind a docker container and run in
5. return the results with pre-signed S3 URLs
I'm currently struggeling to transition from downloading large binary files and binding them into a docker container, because I get to either run
bun
for file retrieval or bash
for docker execution.
An alternative may be to register specific works that are built for individual tasks, however, I'd like to avoid that currently.1 Reply
I've noticed that the worker process for the
docker
workflow is the same worker for bun
and thus have both access to docker:
good enough for us, I think