Mio Moto•3w ago

How to build a media pipeline around a S3 bucket and Docker

Hi, we're evaluating windmill.dev as an option to build a trigger based media pipeline. We have a bunch of locally maintaned docker containers doing different task (such as taking FBX files and doing format conversions for vendors) or transcribing media contents. I would like to achieve the following: 1. trigger a process with an input (which is a remote URL) 2. download the remote URLs contents into a folder 3. download some contents from an S3 bucket (possibly multiple GB in size sometimes) 4. bind a docker container and run in 5. return the results with pre-signed S3 URLs I'm currently struggeling to transition from downloading large binary files and binding them into a docker container, because I get to either run bun for file retrieval or bash for docker execution. An alternative may be to register specific works that are built for individual tasks, however, I'd like to avoid that currently.

1 Reply

Mio MotoOP•3w ago

I've noticed that the worker process for the docker workflow is the same worker for bun and thus have both access to docker:

import { $ } from "bun";
export async function main() {
  for await (const line of $`docker run --name ${process.env["WM_JOB_ID"]} hello-world`.lines()) {
    console.log(line)
  } 
}

import { $ } from "bun";
export async function main() {
  for await (const line of $`docker run --name ${process.env["WM_JOB_ID"]} hello-world`.lines()) {
    console.log(line)
  } 
}

job=0198cc6d-53ed-9548-1cc9-eb647f4edcac tag=bun worker=wk-default-e77d04fa16ce-LC4oN hostname=e77d04fa16ce


--- BUN INSTALL ---

bun install v1.2.18 (0d4089ea)
Saved lockfile
+ bun@1.2.20
5 packages installed [166.00ms]

--- BUN CODE EXECUTION ---

Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/
For more examples and ideas, visit:
 https://docs.docker.com/get-started/

job=0198cc6d-53ed-9548-1cc9-eb647f4edcac tag=bun worker=wk-default-e77d04fa16ce-LC4oN hostname=e77d04fa16ce


--- BUN INSTALL ---

bun install v1.2.18 (0d4089ea)
Saved lockfile
+ bun@1.2.20
5 packages installed [166.00ms]

--- BUN CODE EXECUTION ---

Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/
For more examples and ideas, visit:
 https://docs.docker.com/get-started/

good enough for us, I think

How to build a media pipeline around a S3 bucket and Docker

Did you find this page helpful?