Nevets
Nevets•2mo ago

Recommendation for Using a Private Git Repository as a Python Dependency

Hi everyone, I’m looking for best practices on how to use a private Git repository as a dependency in a Python project. The repository is secured either via password protection or SSH keys. What’s the recommended approach for adding the dependency to a script? Thanks in advance for your help! Stephen
3 Replies
grantith
grantith•2mo ago
We have the ssh key installed on our workers and use the git+ssh link https://www.windmill.dev/docs/advanced/dependencies_in_python#pinning-dependencies-and-requirements If you can't pre-install the ssh key then you might be able to get away with $GITHUB_TOKEN (& git+https) as an env var dispatched via the worker config or script runtime but I haven't tried that thoroughly
Dependencies in Python | Windmill
In Windmill standard mode, dependencies in Python are handled directly within their scripts without the need to manage separate dependency files.
Nevets
NevetsOP•2mo ago
Hi @grantith, I've mounted and bound the SSH agent socket into the worker containers. When I exec into one of the container and manually create and run the script, both Git and UV correctly use the SSH agent, the Git dependencies are pulled successfully, and the script runs as expected. However, when I trigger the script through the user interface, it fails to pull from Git, returning a "Permission denied (publickey)" error. Am I missing something? Thanks heaps! Stephen 
  
windmill_worker:
    image: ${WM_IMAGE}
    pull_policy: always
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '1'
          memory: 2048M
        # for GB, use syntax '2Gi'
    restart: unless-stopped
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - MODE=worker
      - WORKER_GROUP=default
      - SSH_AUTH_SOCK=/tmp/ssh-agent
    networks:
      - windmill
    depends_on:
      db:
        condition: service_healthy
    # to mount the worker folder to debug, KEEP_JOB_DIR=true and mount /tmp/windmill
    volumes:
      # mount the docker socket to allow to run docker containers from within the workers
      - /var/run/docker.sock:/var/run/docker.sock
      - worker_dependency_cache:/tmp/windmill/cache
      - type: bind
        source: ${SSH_AUTH_SOCK}
        target: /tmp/ssh-agent
        bind:
          create_host_path: false
  
windmill_worker:
    image: ${WM_IMAGE}
    pull_policy: always
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '1'
          memory: 2048M
        # for GB, use syntax '2Gi'
    restart: unless-stopped
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - MODE=worker
      - WORKER_GROUP=default
      - SSH_AUTH_SOCK=/tmp/ssh-agent
    networks:
      - windmill
    depends_on:
      db:
        condition: service_healthy
    # to mount the worker folder to debug, KEEP_JOB_DIR=true and mount /tmp/windmill
    volumes:
      # mount the docker socket to allow to run docker containers from within the workers
      - /var/run/docker.sock:/var/run/docker.sock
      - worker_dependency_cache:/tmp/windmill/cache
      - type: bind
        source: ${SSH_AUTH_SOCK}
        target: /tmp/ssh-agent
        bind:
          create_host_path: false
grantith
grantith•2mo ago
Are you using an environment where you can install the key to a derived image? That is what worked for me when i was exploring self hosting. Using cloud now it works for me by installing the key to each worker using the worker group config via init script.

Did you find this page helpful?