Deploy new workers to Windmill installation
I've used the Docker Compose deployment of Windmill on an LXD container. It's working just fine, but I need to know how to deploy additional workers and join them to my Windmill server.
On the Workers page, I can see that 4 distinct workers have been created, but I don't see any buttons to join new workers to my Windmill server.
I searched the documentation and forum for "workers" and cannot find any directions to install / join new workers either.
How can this be accomplished?
34 Replies
you just need to spawn new workers, they're automatically added
on docker-compose that is about increasing the number of worker replicas
But what if I need to deploy workers to other hosts? Create new VMs, join them to my Windmill server.
as long as they all share the same database_url it will work just fine
you need a common postgres database, and that's all you need
Hmmmm, so I would have to expose the Postgres DB from the host, and set up authentication to it.
yes or vpn/tunnel those connections
I would generally expect that workers would access Windmill through a REST API endpoint. Interesting that only the DB is shared.
In theory it could be feasible to front the db with a rest api and make the workers only depend on REST API
Have you ever deployed GitLab before? You can install the GitLab worker package on any node, and then specify a URL to the main server's REST API to join them.
but that's less efficient overall, so we're waiting for a customer request to implement it
It might be a good thing that only the DB is shared. Just saying most software I've come across exposes a REST API for the workers.
Generally DBs are hidden from direct access most of the time, so the REST API can control who accesses it, and how much. I can see why you've chosen this approach for now though. It makes sense.
wrt to auth, db connections and https are not very different
I think we're a lot more low latency than gitlab CI
so we have different requirements
I doubt Gitlab CI can run 1000 jobs/s on 100 workers on the same instance/cluster.
They're optimizing for long running jobs where few secs of latency is no big deal
Maybe there's a documentation gap for deploying workers on separate hosts. Like, where do I get credentials from? Do I need to create a new DB user and assign permissions in Postgres?
Is there a Docker Compose file that I can use to deploy only workers (no server, no DB)?
You can remove the other services from the main docker-compose or set their replicas to 0
you really just need everything to connect to the same database and that's it
everything is left as an exercise to the reader because there is 1000 ways you could do that
I'm a very technical person, but even technical people don't want to spend their time reverse engineering software, to find out how to use it. Prescriptive guidance is still valuable.
Since you chose to use Docker Compose for deploying Windmill, it makes sense to provide opinionated guidance on altering the deployment as well. Just my two cents.
Actually the best way to deploy windmill is helm/kubernetes
I have to find out these answers, so I can provide the answers to my channel viewers. 🙂
But there are mostly 2 buckets of people that deploy windmill
devops and they know their ways around kubes/helm
others that will put windmill on a big node with the docker-compose
Your request which is absolutely legitimate is more uncommon
I agree, for scalability it is best. But many people still do not want to deploy Kubernetes because of how complex it is. Believe me, I built 3x certification training courses on Kubernetes, and even though it's powerful, it creates a lot of technical debt by just creating a production cluster.
I do agree we lack documentation, but the good thing is, our foundation wrt to deployment is very strong. The binary and postgres is sufficient
So we're hoping the community will just be creative with it, for custom requests, it usually come from enterprise customers for whom we give extended guidance
In the PowerShell community especially, there are lots of "IT people" (not necessarily engineers) who know some basics about developing automation scripts, but no clue how to build a UI. I think Windmill will satisfy their needs very well.
Yup, gotta make money.
It's not so much about making money, it's just that people having custom requests don't read docs, they want us on call
We are not gating anything especially docs with enterprise needs, it's just that we have limited bandwidth, so we do what people ask
but we more than welcome any contributions, i'd be happy to make windmill as accessible as possible to everyone!
I am trying to learn enough about it so that I can contribute something back. My best contribution is by creating video training that others can consume and build their own solutions. But maybe I can write some docs if I find extra time. My current "business" is growing my social media channel.
Of course, we don't expect anything from anyone, everyone has their priorities and ours align more with windmill's success than anyone else
Also, by posting on this forum, I am trying to create searchable history about Windmill, so that other people can locate solutions to their questions, before asking. I've been doing the same with the LXD project recently. Found and filed a number of bugs as well, to help improve the user experience.
👍 that's great, i do regret that discord is not so great for searching
FYI I tried to use Budibase a couple weeks ago. Spent a few hours trying to create a basic application. Unfortunately it was way too buggy and unreliable for me to realistically use it. Wasted too many hours. Windmill seems much more reliable and friendlier to use than Budibase.
Discourse is used by many communities, such as the LXC, Golang, Rustlang, etc. Discourse has a nice UX and is excellent for searching on Google. I would highly recommend it for forums.
I'm somewhat glad to hear that! We still have a few bugs there and there but @sindresvendby is a real bug sniper so i'm hopeful to get 100% bug-free pretty quickly
Well, the scripts execute reliably in Windmill. In Budibase, half the time the "link" between UI elements and scripts would break, so events would never even fire.
Reason I say somewhat is that I do feel like budibase is fighting the good fight being open-source
for me the ennemy are the proprietary alternatives
Passing data between UI and scripts is easy to understand in Windmill. In Budibase, it is not well-documented or easy to understand. I do like that they are open source also, but unreliable software is frustrating to work with no matter what.
but yes, i'm a backend engineers and I love distributed systems so windmill is pretty much 100% reliable from a backend perspective, and svelte has been a joy to work with to build this frontend builder
You will have no problem competing with proprietary alternatives. You have a solid foundation already.
🙏
@rubenf Do I have to deploy both the
windmill_worker and the windmill_worker_native?