Trevor Sullivan
Trevor Sullivan16mo ago

Deploy new workers to Windmill installation

I've used the Docker Compose deployment of Windmill on an LXD container. It's working just fine, but I need to know how to deploy additional workers and join them to my Windmill server. On the Workers page, I can see that 4 distinct workers have been created, but I don't see any buttons to join new workers to my Windmill server. I searched the documentation and forum for "workers" and cannot find any directions to install / join new workers either. How can this be accomplished?
34 Replies
rubenf
rubenf16mo ago
you just need to spawn new workers, they're automatically added on docker-compose that is about increasing the number of worker replicas
Trevor Sullivan
Trevor Sullivan16mo ago
But what if I need to deploy workers to other hosts? Create new VMs, join them to my Windmill server.
rubenf
rubenf16mo ago
as long as they all share the same database_url it will work just fine you need a common postgres database, and that's all you need
Trevor Sullivan
Trevor Sullivan16mo ago
Hmmmm, so I would have to expose the Postgres DB from the host, and set up authentication to it.
rubenf
rubenf16mo ago
yes or vpn/tunnel those connections
Trevor Sullivan
Trevor Sullivan16mo ago
I would generally expect that workers would access Windmill through a REST API endpoint. Interesting that only the DB is shared.
rubenf
rubenf16mo ago
In theory it could be feasible to front the db with a rest api and make the workers only depend on REST API
Trevor Sullivan
Trevor Sullivan16mo ago
Have you ever deployed GitLab before? You can install the GitLab worker package on any node, and then specify a URL to the main server's REST API to join them.
rubenf
rubenf16mo ago
but that's less efficient overall, so we're waiting for a customer request to implement it
Trevor Sullivan
Trevor Sullivan16mo ago
It might be a good thing that only the DB is shared. Just saying most software I've come across exposes a REST API for the workers. Generally DBs are hidden from direct access most of the time, so the REST API can control who accesses it, and how much. I can see why you've chosen this approach for now though. It makes sense.
rubenf
rubenf16mo ago
wrt to auth, db connections and https are not very different I think we're a lot more low latency than gitlab CI so we have different requirements I doubt Gitlab CI can run 1000 jobs/s on 100 workers on the same instance/cluster. They're optimizing for long running jobs where few secs of latency is no big deal
Trevor Sullivan
Trevor Sullivan16mo ago
Maybe there's a documentation gap for deploying workers on separate hosts. Like, where do I get credentials from? Do I need to create a new DB user and assign permissions in Postgres? Is there a Docker Compose file that I can use to deploy only workers (no server, no DB)?
rubenf
rubenf16mo ago
You can remove the other services from the main docker-compose or set their replicas to 0 you really just need everything to connect to the same database and that's it everything is left as an exercise to the reader because there is 1000 ways you could do that
Trevor Sullivan
Trevor Sullivan16mo ago
I'm a very technical person, but even technical people don't want to spend their time reverse engineering software, to find out how to use it. Prescriptive guidance is still valuable. Since you chose to use Docker Compose for deploying Windmill, it makes sense to provide opinionated guidance on altering the deployment as well. Just my two cents.
rubenf
rubenf16mo ago
Actually the best way to deploy windmill is helm/kubernetes
Trevor Sullivan
Trevor Sullivan16mo ago
I have to find out these answers, so I can provide the answers to my channel viewers. 🙂
rubenf
rubenf16mo ago
But there are mostly 2 buckets of people that deploy windmill devops and they know their ways around kubes/helm others that will put windmill on a big node with the docker-compose Your request which is absolutely legitimate is more uncommon
Trevor Sullivan
Trevor Sullivan16mo ago
I agree, for scalability it is best. But many people still do not want to deploy Kubernetes because of how complex it is. Believe me, I built 3x certification training courses on Kubernetes, and even though it's powerful, it creates a lot of technical debt by just creating a production cluster.
rubenf
rubenf16mo ago
I do agree we lack documentation, but the good thing is, our foundation wrt to deployment is very strong. The binary and postgres is sufficient So we're hoping the community will just be creative with it, for custom requests, it usually come from enterprise customers for whom we give extended guidance
Trevor Sullivan
Trevor Sullivan16mo ago
In the PowerShell community especially, there are lots of "IT people" (not necessarily engineers) who know some basics about developing automation scripts, but no clue how to build a UI. I think Windmill will satisfy their needs very well. Yup, gotta make money.
rubenf
rubenf16mo ago
It's not so much about making money, it's just that people having custom requests don't read docs, they want us on call We are not gating anything especially docs with enterprise needs, it's just that we have limited bandwidth, so we do what people ask but we more than welcome any contributions, i'd be happy to make windmill as accessible as possible to everyone!
Trevor Sullivan
Trevor Sullivan16mo ago
I am trying to learn enough about it so that I can contribute something back. My best contribution is by creating video training that others can consume and build their own solutions. But maybe I can write some docs if I find extra time. My current "business" is growing my social media channel.
rubenf
rubenf16mo ago
Of course, we don't expect anything from anyone, everyone has their priorities and ours align more with windmill's success than anyone else
Trevor Sullivan
Trevor Sullivan16mo ago
Also, by posting on this forum, I am trying to create searchable history about Windmill, so that other people can locate solutions to their questions, before asking. I've been doing the same with the LXD project recently. Found and filed a number of bugs as well, to help improve the user experience.
rubenf
rubenf16mo ago
👍 that's great, i do regret that discord is not so great for searching
Trevor Sullivan
Trevor Sullivan16mo ago
FYI I tried to use Budibase a couple weeks ago. Spent a few hours trying to create a basic application. Unfortunately it was way too buggy and unreliable for me to realistically use it. Wasted too many hours. Windmill seems much more reliable and friendlier to use than Budibase. Discourse is used by many communities, such as the LXC, Golang, Rustlang, etc. Discourse has a nice UX and is excellent for searching on Google. I would highly recommend it for forums.
rubenf
rubenf16mo ago
I'm somewhat glad to hear that! We still have a few bugs there and there but @sindresvendby is a real bug sniper so i'm hopeful to get 100% bug-free pretty quickly
Trevor Sullivan
Trevor Sullivan16mo ago
Well, the scripts execute reliably in Windmill. In Budibase, half the time the "link" between UI elements and scripts would break, so events would never even fire.
rubenf
rubenf16mo ago
Reason I say somewhat is that I do feel like budibase is fighting the good fight being open-source for me the ennemy are the proprietary alternatives
Trevor Sullivan
Trevor Sullivan16mo ago
Passing data between UI and scripts is easy to understand in Windmill. In Budibase, it is not well-documented or easy to understand. I do like that they are open source also, but unreliable software is frustrating to work with no matter what.
rubenf
rubenf16mo ago
but yes, i'm a backend engineers and I love distributed systems so windmill is pretty much 100% reliable from a backend perspective, and svelte has been a joy to work with to build this frontend builder
Trevor Sullivan
Trevor Sullivan16mo ago
You will have no problem competing with proprietary alternatives. You have a solid foundation already.
rubenf
rubenf16mo ago
🙏
Trevor Sullivan
Trevor Sullivan16mo ago
@rubenf Do I have to deploy both the windmill_worker and the windmill_worker_native?