Stephen•14mo ago

Proxmox deployment and OOM(?) errors

Hello everyone, I've been using Windmill for months on an Ubuntu server, with Windmill deployed in a Docker container. It's been working flawlessly. Lately I've been looking into using Proxmox on this server for my home setup. So, I backed up my workspace, installed Promox, created an Ubuntu VM with 16GB of RAM and I deployed Windmill on it in a Docker container. I then pushed my backed up workspace, which means I am now roughly back to how it was in my previous setup. Sadly none of my flows work. Lightweight scripts getting and setting Windmill variables work, but heavier flows involving pulling data from an API and pushing to a PostgreSQL db fail immediately with the following error:

[
    {
        "error": {
            "name": "ExecutionErr",
            "message": "error during execution of the script:\nprocess terminated by signal: Some(\n    4,\n), stopped_signal: None, core_dumped: true"
        }
    }
]

[
    {
        "error": {
            "name": "ExecutionErr",
            "message": "error during execution of the script:\nprocess terminated by signal: Some(\n    4,\n), stopped_signal: None, core_dumped: true"
        }
    }
]

Someone else posted a question with a similar error log, and the answer was that he was getting an Out Of Memory error. I don't see how this could be the case though: - The VM is currently using 800M of mem out of 15G - docker info returns

 
CPUs: 8
Total Memory: 15.61GiB

 
CPUs: 8
Total Memory: 15.61GiB

I'm a bit lost. Not sure what to do. Does anyone have an idea of what could be going wrong here? I really just have a barebone setup with default configs.

19 Replies

rubenf•14mo ago

Signal 4 is illegal instruction which I have no idea how could that happen So before investigating if it's windmill, I would try to run on same hardware/machinme without proxmox

Stephen•14mo ago

The previous setup is actually still there on the same machine. I have Ubuntu Server installed on a different SSD. I tried to boot on this SSD yesterday to test my flows, and they work. So, what I did is just add a new SSD, installed Proxmox on it and booted with that SSD. I'll try again today just in case and report back! - I booted on the same machine running Proxmox, except on the SSD which just runs Ubuntu Server 22.04 and the flows worked as expected. - I happen to have a different machine with Proxmox installed. I deployed Windmill on it in exactly the same way. Tested my flows and I get the same Signal 4 error. So it's not hardware related. It's got to be Proxmox somehow. So to make it clear: - Machine 1 -> Proxmox -> Ubuntu Server 22.04 -> Windmill using Docker -> ❌ - Machine 1 -> Ubuntu Server 22.04 -> Windmill using Docker -> ✅ - Machine 2 -> Proxmox -> Ubuntu Server 22.04 -> Windmill using Docker -> ❌ Note: everything is deployed with default configs. Next thing I'd like to try is: - Machine 1 -> Proxmox -> Fedora 38 -> Windmill using Docker I don't have much hope but at least it's something!

rubenf•14mo ago

All I can tell you is that native jobs and non native jobs use very different runtimes native jobs are pure rust embedded in the backend non native fork a runtime

Stephen•14mo ago

I'm not entirely sure what that entails, but thanks for the swift reply nonetheless! I'll try to mess around a bit more Ok, so most of my flows are Python flows that use Polars, and switching to Pandas solves the issue! I have no clue what's going on!

rubenf•14mo ago

polars use specific hardware acceleration like SIMD that are probably not available on proxmox

Stephen•14mo ago

Alright then I'll see if there's anything I can do about this in the proxmox config

Stephen•14mo ago

Importing Polars doesn't even work at all within the VM. But, it's possible to make it work! So for the future peeps who run into this: SOLUTION 1) Go to the Proxmox UI 2) Go to the Hardware settings of the VM 3) Edit the Processors config 4) Change the processor Type to host 5) Restart your VM Reference

GitHub

"Illegal instruction (core dumped)" with pip installation on non-AV...

I installed polars 0.13.12 with pip in Python 3.10. When I try to import it (e.g. with python -m polars), it segfaults with Illegal instruction (core dumped): $ pip install polars Collecting polars...

rubenf•14mo ago

qq, what's the benefit of using proxmox ?

Tiago Serafim•14mo ago

There are some new changes in Proxmox 8, might want to check this option https://youtu.be/sZcOlW-DwrU?si=kdbD591GDSm3zAIp&t=1009

Craft Computing

YouTube

Let's Install Proxmox 8.0!

Visit https://Linode.com/CraftComputing to start your $100, 60-Day Trial, and move your services to your own personal cloud today! It's been a while since I've updated my tutorial videos, so today, let's install Proxmox 8.0! This is a beginner's guide to the installation, with step-by-step instructions for installing Proxmox, setting up your fi...

Stephen•14mo ago

Not completely sure so far, I'm still testing it. It's quite popular within the self-hosting community, so I thought I'd give it a go. Initially I wanted to upgrade from running the Home Asisstant OS on my pi. But I already had Windmill and a bunch of containers running on my minipc. And running Home Assistant in a container prevents you from using a lot of nice functionalities you otherwise have access to with the Home Assistant OS. So Proxmox sounded like a perfect solution for this: Proxmox is running on my minipc, and I use it to manage a VM running the Home Assistant OS AND another VM running a regular Ubuntu Server with all my containers. I quite like the ability to spin up VMs in a really simple way in basically 1 minute. For the people like me who don't know much about VMs or Linux, it makes it really easy to cross the bridge 🙂 Though I haven't tested it yet, it looks like the functionality to backup VMs via snapshots is very powerful and ridiculously easy to use.

rubenf•14mo ago

Would you be interested in a small PR to the self-host page of windmill to mention proxmox ?

Stephen•14mo ago

Absolutely! I've never contributed to a public repository on GitHub though, but shouldn't be a problem

rubenf•14mo ago

🙏

Stephen•14mo ago

What do you envision for the mention of Proxmox? Because I gave it some thoughts yesterday, and ultimately I'm just spinning up Ubuntu VMs and deploying Windmill on it via Docker containers. There is nothing special here and Windmill works just as expected following the Docker deployment instructions! The issue I encountered was not linked to Windmill, but to Proxmox. Do you think it still makes sense to mention it?

rubenf•14mo ago

I think a note mentionning how to fix your proxmox specific issue might be sufficient

trogdor•13mo ago

I have been running windmill in both proxmox vm and lxc without a hitch. i have a fair bit of experience in using proxmox for almost a decade old

but windmill behaves predictably and configuring it in different environments is actually very ergonomic in my experience. most recently i am trying to patch up the last pieces of a railway template for a one click deploy of all 6 pieces of infra etc. @rubenf such a clean experience so far kudos...

rubenf•13mo ago

Amazing for the railway template. We always wanted to provide good templates for all cloud providers but always lack time to do so. Happy to reference it in our self hosted docs

2little2late•12mo ago

@trogdor did you have any luck with your railway template? please link if so!

floody•12mo ago

@trogdor good job mate, don't keep it to yourself now 🙂