sledge•4mo ago

Bun installer appears to fail during script launch.

I just finished setting up a self-host install of WM. I can run a deno typescript test without problems, but if I run a bun test, the mem just keeps building then eventually I see this: ExecutionErr: error during execution of the script: process terminated by signal: Some( 9, ), stopped_signal: None, core_dumped: false Any tips on what to do to investigate? Thanks, -Chris

13 Replies

rubenf•4mo ago

You can exec into the container and run bun on there windmill does optimization on top of it but that's mostly what it does

sledge•4mo ago

So I did this, and bun runs fine in the worker. I can make a simple js or ts file and run it in the worker without problems. But if I goto the worker filesystem and find the failed job and try to run its build.js, it fails with just "Killed".

# bun run test.ts
Hello typescript world
# pwd
/tmp/windmill/wk-default-f93bad86eef5-FVyHn/01904812-ffea-f1c5-47d2-d3b0157eddd0
# ls -a
.  ..  build.js  main.ts  out  shared  test.js  test.ts
# pwd
/tmp/windmill/wk-default-f93bad86eef5-FVyHn/01904812-ffea-f1c5-47d2-d3b0157eddd0
# ls -l
total 28
-rw-r--r-- 1 root root 4229 Jun 24 02:28 build.js
-rw-r--r-- 1 root root  297 Jun 24 02:28 main.ts
drwxr-xr-x 2 root root 4096 Jun 24 02:28 out
drwxr-xr-x 2 root root 4096 Jun 24 02:28 shared
-rw-r--r-- 1 root root   21 Jun 24 03:08 test.js
-rw-r--r-- 1 root root   38 Jun 24 03:09 test.ts
# bun run build.js
Killed
#

# bun run test.ts
Hello typescript world
# pwd
/tmp/windmill/wk-default-f93bad86eef5-FVyHn/01904812-ffea-f1c5-47d2-d3b0157eddd0
# ls -a
.  ..  build.js  main.ts  out  shared  test.js  test.ts
# pwd
/tmp/windmill/wk-default-f93bad86eef5-FVyHn/01904812-ffea-f1c5-47d2-d3b0157eddd0
# ls -l
total 28
-rw-r--r-- 1 root root 4229 Jun 24 02:28 build.js
-rw-r--r-- 1 root root  297 Jun 24 02:28 main.ts
drwxr-xr-x 2 root root 4096 Jun 24 02:28 out
drwxr-xr-x 2 root root 4096 Jun 24 02:28 shared
-rw-r--r-- 1 root root   21 Jun 24 03:08 test.js
-rw-r--r-- 1 root root   38 Jun 24 03:09 test.ts
# bun run build.js
Killed
#

I watched CPU and mem with docker stats on the worker container while running bun run build.js and CPU pegs at 100, ram gets to around 1.5gb (container is set for 2.0gb). Upped the worker size to 3.0gb and same thing happens - pegs out memory usage and then gets killed. Ran simple script on my cloud instance and bun used 4.75M - so obviously the massive memory consumption when running build.js is a clue. What else should I do to help troubleshoot?

rubenf•4mo ago

Do you have that error on every script or a particular one?

sledge•4mo ago

Every script - here is the one I'm testing:

export function main(
) {
  console.log('Hello world')
}

export function main(
) {
  console.log('Hello world')
}

@rubenf Any other suggestions on what I should try here to determine if this is a bug in windmill or in my config? Using the provided docker-compose with ee image except dropped worker count to 1 - which seems like that should not matter.

rubenf•4mo ago

what tools did you use to assess that command took all the mem ?

sledge•4mo ago

Watching the windmill job execution window And watching docker stats on the worker container

rubenf•4mo ago

can you modify build.js and see what makes it so unhappy

sledge•4mo ago

I'll try ... I'm a docker-inspection newb - and no editor is in that image, so I couldn't easily change it when I tried, and the file is not on a mapped volume so I couldn't see an easy way of editing locally (outside of the container).

rubenf•4mo ago

you can install nano on the worker and nano with bash exec Also can you try reproducing the issue with a local docker-compose I assume you won't be able to otherwise everyone would have that issue so now need to isolate the difference between your setup and that one which is causing the issue

sledge•4mo ago

Yep, makes sense. I'll proceed down that path and report back. Thanks for the tips. I thought about the local docker-compose - but I thought it would be painful 'cause I'd have to go through all the setup in order to get to the point where I could run a script from windmill. This is the offending code:

await fs.readFile("./out/main.js", { encoding: "utf8" });

I tried a few things: 1. Switched to readFileSync - no change 2. Upgraded bun from 1.1.8 to 1.1.17 - no change 3. Upgraded bun to canary - no change Completely removed that readFile - since there are no dependencies it is not necessary for this test. Definitely now the bun run build.js works fine ... so the readFile is the culprit. And sure enough - culling it down to this - still gets killed:

const fs = require("fs/promises");
const content = await fs.readFile("./package.json", { encoding: "utf8" });
console.log(content)

const fs = require("fs/promises");
const content = await fs.readFile("./package.json", { encoding: "utf8" });
console.log(content)

rubenf•4mo ago

Does readfile bug for any file ? Can you try on a raw bun image to reproduce?

sledge•4mo ago

FWIW - my host VPS is running ubuntu 22.04 and I did see this when I started watching its syslog output: Jun 25 18:50:47 server1 kernel: [146005.812829] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-b170dd6c3 1d89c9c6aa7dd565e4683113776f1775.scope,task=bun,pid=3760437,uid=0 Jun 25 18:50:47 server1 kernel: [146005.812854] Out of memory: Killed process 3760437 (bun) total-vm:85378868kB, anon-rss:2447036kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:6384kB o J Interesting - remove the utf8 and it finishes fine. And switching to Bun API also fails:

const content = await Bun.file('./package.json').text()

So I'm guessing a Bun + Ubuntu 22 filesystem issue? And sure enough - running same test on the raw ubuntu 22 independent of windmill container shows the problem as well. So - guess the current solution is "do not run Windmill on ubuntu 22 due to issues with bun". I posted an issue on the Bun discord with the super simple sample. Turns out its not the filesystem function, changed it to read raw then use Buffer.toString('utf8') - that gets OOMed. @rubenf Turns out the problem is that bun requires minimum avx instruction set on processor - for simd operations. The virtual CPU provided by my hoster does not support avx - so basically its a does-not-meet-minimum-requirements issue for running bun.

sledge•4mo ago

There is an open issue for other folks experiencing other strange symptoms tied to what seems to be the same core issue - CPU emulation. https://github.com/oven-sh/bun/issues/3312

GitHub

CPU emulator leads to strange memory issues · Issue #3312 · oven-sh...

The solution edited Download the build of Bun with “baseline” in the name https://github.com/oven-sh/bun/releases/latest/download/bun-linux-x64-baseline.zip if using Proxmor, follow this screenshot...