marko•4mo ago

Bun cache missing windmill-client

One of my flows started suddenly failing with the following error:

job=0190e46d-49f3-598f-e238-e13b0d595088 tag=bun worker=wk-default-eb4f47c94407-YMYN2 hostname=eb4f47c94407

skipping install, using cached buntar based on lockfile hash: xb6AgX-a-bSr6dsMtfSbyDy0oxGu0XSg7uiNtNx4HTs=

--- BUN CODE EXECUTION ---

error: Cannot find package "windmill-client" from "/tmp/windmill/wk-default-eb4f47c94407-YMYN2/0190e46d-49f3-598f-e238-e13b0d595088/main.ts"
Bun v1.1.18 (Linux x64 baseline)

job=0190e46d-49f3-598f-e238-e13b0d595088 tag=bun worker=wk-default-eb4f47c94407-YMYN2 hostname=eb4f47c94407

skipping install, using cached buntar based on lockfile hash: xb6AgX-a-bSr6dsMtfSbyDy0oxGu0XSg7uiNtNx4HTs=

--- BUN CODE EXECUTION ---

error: Cannot find package "windmill-client" from "/tmp/windmill/wk-default-eb4f47c94407-YMYN2/0190e46d-49f3-598f-e238-e13b0d595088/main.ts"
Bun v1.1.18 (Linux x64 baseline)

If I test the node, it works fine. It only fails when I run the deployed flow. It looks like it's just a caching issue, but I can't find any functionality in Windmill that would allow me to wipe the build cache. I did try to create a whole new node where I copy/pasted the code, but the problem persists. Any ideas?

30 Replies

rubenf•4mo ago

it is indeed a caching issue, on EE you can wipe the cache on the worker groups but if you can bash exec into your container and live investigate that would be great. We improved the caching mechanism for bun recently and that might be a by-product of it on what version of windmill are you ?

invakid404•4mo ago

I just ran into the same issue today, I am running CE v1.366.6-15-g8fcda68af

job=0190e4d9-f832-e2fa-2ab9-2dd2debb229d tag=bun worker=wk-default-e82d921f019408-wm1qh hostname=e82d921f019408

skipping install, using cached buntar based on lockfile hash: tIq3Yr9cAUfe4trUyu1nqBu1Ql03fZdnRQiQGxiYh8U=

--- BUN CODE EXECUTION ---

error: Cannot find module "@elastic/elasticsearch" from "/tmp/windmill/wk-default-e82d921f019408-wm1qh/0190e4d9-f832-e2fa-2ab9-2dd2debb229d/main.ts"
Bun v1.1.18 (Linux x64 baseline)

job=0190e4d9-f832-e2fa-2ab9-2dd2debb229d tag=bun worker=wk-default-e82d921f019408-wm1qh hostname=e82d921f019408

skipping install, using cached buntar based on lockfile hash: tIq3Yr9cAUfe4trUyu1nqBu1Ql03fZdnRQiQGxiYh8U=

--- BUN CODE EXECUTION ---

error: Cannot find module "@elastic/elasticsearch" from "/tmp/windmill/wk-default-e82d921f019408-wm1qh/0190e4d9-f832-e2fa-2ab9-2dd2debb229d/main.ts"
Bun v1.1.18 (Linux x64 baseline)

if I wipe the cache, it works for a single run, then it dies again

rubenf•4mo ago

you are using docker-compose ?

invakid404•4mo ago

no, my windmill is deployed on fly.io

rubenf•4mo ago

so you have no shared volume right ?

invakid404•4mo ago

the entirety of my windmill, including workers, is running on a single node, so all workers should have the same cache is what i'd assume

rubenf•4mo ago

do you have a single container ?

invakid404•4mo ago

yes

[env]
  BASE_URL = 'https://xxxxx.fly.dev/'
  KEEP_JOB_DIR = 'false'
  NUM_WORKERS = '4'
  WORKER_TAGS = 'deno,python3,go,bash,powershell,dependency,flow,hub,other,bun,php,postgresql'
  RUST_LOG = 'info'

[[mounts]]
  source = 'worker_cache'
  destination = '/tmp/windmill/cache'
  auto_extend_size_threshold = 80
  auto_extend_size_increment = "1GB"
  auto_extend_size_limit = "10GB"

[http_service]
  internal_port = 80
  force_https = true

[env]
  BASE_URL = 'https://xxxxx.fly.dev/'
  KEEP_JOB_DIR = 'false'
  NUM_WORKERS = '4'
  WORKER_TAGS = 'deno,python3,go,bash,powershell,dependency,flow,hub,other,bun,php,postgresql'
  RUST_LOG = 'info'

[[mounts]]
  source = 'worker_cache'
  destination = '/tmp/windmill/cache'
  auto_extend_size_threshold = 80
  auto_extend_size_increment = "1GB"
  auto_extend_size_limit = "10GB"

[http_service]
  internal_port = 80
  force_https = true

rubenf•4mo ago

can you exec into that container ?

invakid404•4mo ago

here's my fly.toml if it helps yeah, I can

rubenf•4mo ago

can you go into /tmp/windmill/cache/buntar and ls there

invakid404•4mo ago

lemme trigger the issue again

rubenf•4mo ago

you should have a directory tIq3Yr9cAUfe4trUyu1nqBu1Ql03fZdnRQiQGxiYh8U=, cd there and ls it you do not need to trigger the issue to do the above

invakid404•4mo ago

root@e82d921f019408:/tmp/windmill/cache/buntar/tIq3Yr9cAUfe4trUyu1nqBu1Ql03fZdnRQiQGxiYh8U=# ls -la
total 16
drwxr-xr-x 4 root root 4096 Jul 24 13:11 .
drwxr-xr-x 3 root root 4096 Jul 24 13:11 ..
drwxr-xr-x 3 root root 4096 Jul 24 13:11 node_modules
drwxr-xr-x 2 root root 4096 Jul 24 13:11 shared

root@e82d921f019408:/tmp/windmill/cache/buntar/tIq3Yr9cAUfe4trUyu1nqBu1Ql03fZdnRQiQGxiYh8U=# ls -la
total 16
drwxr-xr-x 4 root root 4096 Jul 24 13:11 .
drwxr-xr-x 3 root root 4096 Jul 24 13:11 ..
drwxr-xr-x 3 root root 4096 Jul 24 13:11 node_modules
drwxr-xr-x 2 root root 4096 Jul 24 13:11 shared

yeah, figured

root@e82d921f019408:/tmp/windmill/cache/buntar/tIq3Yr9cAUfe4trUyu1nqBu1Ql03fZdnRQiQGxiYh8U=/node_modules# ls -la
total 12
drwxr-xr-x 3 root root 4096 Jul 24 13:11 .
drwxr-xr-x 4 root root 4096 Jul 24 13:11 ..
drwxr-xr-x 2 root root 4096 Jul 24 13:11 ms

root@e82d921f019408:/tmp/windmill/cache/buntar/tIq3Yr9cAUfe4trUyu1nqBu1Ql03fZdnRQiQGxiYh8U=/node_modules# ls -la
total 12
drwxr-xr-x 3 root root 4096 Jul 24 13:11 .
drwxr-xr-x 4 root root 4096 Jul 24 13:11 ..
drwxr-xr-x 2 root root 4096 Jul 24 13:11 ms

i can't fit it in a message, so here's the lockfile information for my script in a pastebin if it helps: https://pastebin.com/w5gtu2fs it's supposed to have @elastic/elasticsearch and zod installed, but neither are present in node_modules I believe this started happening after upgrading to 1.366.6, so it must be a regression that happened semi-recently

rubenf•4mo ago

yes we changed the way it behaved you should have debug @elastic hpagent ms @opentelemetry secure-json-parse tslib undici in there do you have the issue with a script as simple as:

import * as wmill from "@elastic/elasticsearch"

export async function main(x: string) {
  return wmill
}

import * as wmill from "@elastic/elasticsearch"

export async function main(x: string) {
  return wmill
}

invakid404•4mo ago

let me see yes same issue node_modules only has ms again

rubenf•4mo ago

do you have any persistent volume on that container? yeah /tmp/windmill/cache

invakid404•4mo ago

/tmp/windmill/cache is a persistent volume, yes

rubenf•4mo ago

@invakid404 could you: remove the folder: tIq3Yr9cAUfe4trUyu1nqBu1Ql03fZdnRQiQGxiYh8U= then deploy and run again, and show me the logs in particular, looking at the error logs: Could not create buntar: it would be in your container logs

invakid404•4mo ago

on it

rubenf•4mo ago

also, it will recreate the folder "tIq3Yr9cAUfe4trUyu1nqBu1Ql03fZdnRQiQGxiYh8U=" if you can ls node_modules that would be great

invakid404•4mo ago

(currently waiting for the docker image to get pushed so i can redeploy :D)

rubenf•4mo ago

sorry I meant deploying the script just pressing "deploy" in the script UI

invakid404•4mo ago

oh i see

rubenf•4mo ago

I have a pretty good intuition of what the problem is which is somehow you couldn't do the full copy and we didn't handle well if you had an error while doing the copy (which I just fixed) but if i'm right then we should some interesting things in your ocntainer logs

invakid404•4mo ago

2024-07-24T13:36:17Z app[e82d921f019408] ams [info]2024-07-24T13:36:17.528422Z ERROR worker:job: windmill-worker/src/bun_executor.rs:585: Could not create buntar: Invalid cross-device link (os error 18) worker=wk-default-e82d921f019408-A9Tq8 hostname=e82d921f019408 job_id=0190e4f4-f611-f561-6749-a5df8c97a1db

2024-07-24T13:36:17Z app[e82d921f019408] ams [info]2024-07-24T13:36:17.528422Z ERROR worker:job: windmill-worker/src/bun_executor.rs:585: Could not create buntar: Invalid cross-device link (os error 18) worker=wk-default-e82d921f019408-A9Tq8 hostname=e82d921f019408 job_id=0190e4f4-f611-f561-6749-a5df8c97a1db

is this what you're looking for?

rubenf•4mo ago

yes very interesting ok so my fix does fix it

invakid404•4mo ago

so i should be fine if i update windmill?

rubenf•4mo ago

yes but this error is kinda crazy it would only happen if /tmp/windmill/cache is spread across different partitions ah well fly.io does crazy stuff usually

marko•4mo ago

Sorry, I was in meetings. My instance is on docker compose on single VPS. And yeah, the issue has now broken every node that uses Bun. I'll try to delete it as well. Looks like deleting the cache folder worked for me as well.