Bun cache missing windmill-client
One of my flows started suddenly failing with the following error:
If I test the node, it works fine. It only fails when I run the deployed flow. It looks like it's just a caching issue, but I can't find any functionality in Windmill that would allow me to wipe the build cache.
I did try to create a whole new node where I copy/pasted the code, but the problem persists.
Any ideas?
30 Replies
it is indeed a caching issue, on EE you can wipe the cache on the worker groups but if you can bash exec into your container and live investigate that would be great. We improved the caching mechanism for bun recently and that might be a by-product of it
on what version of windmill are you ?
I just ran into the same issue today, I am running CE v1.366.6-15-g8fcda68af
if I wipe the cache, it works for a single run, then it dies again
you are using docker-compose ?
no, my windmill is deployed on fly.io
so you have no shared volume right ?
the entirety of my windmill, including workers, is running on a single node, so all workers should have the same cache is what i'd assume
do you have a single container ?
yes
can you exec into that container ?
here's my fly.toml if it helps
yeah, I can
can you go into /tmp/windmill/cache/buntar and ls there
lemme trigger the issue again
you should have a directory
tIq3Yr9cAUfe4trUyu1nqBu1Ql03fZdnRQiQGxiYh8U=
, cd there and ls it
you do not need to trigger the issue to do the above
yeah, figured
i can't fit it in a message, so here's the lockfile information for my script in a pastebin if it helps: https://pastebin.com/w5gtu2fs
it's supposed to have
@elastic/elasticsearch
and zod
installed, but neither are present in node_modules
I believe this started happening after upgrading to 1.366.6, so it must be a regression that happened semi-recentlyyes we changed the way it behaved
you should have
debug @elastic hpagent ms @opentelemetry secure-json-parse tslib undici
in there
do you have the issue with a script as simple as:
let me see
yes
same issue
node_modules
only has ms
againdo you have any persistent volume on that container?
yeah /tmp/windmill/cache
/tmp/windmill/cache
is a persistent volume, yes@invakid404 could you:
remove the folder:
tIq3Yr9cAUfe4trUyu1nqBu1Ql03fZdnRQiQGxiYh8U=
then deploy and run again, and show me the logs
in particular, looking at the error logs:
Could not create buntar:
it would be in your container logson it
also, it will recreate the folder "tIq3Yr9cAUfe4trUyu1nqBu1Ql03fZdnRQiQGxiYh8U=" if you can ls node_modules that would be great
(currently waiting for the docker image to get pushed so i can redeploy :D)
sorry I meant deploying the script
just pressing "deploy" in the script UI
oh
i see
I have a pretty good intuition of what the problem is
which is somehow you couldn't do the full copy
and we didn't handle well if you had an error while doing the copy (which I just fixed)
but if i'm right then we should some interesting things in your ocntainer logs
is this what you're looking for?
yes
very interesting
ok so my fix does fix it
so i should be fine if i update windmill?
yes
but this error is kinda crazy
it would only happen if /tmp/windmill/cache is spread across different partitions
ah well fly.io does crazy stuff usually
Sorry, I was in meetings. My instance is on docker compose on single VPS.
And yeah, the issue has now broken every node that uses Bun. I'll try to delete it as well.
Looks like deleting the cache folder worked for me as well.