Worker caching weirdness
We are using a web KV store. For a specific key we have written three blobs of JSON - each write with a different number of array elements in this order (5, 8, and 16). We have a simple windmill typescript script that fetches the data and logs the number of array elements. When we run that script on the windmill deploy (self-hosted EE) the workers get a variety of responses from the fetch (sometimes 5, someimtes 8, sometimes 16). Its like windmill has cached the internal fetch results and is returning that instead of doing the actual fetch. Which seems weird to me - since its just running our script on a worker. However, if I run the exact same fetch on a cloud-hosted windmill, the script always gets 16 elements (which is all that instance has ever seen). Is WM doing something like this or have we just hit some weird distributed caching problem with the KV store we're using?
The fetch is simple:
let result = await fetch(url)
let data = await result.json()
console.log(data.length)
2 Replies
Are you using native scripts or normal deno scripts?
if normal deno scripts, we're literally just running
deno runMaybe a bug in the response headers from the KV store?
HTTP/2 200
server: Fly/d42d3a7f1 (2024-11-25)
date: Wed, 27 Nov 2024 15:09:05 GMT
content-type: text/plain
kvdb-edge-region: iad
vary: Accept-Encoding
vary: Origin
access-control-allow-origin: *
expires: Fri Dec 27 14:01:15 2024 GMT
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-response-time: 0.467ms
x-xss-protection: 1; mode=block
kvdb-cache-status: HIT
via: 2 fly.io
fly-request-id: 01JDQ2PCK6MXMMG9KAP8ZJAQJ2-iad
This is what I see from curl.
I tried both as a normal deno and normal bun.
That is what I thought too ... how could this be a WM issue if they're just launching the script on a worker using the specified runtime.
Those headers are from running curl here locally on my machine. I'm gonna see what the response headers look like on the weird responses.
Well the response headers look pretty much the same from the deno requests as from curl:
Response Headers:
access-control-allow-origin: *
content-type: text/plain
date: Wed, 27 Nov 2024 15:21:36 GMT
expires: Fri Dec 27 13:33:55 2024 GMT
fly-request-id: 01JDQ3DAFZC30W8XJXY9HJGJ6D-sea
kvdb-cache-status: HIT
kvdb-edge-region: sjc
server: Fly/d42d3a7f1 (2024-11-25)
strict-transport-security: max-age=31536000; includeSubDomains; preload
vary: Origin
via: 2 fly.io
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-response-time: 0.490ms
x-xss-protection: 1; mode=block
Got 5 messages
And I agree - since this is just fetch getting executed by deno - this has to be a caching issue with the KV store.
One interesting tidbit - the bun requests used http 1.1 - deno used http 2.0