invakid404•3w ago

Internal: Object store client not present, cannot stream logs from store when clicking Show more...

When Windmill truncates the logs of a particular run and shows the "Show more..." button, clicking it causes the error from the title. We do have "Instance object storage" configured in the Instance settings, and clicking both "Test from a server" and "Test from a worker" results in a success. Any ideas?

20 Replies

rubenf•3w ago

Not sure no, it works fine on prod on latest

invakid404OP•3w ago

We're on the Windmill Pro plan and not EE if that makes any difference I'm not sure if the S3 logging is supposed to be working for us

rubenf•3w ago

yeah I think it doesn't will improve the error message

invakid404OP•3w ago

I guess the real question is whether Windmill would fall back to trying to read the logs from the filesystem or it'd keep failing unless we disable the instance object storage

rubenf•3w ago

It will fallback to logs from filesystem actually, not fully certain

Linear•3w ago

Issue WIN-677 created.

WIN-677 - improve error message for lack of instance s3 on pro plan

Status

Triage

Windmill

invakid404OP•3w ago

@rubenf btw, I checked on my community edition instance at home and the error I get is different, so my assumption is that windmill is trying to use S3 for logs even though we're not supposed to have it included in the Pro plan (or so you've told me), so it looks like there is a bug here, as I'd expect it to fall back to logs from filesystem, but the error indicates it's trying S3 only I set up a shared mount for the server and workers mounted on /tmp/windmill/logs, disabled the instance object store, yet I still get Internal: Object store client not present, cannot stream logs from store Windmill appears to be writing stuff to the shared volume already, I'm not sure why it's trying to read from an object store

rubenf•3w ago

I will investigate

invakid404OP•3w ago

I can confirm the log I'm trying to read is located in /tmp/windmill/logs/ under the job id, so it's just trying to read from the wrong location it seems, writing is fine

rubenf•3w ago

👍

invakid404OP•2w ago

@rubenf sorry for the noise, just wanted to check if this issue is being worked on/planned. We currently have to manually SSH and piece together the logs each time we need them, which is very cumbersome and makes debugging very painful

rubenf•2w ago

I'm on it today I believe the shared mount must not be working properly @invakid404

async fn get_log_file(Path((_w_id, file_p)): Path<(String, String)>) -> error::Result<Response> {
    let local_file = format!("{TMP_DIR}/logs/{file_p}");
    if tokio::fs::metadata(&local_file).await.is_ok() {
        let mut file = tokio::fs::File::open(local_file).await.map_err(to_anyhow)?;
        let mut buffer = Vec::new();
        file.read_to_end(&mut buffer).await.map_err(to_anyhow)?;
        let res = Response::builder()
            .header(http::header::CONTENT_TYPE, "text/plain")
            .body(Body::from(bytes::Bytes::from(buffer)))
            .unwrap();
        return Ok(res);
    }

    #[cfg(all(feature = "enterprise", feature = "parquet"))]
    if let Some(os) = OBJECT_STORE_CACHE_SETTINGS.read().await.clone() {
        let file = os
            .get(&object_store::path::Path::from(format!("logs/{file_p}")))
            .await;
        if let Ok(file) = file {
            if let Ok(bytes) = file.bytes().await {
                use axum::http::header;
                let res = Response::builder()
                    .header(header::CONTENT_TYPE, "text/plain")
                    .body(Body::from(bytes::Bytes::from(bytes)))
                    .unwrap();
                return Ok(res);
            } else {
                return Err(error::Error::InternalErr(format!(
                    "Error getting bytes from file: {}",
                    file_p
                )));
            }
        } else {
            return Err(error::Error::NotFound(format!(
                "File not found: {}",
                file_p
            )));
        }
    } else {
        return Err(error::Error::InternalErr(
            "Object store client not present, cannot stream logs from store".to_string(),
        ));
    }

async fn get_log_file(Path((_w_id, file_p)): Path<(String, String)>) -> error::Result<Response> {
    let local_file = format!("{TMP_DIR}/logs/{file_p}");
    if tokio::fs::metadata(&local_file).await.is_ok() {
        let mut file = tokio::fs::File::open(local_file).await.map_err(to_anyhow)?;
        let mut buffer = Vec::new();
        file.read_to_end(&mut buffer).await.map_err(to_anyhow)?;
        let res = Response::builder()
            .header(http::header::CONTENT_TYPE, "text/plain")
            .body(Body::from(bytes::Bytes::from(buffer)))
            .unwrap();
        return Ok(res);
    }

    #[cfg(all(feature = "enterprise", feature = "parquet"))]
    if let Some(os) = OBJECT_STORE_CACHE_SETTINGS.read().await.clone() {
        let file = os
            .get(&object_store::path::Path::from(format!("logs/{file_p}")))
            .await;
        if let Ok(file) = file {
            if let Ok(bytes) = file.bytes().await {
                use axum::http::header;
                let res = Response::builder()
                    .header(header::CONTENT_TYPE, "text/plain")
                    .body(Body::from(bytes::Bytes::from(bytes)))
                    .unwrap();
                return Ok(res);
            } else {
                return Err(error::Error::InternalErr(format!(
                    "Error getting bytes from file: {}",
                    file_p
                )));
            }
        } else {
            return Err(error::Error::NotFound(format!(
                "File not found: {}",
                file_p
            )));
        }
    } else {
        return Err(error::Error::InternalErr(
            "Object store client not present, cannot stream logs from store".to_string(),
        ));
    }

the code is very clear, it's checking first if the file exist on the expected location

invakid404OP•2w ago

this is weird, we have a shared mount on both the workers and the server I'm sure it works on the workers' end, as logs are indeed being written to said shared mount

invakid404OP•2w ago

and I can see that it's mounted on the server as well: https://img.qilin-qilin.ts.net/2024-11-25_10-04-51_BKyZj.webp

rubenf•2w ago

on very latest, the error message will indicates what file is missing

invakid404OP•2w ago

I'll update once the image is out and I'll let you know @rubenf

Internal: Object store client not present and file not found on server logs volume at /tmp/windmill/logs/logs/019362d9-bf87-ca69-d0a8-58e7d0535117/1732530257874_9276.txt

Internal: Object store client not present and file not found on server logs volume at /tmp/windmill/logs/logs/019362d9-bf87-ca69-d0a8-58e7d0535117/1732530257874_9276.txt

there's logs twice in the path for whatever reason

rubenf•2w ago

can you show me the request being done please

invakid404OP•2w ago

it's hitting /api/w/mandel-staging/jobs_u/get_log_file/logs/019362d9-bf87-ca69-d0a8-58e7d0535117/1732530257874_9276.txt the log_file_index column in postgres appears to contain the logs/ prefix as well, so i'd assume it's coming from there

rubenf•2w ago

Will investigate, thanks Ok found the issue and fixed on very latest

invakid404OP•2w ago

thanks! I'll update once the image is out and I'll let you know if it's fixed on our end