andness•11mo ago

Catching errors

I'm trying out Windmill to run some long running nightly tests, and I'm wondering about how to deal with tests that throw. - All the tests are run in parallell using "branch to all" with each branch set to skip on failure - As a result the "suite" as whole thus completes even if one test explodes, good - But at the end of the run I want to collect all the results to produce a test report - Each test has a name, and each test should either return "success" or "failure" - But a test that throws an exception is inconclusive - I would like to report the failing test alongside the other tests - But there is no way to get the name of the branch/script that caused the failure (apart from maybe parsing the stack trace which feels a little ugly) The obvious solution is of course just to wrap the whole body of the test in a try catch and return "inconclusive" when we catch the exception, but I'm curious if there's some feature I'm missing. I thought maybe I could create a generic failure handler script which would transform the error payload into an inconclusive result and place that after the test, but this won't work since Windmill stops running the branch on failure. Having a global failure handler also won't work since that sits outside of the flow and can't affect the final result. What I think would work well is an exception handler tied to the script, sort of like a try/catch as a workflow node.

7 Replies

rubenf•11mo ago

It's a good point. I would need to explore in terms of DX what is optimal but just so you know, errors are treated like normal values and collected as part of the forloop/branchall so I would process them there so you do not need to try/catch your individual script, you can do that at the step right after your branchall/forloop and iterate over the result of that forloop/branchall itself, then build a report by filtering the items that are errors

rubenf•11mo ago

rubenf•11mo ago

# import wmill


def main(x: str):
    if x == 2:
        raise Exception("x")
    return x

# import wmill


def main(x: str):
    if x == 2:
        raise Exception("x")
    return x

that's the script in the forloop

andnessOP•11mo ago

Yeah I do get the error in the collect after the branch all, but the information about which test it was is lost.

rubenf•11mo ago

Aaah, got it. I think best would be for us to add that as metadata to the error itself

windmill-todo-bot•11mo ago

Issue created: https://github.com/windmill-labs/windmill/issues/2978

GitHub

feature: when error happens in script within flows, supplement err...

[MESSAGE_ID=1194201797435346974]

andnessOP•11mo ago

Yeah I think that would work well Oh great 😄 Not a show-stopper for sure, but nevertheless seems like it could be useful I just realized another good reason for having this is that when I catch within the test script itself it looks green in the workflow view.