Catching errors
I'm trying out Windmill to run some long running nightly tests, and I'm wondering about how to deal with tests that throw.
- All the tests are run in parallell using "branch to all" with each branch set to skip on failure
- As a result the "suite" as whole thus completes even if one test explodes, good
- But at the end of the run I want to collect all the results to produce a test report
- Each test has a name, and each test should either return "success" or "failure"
- But a test that throws an exception is inconclusive
- I would like to report the failing test alongside the other tests
- But there is no way to get the name of the branch/script that caused the failure (apart from maybe parsing the stack trace which feels a little ugly)
The obvious solution is of course just to wrap the whole body of the test in a try catch and return "inconclusive" when we catch the exception, but I'm curious if there's some feature I'm missing.
I thought maybe I could create a generic failure handler script which would transform the error payload into an inconclusive result and place that after the test, but this won't work since Windmill stops running the branch on failure. Having a global failure handler also won't work since that sits outside of the flow and can't affect the final result.
What I think would work well is an exception handler tied to the script, sort of like a try/catch as a workflow node.
- All the tests are run in parallell using "branch to all" with each branch set to skip on failure
- As a result the "suite" as whole thus completes even if one test explodes, good
- But at the end of the run I want to collect all the results to produce a test report
- Each test has a name, and each test should either return "success" or "failure"
- But a test that throws an exception is inconclusive
- I would like to report the failing test alongside the other tests
- But there is no way to get the name of the branch/script that caused the failure (apart from maybe parsing the stack trace which feels a little ugly)
The obvious solution is of course just to wrap the whole body of the test in a try catch and return "inconclusive" when we catch the exception, but I'm curious if there's some feature I'm missing.
I thought maybe I could create a generic failure handler script which would transform the error payload into an inconclusive result and place that after the test, but this won't work since Windmill stops running the branch on failure. Having a global failure handler also won't work since that sits outside of the flow and can't affect the final result.
What I think would work well is an exception handler tied to the script, sort of like a try/catch as a workflow node.
