Single script with long running loop vs flow loop feature
Context:
- Single Python script with an internal loop that iterates over several dozen items
- Each item interacts with a pretty slow API endpoint (i.e., 1-2 seconds for each request)
- The process in question is not super time sensitive (doesn't matter if it takes 30 seconds or 3 minutes to run)
Option 1:
- Single script that internally iterates over the list. If there are 30 items, it could run for 30-60 seconds, which means it occupies a worker for the full 30-60 seconds
Option 2:
- Flow that interates over the list, running a new script execution for each item. This will, overall, take more time than a single script, but I assume it would allow other items waiting for available Workers to slip in between each iteration (rather than waiting for up to a minute -- in the case that other workers were occupied as well).
Do folks have opinions about which option they've selected in their own approaches?
- Single Python script with an internal loop that iterates over several dozen items
- Each item interacts with a pretty slow API endpoint (i.e., 1-2 seconds for each request)
- The process in question is not super time sensitive (doesn't matter if it takes 30 seconds or 3 minutes to run)
Option 1:
- Single script that internally iterates over the list. If there are 30 items, it could run for 30-60 seconds, which means it occupies a worker for the full 30-60 seconds
Option 2:
- Flow that interates over the list, running a new script execution for each item. This will, overall, take more time than a single script, but I assume it would allow other items waiting for available Workers to slip in between each iteration (rather than waiting for up to a minute -- in the case that other workers were occupied as well).
Do folks have opinions about which option they've selected in their own approaches?
