Rico
Rico11mo ago

How to handle a paginated graphQL request?

Hello, I have been really enjoying using windmill and I figure I am missing something but how should I handle paginated graphQL requests? I have a a graphQL request which takes a "first" prop to make paginated requests and I have used a while loop in the past to keep fetching pages until I retrieved all necessary records. However, I am unsure how to do it the Windmill way. To be clear, this is being used for a data pipeline so all the data needs to be fetched and processed so I need to paginate through it all. Any suggestions would be great. Thanks.
4 Replies
Sindre
Sindre11mo ago
As long as you need everything and its not a lot of request. I like to just use a while loop in the code. If you want to use windmill features like retry or throttling. You could use a loop in the workflow. But I rarely do that
Rico
Rico11mo ago
Thanks @sindre . Thats where I am at with that as well. I like the built-in graphQL scripts but not necessary. I was wondering because it seems like it would scale better doing it as a for loop flow type. I almost got it working using an early abort but I can't find a way to use the results of previous iterations a you may need with a paginated cursor.
rubenf
rubenf11mo ago
@Rico it seems you would want to use the setState/getState to propagate the index between the iterations
Rico
Rico11mo ago
Thanks @rubenf I will try that!