Working with (CSV) file output and python in scripts/workflows
Hello, I am working with an API that returns a large JSON response (like ~2MB). I am doing some transformation in python but then I want my python script to return this as a CSV file object (for use in a workflow step - to email or put in S3 bucket etc) which is proving a bit challenging for me.
Is this achievable and if so, could anyone please share an example or pointer for how to have a python script return a (CSV) file? Thanks in advance!
3 Replies
you can share it directly as a result, store it on disk using the shared directoyy or use s3 directly
Thanks, I'll keep working at it. I see another user had the exact question too. I will try to use the previous answer.
Have you tried to use parquet format?