stair_spirit
stair_spirit12mo ago

Working with (CSV) file output and python in scripts/workflows

Hello, I am working with an API that returns a large JSON response (like ~2MB). I am doing some transformation in python but then I want my python script to return this as a CSV file object (for use in a workflow step - to email or put in S3 bucket etc) which is proving a bit challenging for me. Is this achievable and if so, could anyone please share an example or pointer for how to have a python script return a (CSV) file? Thanks in advance!
3 Replies
rubenf
rubenf12mo ago
you can share it directly as a result, store it on disk using the shared directoyy or use s3 directly
stair_spirit
stair_spiritOP12mo ago
Thanks, I'll keep working at it. I see another user had the exact question too. I will try to use the previous answer.
GoldenChain
GoldenChain12mo ago
Have you tried to use parquet format?