Joldz 🔮
Joldz 🔮•2mo ago

Passing Pandas Dataframe Object between componenets.

I feel like I'm missing something but when I try to pass a Pandas Dataframe object from one script to another it becomes a different datatype. Do I need to setup S3 Storage in order to pass objects between scripts?
4 Replies
rubenf
rubenf•2mo ago
You can only pass json serializable data between steps Also for large data we really recommend using S3 indeed
Alper
Alper•2mo ago
Joldz 🔮
Joldz 🔮OP•2mo ago
thanks!
daskindimteich
daskindimteich•2mo ago
if u use dataframe i recommend parquet files: https://arrow.apache.org/docs/python/parquet.html#reading-from-cloud-storage Wmill has docs samples for it too.