brianthegreat•3mo ago

DataFrame in Script and Flow Showing Incorrect Column Order

I created a dataframe firstly with these 3 columns in this order: ["ID", "Amount", "Status"] Then I reorder the column to ["Status", "ID", "Amount"] However when shown in Windmill, it doesn't get reordered. This is just a simplification of my code, in reality I'm working with more complex column management, but this issue makes me cannot proceed. Here is the full code

import pandas as pd
from typing import Any
import csv
import io

def main() -> Any:
    # Create a dummy CSV data string with 3 columns
    csv_data = """ID;Amount;Status
    abc123;1000;completed
    def456;2000;pending
    ghi789;3000;failed"""

    # Use StringIO to treat the string as a file object
    csv_file_like_object = io.StringIO(csv_data)

    # Read the CSV data into a DataFrame directly
    df = pd.read_csv(csv_file_like_object, delimiter=';')
    df2 = df[['Status', 'ID', 'Amount']]

    # Debugging: Print DataFrame information
    print("DataFrame columns:", df2.columns)
    print("DataFrame head:\n", df2.head())
    
    # Return the DataFrame
    return df2.to_dict('records')

import pandas as pd
from typing import Any
import csv
import io

def main() -> Any:
    # Create a dummy CSV data string with 3 columns
    csv_data = """ID;Amount;Status
    abc123;1000;completed
    def456;2000;pending
    ghi789;3000;failed"""

    # Use StringIO to treat the string as a file object
    csv_file_like_object = io.StringIO(csv_data)

    # Read the CSV data into a DataFrame directly
    df = pd.read_csv(csv_file_like_object, delimiter=';')
    df2 = df[['Status', 'ID', 'Amount']]

    # Debugging: Print DataFrame information
    print("DataFrame columns:", df2.columns)
    print("DataFrame head:\n", df2.head())
    
    # Return the DataFrame
    return df2.to_dict('records')

7 Replies

rubenf•3mo ago

We store the result in postgresql JSONB which doesn't store the keys orders fortunately there is a solution which is to add the columns in the right order at top we could do that automatically but most people would expect the result to not have the headers as first row

brianthegreatOP•3mo ago

can you please explain what you mean by

add the columns in the right order at top

rubenf•3mo ago

see second part of https://www.windmill.dev/docs/core_concepts/rich_display_rendering#table-row-object

Rich Display Rendering | Windmill

Windmill processes some outputs (from scripts or flows) intelligently to provide rich display rendering, allowing you to customize the display format of your results.

brianthegreatOP•3mo ago

@rubenf sorry I don't really understand the docs. I tried this but still doesn't work

[...]
    # Return the DataFrame
    return df2[['Status', 'ID', 'Amount']].to_dict('records')

[...]
    # Return the DataFrame
    return df2[['Status', 'ID', 'Amount']].to_dict('records')

Can you please help edit my code to follow your solution?

rubenf•3mo ago

make sure the columns are at the top with the right order println to make sure that's the case if not, edit your code so that it is

brianthegreatOP•3mo ago

I'm so sorry I'm so dumb... And I'm still a python beginner. Can you please help to edit the code for me? I don't understand what you mean...

brianthegreatOP•3mo ago

This code works. Hopefully it helps the others who has the same problem. Thanks

import pandas as pd
from typing import Any
import io

def main() -> Any:
    # Create a dummy CSV data string with 3 columns
    csv_data = """ID;Amount;Status
    abc123;1000;completed
    def456;2000;pending
    ghi789;3000;failed"""

    # Use StringIO to treat the string as a file object
    csv_file_like_object = io.StringIO(csv_data)

    # Read the CSV data into a DataFrame directly
    df = pd.read_csv(csv_file_like_object, delimiter=';')
    
    # Reorder columns
    df = df[['Status', 'ID', 'Amount']]
    
    # Convert the DataFrame to a list of lists with column names at the top
    result = [df.columns.tolist()] + df.values.tolist()
    
    # Debugging: Print to ensure correct format
    print(result)
    
    # Return the result
    return result

import pandas as pd
from typing import Any
import io

def main() -> Any:
    # Create a dummy CSV data string with 3 columns
    csv_data = """ID;Amount;Status
    abc123;1000;completed
    def456;2000;pending
    ghi789;3000;failed"""

    # Use StringIO to treat the string as a file object
    csv_file_like_object = io.StringIO(csv_data)

    # Read the CSV data into a DataFrame directly
    df = pd.read_csv(csv_file_like_object, delimiter=';')
    
    # Reorder columns
    df = df[['Status', 'ID', 'Amount']]
    
    # Convert the DataFrame to a list of lists with column names at the top
    result = [df.columns.tolist()] + df.values.tolist()
    
    # Debugging: Print to ensure correct format
    print(result)
    
    # Return the result
    return result