pythonpython-itertoolsexperimental-design

Fill a dataframe with Carthesian product of variably shaped input lists


I want to create a script that fills a dataframe with values that are the Carthesian product of parameters I want to vary in a series of experiments. My first thought was to use the product function of itertools, however it seems to require a fixed set of input lists. The output I'm looking for can be generated using this sample:

cols = ['temperature','pressure','power']

l1 = [1, 100, 50.0 ]
l2 = [1000, 10, np.nan] 
l3 = [0, 100, np.nan]


data = []
for val in itertools.product(l1,l2,l3): #use itertools to get the Carthesian product of the lists
    data.append(val) #make a list of lists to store each variation

df = pd.DataFrame(data, columns=cols).dropna(0) #make a dataframe from the list of lists (dropping NaN values)

However, I would like instead to extract the parameters from dataframes of arbitrary shape and then fill up a dataframe with the product, like so (code doesn't work):

data = [{'parameter':'temperature','value1':1,'value2':100,'value3':50},
        {'parameter':'pressure','value1':1000,'value2':10},
        {'parameter':'power','value1':0,'value2':100},
        ]

df = pd.DataFrame(data)
l = []
cols = []
for i in range(df.shape[0]):
    l.append(df.iloc[i][1:].to_list()) #store the values of each df row to a separate list
    cols.append(df.iloc[i][0]) #store the first value of the row as column header

data = []
for val in itertools.product(l): #ask itertools to parse a list of lists
    data.append(val)

df2 = pd.DataFrame(data, columns=cols).dropna(0)

Can you recommend a way about this? My goal is creating the final dataframe, so it's not a requirement to use itertools.


Solution

  • Another alternative without product (nothing wrong with product, though) could be to use .join() with how="cross" to produce successive cross-products:

    df2 = df.T.rename(columns=df.iloc[:, 0]).drop(df.columns[0])
    df2 = (
        df2.iloc[:, [0]]
        .join(df2.iloc[:, [1]], how="cross")
        .join(df2.iloc[:, [2]], how="cross")
        .dropna(axis=0)
    )
    

    Result:

       temperature pressure power
    0            1     1000     0
    1            1     1000   100
    3            1       10     0
    4            1       10   100
    9          100     1000     0
    10         100     1000   100
    12         100       10     0
    13         100       10   100
    18        50.0     1000     0
    19        50.0     1000   100
    21        50.0       10     0
    22        50.0       10   100
    

    A compacter version with product:

    from itertools import product
    
    df2 = pd.DataFrame(
        product(*df.set_index("parameter", drop=True).itertuples(index=False)),
        columns=df["parameter"]
    ).dropna(axis=0)