import pandas as pd
import numpy as np
d = {'l': ['left', 'right', 'left', 'right', 'left', 'right'],
'r': ['right', 'left', 'right', 'left', 'right', 'left'],
'v': [-1, 1, -1, 1, -1, np.nan]}
df = pd.DataFrame(d)
When a grouped dataframe contains a value of np.NaN
I want the grouped sum to be NaN
as is given by the skipna=False
flag for pd.Series.sum
and also pd.DataFrame.sum
however, this
In [235]: df.v.sum(skipna=False)
Out[235]: nan
However, this behavior is not reflected in the pandas.DataFrame.groupby
object
In [237]: df.groupby('l')['v'].sum()['right']
Out[237]: 2.0
and cannot be forced by applying the np.sum
method directly
In [238]: df.groupby('l')['v'].apply(np.sum)['right']
Out[238]: 2.0
I can workaround this by doing
check_cols = ['v']
df['flag'] = df[check_cols].isnull().any(axis=1)
df.groupby('l')['v', 'flag'].apply(np.sum).apply(
lambda x: x if not x.flag else np.nan,
axis=1
)
but this is ugly. Is there a better method?
I think it's inherent to pandas. A workaround can be :
df.groupby('l')['v'].apply(array).apply(sum)
to mimic the numpy way,
or
df.groupby('l')['v'].apply(pd.Series.sum,skipna=False) # for series, or
df.groupby('l')['v'].apply(pd.DataFrame.sum,skipna=False) # for dataframes.
to call the good function.