pysparkqubole

Pyspark error- Invalid argument, not a string or column

I have a dataframe in Pyspark - df_all. It has some data and need to do the following

count = ceil(df_all.count()/1000000)

It gives the following error

TypeError: Invalid argument, not a string or column: 0.914914 of type <class ‘float’>. For column literals, use ‘lit’, ‘array’, ‘struct’ or ‘create_map’ function.

How can I use ceil function in pyspark?

Solution

  • Looks like for your requirement, this would be suitable:

    import math

count = math.ceil(df_all.count()/1000000)