pythonnumpymachine-learninggraphlabsframe

Graphlab Sframe, retrieve multiple rows

I am trying to access multiple rows from a graphlab SFrame and convert them into a numpy array.

I have a database fd of 96000 rows and 4096 columns and need to retrieve the row numbers that are stored in a numpy array. The method that I have come up with is very slow. I suspect that it is because I keep increasing the size of the sframe at every iteration, but I don't know if any method of pre-allocating the values. I need to grab 20000 rows and the current method does not finish. 

fd=fd.add_row_number()
print(indexes)
xs=fd[fd['id'] == indexes[0]] #create the first entry

t=time.time()
for i in indexes[1:]: #Parse through and get indeces
    t=time.time()
    xtemp=fd[fd['id'] == i]
    xs=xs.append(xtemp) #append the new row to the existing sframe
    print(time.time()-t)

xs.remove_column('id') #remove the ID Column
print(time.time()-t)
x_sub=xs.to_numpy() #Convert the sframe to numpy
Solution

  • You can convert your SFrame to pandas.DataFrame, find rows with ids from indexes, remove DataFrame's column 'id' and convert this DataFrame to numpy.ndarray.

    For example:

    import numpy as np

fd=fd.add_row_number()
df = fd.to_dataframe()
df_indexes = df[df.id.isin(indexes)]
df_indexes = df_indexes.drop(labels='id', axis=1)
x_sub = np.array(df_indexes)