Pandas: How to split a column of a dataframe, which follows powerlaw into two based on frequency distribution?
I have a data frame of 1 million records with 5 columns.
unique_index,name,company_name,city_id,state_id
Column, company_name, has 100k unique records. This follows a power law. Top 5000 company_names cover 70% of the records.
I want to take equal number of samples from the companies which contribute to the top 5000 of the data and from the remaining set.
I tried pd.qcut(df['company_name'],[0.25,1]. This gave me the below error:
TypeError: unorderable types: str() <= float(). Can qcut not be applied to strings?
You could try grabbing the top companies by value_counts() and then creating a new column with True/False if it's in/out of the top companies. I think it would look something like this:
top5000 = df['company_name'].value_counts().index[0:5000].tolist()
df['InTop'] = df['company_name'].isin(top5000)
This would allow you to sample from the group where df['InTop'] == True and the group where df['InTop'] == False
- Numerically stable softmax
- ModuleNotFoundError, even after pip install
- how to extract the texts after the first h1 Tag?
- Requests and BeautifulSoup to get video length from YouTube
- Python Multiple Turtles Moving (seemingly) Simultaneously
- Stop pyzmq receiver by KeyboardInterrupt
- pandas dataframe update with filter_func
- Getting the name of a variable as a string
- using bottlenose in python to extract product price from Amazon in different locale
- matplotlib multiple axes mixups
- ZMQ socket gracefully termination in Python
- gdown command not working in jupyter notebook in anaconda.cloud
- Can dependency injection be used with django?
- Simple way to measure cell execution time in ipython notebook
- Asyncio future, running Future objects
- Prime factorization using list comprehension in Python
- Store output of subprocess.Popen call in a string
- sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column could not find table
- How to get the Exception (Message and StackStrace) of a Python Azure Function
- Python - regex to grab specific lines from text
- python: How to get real feature name from feature_importances
- Python: Implement a PCA using SVD
- Apply function only on slice of array under jit
- symbol not found in flat namespace '_bcp_batch'
- Print function not working without flush=True when used with the While loop
- SciPy minimise to find inverse function?
- Microservices architecture with Django
- how to escape texts for formatting in python
- How should I read a file line-by-line in Python?
- Sort columns of numpy unitary matrix such that highest element of each column is on the diagonal