Error in `python': free(): invalid pointer: 0x00007fc3c90dc98e
Got this error on an ubuntu server (100GB RAM) in anacondas py3.6 installation after it opened multiple files this way in a for loop (7 files * 4GB):
temp_df = pd.read_csv(datafolder + str(file), encoding="ISO-8859-1", delimiter=';',low_memory=False)
this is the error, it appeared after i set
low_memory=False
Doesnt happen when low_memory = True
* Error in `python': free(): invalid pointer: 0x00007fc3c90dc98e *
Anyone ideas? Thanks
Solution
This issue is caused due to multiple threads performed parallelly when opening a file , this causes sometimes segmentation fault as well. As you said your file consist of 4 gb each so multiple threads which are executed parallelly to read up your file is causing the issue.
- Python: __add__ and +, different behavior with float and integer
- how to use sklearn when target variable is a proportion
- Need help to extract this XML node - Excel connection strings in Python
- How to use face_landmarks in face_recognition library
- AWS Lambda No module named 'regex._regex'
- from . import XXXX
- Comparing schema of dataframe using Pyspark
- Cannot add data to a ComplexField using Python SDK for Azure AI Search
- SQLAlchemy cannot find referenced table via foreignkey in many-to-many relationship
- Better way to insert a very large data into PostgreSQL tables
- How to retrieve customer id from create customer method in Square using Python
- Using module 'subprocess' with timeout
- Reading SQL-table by pandas raises NotImplementedError
- Numpy: apply mask to values, then take mean, but in parallel
- polars: n_unique(), but as a window function
- matplotlib two different colors in the same annotate
- What does args[0]['valid'] mean here in decorator
- Conditional assignment in polars dataframe
- Is it possible to drop a unique constraint from a column?
- Circle detection in microscopy images (noisy)
- Polars groupby concat on multiple cols returning a list of unique values
- Pytends API throwing 429 error even if the request was made very first time
- PyPlot plots are bigger with high DPI, but still blurry
- Which version of Python do I have installed?
- Unable to make a remote api call to flask app(for MySQL connection) inside my apache server
- How to calculate probability in a normal distribution given mean & standard deviation?
- Unable to resolve " not a valid key=value pair (missing equal-sign) in Authorization header" when POSTing to api gateway
- How to get test cases list in Robot Framework without launching the actual tests?
- Is it possible to stop VS Code from opening new instances of Python for every Jupyter notebook?
- Pytorch Dataloader adding a batch dimension