I am trying to extract data from a ".dat" file by using asammdf.
After extracting the data using asammdf, I am trying to convert the data into a dataframe that can be analyzed using pandas and matplotlib.
Following is the code that I am using for extracting the data and converting to dataframe:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import asammdf
import tkinter as tk
from tkinter.ttk import *
Data_File_01 = asammdf.MDF(r"C:\Users\hsr4ban\Desktop\16_MC Dynamic.dat")
Data_01 = Data_File_01.to_dataframe()
However, I am getting the memory error as below:
runfile('C:/Users/hsr4ban/Desktop/untitled0.py', wdir='C:/Users/hsr4ban/Desktop')
Traceback (most recent call last):
File ~\Desktop\untitled0.py:11 in <module>
Data_File_01 = asammdf.MDF(r"C:\Users\hsr4ban\Desktop\16_MC Dynamic.dat").to_dataframe()
File ~\AppData\Roaming\Python\Python39\site-packages\asammdf\mdf.py:4466 in to_dataframe
df = pd.DataFrame(nonstrings, index=master)
File ~\AppData\Roaming\Python\Python39\site-packages\pandas\core\frame.py:636 in __init__
mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
File ~\AppData\Roaming\Python\Python39\site-packages\pandas\core\internals\construction.py:502 in dict_to_mgr
return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)
File ~\AppData\Roaming\Python\Python39\site-packages\pandas\core\internals\construction.py:156 in arrays_to_mgr
return create_block_manager_from_column_arrays(
File ~\AppData\Roaming\Python\Python39\site-packages\pandas\core\internals\managers.py:1959 in create_block_manager_from_column_arrays
mgr._consolidate_inplace()
File ~\AppData\Roaming\Python\Python39\site-packages\pandas\core\internals\managers.py:1685 in _consolidate_inplace
self.blocks = tuple(_consolidate(self.blocks))
File ~\AppData\Roaming\Python\Python39\site-packages\pandas\core\internals\managers.py:2084 in _consolidate
merged_blocks = _merge_blocks(
File ~\AppData\Roaming\Python\Python39\site-packages\pandas\core\internals\managers.py:2111 in _merge_blocks
new_values = np.vstack([b.values for b in blocks]) # type: ignore[misc]
File <__array_function__ internals>:180 in vstack
File ~\AppData\Roaming\Python\Python39\site-packages\numpy\core\shape_base.py:282 in vstack
return _nx.concatenate(arrs, 0)
File <__array_function__ internals>:180 in concatenate
MemoryError: Unable to allocate 5.46 GiB for an array with shape (332, 2207220) and data type float64
I checked in stack overflow. There are few answers suggested in case of ".csv" file but in this case it is ".dat" file and I could not find much help.
Can someone please suggest how can it be resolved?
Thanks in advance.
You need to use the raster argument whenc alling to_dataframe because you have too many individual timestamps in the file (see https://asammdf.readthedocs.io/en/master/api.html#asammdf.mdf.MDF.iter_to_dataframe)