pythonpython-3.xdataframetextread-text

How to read a large number of text files in Python software and convert data into a sample file?


I receive information from an application. I have given a very small example as follows.

import pandas as  pd
df = [{'etelat':{'name' : 'sajjad1' , 'last_name' : 'esma1' },

       'class_id':{'id':101 , 'name_os' : 'win'}}, 

       {'etelat':{'name' : 'sajjad2' , 'last_name' : 'esma2' },

       'class_id':{'id':102 , 'name_os' : 'mac'}}]
df
type(df)
# Import pandas library
import pandas as pd
# initialize list of lists
data = [['sajjad1', 'esma1',101,'win'], ['sajjad2', 'esma2',102,'mac'] ] 
# Create the pandas DataFrame
df_new = pd.DataFrame(data, columns = ['etelat_name', 'etelat_last_name','class_id_id', 'class_id_name_os'])
# print dataframe.
df

1- In this case, it answers correctly, but if we want the data, which is in the form of text, I will have a problem.

2- If the data is in a folder and the number of text files is 50, if we want to call the desire of the files with a code. How is this code? enter image description here


Solution

  • If you have your list of lists in a file like the following:

    example.txt

    [['sajjad1', 'esma1',101,'win'], ['sajjad2', 'esma2',102,'mac'],
    ['sajjad3', 'esma1',103,'win'], ['sajjad4', 'esma2',104,'mac'] ] 
    
    

    you can read the file and convert it to a pandas dataframe:

    import pandas as pd
    
    columns = ['etelat_name', 'etelat_last_name','class_id_id', 'class_id_name_os']
    with open('example.txt', 'rb') as file:
        data = eval(file.read())
      
    df = pd.DataFrame(data=data, columns=columns)
    print(df)
    

    output:

      etelat_name etelat_last_name  class_id_id class_id_name_os
    0     sajjad1            esma1          101              win
    1     sajjad2            esma2          102              mac
    2     sajjad3            esma1          103              win
    3     sajjad4            esma2          104              mac
    

    Edit:

    you can load it and convert it to utf-8 and then you get a list of dictionaries that you can deal with.

    import ast
    lst = []
    with open('example.txt', 'rb') as file:
        content = file.readlines()
    for item in content:
        temp = item.decode('UTF-8')
        lst.append(ast.literal_eval(temp))
    print(last)
    

    output:

    [({'etelat': {'name': 'sajjad1', 'last_name': 'esma1'}, 'class_id': {'id': 101, 'name_os': 'win'}},), ({'etelat': {'name': 'sajjad2', 'last_name': 'esma2'}, 'class_id': {'id': 102, 'name_os': 'mac'}},)]
    

    tested file with:

    {'etelat':{'name' : 'sajjad1' , 'last_name' : 'esma1' },'class_id':{'id':101 , 'name_os' : 'win'}}, 
    {'etelat':{'name' : 'sajjad2' , 'last_name' : 'esma2' },'class_id':{'id':102 , 'name_os' : 'mac'}},