pythonexport-to-csvcacti

loop over mulitple rrd files in directory


I need to iterate through all .rrd files inside a given directory and do fetch data inside each rrd database and do some actions and export them into single csv file in python script!

How can this be done in a efficient way? it's ok to advice me how to loop and access database data over multiple files?


Solution

  • I assume you have a rrdtool installation with python bindings already on your system. If not, here is an installation description.

    Then, to loop over .rrd files in a given directory and performing a fetch:

    import os
    import rrdtool
    
    target_directory = "some/directory/with/rrdfiles"
    rrd_files = [os.path.join(target_directory, f) for f in os.listdir(target_directory) if f.endswith('.rrd')]
    data_container = []
    for rrd_file in rrd_files:
        data = rrdtool.fetch(rrd_file, 'AVERAGE'  # or whichever CF you need
            '--resolution', '200',                # for example
        )                                         # and any other settings you need
        data_container.append(data)
    

    The parameter list follows rrdfetch.

    Once you have whatever data you need inside the rrd_files loop, you should accumulate it in a list of lists, with each sublist being one row of data. Writing them to a csv is as easy as this:

    import csv
    
    # read the .rrd data as above into a 'data_container' variable
    
    with open('my_data.csv', 'w', newline='') as csv_file:
        rrd_writer = csv.writer(csv_file)
        for row in data_container:
            rrd_writer.writerow(row)
    

    This should outline the general steps you have to follow, you will probably need to adapt them (the rrdfetch in particular) to your case.