python-2.7windows-7postgresql-9.3import-from-csv

Postgres Copy select rows from CSV table


This is my first post to stackoverflow. Your forum has been SO very helpful as I've been learning Python and Postgres on the fly for the last 6 months, that I haven't needed to post yet. But this task is tripping me up and I figure I need to start earning reputation points:

I am creating a python script for backing up data into an SQL database daily. I have a CSV file with an entire months worth of hourly data, but I only want to select a single day of data from from the file and copy those select rows into my database. Am I able to query the CSV table and append the query results into my database? For example:

        sys.stdin = open('file.csv', 'r')    
        cur.copy_expert("COPY table FROM STDIN 
                         SELECT 'yyyymmddpst LIKE 20140131' 
                         WITH DELIMITER ',' CSV HEADER", sys.stdin)

This code and other variations aren't working out - I keep getting syntax errors. Can anyone help me out with this task? Thanks!!


Solution

  • You need create temporary table at first:

    cur.execute('CREATE TEMPORARY TABLE "temp_table" (LIKE "your_table") WITH OIDS')
    

    Than copy data from csv:

    cur.execute("COPY temp_table FROM '/full/path/to/file.csv' WITH CSV HEADER DELIMITER ','")
    

    Insert necessary records:

    cur.execute("INSERT INTO your_table SELECT * FROM temp_table WHERE yyyymmddpst LIKE 20140131")
    

    And don't forget do conn.commit() Temp table will destroy after cur.close()