[SOLVED] Copy output of WITH subquery to CSV in postgres

Copy output of WITH subquery to CSV in postgres

I am trying to save the output of below 'WITH' sub-query to a csv file.

 WITH mer9 AS (
                SELECT *, 
                        substring(seq_window_mut_9mers, split9.start, 9)
                FROM split9
        ),

    mer23 AS (
                  SELECT *, 
                        substring(seq_window_mut_23mers, split23.start, 23)
                   FROM split23
        ),

    dataset AS (
                SELECT *
                    FROM table 
                    INNER JOIN mer9 ON mer9.seq_window_mut_9mers = table.seq_window_mut_9mers
                    INNER JOIN mer23 ON mer23.seq_window_mut_23mers = table.seq_window_mut_23mers

        )

COPY (SELECT * FROM dataset) TO '/tmp/filename.csv' (format CSV);

After running the query, I am getting an error:

[Code: 0, SQL State: 42601]  ERROR: syntax error at or near "COPY"
  Position: 3566  [Script position: 3566 - 3570]

Solution

Result sets generated from a CTE cannot be accessed in a different query. A CTE creates a sort of "temporary table" that only exists in the current query. That being said, put your CTE inside of the COPY statement and it should work, e.g.

COPY (
 WITH mer9 AS (
  SELECT *, substring(seq_window_mut_9mers, split9.start, 9)
  FROM split9),
 mer23 AS (
  SELECT *, substring(seq_window_mut_23mers, split23.start, 23)
  FROM split23),
 dataset AS (
  SELECT * FROM table 
  INNER JOIN mer9 ON mer9.seq_window_mut_9mers = table.seq_window_mut_9mers
  INNER JOIN mer23 ON mer23.seq_window_mut_23mers = table.seq_window_mut_23mers
 )
) TO '/tmp/filename.csv' (format CSV);

Edit. As pointed out by @a_horse_with_no_name:

Keep in mind that this command will create a file in the server. If you wish to create a file with the output in the client, consider using STDOUT in your COPY command, e.g. using psql:

$ psql -d yourdb -h yourdbhost -U your_user -c "COPY (WITH..) TO STDOUT" > file.csv