databasepostgresqlcurlpsqlpostgresql-copy

ERROR: missing data for column when using COPY FROM PROGRAM in psql


i'm trying to import the data from a cURL with the next command in psql:

COPY testtable FROM PROGRAM 'curl https://.....'

This is the data in the URL:

[{"date":"20201006T120000Z","uri":"secret","val":"1765.756"},{"date":"20201006T120500Z","uri":"secret","val":"2015.09258"},{"date":"20201006T121000Z","uri":"secret","val":"2283.0885"}]

But psql returns

ERROR: missing data for column "uri"

I've tried copying it on tables with the columns as text and json format. Also tried adding (DELIMITER ',') but that returns

ERROR: extra data after last expected column

i feel like the problem could be caused from the "[]" in the start and the end of the data, but i im not sure.

These are the definitions of the tables that i used.

Table "public.test_table"

Column Type Modifiers
date text not null
uri text
val text

Indexes: "test_table_pkey" PRIMARY KEY, btree (date)

Table "public.test_table2"

Column Type Modifiers
date json
uri json
val json

Solution

  • COPY only supports csv, text, and binary formats. It does not support JSON. It will import or export data into or out of json fields as a whole, but will not assemble or parse them.

    You could use a staging table with one row and one column.

    create temp table stage(x jsonb);
    
    COPY stage FROM PROGRAM 'curl https://.....';
    
    insert into test_table select f.* from stage,
       jsonb_populate_recordset(null::test_table, x) f;
    

    If PostgreSQL offered a pg_read_program() function, you could use that directly rather than creating a stage table. But it doesn't (but you could make one in C, or plpythonu or plperlu)