google-cloud-platformgoogle-cloud-data-fusioncdap

GCP Data Fusion multiple table import

I'm trying to use Multiple Database Tables and BigQuery Multi Table Data Fusion plugin to import multiple table in one pipeline

Pipeline

But when I try to execute I get the following error

java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: BigQuery Multi Table has no outputs. Please check that the sink calls addOutput at some point.

I'm using Data Fusion version 6.1.4 Multiple Database Tables version 1.2.0 and BigQuery Multi Table version 0.14.8.

Any suggestion on what may be the problem?

Edit:

following the configuration of multiple table database source

{
            "name": "Multiple Database Tables",
            "plugin": {
                "name": "MultiTableDatabase",
                "type": "batchsource",
                "label": "Multiple Database Tables",
                "artifact": {
                    "name": "multi-table-plugins",
                    "version": "1.2.0",
                    "scope": "USER"
                },
                "properties": {
                    "splitsPerTable": "1",
                    "referenceName": "multiTable",
                    "connectionString": "${secure(connection)}",
                    "jdbcPluginName": "netezza",
                    "user": "${secure(username)}",
                    "password": "${secure(password)}",
                    "whiteList": "categoria_l,cliente_l,regione_l"
                }
            },
            "outputSchema": [
                {
                    "name": "etlSchemaBody",
                    "schema": ""
                }
            ]
        },

After further test the problem is that the source response is empty because data fusion is not reading view from source database but only tables

Solution

  • It seems like the Multiple Database Tables source produced no records ("Out 0"). I'd check there first. You can do a quick check using the Preview mode. Plugin doc here.

    Related answer here.