Complete newbie with JavaScript, Dataform and BiqQuery so my current level right now is copy/paste from internet. Now however I have come across a problem I can't find any suitable tips for.
I have added the package @google-cloud/bigquery to the package.json file, and installed it successfully by clicking "Install packages":
{
"dependencies": {
"@dataform/core": "3.0.0",
"@google-cloud/bigquery": "8.1.0"
}
}
To test I have created a simple function in the file includes/columnFunctions.js:
function firstTest() {
const {BigQuery} = require('@google-cloud/bigquery');
const bigquery = new BigQuery();
return "XX";
}
module.exports = { firstTest }
Then calling the function in an SQLX file:
config {type: "view"}
select ${columnFunctions.firstTest()} as X
from ${ref("dummysrc1")}
This returns an error: "Failed to resolve events"
The error is generated by the first statement const {BigQuery} = require('@google-cloud/bigquery'); so I'm suspecting some sort of access or authorization problem, but have no idea where to start looking...
You cannot use Node.js packages that perform I/O inside the JavaScript that Dataform uses to compile your SQLX files. Also the @google-cloud/bigquery library is designed for Node.js environments and often depends on Node.js modules like fs or child_process .
The ideal solution is to perform the BigQuery-related operations outside of Dataform and then feed the results into your Dataform project. Dataform is excellent for data transformations, but less suited for direct interaction with external services like BigQuery within transformation logic. You can use other tools like Cloud Functions, Cloud Run, or a separate Node.js application to interact with BigQuery, and then store the results in a BigQuery table that Dataform can then process.
Adding this Dataform common issues that might be helpful to you.