node.jspostgresqlfirebasegoogle-cloud-functionsgoogle-cloud-sql

Why I get connection timeout to Cloud SQL in GCP Cloud Functions Gen 2 randomly on some requests?


Here’s a simplified version of the code:

import { onRequest } from 'firebase-functions/v2/https';
import { getBdConfig } from '../../submodules/for-backend/postgres';
import { Pool } from 'pg';

let pool: Pool | undefined;

export const exampleCloudFunction = onRequest(async (req, res) => {
    if (!pool) {
        pool = new Pool(getBdConfig(postgresSecrets));
    }
    const client = await pool.connect();

    try {
        // Business logic and queries
    } finally {
        client.release();
    }
});

Problem

After some successful requests, I encounter connection timeouts when calling pool.connect().
Even after disabling private IP restrictions and retrying the connection and queries 3 times, the issue persists.

From logs at the moment of failure:

Research and Related Issues

I reviewed the following discussions but didn’t find a solution that matches my case exactly:

Some discussions mentioned TCP connections being closed (e.g., in AWS Lambda), but I don't think that applies here because of the small time between function start and error (<2min) and the different error type (connection timeout).

What could be causing pool.connect() to timeout randomly, even with low connection usage and with retries?
Are there Cloud Run or Cloud SQL configurations I might be missing?


Solution

  • I figured out what was happening on my side. It was a problem with the VPC connector from GCP and not related to node-postgres. As soon as I removed the VPC connector, the connection timeout was reduced from 1 in every 6 connection attempt to 1 in every 600 connection attempt. Then a simple retry worked fine.

    I hope this helps you and future developers.