Run NestJS worker in a separate process

I am implementing NestJS worker, queues, using Bull.

According to the documentation, both the worker and the server (will) run in a same "process", but I want to run the worker in a separate process, so as to not block the main event loop.

I think it's called "running a task in a separate binary" or something else.

Anyway, I tried googling it, went through the documentation of NestJS, but couldn't find something similar.

++ In other words:

I have a main project (my current), and I want to create the worker in a separate process (standalone application) and want to connect both my current main project and worker. And I can't really find it in the documentation.

In which module should I instantiate my Bull's instance? I am assuming I'll keep my producer in my main module and consumer in my worker module.

How can I do so?

Please note, by "separate process", I do not mean running a specific task in a separate process, as defined in Bull's documentation. I want to deploy the whole worker module in a separate process or whatever the term should be used.

++ [Extra, if possible]

Before running my server and worker, I also want to check whether my worker (bull instance) is successfully connected to my Redis server. I couldn't find anything on the Bull's documentation... do you think there is a good workaround for that?

Goal: Horizontal Scaling ✅

While the answer provided by Isolated should work, I didn't want to run a whole new project and import my modules as they suggested. So after some more R&D, I've figured out a different way to do so.

Just as we have our index.ts or main.ts file for our "parent" project, in the same dir (doesn't have to be), create a worker.ts and worker.module.ts.

In the worker.module.ts, make sure you register your Bull module again [BullModule.forRoot({})] and include all the imports that you'd need for your consumer.

In the providers, you should add our consumers and you're good to go.

The worker.ts would look like this (nothing fancy):

import { NestFactory } from '@nestjs/core';
import { NestExpressApplication } from '@nestjs/platform-express';
import { WINSTON_MODULE_NEST_PROVIDER } from 'nest-winston';
import { WorkerModule } from './worker/worker.module';

async function bootstrap() {
  const app = await NestFactory.create<NestExpressApplication>(WorkerModule);
  app.useLogger(app.get(WINSTON_MODULE_NEST_PROVIDER));
  process.env.WORKER_HTTP_PORT = process.env.WORKER_HTTP_PORT ?? '4001';
  await app.listen(process.env.WORKER_HTTP_PORT);
  console.debug(`Worker is running on ${await app.getUrl()}`);
}
bootstrap();

Your nest-cli.json should like something like this

{
  "collection": "@nestjs/schematics",
  "sourceRoot": "src",
  "entryFile": "main",
  "compilerOptions": {
    "assets": ["**/*.graphql"],
    "watchAssets": true
  }
}

and create a new nest-cli-worker.json

{
    "collection": "@nestjs/schematics",
    "sourceRoot": "src",
    "entryFile": "worker",
    "compilerOptions": {
        "watchAssets": true
    }
}

Now, the question is, how to run it?

I am using yarn commands to start my server (defining them in package.json)

To start my server, I'd

"start:dev": "yarn nest start --watch -e 'node -r dotenv/config -r source-map-support/register'"

or

"start:prod": "node -r dotenv/config -r ./tsconfig-paths-bootstrap.js dist/src/main.js"

and to start my worker, I'd run the following command(s) in another (terminal) shell...

Dev

"worker:start:dev": "yarn nest start --config nest-cli-worker.json --watch -e 'node -r dotenv/config -r source-map-support/register'"

or

Prod

"worker:start:prod": "node -r dotenv/config -r ./tsconfig-paths-bootstrap.js dist/src/worker.js"

P.S You don't necessarily have to add dotenv/config.

Bonus:

If you want to run your server(s) in docker

Here's my docker-compose.yaml file

version: '3.8'

services:
  main:
    container_name: my-server
    image: xxx.amazonaws.com/xx/xxx:${CONTAINER_IMAGE_TAG:-latest}
    ports:
      - 80:80
    command: node -r dotenv/config -r ./tsconfig-paths-bootstrap.js dist/src/main.js #My `prod` command for main server
    volumes:
      - xxx
    links:
      - xxx
    environment:
      xxx
    # .env is generated by Elastic Beanstalk, don't provide one
    env_file:
      - .env
  worker:
    container_name: worker-server #YOUR WORER
    image: xxx.us-west-2.amazonaws.com/xxx:${CONTAINER_IMAGE_TAG:-latest}
    ports:
      - 90:90
    links:
      - xxx
    command: node -r dotenv/config -r ./tsconfig-paths-bootstrap.js dist/src/worker.js  #prod command for Worker
    volumes:
      - xxx
    environment:
      xxx
    # .env is generated by Elastic Beanstalk, don't provide one
    env_file:
      - .env