nestjstypeormbundling-and-minification

Bundle a NestJS + TypeORM application (with webpack)

I recently have to think about a deployment method for a new piece of software that is written with:

The software will be deployed on more than 160 servers, distributed all across Europe, and some of them have very bad Internet connections.

I did some research and a lot of people explicitly advices against bundling. The main argument is that native extension will fails with bundlers like webpack or rollup (Spoiler: it's true, but there is a solution). In my opinion, it's largely due to the fact that people don't care for this: the author of node-pre-gyp used nearly the same words for this use case. So usually, I was told to either use yarn install or sync the node_modules/ folder.

The project is new, but the node_modules/ folder is already more than 480 MB. Using XZ with maximum compression gave me an archive of 20 MB. This is still way too large for me, and seems like a huge waste of resources.

I also had a look at the following Q&A:

There are also some separate Q&A for TypeORM, but all of them seems to require the installation of ts-node or typescript:

Solution

  • UPDATE 2022-09-05

    Now that ormconfig.json have been seriously deprecated from TypeORM in version 0.3.0, the data source definition is a separate JS (or even TS) file that must be bundled by Webpack.

    This was tricky, since this will be a dynamic require(), so Webpack can't know how to handle it in TypeORM code. The hint is the library key in entry point definition, without a name. You can go to the code above, but here is the excerpt:

      entry: {
    main: './src/main.ts',
    console: "./src/console.ts",
    'data-source': {
      import: './src/data-source.ts',
      library: {
        type: 'commonjs2'
      }
    },
    typeorm:  './node_modules/typeorm/cli.js',
  },

    UPDATE 2021-08-25

    The original response was done with NestJS 6, that used Webpack 4. Since NestJS 8 uses Webpack 5, chunk split is available, and provides a better solution.

    I also integrated the use of webpack-merge to have only one configuration file. This only changes the Webpack configuration that you'll see while reading.

    Original answer

    I managed to have a good solution, that generate a self-contained RPM of 2.7 MB with the following tools:

    The software is an API server, using PostgreSQL for persistence. Users are usually authenticated using an external servers, but we can have local (emergency) users, so we use bcrypt to store and check passwords.

    I have to insist: my solution does not work with native extensions. Fortunately, the popular bcrypt can be replaced with a pure JS implementation, and the most popular postgresql package is able of using both compiled or pure JS.

    If want to bundle with native extension, you can try to use ncc. They managed to implement a solution for node-pre-gyp dependent packages that worked for me in some preliminary tests. Of course, the compiled extensions should match your target platform, as always for compiled stuff.

    I personally chose webpack because NestJS support this in it's build command. This is merely a passthrough to the webpack compiler, but it seems to adjust some paths, so it was kind of easier.

    So, how to achieve this? webpack can bundle everything in a single file, but in this use case, I need three of them:

    And since each bundling required different options… I used 3 webpack files. Here is the layout:

    webpack.config.js
webpack
├── migrations.config.js
└── typeorm-cli.config.js

    All those files were based of the same template kindly provided by ZenSoftware. The main difference is that I switched from IgnorePlugin to externals because that is simpler to read, and fits the use case perfectly.

    // webpack.config.js
const { NODE_ENV = 'production' } = process.env;

console.log(`-- Webpack <${NODE_ENV}> build --`);

module.exports = {
  target: 'node',
  mode: NODE_ENV,
  externals: [
    // Here are listed all optional dependencies of NestJS,
    // that are not installed and not required by my project
    {
      'fastify-swagger': 'commonjs2 fastify-swagger',
      'aws-sdk': 'commonjs2 aws-sdk',
      '@nestjs/websockets/socket-module': 'commonjs2 @nestjs/websockets/socket-module',
      '@nestjs/microservices/microservices-module': 'commonjs2 @nestjs/microservices/microservices-module',
      
      // I'll skip pg-native in the production deployement, and use the pure JS implementation
      'pg-native': 'commonjs2 pg-native'
    }
  ],
  optimization: {
    // Minimization doesn't work with @Module annotation
    minimize: false,
  }
};

    Configuration files for TypeORM are more verbose, because we need to explicit the use of TypeScript. Fortunately, they have some advices for this in their FAQ. However, bundling the migration tool required two more hacks:

    // webpack/typeorm-cli.config.js

const path = require('path');
// TypeScript compilation option
const TsconfigPathsPlugin = require('tsconfig-paths-webpack-plugin');
// Don't try to replace require calls to dynamic files
const IgnoreDynamicRequire = require('webpack-ignore-dynamic-require');

const { NODE_ENV = 'production' } = process.env;

console.log(`-- Webpack <${NODE_ENV}> build for TypeORM CLI --`);

module.exports = {
  target: 'node',
  mode: NODE_ENV,
  entry: './node_modules/typeorm/cli.js',
  output: {
    // Remember that this file is in a subdirectory, so the output should be in the dist/
    // directory of the project root
    path: path.resolve(__dirname, '../dist'),
    filename: 'migration.js',
  },
  resolve: {
    extensions: ['.ts', '.js'],
    // Use the same configuration as NestJS
    plugins: [new TsconfigPathsPlugin({ configFile: './tsconfig.build.json' })],
  },
  module: {
    rules: [
      { test: /\.ts$/, loader: 'ts-loader' },
      // Skip the shebang of typeorm/cli.js
      { test: /\.[tj]s$/i, loader: 'shebang-loader' }
    ],
  },
  externals: [
    {
      // I'll skip pg-native in the production deployement, and use the pure JS implementation
      'pg-native': 'commonjs2 pg-native'
    }
  ],
  plugins: [
    // Let NodeJS handle are requires that can't be resolved at build time
    new IgnoreDynamicRequire()
  ]
};

    // webpack/migrations.config.js

const glob = require('glob');
const path = require('path');
// TypeScript compilation option
const TsconfigPathsPlugin = require('tsconfig-paths-webpack-plugin');
// Minimization option
const TerserPlugin = require('terser-webpack-plugin');

const { NODE_ENV = 'production' } = process.env;

console.log(`-- Webpack <${NODE_ENV}> build for migrations scripts --`);

module.exports = {
  target: 'node',
  mode: NODE_ENV,
  // Dynamically generate a `{ [name]: sourceFileName }` map for the `entry` option
  // change `src/db/migrations` to the relative path to your migration folder
  entry: glob.sync(path.resolve('src/migration/*.ts')).reduce((entries, filename) => {
    const migrationName = path.basename(filename, '.ts');
    return Object.assign({}, entries, {
      [migrationName]: filename,
    });
  }, {}),
  resolve: {
    // assuming all your migration files are written in TypeScript
    extensions: ['.ts'],
    // Use the same configuration as NestJS
    plugins: [new TsconfigPathsPlugin({ configFile: './tsconfig.build.json' })],
  },
  module: {
    rules: [
      { test: /\.ts$/, loader: 'ts-loader' }
    ]
  },
  output: {
    // Remember that this file is in a subdirectory, so the output should be in the dist/
    // directory of the project root
    path: __dirname + '/../dist/migration',
    // this is important - we want UMD (Universal Module Definition) for migration files.
    libraryTarget: 'umd',
    filename: '[name].js',
  },
  optimization: {
    minimizer: [
      // Migrations rely on class and function names, so keep them.
      new TerserPlugin({
        terserOptions: {
          mangle: true, // Note `mangle.properties` is `false` by default.
          keep_classnames: true,
          keep_fnames: true,
        }
      })
    ],
  },
};

    Update 2021-08-25 Nest 8/Webpack 5

    Since nest-cli moved to webpack 5, an interesting feature is now available: chunk split for node target.

    I was also upset to manage multiple files with the same logic, so I decided to use webpack-merge to have only one configuration file.

    You'll have to yarn add -D webpack-merge and have the following webpack.config.js

    // webpack.config.js
const { merge } = require("webpack-merge")
const path = require('path')
const glob = require('glob')
const TsconfigPathsPlugin = require('tsconfig-paths-webpack-plugin')
const TerserPlugin = require('terser-webpack-plugin')
const MomentLocalesPlugin = require('moment-locales-webpack-plugin')
const IgnoreDynamicRequire = require('webpack-ignore-dynamic-require')

const { NODE_ENV = 'production', ENTRY, npm_lifecycle_event: lifecycle } = process.env

// Build platform don't support ?? and ?. operators
const entry = ENTRY || (lifecycle && lifecycle.match(/bundle:(?<entry>\w+)/).groups["entry"])

if (entry === undefined) {
  throw new Error("ENTRY must be defined")
}

console.log(`-- Webpack <${NODE_ENV}> build  for <${entry}> --`);

const BASE_CONFIG = {
  target: 'node',
  mode: NODE_ENV,
  resolve: {
    extensions: ['.ts', '.js'],
    plugins: [new TsconfigPathsPlugin({ configFile: './tsconfig.build.json' })],
  },
  module: {
    rules: [
      { test: /\.ts$/, loader: 'ts-loader' }
    ]
  },
  output: {
    path: path.resolve(__dirname, 'dist/'),
    filename: '[name].js',
  },
}

const MIGRATION_CONFIG = {
  // Dynamically generate a `{ [name]: sourceFileName }` map for the `entry` option
  // change `src/db/migrations` to the relative path to your migration folder
  entry: glob.sync(path.resolve('src/migration/*.ts')).reduce((entries, filename) => {
    const migrationName = path.basename(filename, '.ts')
    return Object.assign({}, entries, {
      [migrationName]: filename,
    })
  }, {}),
  output: {
    path: path.resolve(__dirname, 'dist/migration'),
    // this is important - we want UMD (Universal Module Definition) for migration files.
    libraryTarget: 'umd',
    filename: '[name].js',
  },
  optimization: {
    minimizer: [
      new TerserPlugin({
        terserOptions: {
          mangle: true, // Note `mangle.properties` is `false` by default.
          keep_classnames: true,
          keep_fnames: true,
        }
      })
    ],
  }
}

const TYPEORM_CONFIG = {
  entry: {
    typeorm: './node_modules/typeorm/cli.js'
  },
  externals: [
    {
      'pg-native': 'commonjs2 pg-native',
    }
  ],
  plugins: [
    new IgnoreDynamicRequire(),
  ],
  module: {
    rules: [
      { test: /\.[tj]s$/i, loader: 'shebang-loader' }
    ],
  },
}

const MAIN_AND_CONSOLE_CONFIG = {
  entry: {
    main: './src/main.ts',
    console: "./src/console.ts"
  },
  externals: [
    {
      'pg-native': 'commonjs2 pg-native',
      'fastify-swagger': 'commonjs2 fastify-swagger',
      '@nestjs/microservices/microservices-module': 'commonjs2 @nestjs/microservices/microservices-module',
      '@nestjs/websockets/socket-module': 'commonjs2 @nestjs/websockets/socket-module',
      // This one is a must have to generate the swagger document, but we remove it in production
      'swagger-ui-express': 'commonjs2 swagger-ui-express',
      'aws-sdk': 'commonjs2 aws-sdk',
    }
  ],
  plugins: [
    // We don't need moment locale
    new MomentLocalesPlugin()
  ],
  optimization: {
    // Full minization doesn't work with @Module annotation
    minimizer: [
      new TerserPlugin({
        terserOptions: {
          mangle: true, // Note `mangle.properties` is `false` by default.
          keep_classnames: true,
          keep_fnames: true,
        }
      })
    ],
    splitChunks: {
      cacheGroups: {
        commons: {
          test: /[\\/]node_modules[\\/]/,
          name: 'vendors',
          chunks: 'all'
        }
      }
    }
  }
}

const withPlugins = (config) => (runtimeConfig) => ({
  ...config,
  plugins: [
    ...runtimeConfig.plugins,
    ...(config.plugins || [])
  ]
})

const config = entry === "migrations" ? merge(BASE_CONFIG, MIGRATION_CONFIG)
  : entry === "typeorm" ? merge(BASE_CONFIG, TYPEORM_CONFIG)
  : entry === "main" ? merge(BASE_CONFIG, MAIN_AND_CONSOLE_CONFIG)
    : undefined


module.exports = withPlugins(config)

    With this file, the webpack configuration is chosen from the current command: bundle:main will select the configuration for the main entry point.

    You'll notice also that there are now multiple entry points in the main: main and console. The former is for the main application, and the latter is for a CLI helper. But they all share the same (and huge) amount of code, and Webpack 5 is able to do that with splitChunks section. This was available in Webpack 4 but not working for node targets.

    Finally, some optimization are now fully working even with decorators (that uses reflection), when you keep class and function names.

    Bundle is smaller, code is shared, package.json is clearer, everyone is happy.

    End of update

    Update for TypeORM >= 0.3

    Since we now have to use a JS data source, it must be discoverable from TypeORM CLI.

    
const TYPEORM_CONFIG = {
  entry: {
    'data-source': {
      // Are you ready? You must provide a data source as TypeORM cli >= 0.3
      // But this will be a dynamic require(), so Webpack can't know how to handle it
      // in TypeORM code. Instead, export this data source as a library.
      // BUT, TypeORM expect it to be at the top level instead of a module variable, so
      // we MUST remove the library name to make webpack export each variable from data source into the module
      import: './src/data-source.ts',
      library: {
        type: 'commonjs2'
      }
    },
    typeorm: './node_modules/typeorm/cli.js'
  },
  externals: [
    {
      'pg-native': 'commonjs2 pg-native',
    }
  ],
  plugins: [
    new IgnoreDynamicRequire(),
  ],
  module: {
    rules: [
      { test: /\.[tj]s$/i, loader: 'shebang-loader' }
    ],
  },
}

    End of update for TypeORM >= 0.3

    After that, to simplify the build process I added some targets in package.json:

    {
  "scripts": {
    "bundle:application": "nest build --webpack",
    "bundle:migrations": "nest build --webpack --webpackPath webpack/typeorm-cli.config.js && nest build --webpack --webpackPath webpack/migrations.config.js",
    "bundle": "yarn bundle:application && yarn bundle:migrations"
  },
}

    And… you're nearly done. You can call yarn bundle, and the output will kindly be built in the dist/ directory. I didn't managed to remove some TypeScript definition files that were generated, but that wasn't a real issue.

    The final step was writing the RPM specification file:

    %build
mkdir yarncache
export YARN_CACHE_FOLDER=yarncache

# Setting to avoid node-gype trying to download headers
export npm_config_nodedir=/opt/rh/rh-nodejs10/root/usr/

%{_yarnbin} install --offline --non-interactive --frozen-lockfile
%{_yarnbin} bundle

rm -r yarncache/

%install
install -D -m644 dist/main.js $RPM_BUILD_ROOT%{app_path}/main.js

install -D -m644 dist/migration.js $RPM_BUILD_ROOT%{app_path}/migration.js
# Migration path have to be changed, let's hack it.
sed -ie 's/src\/migration\/\*\.ts/migration\/*.js/' ormconfig.json
install -D -m644 ormconfig.json $RPM_BUILD_ROOT%{app_path}/ormconfig.json
find dist/migration -name '*.js' -execdir install -D -m644 "{}" "$RPM_BUILD_ROOT%{app_path}/migration/{}" \;

    If you use TypeORM >= 0.3 with a JS data source, the sed line doesn't have to be done, since a DataSource can accept production and development sources. However, the ormconfig.json must be replaced by the data-source.js.

    And the systemd service file can give you how to launch this. The target platform is CentOS7, so I have to use NodeJS 10 from software collections. You can adapt the path to your NodeJS binary.

    [Unit]
Description=NestJS Server
After=network.target

[Service]
Type=simple
User=nestjs
Environment=SCLNAME=rh-nodejs10
ExecStartPre=/usr/bin/scl enable $SCLNAME -- /usr/bin/env node migration migration:run
ExecStart=/usr/bin/scl enable $SCLNAME -- /usr/bin/env node main
WorkingDirectory=/export/myapplication
Restart=on-failure

# Hardening
PrivateTmp=true
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=read-only

[Install]
WantedBy=multi-user.target

    If you use a TypeORM JS datasource, you must add -f data-source.js to ExecStartPre.

    Final statistics: