Error: connect EHOSTUNREACH 52.94.177.94:443 On AWS KMS

New to AWS here so please bear that in mind.

I want to encrypt/decrypt data using AWS KMS.

Here is the code I have

const { KMS } = require('@aws-sdk/client-kms');
const { KmsKeyringNode, buildClient } = require('@aws-crypto/client-node');

const kmsAccessKeyId = process.env.KMS_ACCESS_KEY_ID;
const kmsSecretAccessKey = process.env.KMS_SECRET_ACCESS_KEY;
const kmsGeneratorKeyId = process.env.KMS_GENERATOR_KEY_ID;
const kmsKeyId = process.env.KMS_KEY_ID;

const REGIONS = {
  REGION: 'region-here', // Contains AWS region. Value here is just for illustration purposes.
};

const clientProvider = () => new KMS({ region: REGIONS.REGION, credentials: { accessKeyId: kmsAccessKeyId, secretAccessKey: kmsSecretAccessKey } });
const keyring = new KmsKeyringNode({ generatorKeyId: kmsGeneratorKeyId , clientProvider, keyIds: [kmsKeyId ] });
const { encrypt, decrypt } = buildClient();


/**
 * Convert buffer to string.
 */
const convertU8toString = (u8Arr) => {
  const textDecoder = new TextDecoder();

  const str = textDecoder.decode(u8Arr);

  return str;
};

/**
 * Receives a plaintext message typeof string
 * encrypt the string
 * receive back a buffer
 * convert the buffer to base64
 * return the string-version of the base64.
 */
const encryptData = async (value) => {
  try {
    if (!value) throw new Error('No value provided');

    if (typeof value !== 'string') throw new Error('Only value type of string is supported');

    const { result } = await encrypt(keyring, value);
    const storableMessage = result.toString('base64');

    return storableMessage;
  } catch (err) {
    throw err;
  }
};

/**
 * Receives a base64 encoded message typeof string
 * convert the base64 back to a buffer.
 * decrypt the buffer
 * receive back a buffer
 * return the plaintext message typeof string.
 */
const decryptData = async (value) => {
  try {
    if (!value) throw new Error('No value provided');

    if (typeof value !== 'string') throw new Error('Only value type of string is supported');

    const newBuff = Buffer.from(value, 'base64');
    const { plaintext } = await decrypt(keyring, newBuff);

    return convertU8toString(plaintext);
  } catch (err) {
    throw err; // Our own custom error. Just re-throw.
  }
};

So the code above seemed to work. I was able to encrypt and decrypt data no problem.

The problem arises when you use encrypt and decrypt consicutively in a set period of time. The error:

Error: connect EHOSTUNREACH 52.94.177.94:443
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16) {
  errno: -113,
  code: 'EHOSTUNREACH',
  syscall: 'connect',
  address: '52.94.177.94',
  port: 443,
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

randomly occurs. You have to wait for a couple of time to use encrypt and decrypt data again before it will work again properly.

This issue is very evident when running automated tests.

Even if I run only 1 test test.only sometimes the error randomly pops up. Sometimes it doesn't.

It is very confusing.

It looks like encrypt and decrypt connects to AWS everytime it is run because the ip seemed to belong from AWS (ipinfo:52.94.177.94) and for a random reason it fails and throws that error.

Any help to resolve the issue or point out where or what I am doing wrong is highly appreciated.

Thanks in advance.

Quick Background:

The code I have above, encrypts/decrypts locally (on my server). It does not encrypt/decrypt on AWS side (Which is the case if you use KMSClient as opposed to what I am using which is KMS).

Answer

The main issue is related to GenerateDataKey of AWS KMS. https://docs.aws.amazon.com/kms/latest/APIReference/API_GenerateDataKey.html

Basically, everytime you run encrypt and decrypt, the KMS client will ALWAYS generate data key as stated in this documentation: https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/data-key-caching.html

By default, the AWS Encryption SDK generates a new data key for every encryption operation.

Because of this, it may result in exceeding your request quota as stated here: https://docs.aws.amazon.com/kms/latest/developerguide/requests-per-second.html

If you are exceeding the request quota for the GenerateDataKey operation, consider using the data key caching feature of the AWS Encryption SDK. Reusing data keys might reduce the frequency of your requests to AWS KMS.

Quick Note:

As stated, I am encrypting and decrypting locally on my server, so it does not contact AWS servers for these operations. However, generateDataKey function, ALWAYS communicate to AWS server to get these data keys. These data keys are not generated locally, you get it from AWS server.

This is very evident if you are running automated tests which triggers encrypt and decrypt multiple times in a short period.

As stated above, the solution is to cache the data keys used for encrypt and decrypt operations. This is so these operations does not generate these keys over and over again. It will use the cached keys if they are available.

Example for caching data key locally are documented here: https://github.com/aws/aws-encryption-sdk-javascript/blob/master/modules/example-node/src/caching_cmm.ts

Here is my updated code

const { KMS } = require('@aws-sdk/client-kms');
const { KmsKeyringNode, buildClient, NodeCachingMaterialsManager, getLocalCryptographicMaterialsCache } = require('@aws-crypto/client-node');

const kmsAccessKeyId = process.env.KMS_ACCESS_KEY_ID;
const kmsSecretAccessKey = process.env.KMS_SECRET_ACCESS_KEY;
const generatorKeyId = process.env.KMS_GENERATOR_KEY_ID;
const kmsKeyId = process.env.KMS_KEY_ID;
const REGIONS = { REGION: 'region-here' };

const clientProvider = () => new KMS({ region: REGIONS.REGION, credentials: { accessKeyId: kmsAccessKeyId, secretAccessKey: kmsSecretAccessKey } });
const keyIds = [kmsKeyId];
const keyring = new KmsKeyringNode({ clientProvider, generatorKeyId, keyIds });
const { encrypt, decrypt } = buildClient();

const capacity = 100
const cache = getLocalCryptographicMaterialsCache(capacity)
const partition = 'local partition name'
const maxAge = 1000 * 60
const maxBytesEncrypted = 100
const maxMessagesEncrypted = 10

const cachingCMM = new NodeCachingMaterialsManager({
  backingMaterials: keyring,
  cache,
  partition,
  maxAge,
  maxBytesEncrypted,
  maxMessagesEncrypted,
})

const convertU8toString = (u8Arr) => {
  const textDecoder = new TextDecoder();

  const str = textDecoder.decode(u8Arr);

  return str;
};

const encryptData = async (value) => {
  try {
    if (!value) throw new Error('No value provided');

    if (typeof value !== 'string') throw new Error('Only value type of string is supported');

    const { result } = await encrypt(cachingCMM, value);
    const storableMessage = result.toString('base64');

    return storableMessage;
  } catch (err) {
    throw err;
  }
};

const decryptData = async (value) => {
  try {
    if (!value) throw new Error('No value provided');

    if (typeof value !== 'string') throw new Error('Only value type of string is supported');

    const newBuff = Buffer.from(value, 'base64');
    const { plaintext } = await decrypt(cachingCMM, newBuff);

    return convertU8toString(plaintext);
  } catch (err) {
    throw err; // Our own custom error. Just re-throw.
  }
};

That's it, it works ok now. I don't get that random error everytime I run my automated tests. It does improve the encryption and decryption performance too quite a lot.

I suggest thoroughly reading the example code here:

https://github.com/aws/aws-encryption-sdk-javascript/blob/master/modules/example-node/src/caching_cmm.ts

Hope this helps anyone, and also if I have something in my answer that I explained incorrectly, feel free to update/edit and comment.

Thanks.