Using the nonce and counter correctly for AES-CTR mode

I understand that in AES Counter mode I need to use a 128 bit nonce. The naïve way to do that would be to use a random 128 bit nonce, but I'm not sure the algorithm will be able to increment the counter correctly if it's passed as all random bits. I thought the correct way to do it is to use a 96 bit nonce and also a 32 bit counter starting at 0, for example:

var key = CryptoJS.enc.Hex.parse('01ab23cd45ef67089a1b2c3d4e5f6a7b'); // 128 bits / 16 bytes
var nonce = '2301cd4ef785690a1b2c3dab'; // 96 bits / 12 bytes
var counter = '00000000'; // 32 bits / 4 bytes
var nonceAndCounter = nonce + counter;
    nonceAndCounter = CryptoJS.enc.Hex.parse(nonceAndCounter);
var plaintext = 'The quick brown fox jumps over the lazy dog.';

var encryption = CryptoJS.AES.encrypt(plaintext, key, { iv: nonceAndCounter, mode: CryptoJS.mode.CTR, padding: CryptoJS.pad.NoPadding });
var ciphertext = encryption.ciphertext.toString(CryptoJS.enc.Hex);

Is this the correct way to do it with the CryptoJS library? Or what is the correct way?

I'm going to answer my own question as I went digging into the library code to see what it really does.

Summary:

The answer is you can use either of two methods and it will work as expected:

1) Pass in a random nonce of 96 bits in length and the library itself will add the 32 bit counter automatically and increment it with every keystream block generated. E.g.

var nonce = CryptoJS.enc.Hex.parse('2301cd4ef785690a1b2c3dab'); // 12 Bytes
var encryption = CryptoJS.AES.encrypt(plaintext, key, { iv: nonce, mode: CryptoJS.mode.CTR, padding: CryptoJS.pad.NoPadding });

2) Pass in a random nonce of 96 bits in length and explicitly specify the 32 bit counter as well if you want to. You can even specify a counter like 00000009 if you want to start encrypting/decrypting from the 9th block. Below is an example starting from counter 0:

var nonce = '2301cd4ef785690a1b2c3dab';  // 12 Bytes
var counter = '00000000';                // 4 Bytes, start at counter 0
var nonceAndCounter = CryptoJS.enc.Hex.parse(nonce + counter);  // 16 Bytes
var encryption = CryptoJS.AES.encrypt(plaintext, key, { iv: nonceAndCounter, mode: CryptoJS.mode.CTR, padding: CryptoJS.pad.NoPadding });

Explanation:

Using the code in the question with 32 bit counter of 00000000, the relevant code is in this file mode-ctr.js:

/**
 * Counter block mode.
 */
CryptoJS.mode.CTR = (function () {
    var CTR = CryptoJS.lib.BlockCipherMode.extend();

    var Encryptor = CTR.Encryptor = CTR.extend({
        processBlock: function (words, offset) {
            // Shortcuts
            var cipher = this._cipher
            var blockSize = cipher.blockSize;
            var iv = this._iv;
            var counter = this._counter;

            // Generate keystream
            if (iv) {
                counter = this._counter = iv.slice(0);

                // Remove IV for subsequent blocks
                this._iv = undefined;
            }
            var keystream = counter.slice(0);
            cipher.encryptBlock(keystream, 0);

            // Increment counter
            counter[blockSize - 1] = (counter[blockSize - 1] + 1) | 0

            // Encrypt
            for (var i = 0; i < blockSize; i++) {
                words[offset + i] ^= keystream[i];
            }
        }
    });

    CTR.Decryptor = Encryptor;

    return CTR;
}());

When running this code in a browser JS debugger using a breakpoint, it converts the nonceAndCounter into a WordArray consisting of 32 bit elements:

[587320654, -142251766, 455884203, 0]

This is used to encrypt a block. To encrypt the next block it runs this line:

counter[blockSize - 1] = (counter[blockSize - 1] + 1) | 0

Which evaluates to take the counter[3] element i.e. the integer 0 and increments it to:

[587320654, -142251766, 455884203, 1]

With subsequent blocks and nonces I can see...

[587320654, -142251766, 455884203, 2]

[587320654, -142251766, 455884203, 3]

[587320654, -142251766, 455884203, 4]

And so on. So it appears to be working correctly this way.

Contrast this with how it works if you pass a 128 bit random nonce e.g.

var nonceAndCounter = CryptoJS.enc.Hex.parse('2301cd4ef785690a1b2c3dabdf99a9b3');

This produces a nonce of:

[587320654, -142251766, 455884203, -543577677, 0]

So it creates 5 array elements!? Then the function increments the fourth element from -543577677 to -543577676, then -543577675, then -543577674 and so on. So it still works in a way, but but does not increment as nicely as starting from 0 and is perhaps more error prone.

When I passed in just a 96 bit random nonce, the library automatically added the start counter as 0 to the end of the counter array and incremented it correctly for subsequent blocks. e.g.

[587320654, -142251766, 455884203, 0]
[587320654, -142251766, 455884203, 1]
[587320654, -142251766, 455884203, 2]