I have seen many examples of AES-GCM implementation in OpenSSL, but all of them are encrypting and decrypting one message.
I want to encrypt/decrypt multiple messages and as I read about AES-GCM, there must be unique IV-KEY pair.
So my question is:
How to properly change IV after decryption (using OpenSSL)?
Is it safe to have predefined IV and just increment it by one after decryption?
If so, is there a function in OpenSSL that allows to do that?
You need to use a different IV each time you encrypt with the same key. Decryption is not relevant here, it just uses whatever IV it's given and doesn't “consume” the IV. What “consumes” an IV value is encryption.
GCM only requires the IV to be unique (for a given key). So starting with 0 and incrementing by 1 each time you encrypt a message is perfectly fine.
Incrementing is easy if the key is only ever used in a single session which is managed by a single thread. If your program is multithreaded and multiple threads may encrypt with the same key, you need to ensure that there's no race condition where the different threads might use the same IV around the same time. One way to do it is to put a lock around IV read-and-increment. Another is to use thread ID + per-thread counter as the IV (but note that it has to fit in GCM's IV size which is 12 bytes). If the same key is used across multiple invocations of the program, it gets more difficult because you need to ensure the IV is stored reliably (even if the program or the whole machine crashes just after consuming an IV value) — in this case you should generally avoid using the same key.
I don't think OpenSSL has a function to increment a 12-byte counter (but maybe it has and I don't know it). You can easily make your own though:
uint64_t counter = 0;
encrypt() {
unsigned char iv[12] = {0};
++counter;
memcpy(iv, counter, sizeof counter);
}
This increments a 64-bit counter, which should be enough in practice. The representation of the counter is platform-dependent (depends on the endianness), but that's not a problem as long as you send the IV as part of each ciphertext. If you're using a network protocol which avoids sending explicit IVs, it will define the precise way of incrementing IVs anyway.
An alternative approach is to use a random IV. (Of course using OpenSSL's random, not some non-cryptographic random.) Using 12 random bytes as an IV is fine as long as the number of messages is small. You need to stay well below the birthday bound which is around 2^48 (square root of the number of possible IVs). As you approach the birthday bound, the probability of a repeat becomes non-negligible. Do beware of possible attacks where an adversary somehow convinces your application to generate a very large number of messages though (for example by faking or triggering “message not received, please resend” errors).
GCM uses a 12-byte IV internally. There is a well-defined interface to take an arbitrary-length IV input and convert it to an internal 12-byte IV, but it's best to avoid this because the conversion to 12 bytes has a small chance of introducing a collision. The odds are better with a 12-byte random IV than with a longer random IV.
Final note: if you can, use AES-SIV or AES-GCM-SIV in preference to GCM. SIV makes the internal IV dependent on the message, so reusing the same value as IV input does not result in a catastrophic failure: for AES-SIV or AES-GCM-SIV, the only reason to have a different IV each time is that otherwise it's possible to see when the same message is encrypted multiple times). The downside of SIV is that you need to have the whole message before you can start encrypting, i.e. you can't do streaming encryption. It's also newer and so less widely supported. OpenSSL supports AES-SIV since version 3.0.0, but does not appear to support AES-GCM-SIV yet. AES-GCM-SIV has slightly better performance on modern PCs and smartphones that have hardware acceleration for the GHASH (GCM authentication) calculation, but other than that I'm not aware of any reason to prefer it over AES-SIV.