How to compress to memory with libjpeg-turbo using jpeg_mem_dest

I've tried following another answer but just can't seem to get this right. I have about 8MiB of RBGX bitmap to convert to a jpeg in memory using libjpeg-turbo. If I use jpeg_stdio_dest I can write the whole thing to a file, then read the file back in, and it's all right. However, trying to use jpeg_mem_dest has been a puzzle. I have all the same set up as jpeg_stdio_dest, but using mem seems to only do one allocation of 4KiB and then never allocate any more space.

I can't find docs with further instruction on how to use jpeg_mem_dest, and could really use some direction.

void compress(std::vector<unsigned char>& input) {
  jpeg_compress_struct cinfo{};
  jpeg_error_mgr err{};

  cinfo.err = jpeg_std_error(&err);
  jpeg_create_compress(&cinfo);
#if 0 // using this with an open FILE* out works
  jpeg_stdio_dest(&cinfo, out);
#endif


  cinfo.image_width = kWidth; // constants defined somewhere
  cinfo.image_height = kHeight;
  cinfo.input_components = 4;
  cinfo.in_color_space = JCS_EXT_RGBX;

  // what's wrong with this?
  unsigned char* buf{};
  unsigned long buf_sz{};
  jpeg_mem_dest(&cinfo, &buf, &buf_sz);

  jpeg_set_defaults(&cinfo);
  jpeg_set_quality(&cinfo, 70, true);

  jpeg_start_compress(&cinfo, true);   
  while (cinfo.next_scanline < cinfo.image_height) {
    auto row = static_cast<JSAMPROW>(&input[cinfo.next_scanline  * 4  * kWidth]);
    jpeg_write_scanlines(&cinfo, &row, 1);
    // Always prints 4096, and buf never changes 
    std::cout << "buf_sz: " << buf_sz 
              << " buf: " << static_cast<void*>(buf) << '\n';
  }
  jpeg_finish_compress(&cinfo);

  // ...
  // in reality, return the compressed data
}

Solution

Yeah, that is not intuitive at all. The programmer that proposed the jpeg_mem_dest() tweak did not have much choice, extending an existing api is not that easy when it wasn't designed to support a feature in the first place. What is completely unobvious is that your variables don't get updated until after the jpeg_finish_compress() call. Relevant code in the library is:

METHODDEF(void)
term_mem_destination (j_compress_ptr cinfo)
{
  my_mem_dest_ptr dest = (my_mem_dest_ptr) cinfo->dest;

  *dest->outbuffer = dest->buffer;
  *dest->outsize = (unsigned long)(dest->bufsize - dest->pub.free_in_buffer);
}

Note the word "term". This function is called indirectly through a function pointer:

GLOBAL(void)
jpeg_finish_compress (j_compress_ptr cinfo)
{
  //...
  /* Write EOI, do final cleanup */
  (*cinfo->marker->write_file_trailer) (cinfo);
  (*cinfo->dest->term_destination) (cinfo);
  //...
}

Nothing much you can do about it. Just tweak your std::cout code, move it after the loop to accommodate the way the library works.

Beware the other gritty detail of this function, also not obvious. You have to free() the buffer that it created. Visible in the provided cjpeg.c sample program, end of main().