Goal:
To modify the libcurl example for HTTP/2 multiplexing that can be found here to save a payload response as it arrives into some buffer, instead of writing it to a file like the aforementioned example currently does. The payload within the buffer would then be available for things such as printing, searching for strings etc.
Expected output:
Program should print out each received payload response to stdout whenever the callback function detects that one has been delivered.
Actual output:
Sometimes the program works as expected for a small number of transfers (see line of code int num_transfers = 3 in main() further below). If the number of transfers is increased to say 8 or 10, sometimes the program doesn't function properly and the program will still print the output to stdout, but in the default format that libcurl will do if no CURL_WRITEFUNCTION/CURL_WRITEDATA has been included in the code, possibly suggesting nothing is being received by the callback function? Also in this scenario, an incorrect number of responses will be printed.
In the main do...while loop within main(), I set chunk.memory and chunk.size equal to 0 after they had been printed out. Without doing this, every time a new response was received these would continue growing. I'm unsure if this was the correct approach, however.
Current attempt:
Using the libcurl example that can be found here, I have attempted to mimic the functionality of writing the output to a callback function as seen below (instead of writing each response payload to a file).
#include <iostream>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
/* somewhat unix-specific */
#include <sys/time.h>
#include <unistd.h>
/* curl stuff */
#include <curl/curl.h>
#include <curl/mprintf.h>
#ifndef CURLPIPE_MULTIPLEX
#define CURLPIPE_MULTIPLEX 0
#endif
struct CURLMsg *msg;
struct transfer {
CURL *easy;
unsigned int num;
FILE *out;
};
struct MemoryStruct {
char *memory;
size_t size;
};
struct MemoryStruct chunk;
#define NUM_HANDLES 1000
static size_t WriteMemoryCallback(void *contents, size_t size, size_t nmemb, void *userp)
{
size_t realsize = size * nmemb;
struct MemoryStruct *mem = (struct MemoryStruct *)userp;
char *ptr = (char*)realloc(mem->memory, mem->size + realsize + 1);
if(!ptr) {
/* out of memory! */
std::cout << "not enough memory (realloc returned NULL)" << std::endl;
return 0;
}
mem->memory = ptr;
memcpy(&(mem->memory[mem->size]), contents, realsize);
mem->size += realsize;
mem->memory[mem->size] = 0;
return realsize;
}
static void setup(struct transfer *t, int num)
{
CURL *hnd;
hnd = t->easy = curl_easy_init();
curl_easy_setopt(hnd, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
curl_easy_setopt(hnd, CURLOPT_WRITEDATA, (void *)&chunk);
/* set the same URL */
curl_easy_setopt(hnd, CURLOPT_URL, "https://someurl.xyz");
/* HTTP/2 please */
curl_easy_setopt(hnd, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_2_0);
/* we use a self-signed test server, skip verification during debugging */
curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYPEER, 0L);
curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYHOST, 0L);
#if (CURLPIPE_MULTIPLEX > 0)
/* wait for pipe connection to confirm */
curl_easy_setopt(hnd, CURLOPT_PIPEWAIT, 1L);
#endif
}
int main() {
struct transfer trans[NUM_HANDLES];
CURLM *multi_handle;
int i;
int still_running = 0; /* keep number of running handles */
int num_transfers = 3;
chunk.memory = (char*)malloc(1);
chunk.size = 0;
/* init a multi stack */
multi_handle = curl_multi_init();
for(i = 0; i < num_transfers; i++) {
setup(&trans[i], i);
/* add the individual transfer */
curl_multi_add_handle(multi_handle, trans[i].easy);
}
curl_multi_setopt(multi_handle, CURLMOPT_PIPELINING, CURLPIPE_MULTIPLEX);
// Main loop
do {
CURLMcode mc = curl_multi_perform(multi_handle, &still_running);
if(still_running) {
/* wait for activity, timeout or "nothing" */
mc = curl_multi_poll(multi_handle, NULL, 0, 1000, NULL);
}
if(mc) {
break;
}
// Get response
do {
int queued;
msg = curl_multi_info_read(multi_handle, &queued);
if ((msg) && (msg->msg == CURLMSG_DONE) && (msg->data.result == CURLE_OK)) {
// Print the response payload
std::cout << "size: " << chunk.size << std::endl;
std::cout << chunk.memory << std::endl;
chunk.memory = 0;
chunk.size = 0;
}
} while (msg);
} while (still_running);
for(i = 0; i < num_transfers; i++) {
curl_multi_remove_handle(multi_handle, trans[i].easy);
curl_easy_cleanup(trans[i].easy);
}
free(chunk.memory);
curl_multi_cleanup(multi_handle);
return 0;
}
Summary question:
Q1. How can I modify the above program to correctly save a received payload response into a struct or a buffer asynchronously so that it can be available for functionality such as printing to stdout or searching for strings?
First, you should associate the transfer struct with a way of accessing the output:
struct transfer {
CURL *easy;
unsigned int num;
std::string contents;
};
and associate the CURLOPT_WRITEDATA with the pointer:
curl_easy_setopt(hnd, CURLOPT_WRITEDATA, (void *)t);
Then, WriteMemoryCallback becomes:
static size_t WriteMemoryCallback(void *contents, size_t size, size_t nmemb, void *userp) {
transfer *t = (transfer *)userp;
size_t realsize = size * nmemb;
t->contents.append((const char *)contents, realsize);
return realsize;
}
Afterwards you can find the contents in the trans[i].contents variables.