My goal is to be able to send a file of around 150MB over mosquitto. The matters of efficiency of such transfer are not important at the moment, it's just to find out if such a transfer is possible.
MQTT specification mentions a message payload limit of 256MB, however I was only able to send a file of size up to 4095KB and nothing more. The publisher is implemented with C mosquitto library, dynamically allocating a vector with binary data. The publisher program does not run into any errors, however after the data is published on the client side, no notifications of publishing are present on the broker side if the sent file is larger than 4065KB, nor the mosquitto_sub receives any messages, what happens though is a message from the broker that the publishing client disconnected with success:
1711360340: New connection from 127.0.0.1:50002 on port 1883.
1711360340: New client connected from 127.0.0.1:50002 as auto-A51C5B5B-F445-5952-5EA5-49B12F2E614D (p2, c1, k5).
1711360340: No will message specified.
1711360340: Sending CONNACK to auto-A51C5B5B-F445-5952-5EA5-49B12F2E614D (0, 0)
1711360340: Client auto-A51C5B5B-F445-5952-5EA5-49B12F2E614D disconnected: Success.
Is it really possible to send those large files over mosquitto? If so, what are the best practices regarding the implementation?
EDIT: Appending the file source code
#include <cstdio>
#include <mosquitto.h>
#include <iostream>
#include <fstream>
#include <vector>
#include <memory>
struct mosquitto *mosq;
void publishImage(const char *topic, const char *image_path)
{
std::ifstream fs(image_path, std::ios::in | std::ios::binary | std::ios::app);
std::unique_ptr<std::vector<uint8_t>> v_buf;
void *msgPayload = nullptr;
int msgPayloadLen;
if (fs.good())
{
v_buf.reset(new std::vector<uint8_t>((std::istreambuf_iterator<char>(fs)), (std::istreambuf_iterator<char>())));
fs.close();
msgPayload = static_cast<void*>(v_buf.get()->data());
msgPayloadLen = v_buf.get()->size();
}
if(msgPayload == nullptr)
{
fprintf(stderr, "image could not be copied to memory\n");
return;
}
int rc = mosquitto_publish(mosq, NULL, topic, msgPayloadLen, msgPayload, 1, false);
if (rc != MOSQ_ERR_SUCCESS)
{
fprintf(stderr, "mosquitto publish error: %s\n", mosquitto_strerror(rc));
}
return;
};
int main(int argc, const char* argv[])
{
if (argc < 2)
{
fprintf(stderr, "Not enough input arguments!\nUsage: imageUpdater <topic> <path_to_image>\n");
return 1;
}
const char *topic = argv[1];
const char *pathToImage = argv[2];
mosquitto_lib_init();
mosq = mosquitto_new(NULL, true, NULL);
if (mosq == NULL)
{
fprintf(stderr, "Error: Out of memory.\n");
return 1;
}
int rc = mosquitto_connect(mosq, "127.0.0.1", 1883, 5);
if (rc != MOSQ_ERR_SUCCESS)
{
mosquitto_destroy(mosq);
fprintf(stderr, "Error: %s\n", mosquitto_strerror(rc));
return 1;
}
else
{
printf("Connected!\n");
}
rc = mosquitto_loop_start(mosq);
if (rc != MOSQ_ERR_SUCCESS)
{
mosquitto_destroy(mosq);
fprintf(stderr, "Error: %s\n", mosquitto_strerror(rc));
return 1;
}
publishImage(topic, pathToImage);
mosquitto_lib_cleanup();
return 0;
}
EDIT2: log output for a successful publish (small file)
1711376922: New client connected from 127.0.0.1:38426 as auto-33E25100-81EB-0583-E540-C73A5D29E267 (p2, c1, k5).
1711376922: No will message specified.
1711376922: Sending CONNACK to auto-33E25100-81EB-0583-E540-C73A5D29E267 (0, 0)
1711376922: Received PUBLISH from auto-33E25100-81EB-0583-E540-C73A5D29E267 (d0, q1, r0, m1, 'fota/update', ... (4192256 bytes))
1711376922: Sending PUBACK to auto-33E25100-81EB-0583-E540-C73A5D29E267 (m1, rc0)
1711376922: Client auto-33E25100-81EB-0583-E540-C73A5D29E267 closed its connection.
Using QOS0 instead of QOS1 makes it possible to transfer larger file, however the reliability is much lower. 10MB file transmission was successful, but only at the 5th try. Actually checking again, QOS1 also succeeded at 10MB file transmission after multiple tries. QoS0 is less reliable for the files around 4000KB than QoS1.
The correct solution here is probably to use mosquitto_loop_stop() before mosquitto_lib_cleanup()
As the doc explains, this should wait for outstanding messages to be sent before exiting.
This is part of the threaded client interface. Call this once to stop the network thread previously created with mosquitto_loop_start. This call will block until the network thread finishes.
Docs: https://mosquitto.org/api/files/mosquitto-h.html#mosquitto_loop_stop