c++httpboostbeast

Handling large http response using boost::beast


The following code use to get http response message:

  boost::beast::tcp_stream stream_;

  boost::beast::flat_buffer buffer;
  boost::beast::http::response<boost::beast::http::dynamic_body> res;
  boost::beast::http::read(stream_, buffer, res);

However, In some cases, based on the preceding request, I can expect that the response message body will include large binary file.

Therefore, I’d like to read it directly to the filesystem and not through buffer variable to avoid excessive use of process memory. How can it be done ?

in Objective-c framework NSUrlSession there's an easy way to do it using NSURLSessionDownloadTask instead of NSURLSessionDataTask, so I wonder if it's also exist in boost.

Thanks !


Solution

  • In general, you can use the http::buffer_body to handle arbitrarily large request/response messages.

    If you specifically want to read/write from a filesystem file, you can have the http::file_body instead.

    Full Demo buffer_body

    The documentation sample for buffer_body is here https://www.boost.org/doc/libs/1_77_0/libs/beast/doc/html/beast/using_http/parser_stream_operations/incremental_read.html.

    Using it to write to std::cout: Live On Coliru

    #include <boost/asio.hpp>
    #include <boost/asio/ip/tcp.hpp>
    #include <boost/beast.hpp>
    #include <boost/beast/websocket.hpp>
    #include <iostream>
    
    namespace net       = boost::asio;
    namespace beast     = boost::beast;
    namespace http      = beast::http;
    using tcp           = net::ip::tcp;
    using socket_t      = tcp::socket;
    
    /*  This function reads a message using a fixed size buffer to hold
        portions of the body, and prints the body contents to a `std::ostream`.
    */
    
    template<
        bool isRequest,
        class SyncReadStream,
        class DynamicBuffer>
    void
    read_and_print_body(
        std::ostream& os,
        SyncReadStream& stream,
        DynamicBuffer& buffer,
        beast::error_code& ec)
    {
        http::parser<isRequest, http::buffer_body> p;
        http::read_header(stream, buffer, p, ec);
        if(ec)
            return;
        while(! p.is_done())
        {
            char buf[512];
            p.get().body().data = buf;
            p.get().body().size = sizeof(buf);
            http::read(stream, buffer, p, ec);
            if(ec == http::error::need_buffer)
                ec = {};
            if(ec)
                return;
            os.write(buf, sizeof(buf) - p.get().body().size);
        }
    }
    
    int main() {
        std::string host = "173.203.57.63"; // COLIRU 20210901
        auto const port  = "80";
    
        net::io_context ioc;
        tcp::resolver   resolver{ioc};
    
        socket_t s{ioc};
        net::connect(s, resolver.resolve(host, port));
    
        write(s, http::request<http::empty_body>{http::verb::get, "/", 11});
    
        beast::error_code  ec;
        beast::flat_buffer buf;
    
        read_and_print_body<false>(std::cout, s, buf, ec);
    }
    

    Full file_body example

    This is much shorter, writing to body.html:

    Live On Coliru

    #include <boost/asio.hpp>
    #include <boost/asio/ip/tcp.hpp>
    #include <boost/beast.hpp>
    #include <boost/beast/websocket.hpp>
    #include <iostream>
    
    namespace net       = boost::asio;
    namespace beast     = boost::beast;
    namespace http      = beast::http;
    using tcp           = net::ip::tcp;
    using socket_t      = tcp::socket;
    
    int main() {
        std::string host = "173.203.57.63"; // COLIRU 20210901
        auto const port  = "80";
    
        net::io_context ioc;
        tcp::resolver   resolver{ioc};
    
        socket_t s{ioc};
        net::connect(s, resolver.resolve(host, port));
    
        write(s, http::request<http::empty_body>{http::verb::get, "/", 11});
    
        beast::error_code  ec;
        beast::flat_buffer buf;
        http::response<http::file_body> res;
        res.body().open("body.html", beast::file_mode::write_new, ec);
        if (!ec.failed())
        {
            read(s, buf, res, ec);
        }
    
        std::cout << "Wrote 'body.html' (" << ec.message() << ")\n";
        std::cout << "Headers " << res.base() << "\n";
    }
    

    Prints

    Wrote 'body.html' (Success)
    Headers HTTP/1.1 200 OK 
    Content-Type: text/html;charset=utf-8
    Content-Length: 8616
    Server: WEBrick/1.4.2 (Ruby/2.5.1/2018-03-29) OpenSSL/1.0.2g
    Date: Wed, 01 Sep 2021 19:52:20 GMT
    Connection: Keep-Alive
    

    With file body.html; wc body.html showing:

    body.html: HTML document, ASCII text, with very long lines
     185  644 8616 body.html
    

    Beyond: streaming to child processes and streaming processing

    I have an advanced example of that here: How to read data from Internet using muli-threading with connecting only once?.