basic example with boost beast and coroutines how to co_await a response

Finally I made boost and beast compile, this is my first attempt and I have no clue about asio. It does not work, and I don't know where to start looking. Debugging shows the code does not even reach the task_main_GET or do_session coroutines.

As baseline I took the example here: Can beast be used with C++20's co_await keyword?

As @sehe suggested this time (last time) I don't try to use any custom coroutine stuff but I reckon I don't know how to use asio.

I am also wondering how to co_spawn with the executor, looking at the function signature I thought it would compile, but I could only make it run by passing the io_context which I believe is not recommended.

To be honest, and this is offtopic, I don't get yet the point of the blocking run() function. What I want is an async http client with coroutines, so I can co_await the response in my application. But how, if it is blocking? I basically don't care how the http client is implemented internally, it almost feels like the "asio is async" stuff is misleading because that seems to refer to the internals, but I only care from the outside so I can nicely chain it in my application logic without callbacks. On the client, I could even use a blocking library and throw it at some threads, does not matter for performance, unlike on a server. But yes, I know it is capable, and I have to learn more about it.

#include "openssl/conf.h"

#include <boost/beast/core.hpp>
#include <boost/beast/http.hpp>
#include <boost/beast/ssl.hpp>
#include <boost/beast/version.hpp>
#include <boost/asio/co_spawn.hpp>  // added
#include <boost/asio/awaitable.hpp>  // added
#include <boost/asio/experimental/promise.hpp> // added
#include <boost/asio/experimental/use_promise.hpp> // added
#include <boost/asio/as_tuple.hpp> // added
#include <boost/asio/detached.hpp> // added
#include <cstdlib>
#include <functional>
#include <iostream>
#include <string>

namespace this_coro = boost::asio::this_coro; // added

#include <boost/certify/extensions.hpp>   // added
#include <boost/certify/https_verification.hpp>   // added

#include <syncstream>  // added

namespace beast = boost::beast;         // from <boost/beast.hpp>
namespace http = beast::http;           // from <boost/beast/http.hpp>
namespace net = boost::asio;            // from <boost/asio.hpp>
namespace ssl = boost::asio::ssl;       // from <boost/asio/ssl.hpp>
using tcp = boost::asio::ip::tcp;       // from <boost/asio/ip/tcp.hpp>
using boost::asio::use_awaitable; // added

//------------------------------------------------------------------------------

#define use_void
#ifdef use_void
using T = void;
#else
using T = std::string;
#endif

// Performs an HTTP GET and prints the response
boost::asio::awaitable<T>
do_session(
    std::string host,
    std::string port,
    std::string target,
    ssl::context& ctx)
{

    beast::error_code ec;

    auto ex = co_await net::this_coro::executor;

    // These objects perform our I/O
    tcp::resolver resolver(ex);
    beast::ssl_stream<beast::tcp_stream> stream(ex, ctx);

    // Set SNI Hostname (many hosts need this to handshake successfully)
    if(! SSL_set_tlsext_host_name(stream.native_handle(), host.c_str()))
    {
        ec.assign(static_cast<int>(::ERR_get_error()), net::error::get_ssl_category());
        std::cerr << ec.message() << "\n";
        co_return;
    }

    try{    
        // Look up the domain name
        auto const results = co_await resolver.async_resolve(host, port, use_awaitable);

        // Set the timeout.
        beast::get_lowest_layer(stream).expires_after(std::chrono::seconds(30));

        // Make the connection on the IP address we get from a lookup
        co_await beast::get_lowest_layer(stream).async_connect(results, use_awaitable);

        // Set the timeout.
        beast::get_lowest_layer(stream).expires_after(std::chrono::seconds(30));

        // Perform the SSL handshake
        co_await stream.async_handshake(ssl::stream_base::client, use_awaitable);

        // Set up an HTTP GET request message
        http::request<http::string_body> req{http::verb::get, target, 11};
        req.set(http::field::host, host);
        req.set(http::field::user_agent, BOOST_BEAST_VERSION_STRING);

        // Set the timeout.
        beast::get_lowest_layer(stream).expires_after(std::chrono::seconds(30));

        // Send the HTTP request to the remote host
        co_await http::async_write(stream, req, use_awaitable);

        // This buffer is used for reading and must be persisted
        beast::flat_buffer b;

        // Declare a container to hold the response
        http::response<http::dynamic_body> res;

        // Receive the HTTP response
        auto [ec, bytes] =
            co_await http::async_read(stream, b, res, boost::asio::as_tuple(use_awaitable));

        // Write the message to standard out
        std::cout << res << std::endl;

        // Set the timeout.
        beast::get_lowest_layer(stream).expires_after(std::chrono::seconds(30));

        if(ec == net::error::eof)
        {
            // eof is to be expected for some services
            if(ec != net::error::eof)
                throw beast::system_error(ec);
        }
        else
        {
            std::cout << res << std::endl;

            // Gracefully close the stream
            co_await stream.async_shutdown(use_awaitable);
        }

    }
    catch(beast::system_error const& se)
    {
        //std::cerr << "Handled: " << se.code().message() << "\n";
        throw; // handled at the spawn site instead
    }

    // If we get here then the connection is closed gracefully
}

//------------------------------------------------------------------------------


boost::asio::awaitable<T> task_main_GET(net::io_context &ioc)
{
    auto const host = "https://microsoftedge.github.io/Demos/json-dummy-data/64KB.json";
    auto const port = "443";
    auto const target = "/";

    std::osyncstream(std::cout) << "GET " << host << ":" << port << target << std::endl;
    // The SSL context is required, and holds certificates
    ssl::context ctx{ssl::context::tlsv12_client};

    // https://stackoverflow.com/a/61429566/2366975
    ctx.set_verify_mode(ssl::context::verify_peer );
    boost::certify::enable_native_https_server_verification(ctx);


    auto ex = this_coro::executor;
    auto task = boost::asio::co_spawn(
        //ex,
        ioc,
        [&]() mutable -> boost::asio::awaitable<std::string>
        {
            co_await do_session(host, port, target, ctx);
        },
        use_awaitable
    );


#ifdef use_void
    co_return;
#else
    std::string responseText = co_await task;
    std::osyncstream(std::cout) << "response:\n" << responseText << std::endl;
    co_return responseText ;
#endif
}

int main()
{
    // The io_context is required for all I/O
    net::io_context ioc;

    // https://stackoverflow.com/questions/79177275/how-can-co-spawn-be-used-with-co-await
    auto task = boost::asio::co_spawn(ioc, task_main_GET(ioc), use_awaitable);

    // Run the I/O service. The call will return when the get operation is complete.
    ioc.run();

    return EXIT_SUCCESS;
}

CMakeLists.txt

cmake_minimum_required(VERSION 3.5)

project(boost_beast_example LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

# Fix compiler error C1128
# https://learn.microsoft.com/en-us/cpp/error-messages/compiler-errors-1/fatal-error-c1128?view=msvc-170
if (MSVC)
    add_compile_options(/bigobj)
else ()
    add_compile_options(-Wa,-mbig-obj)   # not tested
endif ()

# OPENSSL =======================================================
# https://cmake.org/cmake/help/v3.31/module/FindOpenSSL.html
set(OPENSSL_ROOT_DIR "C:/OpenSSL-Win64")
set(OPENSSL_INCLUDE_DIR "C:/OpenSSL-Win64/include")
#find_package(OpenSSL REQUIRED PATHS "C:/OpenSSL-Win64/")
find_package(OpenSSL REQUIRED)
link_directories("C:/OpenSSL-Win64/")
include_directories(${OPENSSL_INCLUDE_DIR})
message("OPENSSL_FOUND: " ${OPENSSL_FOUND})
message("OPENSSL_INCLUDE_DIR: " ${OPENSSL_INCLUDE_DIR})
message("OPENSSL_CRYPTO_LIBRARY: " ${OPENSSL_CRYPTO_LIBRARY})
message("OPENSSL_SSL_LIBRARY: " ${OPENSSL_SSL_LIBRARY})

# BOOST =========================================================
# https://cmake.org/cmake/help/v3.31/module/FindBoost.html
set(Boost_DEBUG 1)
SET(CMAKE_INCLUDE_PATH ${CMAKE_INCLUDE_PATH} "G:/SoftwareDev/libs/boost")
SET(CMAKE_LIBRARY_PATH ${CMAKE_LIBRARY_PATH} "G:/SoftwareDev/libs/boost/bin/x64/lib/cmake")
find_package(Boost REQUIRED context system coroutine regex PATHS "G:/SoftwareDev/libs/boost/bin/x64/lib/cmake")
include_directories("G:/SoftwareDev/libs/boost/libs/beast")
include_directories(${Boost_INCLUDE_DIRS})
message("Boost_FOUND: " ${Boost_FOUND})
message("Boost_INCLUDE_DIRS: " ${Boost_INCLUDE_DIRS})
message("Boost_LIBRARY_DIRS: " ${Boost_LIBRARY_DIRS})
message("Boost_LIBRARIES: " ${Boost_LIBRARIES})
message("Boost_CONTEXT_LIBRARY: " ${Boost_CONTEXT_LIBRARY})
message("Boost_SYSTEM_LIBRARY: " ${Boost_SYSTEM_LIBRARY})

find_package(Threads REQUIRED)

add_executable(${PROJECT_NAME} main.cpp)

# CERTIFY FOR BEAST==============================================
include_directories(libs/certify/include)
if(MSVC)
    target_link_libraries(${PROJECT_NAME} Crypt32.lib)
endif()
if(APPLE)
    target_link_libraries(${PROJECT_NAME} INTERFACE "-framework CoreFoundation" "-framework Security")
    set_target_properties(${PROJECT_NAME} PROPERTIES LINK_FLAGS "-Wl,-F/Library/Frameworks")
endif()


target_link_libraries(${PROJECT_NAME}  OpenSSL::Crypto OpenSSL::SSL)
# How to link boost libraries
# https://stackoverflow.com/a/43885372/2366975
target_link_libraries(${PROJECT_NAME} ${Boost_LIBRARIES} Threads::Threads)

Solution

There were just a few coro-beginner mistakes:

missing co_await on the this_coro::executor transform keyword
missing std::move on the task awaitable in co_await std::move(task)

I think that was basically that. So, then I removed the unncessary io_context& reference, and arrive at the following cleaned-up listing. Among the changes I made is replacing use_awaitable with the more efficient asio::deferred -- which is the default completion token in recent boost.

#include <boost/asio.hpp>
#include <boost/beast.hpp>
#include <boost/beast/ssl.hpp>
#include <boost/certify/extensions.hpp>         // added
#include <boost/certify/https_verification.hpp> // added
#include <iostream>
#include <syncstream> // added

namespace asio      = boost::asio;
namespace this_coro = asio::this_coro; // added
namespace beast     = boost::beast;    // from <boost/beast.hpp>
namespace http      = beast::http;     // from <boost/beast/http.hpp>
namespace ssl       = asio::ssl;       // from <boost/asio/ssl.hpp>
using asio::ip::tcp;                   // from <boost/asio/ip/tcp.hpp>
using beast::error_code;               // from <boost/beast/core.hpp>
using namespace std::chrono_literals;

//------------------------------------------------------------------------------

using SessionResult = void;

// Performs an HTTP GET and prints the response
asio::awaitable<SessionResult> do_session( //
    std::string host, std::string port, std::string target, ssl::context& ctx) {

    auto ex = co_await this_coro::executor;

    // These objects perform our I/O
    tcp::resolver resolver(ex);
    beast::ssl_stream<beast::tcp_stream> stream(ex, ctx);

    // Set SNI Hostname (many hosts need this to handshake successfully)
    if (!::SSL_set_tlsext_host_name(stream.native_handle(), host.c_str())) {
        std::cerr << beast::error_code(static_cast<asio::error::ssl_errors>(::ERR_get_error())).message()
                  << "\n";
        co_return;
    }

    try {
        // Look up the domain name
        auto const results = co_await resolver.async_resolve(host, port);

        // Set the timeout.
        beast::get_lowest_layer(stream).expires_after(30s);

        // Make the connection on the IP address we get from a lookup
        co_await beast::get_lowest_layer(stream).async_connect(results);

        // Set the timeout.
        beast::get_lowest_layer(stream).expires_after(30s);

        // Perform the SSL handshake
        co_await stream.async_handshake(ssl::stream_base::client);

        // Set up an HTTP GET request message
        http::request<http::string_body> req{http::verb::get, target, 11};
        req.set(http::field::host, host);
        req.set(http::field::user_agent, BOOST_BEAST_VERSION_STRING);

        // Set the timeout.
        beast::get_lowest_layer(stream).expires_after(30s);

        // Send the HTTP request to the remote host
        co_await http::async_write(stream, req);

        // This buffer is used for reading and must be persisted
        beast::flat_buffer b;

        // Declare a container to hold the response
        http::response<http::dynamic_body> res;

        // Receive the HTTP response
        auto [ec, bytes] =
            co_await http::async_read(stream, b, res, asio::as_tuple(asio::deferred));

        // Write the message to standard out
        std::cout << res << std::endl;

        // Set the timeout.
        beast::get_lowest_layer(stream).expires_after(30s);

        if(ec == asio::error::eof)
        {
            // eof is to be expected for some services
            if(ec != asio::error::eof)
                throw beast::system_error(ec);
        }
        else
        {
            std::cout << res << std::endl;

            // Gracefully close the stream
            co_await stream.async_shutdown();
        }

    } catch (beast::system_error const& se) {
        //std::cerr << "Handled: " << se.code().message() << "\n";
        throw; // handled at the spawn site instead
    }

    // If we get here then the connection is closed gracefully
}

//------------------------------------------------------------------------------

asio::awaitable<SessionResult> task_main_GET() {
    auto const host = "https://microsoftedge.github.io/Demos/json-dummy-data/64KB.json";
    auto const port = "443";
    auto const target = "/";

    std::osyncstream(std::cout) << "GET " << host << ":" << port << target << std::endl;
    // The SSL context is required, and holds certificates
    ssl::context ctx{ssl::context::tlsv12_client};

    // https://stackoverflow.com/a/61429566/2366975
    ctx.set_verify_mode(ssl::context::verify_peer);
    boost::certify::enable_native_https_server_verification(ctx);

    auto ex   = co_await this_coro::executor;
    auto task = co_spawn(
        ex,
        [&] -> asio::awaitable<std::string> {
            co_await do_session(host, port, target, ctx);
            co_return "some value, then";
        },
        asio::deferred);

    std::osyncstream(std::cout) << "response:\n" << co_await std::move(task) << std::endl;

    co_return; // TODO SessionResult
}

int main() try {
    asio::io_context ioc;

    co_spawn(ioc, task_main_GET, asio::detached);

    ioc.run();
} catch (std::exception const& e) {
    std::cerr << "Error: " << e.what() << "\n";
} catch (...) {
    std::cerr << "Unknown error\n";
}

This compiles (GCC 14, Boost 1.88). However, there are some questions I have:

your eof check is never reached because it's in the opposite if branch

your host is not a host but the entire https://.... url. oops, that cannot resolve. Fixing:

 auto const host   = "microsoftedge.github.io";
 auto const port   = "443";
 auto const target = "/Demos/json-dummy-data/64KB.json";

why the co_spawn when you immediately co_await it anyways? You could simply replace

 auto ex   = co_await this_coro::executor;
 auto task = co_spawn(
     ex,
     [&] -> asio::awaitable<std::string> {
         co_await do_session(host, port, target, ctx);
         co_return "some value, then";
     },
     asio::deferred);

 std::osyncstream(std::cout) << "response:\n" << co_await std::move(task) << std::endl;

With

SessionResult data = co_await do_session(host, port, target, ctx);
std::osyncstream(std::cout) << "response:\n" << data << std::endl;

On my system that host's certificate doesn't verify with enable_native_https_server_verification. However, it does so with Asio's own
```
// boost::certify::enable_native_https_server_verification(ctx);
ctx.set_default_verify_paths();
```

Fixing those (and probably some more minor things I forgot):

Live On Coliru


#include <boost/asio.hpp>
#include <boost/beast.hpp>
#include <boost/beast/ssl.hpp>
#include <iostream>
#include <syncstream> // added

namespace asio      = boost::asio;
namespace this_coro = asio::this_coro; // added
namespace beast     = boost::beast;    // from <boost/beast.hpp>
namespace http      = beast::http;     // from <boost/beast/http.hpp>
namespace ssl       = asio::ssl;       // from <boost/asio/ssl.hpp>
using asio::ip::tcp;                   // from <boost/asio/ip/tcp.hpp>
using beast::error_code;               // from <boost/beast/core.hpp>
using namespace std::chrono_literals;

//------------------------------------------------------------------------------

using SessionResult = std::string;

// Performs an HTTP GET and prints the response
asio::awaitable<SessionResult> do_session( //
    std::string host, std::string port, std::string target, ssl::context& ctx) {

    auto ex = co_await this_coro::executor;

    // These objects perform our I/O
    tcp::resolver resolver(ex);
    beast::ssl_stream<beast::tcp_stream> stream(ex, ctx);

    // Set SNI Hostname (many hosts need this to handshake successfully)
    if (!::SSL_set_tlsext_host_name(stream.native_handle(), host.c_str()))
        throw beast::system_error(static_cast<asio::error::ssl_errors>(::ERR_get_error()));

    // Look up the domain name
    auto const results = co_await resolver.async_resolve(host, port);

    // Set the timeout.
    beast::get_lowest_layer(stream).expires_after(30s);

    // Make the connection on the IP address we get from a lookup
    co_await beast::get_lowest_layer(stream).async_connect(results);

    // Perform the SSL handshake
    co_await stream.async_handshake(ssl::stream_base::client);

    // Set up an HTTP GET request message
    http::request<http::string_body> req{http::verb::get, target, 11};
    req.set(http::field::host, host);
    req.set(http::field::user_agent, BOOST_BEAST_VERSION_STRING);
    req.set(http::field::connection, "close");

    // Send the HTTP request to the remote host
    co_await http::async_write(stream, req);

    // This buffer is used for reading and must be persisted
    beast::flat_buffer b;

    // Declare a container to hold the response
    http::response<http::string_body> res;

    // Receive the HTTP response
    auto [ec, bytes] = co_await http::async_read(stream, b, res, asio::as_tuple);

    // eof is to be expected for some services
    if (ec && ec != asio::error::eof)
        throw beast::system_error(ec);

    // Write the message to standard out
    // std::cout << res << std::endl;

    // Gracefully close the stream
    std::tie(ec) = co_await stream.async_shutdown(asio::as_tuple);
    if (ec)
        std::cerr << "Shutdown: " << ec.message() << "\n";

    // If we get here then the connection is closed gracefully
    co_return std::move(res).body();
}

//------------------------------------------------------------------------------

asio::awaitable<SessionResult> GET() {
    auto const host   = "microsoftedge.github.io";
    auto const port   = "443";
    auto const target = "/Demos/json-dummy-data/64KB.json";

    std::osyncstream(std::cerr) << "GET " << host << ":" << port << target << std::endl;
    // The SSL context is required, and holds certificates
    ssl::context ctx{ssl::context::tlsv12_client};

    // https://stackoverflow.com/a/61429566/2366975
    ctx.set_verify_mode(ssl::context::verify_peer);
    // boost::certify::enable_native_https_server_verification(ctx);
    ctx.set_default_verify_paths();

    co_return co_await do_session(host, port, target, ctx);
}

int main() try {
    asio::io_context ioc;

    co_spawn(ioc, GET, [](std::exception_ptr ep, SessionResult result) {
        if (ep)
            rethrow_exception(ep);
        std::cout << result << "\n";
    });

    ioc.run();
} catch (std::exception const& e) {
    std::cerr << "Main Error: " << e.what() << "\n";
} catch (...) {
    std::cerr << "Main Unknown error\n";
}

Local demo: