c++boostwebsocketbeast

Boost inflate algorithm decompress


I am developing a simple test code using Websocket client using c++ boost. A server I get response from says I need to decompress messages using inflate algorithm. I found out there is deflate option in boost Websocket library but it did not work. Please let me know how to convert data to decompressed string.

#include <iostream>
#include <string>
#include <boost/beast/core.hpp>
#include <boost/beast/websocket.hpp>
#include <boost/asio/connect.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/beast/websocket/ssl.hpp>
#include <boost/asio/ssl.hpp>
#include <chrono>

using tcp = boost::asio::ip::tcp;
namespace websocket = boost::beast::websocket;

int main()
{
    std::ostringstream stream;
    std::string host = "real.okex.com";
    auto const port = "8443";
    auto const path = "/ws/v3";
    boost::beast::multi_buffer buffer;
    boost::asio::io_context ioc;
    boost::asio::ssl::context ctx{boost::asio::ssl::context::sslv23};
    tcp::resolver resolver{ioc};
    websocket::stream<boost::asio::ssl::stream<boost::asio::ip::tcp::socket>> wss{ioc, ctx};

    ctx.set_verify_mode(boost::asio::ssl::verify_none);
    tcp::resolver::results_type results = resolver.resolve(host, port);
    boost::asio::connect(wss.next_layer().next_layer(), results.begin(), results.end());

    // SSL handshake
    wss.next_layer().handshake(boost::asio::ssl::stream_base::client);

    // websocket handshake
    wss.handshake(host, path);

    std::cout << "connected" << std::endl;

    // send request to the websocket
    wss.write(boost::asio::buffer("{'op':'subscribe', 'args':['spot/ticker:ETH-USDT']}"));

    // read message
    wss.read(buffer);
    std::cout << buffer.size() << std::endl;
    buffer.consume(buffer.size());

    /*
    stream << boost::beast::buffers(buffer.data());
    buffer.consume(buffer.size());
    std::string incoming = stream.str();
    std::cout << incoming << std::endl;
    */
}

Thanks !


Solution

  • I struggled for a long time, then I figured, what if I try with a different server?

    That helped. I took echo_compressed/server.py from Autobahn:

    wget 'https://github.com/crossbario/autobahn-python/raw/master/examples/twisted/websocket/echo_compressed/server.py'
    virtualenv venv && . venv/bin/activate && pip install autobahn twisted
    python server.py
    

    That starts a WS server on port 9000. It's not using SSL though, so I disabled that in the code (see #ifdef SSL below).

    Now the key is to set the permessage_deflate extension option before WS handshake:

    websocket::permessage_deflate opt;
    opt.client_enable = true; // for clients
    opt.server_enable = true; // for servers
    s.set_option(opt);
    

    Also noted that some servers require the port name be present in the Host header when not running on standard ports:

    s.handshake(host + ":" + port, path);
    

    Now reading works just fine and deflates as you'd expect, e.g. write it to response.txt:

    beast::multi_buffer buffer;
    s.read(buffer);
    {
        std::ofstream ofs("response.txt", std::ios::binary);
        std::copy(
                net::buffers_begin(buffer.data()),
                net::buffers_end(buffer.data()),
                std::ostreambuf_iterator<char>(ofs));
    }
    

    Or, when replacing the multi_buffer with an Asio streambuf, it's easy to just stream it:

    net::streambuf buffer;
    s.read(buffer);
    std::cout << &buffer;
    

    Proof That It Was Deflating

    Inspecting the traffic with tcpdump/Wireshark shows this. Also, the Autobahn logging confirms it:

    2020-06-22 02:12:05+0200 [-] Log opened.
    2020-06-22 02:12:05+0200 [-] WebSocketServerFactory starting on 9000
    2020-06-22 02:12:05+0200 [-] Starting factory <autobahn.twisted.websocket.WebSocketServerFactory object at 0x7f7af3fa5710>
    2020-06-22 02:12:05+0200 [-] Site starting on 8080
    2020-06-22 02:12:05+0200 [-] Starting factory <twisted.web.server.Site instance at 0x7f7af3850910>
    2020-06-22 02:12:11+0200 [-] WebSocket connection request by tcp4:127.0.0.1:48658
    2020-06-22 02:12:11+0200 [-] WebSocket extensions in use: [PerMessageDeflate(is_server = True, server_no_context_takeover = False, client_no_context_takeover = False, server_max_window_bits = 15, client_max_window_bits = 15, mem_level = 8)]
    

    The Problem With That Server (real.okex.com)

    I don't know what about it, really, but it seems that server is not sending standard responses. Perhaps someone else can tell. Writing the responses to a file did NOT result in a file that looks like it is zlib compressed.

    Other tools tried ALSO fail to decode the data:

    Full Listing

    As I tested it with:

    #include <boost/asio/ip/tcp.hpp>
    #include <boost/asio/ssl.hpp>
    #include <boost/asio.hpp>
    #include <boost/beast.hpp>
    #include <boost/beast/websocket.hpp>
    #include <boost/beast/websocket/ssl.hpp>
    #include <iostream>
    #include <string>
    #include <fstream>
    
    namespace net       = boost::asio;
    namespace ssl       = net::ssl;
    namespace beast     = boost::beast;
    namespace http      = beast::http;
    namespace websocket = beast::websocket;
    using tcp = net::ip::tcp;
    //#define SSL
    #ifdef SSL
    using stream_t = websocket::stream<ssl::stream<tcp::socket>>;
    #else
    using stream_t = websocket::stream<tcp::socket/*, true*/>;
    #endif
    
    int main(int argc, char** argv) {
        if (argc<4) {
            std::cerr << "Usage: " << argv[0] << " host port path\n";
            return 1;
        }
        std::string host = argc>=2? argv[1] : "real.okex.com";
        auto const port  = argc>=3? argv[2] : "8443";
        auto const path  = argc>=3? argv[3] : "/ws/v3";
    
        net::io_context ioc;
        ssl::context ctx{ ssl::context::sslv23 };
        tcp::resolver resolver{ ioc };
    #ifdef SSL
        stream_t s{ ioc, ctx };
    #else
        stream_t s{ ioc };
    #endif
    
        ctx.set_verify_mode(ssl::verify_none);
        tcp::resolver::results_type results = resolver.resolve(host, port);
        net::connect(
                beast::get_lowest_layer(s),
                //s.next_layer().next_layer(),
                results.begin());
    
    #ifdef SSL
        // SSL handshake
        s.next_layer().handshake(ssl::stream_base::client);
    #endif
    
        // websocket handshake
        websocket::permessage_deflate opt;
        opt.client_enable = true; // for clients
        opt.server_enable = true; // for servers
        s.set_option(opt);
    
        s.handshake(host + ":" + port, path);
    
        std::cout << "connected" << std::endl;
    
        // send request to the websocket
        s.write(net::buffer("{'op':'subscribe', 'args':['spot/ticker:ETH-USDT']}"));
    
        {
            net::streambuf buffer;
            s.read(buffer);
            std::cout << &buffer << std::endl;
        }
    }
    

    Then I ran with

    enter image description here