c++macosboostcpp-netlib

Why does my simple cpp-netlib program take so long to compile?

I recently started learning cpp-netlib and am testing out one of netlibs example programs

#include <boost/network/protocol/http/client.hpp>
#include <iostream>
int main()
{
    using namespace boost::network;
    http::client client;
    http::client::request request("http://www.boost.org");
    request << header("Connection", "close");
    http::client::response response = client.get(request);
    std::cout << body(response) << std::endl;
    return 0;
}

After many hours of research I found that the proper command I need to use to compile my program is 

clang++ -std=c++11 -I /usr/local/Cellar/openssl/1.0.2e/include test.cpp -L /usr/local/Cellar/openssl/1.0.2e/lib -lboost_system-mt -lboost_thread-mt -lcppnetlib-client-connections -lcppnetlib-uri -lcppnetlib-server-parsers -lssl -lcrypto

Here is a link to an older question that I posted detailing how I came about finding everything I needed for this program to compile cpp-Netlib with Boost Mac OSX seg fault when using HTTP Client body

I find that it takes about 15 seconds to compile and was wondering if there was a way for me to speed up this process? Is it really this slow to compile this code or is the linker taking a good amount of time to go and get all these libraries and if so can I speed this up?

Solution

  • I seem to remember cpp-netlib uses Spirit Qi grammarts for URL parsing e.g.

    network/uri/accessors.hpp
network/uri/uri.ipp

    In this case the slowdown seems to be the key_value_sequence parser in accessors.hpp

    These are very template heavy and take considerable time to compile, depending only slightly on the compiler used (MSVC being worst, in my experience).

    You could prevent inclusion of these headers. At the very least only include them in the translation units (cpps) that require it; don't ever make it get caught up in your "common" header dependencies. That would mean the compiler has to recompile that stuff on each iteration (even using precompiled headers the cost would likely be significant).

    Depending on your compiler version, these could help:

    Really, if you use c++14 capable clang, I'd be interested in testing a patch to use Spirit X3 instead of Qi.

    At least replacing this Qi parser:

    #include <boost/spirit/include/qi.hpp>

// ...
namespace details {
template <typename Map>
struct key_value_sequence : spirit::qi::grammar<uri::const_iterator, Map()> {
  typedef typename Map::key_type key_type;
  typedef typename Map::mapped_type mapped_type;
  typedef std::pair<key_type, mapped_type> pair_type;

  key_value_sequence() : key_value_sequence::base_type(query) {
    query = pair >> *((spirit::qi::lit(';') | '&') >> pair);
    pair = key >> -('=' >> value);
    key =
        spirit::qi::char_("a-zA-Z_") >> *spirit::qi::char_("-+.~a-zA-Z_0-9/%");
    value = +spirit::qi::char_("-+.~a-zA-Z_0-9/%");
  }

  spirit::qi::rule<uri::const_iterator, Map()> query;
  spirit::qi::rule<uri::const_iterator, pair_type()> pair;
  spirit::qi::rule<uri::const_iterator, key_type()> key;
  spirit::qi::rule<uri::const_iterator, mapped_type()> value;
};
}  // namespace details

template <class Map>
inline Map &query_map(const uri &uri_, Map &map) {
  const uri::string_type range = uri_.query();
  details::key_value_sequence<Map> parser;
  spirit::qi::parse(boost::begin(range), boost::end(range), parser, map);
  return map;
}

    With this X3 variant:

    #include <boost/spirit/home/x3.hpp>

// ...
namespace details {
    namespace kvs_parser {
        namespace x3 = boost::spirit::x3;

        static auto const key    = x3::char_("a-zA-Z_") >> *x3::char_("-+.~a-zA-Z_0-9/%");
        static auto const value  = +x3::char_("-+.~a-zA-Z_0-9/%");

        template <typename Map, typename K = typename Map::key_type, typename V = typename Map::mapped_type, typename Pair = std::pair<K, V> >
        static auto const pair   = x3::rule<struct kvs_pair, Pair> {} 
                                = key >> -('=' >> value);

        template <typename Map>
        static auto const query  = pair<Map> >> *((x3::lit(';') | '&') >> pair<Map>);
    }
}  // namespace details

template <class Map>
inline Map &query_map(const uri &uri_, Map &map) {
    const uri::string_type range = uri_.query();
    spirit::x3::parse(boost::begin(range), boost::end(range), details::kvs_parser::query<Map>, map);
    return map;
}

    Reduces compilation time from ~8s down to ~5s on my system

    BIG FAT WARNING The X3 code is untested (I don't even know what uses it, I just "blindly" translated to X3 to the best of my knowledge)