c++capnproto

Add content of a vector into a Capnproto map object


As answered in this question:

Note that an individual List value in a Cap'n Proto structure has a limit of 2^29-1 elements

Due to this limitation for a single list, I'm trying to split one list that contains more than this amount of items into a map of lists. To do it, I'm using the following schema:

struct Map64UInt{
    entries @0: List(Entry);
    struct Entry{
        key @0: UInt64;
        value @1: List(UInt64);
    }
}

I've been looking into all examples of the Cap'n Proto but I couldn't find one that contains an example on how to create and add itens into a Capn'Proto List, then add this list into a Cap'n Proto Map. As an example, consider the following code:

void addVecToCapnprotoMap(std::vector<uint64_t> &v){
    unsigned int key = 0;
    //Here: how do I create a Capn' Proto List of uint64_t
    //and add it to a Capn Proto map that follows the schema
    //described above?
}

Solution

  • Unfortunately, you cannot dynamically add new elements to a list; you have to specify the full size of the list when you first create it. This is a side effect of Cap'n Proto's zero-copy nature. Since the list is allocated in-place directly into the final message buffer, it can't be resized later.

    What you can do instead is maintain a map separately and then write the final list as a last step, like:

    // WARNING: Not tested, probably has typos.
    class MapHelper {
    public:
      MapHelper(Map64UInt::Builder finalBuilder)
          : finalBuilder(finalBuilder),
            orphanage(capnp::Orphanage::getForMessageContaining(finalBuilder)) {}
    
      void add(uint64_t key, const std::vector<unit64_t>& v) {
        auto orphan = orphanage.newOrphan<capnp::List<uint64_t>>(v.size());
        auto listBuilder = orphan.get();
        for (size_t i = 0; i < v.size(); i++) {
          listBuilder[i] = v[i];
        }
        contents.insert(std::make_pair(key, kj::mv(orphan)));
      }
    
      void finish() {
        // Write all entries to the final map.
        auto entriesBuilder = finalBuilder.initEntries(contents.size());
        size_t i = 0;
        for (auto& entry: contents) {
          auto entryBuilder = entriesBuilder[i];
          entryBuilder.setKey(entry.first);
          entryBuilder.adoptValue(kj::mv(entry.second));
          ++i;
        }
      }
    
    private:
      Map64UInt::Builder finalBuilder;
      capnp::Orphanage orphanage;
      std::map<uint64_t, capnp::Orphan<capnp::List<uint64_t>>> contents;
    };