I am using a boost::interprocess::deque with a memory_mapped_file as a file buffer that lets data survive reboots etc.
I am creating the buffer like this:
typedef boost::interprocess::allocator<char, boost::interprocess::managed_mapped_file::segment_manager> char_allocator_type;
typedef boost::interprocess::basic_string<char, std::char_traits<char>, char_allocator_type> persisted_string_type;
typedef boost::interprocess::allocator<persisted_string_type, boost::interprocess::managed_mapped_file::segment_manager> persisted_string_allocator_type;
typedef boost::interprocess::deque<persisted_string_type , persisted_string_allocator_type> deque_buf;
auto* mmf = new boost::interprocess::managed_mapped_file(boost::interprocess::open_or_create, "file_name", 1000000);
auto* buffer = mmf.find_or_construct<deque_buf>(boost::interprocess::unique_instance)(mmf_->get_segment_manager());
I am writing to the back of the buffer like this:
try {
char_allocator_type ca(mmf->get_segment_manager());
persisted_string_type persisted_string(ca);
persisted_string = "some string";
buffer->push_back(persisted_string);
} catch (const boost::interprocess::bad_alloc &e) {
//buffer full, handle it
}
I am erasing from the front of the buffer like this:
buffer->pop_front();
I use it like a queue. So whenever I cannot add new data to the back I erase data from the front until I can add the new data. But, as it runs I have to erase more and more data from the front in order to add the new data in the back. Ultimately the deque can hardly contain any elements. Increasing mapped file size only postpone the problem.
What am I doing wrong?
Rgds Klaus
Firstly: add some peace and quiet to your code :)
#include <boost/container/scoped_allocator.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/containers/deque.hpp>
#include <boost/interprocess/containers/string.hpp>
#include <boost/interprocess/managed_mapped_file.hpp>
namespace bip = boost::interprocess;
namespace bc = boost::container;
using Segment = bip::managed_mapped_file;
using Mgr = Segment::segment_manager;
template <typename T>
using Alloc = bc::scoped_allocator_adaptor<bip::allocator<T, Mgr>>;
using String =
bip::basic_string<char, std::char_traits<char>, Alloc<char>>;
template <typename T> using Deque = bip::deque<T, Alloc<T>>;
This makes for expressive code. The scoped_allocator_adaptor reduces the number of times you're explicitly (inefficiently) shuffling around allocator instances:
int main() {
using Buffers = Deque<String>;
bip::managed_mapped_file mmf(bip::open_or_create, "file_name", 1 << 20); // 1 MiB
auto& buffer = *mmf.find_or_construct<Buffers>(bip::unique_instance)(
mmf.get_segment_manager());
try {
buffer.emplace_back("some string");
} catch (const boost::interprocess::bad_alloc &e) {
//buffer full, handle it
}
}
The problem is likely fragmentation. There's no real solution here unless you can afford to periodically repopulate the shared memory segment from fresh.
Another part is the segment manager overhead:
One thing to consider is using a pool for the strings. By using a fixed-size allocator you will be largely avoiding fragmentation issues.
Also keep in mind the use of reserve()
and shrink_to_fit()
to guide the implementation on how to manage actual allocation patterns.
But a more direct match to your use-case would seem to be a ring buffer. I'd sketch something like:
using Buffer = boost::container::small_vector<char, 100, Alloc<char>>;
template <typename T> using Ring = boost::circular_buffer<T, Alloc<T>>;
If you know the maximum size of your buffers before hand you could use a static_vector
instead:
using Buffer = boost::container::static_vector<char, 100>;
Static vector never allocates, so it doesn't even require the allocator.
circular_buffer
doesn't quite work as seemlessly with the scoped allocators, so it's a little more painful to use, but probably still more elegant than the manual allocator shuffling you had.
#include <boost/circular_buffer.hpp>
#include <boost/container/small_vector.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/managed_mapped_file.hpp>
namespace bip = boost::interprocess;
namespace bc = boost::container;
using Segment = bip::managed_mapped_file;
using Mgr = Segment::segment_manager;
template <typename T>
using Alloc = bip::allocator<T, Mgr>;
using Buffer = boost::container::small_vector<char, 100, Alloc<char>>;
using BufferAlloc = Buffer::allocator_type;
template <typename T> using Ring = boost::circular_buffer<T, Alloc<T>>;
static inline std::string_view as_sv(Buffer const& b) {
return {b.data(), b.size()};
}
#include <iomanip>
#include <iostream>
#include <ranges>
namespace v = std::ranges::views;
using std::ranges::subrange;
int main()
{
using Buffers = Ring<Buffer>;
bip::managed_mapped_file mmf(bip::open_or_create, "file_name", 1 << 20); // 1 MiB
auto& sequence = *mmf.find_or_construct<size_t>(bip::unique_instance)(0);
auto& buffers = *mmf.find_or_construct<Buffers>(bip::unique_instance)(
1000, // 1000 buffers capacity
mmf.get_segment_manager());
auto to_buffer =
[a = BufferAlloc(buffers.get_allocator())](std::string_view str) {
return Buffer(str.begin(), str.end(), a);
};
int64_t num = buffers.size(); // signed!
std::cout << "Current size: " << buffers.size() << " ...";
auto last5 = subrange(buffers) | v::drop(std::max(0l, num - 5));
for (auto b : last5 | v::transform(as_sv)) {
std::cout << " " << std::quoted(b);
}
std::cout << std::endl;
try {
buffers.push_back(to_buffer("some string #" + std::to_string(++sequence)));
} catch (const boost::interprocess::bad_alloc& e) {
// buffer full, handle it
std::cerr << e.what() << "\n";
return 1;
}
}
Note how this already has the pop_front behaviour without you needing to anything:
rm file_name; for a in {1..10000}; do if ./sotest; then true; else break; fi; done | nl
Prints
1 Current size: 0 ...
2 Current size: 1 ... "some string #1"
3 Current size: 2 ... "some string #1" "some string #2"
4 Current size: 3 ... "some string #1" "some string #2" "some string #3"
5 Current size: 4 ... "some string #1" "some string #2" "some string #3" "some string #4"
6 Current size: 5 ... "some string #1" "some string #2" "some string #3" "some string #4" "some string #5"
7 Current size: 6 ... "some string #2" "some string #3" "some string #4" "some string #5" "some string #6"
8 Current size: 7 ... "some string #3" "some string #4" "some string #5" "some string #6" "some string #7"
9 Current size: 8 ... "some string #4" "some string #5" "some string #6" "some string #7" "some string #8"
10 Current size: 9 ... "some string #5" "some string #6" "some string #7" "some string #8" "some string #9"
11 Current size: 10 ... "some string #6" "some string #7" "some string #8" "some string #9" "some string #10"
12 Current size: 11 ... "some string #7" "some string #8" "some string #9" "some string #10" "some string #11"
13 Current size: 12 ... "some string #8" "some string #9" "some string #10" "some string #11" "some string #12"
14 Current size: 13 ... "some string #9" "some string #10" "some string #11" "some string #12" "some string #13"
...
998 Current size: 997 ... "some string #993" "some string #994" "some string #995" "some string #996" "some string #997"
999 Current size: 998 ... "some string #994" "some string #995" "some string #996" "some string #997" "some string #998"
1000 Current size: 999 ... "some string #995" "some string #996" "some string #997" "some string #998" "some string #999"
1001 Current size: 1000 ... "some string #996" "some string #997" "some string #998" "some string #999" "some string #1000"
1002 Current size: 1000 ... "some string #997" "some string #998" "some string #999" "some string #1000" "some string #1001"
1003 Current size: 1000 ... "some string #998" "some string #999" "some string #1000" "some string #1001" "some string #1002"
1004 Current size: 1000 ... "some string #999" "some string #1000" "some string #1001" "some string #1002" "some string #1003"
1005 Current size: 1000 ... "some string #1000" "some string #1001" "some string #1002" "some string #1003" "some string #1004"
1006 Current size: 1000 ... "some string #1001" "some string #1002" "some string #1003" "some string #1004" "some string #1005"
1007 Current size: 1000 ... "some string #1002" "some string #1003" "some string #1004" "some string #1005" "some string #1006"
...
9998 Current size: 1000 ... "some string #9993" "some string #9994" "some string #9995" "some string #9996" "some string #9997"
9999 Current size: 1000 ... "some string #9994" "some string #9995" "some string #9996" "some string #9997" "some string #9998"
10000 Current size: 1000 ... "some string #9995" "some string #9996" "some string #9997" "some string #9998" "some string #9999"