Trying to migrate counter tables. Reading every row from source table and writing the result to a new table. When writing milions of rows problem with js is running out of heap memory. Syncronous code is being used to make sure row is written before try to write another one. Pinning down the problem observing the heap seems that heap rises when using cassandra-driver to execute UPDATE query.
const query = `UPDATE keyspace.table_name
SET counter1 = counter1 + ${x1}, counter2 = counter2 + ${x2}
WHERE partition_key = ? AND clustering_key = ?`;
await cassandraDestination.execute(query, params, { prepare: true });
How to avoid memory leak in this situation?
==== JS stack trace =========================================
0: ExitFrame [pc: 0x1317672cfc5d]
Security context: 0x192500f1d971 <JSObject>
1: borrowNextConnection [0x2ea411f96091] [/home/asen.chekov/projects/stats/node_modules/cassandra-driver/lib/request-handler.js:~64] [pc=0x131767db1395](this=0x2ea411f95da1 <JSFunction RequestHandler (sfi = 0x2e8feb1ef231)>,0x21125933f379 <Object map = 0x1ddd59acbdb1>,0x21125933f439 <Object map = 0x1ddd59a82521>,0x110a33490991 <ProfileManager map...
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Writing Node.js report to file: report.20191016.122316.23174.001.json
Node.js report completed
1: 0x953b10 node::Abort() [node]
2: 0x9547f4 node::OnFatalError(char const*, char const*) [node]
3: 0xb32bee v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xb32e24 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xf32452 [node]
6: 0xf32558 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
7: 0xf3ec78 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
8: 0xf3f78b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
9: 0xf424c1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xf0c6f4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
11: 0x11c2b3e v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0x1317672cfc5d
Aborted (core dumped)```
This fixed the problem.
The correct query would be:
const query = `UPDATE keyspace.table_name
SET counter1 = counter1 + ?, counter2 = counter2 + ?
WHERE partition_key = ? AND clustering_key = ?`;
await cassandraDestination
.execute(query, [ counter1, counter2, key1, key2 ], {
prepare: true,
});