Currenty in my project, I'm using ExprTk to load a large number of expressions, that will be evaluated at a later point in time. Some of the expressions come locally, but others come from external sources.
ExprTk seems to allocate memory for the local vectors and variables, which is fine. However I don't want to load/compile expressions that would require a large amount of memory.
As an example, when using the double type, the following expression would at least be 8008 bytes given there is 1001 doubles (vars x and v):
var x := 123; var v[1000] := [1]; sum(2 * v / x);
The following is roughly how my loading thread works:
typedef exprtk::symbol_table<double> symbol_table_t;
typedef exprtk::expression<double> expression_t;
typedef exprtk::parser<double> parser_t;
std::vector<expression_t> expressionsList;
while (online)
{
std::string newExprString = getNextExpression(transportConext);
expression_t newExpression;
newExpression.register_symbol_table(globalSymbolTable);
auto usedResidentMemoryBefore = System::Utils::getResidentMemoryUsed();
if (!parser.compile(newExprString,newExpression))
{
handleCompilationError(oarser,newExprString);
continiue;
}
auto usedResidentMemoryAfter = System::Utils::getResidentMemoryUsed();
if ((usedResidentMemoryAfter - usedResidentMemoryBefore) > maxMemoryLimit)
{
handleExpressionError(newExprString);
continue;
}
// all is good add new expression
expressionList.push_back(newExpression);
}
The exprtk expression type doesn't have a size method to call to see what the amount of memory is, so I'm using a function that get the process's resident memory, and call it before and after the compilation steps.
This sort of works, but has issues, such as the program is multi-threaded, and other threads will be allocating and deallocating concurrently, which will lead to false positives and negatives getting through or failing the check.
I've considered also regexing for things like var<variablename>[.*]\:= however it is tricky as the size subscript could itself be made up of variables, such as: "var v[max_elems / 2] := 0;"
In short I want to drop expressions that when compiled will require more than a given limit in memory.
In ExprTk, an expression's size is the amalgamation of the sizes of all the local variables and vectors defined in the expression. This inherently does not include the the sizes of variables used that are sourced from the symbol_table.
There are two specific settings on the exprtk::parser one can use:
Limit to the total amount of memory, in bytes, consumed by the expression (set_max_total_local_symbol_size_bytes)
Limit the size, in terms of elements, any single vector can consume (set_max_local_vector_size)
Option 1 Setting The Total Max Size
const std::size_t max_expression_size_bytes = 1024;
using T = double;
using symbol_table_t = exprtk::symbol_table<T>;
using expression_t = exprtk::expression<T>;
using parser_t = exprtk::parser<T>;
symbol_table_t globalSymbolTable;
parser_t parser;
parser.settings().set_max_total_local_symbol_size_bytes(max_expression_size_bytes);
.
.
.
expression_t newExpression;
newExpression.register_symbol_table(globalSymbolTable);
if (!parser.compile(newExprString,newExpression))
{
handleCompilationError(oarser,newExprString);
continiue;
}
// all is good add new expression
expressionList.push_back(newExpression);
Option 2 Setting The Max Size In Terms Of Elements For Any Vector In An Expression
const std::size_t max_vector_elements_size = 123456;
using T = double;
using symbol_table_t = exprtk::symbol_table<T>;
using expression_t = exprtk::expression<T>;
using parser_t = exprtk::parser<T>;
symbol_table_t globalSymbolTable;
parser_t parser;
parser.settings().set_max_local_vector_size(max_vector_elements_size);
.
.
.
expression_t newExpression;
newExpression.register_symbol_table(globalSymbolTable);
if (!parser.compile(newExprString,newExpression))
{
handleCompilationError(oarser,newExprString);
continiue;
}
// all is good add new expression
expressionList.push_back(newExpression);
Option 3 Setting Both The Total Max Size And The Max Size For Any Vector In An Expression
const std::size_t max_expression_size_bytes = 1024;
const std::size_t max_vector_elements_size = 123;
using T = double;
using symbol_table_t = exprtk::symbol_table<T>;
using expression_t = exprtk::expression<T>;
using parser_t = exprtk::parser<T>;
symbol_table_t globalSymbolTable;
parser_t parser;
parser.settings().set_max_total_local_symbol_size_bytes(max_expression_size_bytes);
parser.settings().set_max_local_vector_size(max_vector_elements_size);
.
.
.
expression_t newExpression;
newExpression.register_symbol_table(globalSymbolTable);
if (!parser.compile(newExprString,newExpression))
{
handleCompilationError(oarser,newExprString);
continiue;
}
// all is good add new expression
expressionList.push_back(newExpression);
For more details, have a read of section 11.9/11.10 of the readme.txt