Having WSO2 BPS 3.6.0 occasionally we encountered OutOfMemoryError and server stopped. After heap analysis we suspect:
We have a few processes which periodically look-up some information (using a web service) until the business item state is not changed. After some time, some process instances may have MANY events (thousands, some 10k). When trying to see the instance information in the carbon console, the loaded data (instance activities) are able to cause en OutOfMemoryError and drop the server (having 6GB RAM) :(
As a workaround - we use a DB lookup:
select ode_event.event_id, ode_event.detail, ode_event.tstamp, ode_event.type,
ode_event.instance_id, ode_event.process_id,
ode_scope.scope_name
from ode_event, ode_scope where ode_event.instance_id=18204 and
(ode_event.scope_id = ode_scope.scope_id);
though we consider idea to let any business users (even process owners) to access the database directly as very bad.
Is there any (better) way / query to see the activities? Which is the correct github project to place an improvement / feature (to load the activities paginated) ?
Edit:
seeing the source code this "behavior" is inherited from the Apache-ODE implementation (eagerly loading the whole list of scopes and events into memory)
This is the current behavior. We can improve it by adding the pagination. But this is the reason, that is not prioritized.
If you check individual DB table sizes, you will see that event table has consumed most of the spaces. This is because, Process debug level events are enabled by default and it generates lots of events.These events will be useful at development time, But when it comes to production, you have to disable them. Otherwise you are wasting production resources (CPS, Memory, DB spaces). It will affect, performance of the entire BPS engine.
Here are some general recommendations.
If you have larger and long running process, it is recommended to enable Event only for selected scopes that you need to monitor. (Even Better if you can have none. :))
Also, you will need to continuously clean up old data from the database. (Just google BPS Documentation for scripts) Otherwise your DBA will going to complain about DB doesn't have enough space to run in near future. :)
Events are not required for process execution. So you can clean it up. In your scenario, is it Ok to clean up OLD events of the active BPEL process ? Eg: Clean events older than 1/2/7.. days. ?. This will solve your problem for some extend.