I'm using Knowage and have setup a data source, many parameterized datasets, and a cockpit.
I have a detailed report that requires many datasets.
The cockpit started freezing and I have to restart the Knowage server to get it to respond.
I split the report into different pages, and it seemed to help because it only loads some of the data at a time.
But now I added another dataset [14 total] and the server crashed again.
I'm monitoring the server through a jmx agent to understand why it's hanging, but the utilization seems fine.
Tomcat just hangs and never responds.
It appears the servlet has stopped processing requests, but I don't see any errors in the logs that look fatal.
The heap utilization is not bad, but does look anomalous.
Errors in log -
[http-nio-8080-exec-17] 09 Mar 2021 15:46:49,390 ERROR it.eng.spagobi.api.v3.DataSetMainDTO.isGeoDataSet:51 - Error during check of Geo spatial column
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: (String)"<?xml version="1.0" encoding="UTF-8"?>
<META version="1">
<COLUMNLIST>
<COLUMN alias="label" fieldType="ATTRIBUTE" multivalue="false" name="label" type="java.lang.String"/>
<COLUMN alias="amount" fieldType="MEASURE" multivalue="false" name="amount" type="java.math.BigDecimal"/>
</COLUMNLIST>
<DATASET>
<PROPERTY name="resultNumber" value="2"/>
</DATASET>
</META>
"; line: 1, column: 2]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1798)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:663)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:561)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1892)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:747)
at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4129)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3988)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2992)
at it.eng.spagobi.utilities.json.JSONUtils.toJSONObject(JSONUtils.java:98)
at it.eng.spagobi.analiticalmodel.execution.service.ExecuteAdHocUtility.hasGeoHierarchy(ExecuteAdHocUtility.java:128)
.......
[http-nio-8080-exec-9] 09 Mar 2021 15:50:48,509 ERROR it.eng.spagobi.rest.interceptors.RestExceptionMapper.toResponse:59 - Catched service error:
javax.ws.rs.NotSupportedException: RESTEASY003065: Cannot consume content type
at org.jboss.resteasy.core.registry.SegmentNode.match(SegmentNode.java:403)
at org.jboss.resteasy.core.registry.SegmentNode.match(SegmentNode.java:137)
at org.jboss.resteasy.core.registry.RootNode.match(RootNode.java:43)
at org.jboss.resteasy.core.registry.RootClassNode.match(RootClassNode.java:48)
at org.jboss.resteasy.core.ResourceMethodRegistry.getResourceInvoker(ResourceMethodRegistry.java:445)
at org.jboss.resteasy.core.SynchronousDispatcher.getInvoker(SynchronousDispatcher.java:260)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:197)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at sun.reflect.GeneratedMethodAccessor973.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:225)
at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at sun.reflect.GeneratedMethodAccessor977.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:191)
at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
....
I believe the issue was caused by the limit of DB connections, and max wait in server.xml
.
I noticed an error about attributes in the jndi configuration, and updated them.
The freezing has not occurred again, yet.
<!-- KNOWAGE -->
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxTotal="20" maxIdle="10"
maxWaitMillis="-1" name="jdbc/knowage" type="javax.sql.DataSource"
url="jdbc:mysql://localhost:3306/knowagedb" username="knowageuser" password="knowagepassword" />
<!-- KNOWAGE CACHE -->
<Resource auth="Container" driverClassName="com.mysql.jdbc.Driver" maxTotal="20" maxIdle="4"
maxWaitMillis="30000" name="jdbc/ds_cache" type="javax.sql.DataSource"
url="jdbc:mariadb://localhost:3310/cache" username="root" password="root" />
I've made these changes, along with jmx export to prometheus, and a few memory tweaks to make knowage run better on kubernetes.
Have a look here -
https://github.com/savantly-net/knowage-docker