For my scala application, I am having high cpu usage and high load average(397) even there is small load.when I tried to analyze threaddump, I see total count of 1471 in which 732 waiting, 700 timed_waiting & 39 runnable. I see reactivemongo-akka.actor 697 threads and reactivemongo-scheduler 687 threads.I see this issue after upgrading reactive mongo driver to "org.reactivemongo" % "play2-reactivemongo_2.11" % "0.12.6-play25". I am using akka default dispatcher.
I ran this command sar -u to check which is exactly what is causing the high cpu and found the context switching(267319 per sec) and I don't see any I/O operations causing the issue.
This is my DB connection set up
def isMongoUp: JsValue = {
var returnValue: JsValue = null
var myresp: ObjectNode = null
val connectedFuture: Future[JsValue] = getDatabase.map { list =>
list match {
case sth: DefaultDB =>
try {
returnValue = statusCheck("", sth)
myresp = returnValue.as[ObjectNode]
myresp.put("status", true)
returnValue = Json.toJson(myresp)
} finally {
sth.connection.close()
}
case _ =>
myresp = new ObjectNode(JsonNodeFactory.instance)
myresp.put("status", false)
returnValue = Json.toJson(myresp)
}
returnValue
}.recover {
case error: Throwable =>
error.printStackTrace()
myresp = new ObjectNode(JsonNodeFactory.instance)
myresp.put("status", false)
returnValue = Json.toJson(myresp)
returnValue
}
val timeout = scala.concurrent.duration.Duration(10, "seconds")
returnValue = Await.result(connectedFuture, timeout)
returnValue
}
def getDatabase: Future[DefaultDB] =
{
val driver = new MongoDriver
val mongoUri = configuration.getString("mongodb.uri").get;
val uri = MongoConnection.parseURI(mongoUri).get;
val con = driver.connection(uri)
val dn = uri.db.get
val db = con.database(dn)
db
}
def statusCheck(dbConn: String = "db", db: DefaultDB): JsValue =
Await.result({
val commandDoc = BSONDocument("serverStatus" -> 1)
val runner = Command.run(BSONSerializationPack)
val futureResult = runner.apply(db, runner.rawCommand(commandDoc)).one[BSONDocument]
futureResult.map {
doc => reactivemongo.play.json.BSONFormats.toJSON(doc.bson)
}
}, Duration.Inf)
}
I don't have any experience with ReactiveMongo but I can see that you are creating a database connection on each request. I suspect that each instance allocates a thread pool internally and they eventually pile up. You should create only one instance of DefaultDB
and reuse it between requests.
Besides that couple of more comments to the code:
1) No need to create fields globalValue
and myResp
. Try to avoid var
s in general.
2) Never ever catch Throwable