scalaplayframeworkakkacpu-usagereactivemongo

I am having high cpu usage in scala(akka) with reactive mongo-driver?


For my scala application, I am having high cpu usage and high load average(397) even there is small load.when I tried to analyze threaddump, I see total count of 1471 in which 732 waiting, 700 timed_waiting & 39 runnable. I see reactivemongo-akka.actor 697 threads and reactivemongo-scheduler 687 threads.I see this issue after upgrading reactive mongo driver to "org.reactivemongo" % "play2-reactivemongo_2.11" % "0.12.6-play25". I am using akka default dispatcher.

I ran this command sar -u to check which is exactly what is causing the high cpu and found the context switching(267319 per sec) and I don't see any I/O operations causing the issue.

This is my DB connection set up

  def isMongoUp: JsValue = {

    var returnValue: JsValue = null
    var myresp: ObjectNode = null

    val connectedFuture: Future[JsValue] = getDatabase.map { list =>
      list match {
        case sth: DefaultDB =>
          try {
            returnValue = statusCheck("", sth)

            myresp = returnValue.as[ObjectNode]
            myresp.put("status", true)

            returnValue = Json.toJson(myresp)
          } finally {
            sth.connection.close()
          }
        case _ =>
          myresp = new ObjectNode(JsonNodeFactory.instance)
          myresp.put("status", false)
          returnValue = Json.toJson(myresp)
      }

      returnValue
    }.recover {

      case error: Throwable =>
        error.printStackTrace()

        myresp = new ObjectNode(JsonNodeFactory.instance)
        myresp.put("status", false)
        returnValue = Json.toJson(myresp)

        returnValue
    }
    val timeout = scala.concurrent.duration.Duration(10, "seconds")

    returnValue = Await.result(connectedFuture, timeout)

    returnValue
  }

  def getDatabase: Future[DefaultDB] =
    {
      val driver = new MongoDriver
      val mongoUri = configuration.getString("mongodb.uri").get;
      val uri  = MongoConnection.parseURI(mongoUri).get;
      val con = driver.connection(uri)
      val dn = uri.db.get
      val db = con.database(dn)

      db
    }

  def statusCheck(dbConn: String = "db", db: DefaultDB): JsValue =
    Await.result({
      val commandDoc = BSONDocument("serverStatus" -> 1)

      val runner = Command.run(BSONSerializationPack)

      val futureResult = runner.apply(db, runner.rawCommand(commandDoc)).one[BSONDocument]

      futureResult.map {
        doc => reactivemongo.play.json.BSONFormats.toJSON(doc.bson)
      }
    }, Duration.Inf)
}

Solution

  • I don't have any experience with ReactiveMongo but I can see that you are creating a database connection on each request. I suspect that each instance allocates a thread pool internally and they eventually pile up. You should create only one instance of DefaultDB and reuse it between requests.

    Besides that couple of more comments to the code:

    1) No need to create fields globalValue and myResp. Try to avoid vars in general.

    2) Never ever catch Throwable