sonarqube amazon-elastic-beanstalk terraform bitnami sonarqube-ops

How to configure sonarqube 7.1 in AWS elasticbeanstalk

I have tried a few ways to get sonarQube running in our AWS environment, all successfully. However, SonarQube is unstable. Whenever Elastic beanstalk recycles an instance, my SonarQube environment is wiped out.

Here is what I tried:

Attempt 1: EC2 instance. I create the EC2 instance off of a bitnami ami imageId: ami-0f9cf81913a6dce27

This seemed like pretty simple process. But I prefer elastic beanstalk environment to manage our sonarQube EC2 instances.

Attempt 2: Create a EB Environment using a single docker instance, with this dockerfile:

{
  "AWSEBDockerrunVersion": "1",
  "Image": {
    "Name": "sonarqube:7.1"
  },
  "Ports": [{
    "ContainerPort": "9000"
  }]
}

This created the EB environment. It creates an RDS instance (with mySql 5.x) to store the scan data (in a database called ebdb). The sonarQube server hosts an internal elasticsearch instance locally for it's search data.

I then have to add a few environment variables to support the RDS instance (jdbc username, password, url endpoint, etc).

I then have to configure the sonarQube security side.

No marketplace features are installed. So I add SonarJava, Groovy, and SonarJS.

I add a login user for scans. All good.

Except, occasionally Elastic Beanstalk will have a health issue and drop the current instance, and re-create a new instance.

In this case, everything is still in tact - security: users, passwords, etc. Except the marketplace features are gone. So code scans will fail until I manually add them back.

The schema for single instance docker container is pretty sparse, I did not see any way to further customize w/ the docker file.

Attempt 3: Use multi-instance docker container. The schema is more robust, perhaps I can configure sonarQube more explicitly. e.g. You can pass environment variables, mysql settings, etc.

I was unable to get this to work. I did learn I needed to set the memory above 2 GB, for elasticsearch to start up. But i was unable to get the sonarQube environment to come up.

I might revisit this later.

Attempt 4: use AMI in elastic beanstalk (with terraform aws provider)

main.tf

resource "aws_elastic_beanstalk_application" "sonarqube" {
    name        = "SonarQube"
    description = "SonarQube for nano-services"
}

resource "aws_elastic_beanstalk_environment" "nonprod" {
    name                = "${var.application-name}"
    application         = "${aws_elastic_beanstalk_application.sonarqube.name}"
    solution_stack_name = "64bit Amazon Linux 2018.03 v2.10.0 running Docker 17.12.1-ce"
    wait_for_ready_timeout = "30m"

    setting {
        namespace = "aws:autoscaling:updatepolicy:rollingupdate"
        name      = "Timeout"
        value     = "PT1H"
      }

     setting {
        namespace = "aws:elasticbeanstalk:environment"
        name      = "ServiceRole"
        value     = "aws-elasticbeanstalk-service-role"
      }

      setting {
        namespace = "aws:elasticbeanstalk:command"
        name      = "DeploymentPolicy"
        value     = "Rolling"
      }

      setting {
        namespace = "aws:elasticbeanstalk:command"
        name      = "BatchSizeType"
        value     = "Fixed"
      }

      setting {
        namespace = "aws:elasticbeanstalk:command"
        name      = "BatchSize"
        value     = "1"
      }

      setting {
        namespace = "aws:elasticbeanstalk:command"
        name      = "IgnoreHealthCheck"
        value     = "true"
      }

      setting {
        namespace = "aws:autoscaling:launchconfiguration"
        name      = "EC2KeyName"
        value     = "web-aws-key"
      }

      setting {
        namespace = "aws:autoscaling:launchconfiguration"
        name      = "IamInstanceProfile"
        value     = "arn:aws:iam::<redacted>:instance-profile/aws-elasticbeanstalk-ec2-role"
      }

      setting {
        namespace = "aws:autoscaling:launchconfiguration"
        name      = "instanceType"
        value     = "t2.xlarge"
      }

      setting {
        namespace = "aws:elb:listener:443"
        name      = "ListenerProtocol"
        value     = "SSL"
      }

      setting {
        namespace = "aws:elb:listener:443"
        name      = "InstanceProtocol"
        value     = "SSL"
      } 

      setting {
        namespace = "aws:elb:listener:443"
        name      = "SSLCertificateId"
        value     = "arn:aws:acm:<redacted>"
      } 

      setting {
        namespace = "aws:elb:listener:443"
        name      = "ListenerEnabled"
        value     = "true"
      } 
}

Initially I included the sonarQube AMI:

  setting {
    namespace = "aws:autoscaling:launchconfiguration"
    name      = "imageId"
    value     = "ami-0f9cf81913a6dce27"
  }

This does create everything. However, the EC2 instances respond too slowly, and EB goes to Grey status. Even though SonarQube is up and running, EB is unaware of it. So I commented this out, and manually modified the image id as a one-off.

wait_for_ready_timeout does assist with this, as that simply keeps terraform from timing out. e.g. It finishes in 22.5 minutes instead of a hard stop at 20 minutes.

In this case, it creates SonarQube with a local mysql database (no RDS instance) w/ elasticsearch being local as well.

SonarQube's market place features are also included, except for Groovy. Which I added.

However, same issue as before. When EB drops an instance and re-creates it, the sonarQube environment is wiped out. This time, the credentials, marketplace features, and everything.

Has anyone run into this problem and figured it out?

Solution

I resolved the issue by using ECS (Fargate), instead of the Elastic Beanstalk container.

Steps:

Create an RDS mysql instance in AWS for sonar
Open a mysql shell for this instance, and configure it for sonar, see: Sonar setup with MySql
Create a dockerfile with the plugins you care about, e.g:

    FROM sonarqube:latest
    
    ENV SONARQUBE_JDBC_USERNAME=[YOUR-USERNAME] \   
        SONARQUBE_JDBC_PASSWORD=[YOUR-PASSWORD] \    
        SONARQUBE_JDBC_URL=jdbc:mysql://[YOUR-RDS-ENDPOINT]:3306/sonar?useSSL=false&useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&useConfigs=maxPerformance
    
    RUN wget "https://sonarsource.bintray.com/Distribution/sonar-java-plugin/sonar-java-plugin-5.7.0.15470.jar" \
        && wget "https://sonarsource.bintray.com/Distribution/sonar-javascript-plugin/sonar-javascript-plugin-4.2.1.6529.jar" \
        && wget "https://sonarsource.bintray.com/Distribution/sonar-groovy-plugin/sonar-groovy-plugin-1.4.jar" \
        && mv *.jar $SONARQUBE_HOME/extensions/plugins \
        && ls -lah $SONARQUBE_HOME/extensions/plugins
    
    EXPOSE 9000
    
    EXPOSE 9092

I exposed 9092 in case i wanted to comment out the mysql connection, and test locally with the internal h2 database at some point.

Verify the docker image runs locally

eval $(docker-machine env)

docker build -t sonar .

docker run -it -d --rm --name sonar -p 9000:9000 -p 9092:9092 sonar:latest

echo $DOCKER_HOST

Open a browser to this ip address, port 9000. e.g. http://192.x.x.x:9000

Create a new ECS repository called sonar to store the docker image.

The AWS interface actually tells you how to publish your docker image, so this should be self-evident.
Tag and push the docker file to the sonar repository

$(aws ecr get-login --no-include-email --region [YOUR-AWS-REGION])

docker tag sonar:latest [YOUR-ECS-DOCKER-IMAGE-URI]/sonar:latest

docker push [YOUR-ECS-DOCKER-IMAGE-URI]/sonar:latest

Create a new fargate cluster, called sonar
Create a new task definition.

For your container, use the ECS docker image URI. I gave mine 6 GB memory and 2 cpus, with 1024 cpu units. Here I exposed port 9000 and 9092. I added the environment vars in the Dockerfile here as well.
Create an ECS service, and include the task. Run it, verify the logs cloudwatch. And hit the public endpoint on port 9000, and done.

I largely borrowed from this: https://www.infralovers.com/en/articles/2018/05/04/sonarqube-on-aws-fargate/

I hope this helps others.