amazon-rdsamazon-data-pipeline

Unable to establish connection to jdbc:mysql communication link failure

I've been trying to set up a data pipeline between an S3 bucket and an Elasitcbeanstalk environment which includes a MySQL RDS instance (all in the same VPC).

I get the failure: 

The last packet sent successfully to the server was 0 milliseconds ago. 
The driver has not received any packets from the server.
amazonaws.datapipeline.database.ConnectionFactory: Unable to establish 
connection to jdbc:mysql://***.us-west-2.rds.amazonaws.com:3306/mydata 
Communications link failure

I believe the problem is that I need to allow the data pipeline to access my MySQL RDS, but can't figure out how. I set myEc2RdsSecurityGrps field to the security group name listed for the RDS instance under EC2 > Security Groups, but that didn't help.

The RDS instance has the value IAM DB Authentication Enabled set to Yes.

Also, very new to IAM roles here but two were created like so: Roles > Create Role > Data Pipeline > EC2 Role for Data Pipeline (Provides access to S3, DynamoDB, and other services for EC2 instances that Data Pipeline launches) and also Roles > Create Role > Data Pipeline > Data Pipeline (Allows Data Pipeline and Data Pipeline managed EMR clusters to call AWS services on your behalf).

Am I missing a step?

Solution

  • The Security for the RDS instance should have the DB port open to the security group of the EC2 task runners in the Datapipeline.

    To create a security group for an EC2 instance in a VPC

    1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
    2. In the navigation pane, click Security Groups.
    3. Click Create Security Group.
    4. Specify a name and description for the security group.
    5. Select your VPC from the list, and then click Create.

    Note the ID of the new security group.

    If you are running Task Runner on your own computer, note its public IP address, in CIDR notation. If the computer is behind a firewall, note the entire address range of its network. You'll need this address later on.

    Next, create rules in the resource security groups that allow inbound traffic for the data sources Task Runner must access. For example, if Task Runner must access a Amazon Redshift cluster, the security group for the Amazon Redshift cluster must allow inbound traffic from the resource.

    To add a rule to the security group for an RDS database

    1. Open the Amazon RDS console at https://console.aws.amazon.com/rds/.
    2. In the navigation pane, click Instances.
    3. Click the details icon for the DB instance. Under Security and Network, click the link to the security group, which takes you to the Amazon EC2 console. If you're using the old console design for security groups, switch to the new console design by clicking the icon that's displayed at the top of the console page.
    4. From the Inbound tab, click Edit and then click Add Rule. Specify the database port that you used when you launched the DB instance. Start typing the ID of the security group or IP address used by the resource running Task Runner in Source.
    5. Click Save.

    http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-resources-vpc.html#dp-vpc-security-groups