amazon-web-servicesnetwork-programmingamazon-ec2nlb

AWS Network Load Balancer without Target in every AZ

In the AWS Network Load Balancer documentation it says that when specifying instances for a Target Group that it must include an instance in every AZ that the Load Balancer is registered in. This is not enforced.

What happens to traffic if you have an NLB registered in 3 AZs, but only a single target EC2 instance in AZ1? What if you enable cross AZ load balancing, does that make any difference?

Solution

  • What happens to traffic if you have an NLB registered in 3 AZs, but only a single target EC2 instance in AZ1? What if you enable cross AZ load balancing, does that make any difference?

    In this particular scenario (NLB in 3 AZs, and single instance in 1 AZ), nothing really happens. There is no apparent difference with, or without, cross-zone load balancing from the perspective of the end-user. The instance will be accessible in either case.

    To verity that, I developed a simple CloudFormation template the creates NLB, with, or without cross-zone load balancing, and 1 instance. The template allows for easy experimentation with different setups of NLB, cross-zone and instance location. I used the template in us-east-1 region and default VPC.

    For the template you specify several parameters, including:

    Once you create the stack from the template and instance health check pass (can take 1 or 2 minutes), you can access NLB DNS in your browser to view a sample webpage hosted on the instance.

    ---

Parameters:

  VpcId:
    Type: AWS::EC2::VPC::Id
    
  NLBSubnetsIds:
    Type: List<AWS::EC2::Subnet::Id>
    
  InstanceSubnetId:
    Type: AWS::EC2::Subnet::Id 
    
  AmazonLinux2AMIId:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
    
  CrossZoneEnabled:
    Type: String  
    Default: false
    AllowedValues: [true, false]

Resources:


  BasicSecurityGroup:                                                        
      Type: AWS::EC2::SecurityGroup                                          
      Properties: 
        GroupDescription: Enable www port
        SecurityGroupIngress:
          - IpProtocol: tcp
            FromPort: 80
            ToPort: 80
            CidrIp: 0.0.0.0/0            
        VpcId:  !Ref VpcId

  MyInstance1:
    Type: AWS::EC2::Instance

    CreationPolicy:
        ResourceSignal:
          Timeout: PT5M
                
    Properties:                
      ImageId: !Ref AmazonLinux2AMIId  
      InstanceType: t2.micro        
      Monitoring: false
      SecurityGroupIds: [!Ref BasicSecurityGroup]
      SubnetId: !Ref InstanceSubnetId
      UserData: 
        Fn::Base64: !Sub |
            #!/bin/bash -xe

            yum install -y httpd aws-cfn-bootstrap

            echo "<h2>Hello world from $(hostname -f)</h2>" \
              > /var/www/html/index.html

            systemctl start httpd

            # check if website is working
            curl -s localhost | grep "Hello"

            # Signal the status from cfn-init
            /opt/aws/bin/cfn-signal -e $? \
                --stack ${AWS::StackName} \
                --resource MyInstance1 \
                --region ${AWS::Region}
                
                
  MyNLB:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties: 
      IpAddressType: ipv4
      LoadBalancerAttributes:  
        - Key: load_balancing.cross_zone.enabled
          Value: !Ref CrossZoneEnabled
      Scheme: internet-facing 
      Subnets: !Ref NLBSubnetsIds
      Type: network
      
  MyListner1:      
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties: 
      DefaultActions: 
        - TargetGroupArn: !Ref MyTargetGroup
          Type: forward 
      LoadBalancerArn: !Ref MyNLB
      Port: 80 
      Protocol: TCP 

  MyTargetGroup: 
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties: 
      HealthCheckEnabled: true
      HealthCheckIntervalSeconds: 10
      HealthCheckPath: /
      HealthCheckProtocol: HTTP 
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 2
      Port: 80
      Protocol: TCP 
      TargetGroupAttributes: 
        - Key: deregistration_delay.timeout_seconds
          Value: 30
      Targets:
        - Id: !Ref MyInstance1
          Port: 80
      TargetType: instance 
      VpcId: !Ref VpcId
      
      
Outputs:
    
  DNSName:
    Value: !GetAtt MyNLB.DNSName

    From the end-user perspective, in your scenario there is no clear difference between enabling or disabling cross-zone in NLB. However, the long-term difference could be in high availability. Namely, if you have cross-zone disabled and if something happens with a NLB node in the AZ where the instance is located, NLB won't be able to route traffic to your instance from other AZ. This is my speculation, as this is not something which you can check manually. The reason is that once you associate an AZ/subnet with your NLB, you can't disassociate it, to check what happens in such scenario.

    In contrast, if cross-zone is enabled, in the above scenario, NLB node from other zone could probably route traffic to the instance across zones.

    The major benefit of having cross-zone traffic enabled, is when you different number of instances in different AZs. In this case, cross-zone balancing enables that all instances will get roughly same amount of traffic. Without, cross-zone balancing, an isolate instance would get much more traffic then the collection of instances in the other AZ.

    You can check the effects of zone-balancing using the second template. The template almost same as before, but now 1 AZ will have 3 instances, while the other one will have 1 AZ.

    ---

Parameters:

  VpcId:
    Type: AWS::EC2::VPC::Id
    
  NLBSubnetsIds:
    Type: List<AWS::EC2::Subnet::Id>
    
  InstanceSubnetId1:
    Type: AWS::EC2::Subnet::Id 

  InstanceSubnetId2:
    Type: AWS::EC2::Subnet::Id     
    
  AmazonLinux2AMIId:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
    
  CrossZoneEnabled:
    Type: String  
    Default: false
    AllowedValues: [true, false]

Resources:


  BasicSecurityGroup:                                                        
      Type: AWS::EC2::SecurityGroup                                          
      Properties: 
        GroupDescription: Enable www port
        SecurityGroupIngress:
          - IpProtocol: tcp
            FromPort: 80
            ToPort: 80
            CidrIp: 0.0.0.0/0            
        VpcId:  !Ref VpcId

  MyInstance1:
    Type: AWS::EC2::Instance

    CreationPolicy:
        ResourceSignal:
          Timeout: PT3M
                
    Properties:                
      ImageId: !Ref AmazonLinux2AMIId  
      InstanceType: t2.micro        
      Monitoring: false
      SecurityGroupIds: [!Ref BasicSecurityGroup]
      SubnetId: !Ref InstanceSubnetId1
      UserData: 
        Fn::Base64: !Sub |
            #!/bin/bash -xe

            yum install -y httpd aws-cfn-bootstrap

            echo "<h2>Hello world from $(hostname -f)</h2>" \
              > /var/www/html/index.html

            systemctl start httpd

            # check if website is working
            curl -s localhost | grep "Hello"

            # Signal the status from cfn-init
            /opt/aws/bin/cfn-signal -e $? \
                --stack ${AWS::StackName} \
                --resource MyInstance1 \
                --region ${AWS::Region}
                

  MyInstance2:
    Type: AWS::EC2::Instance

    CreationPolicy:
        ResourceSignal:
          Timeout: PT3M
                
    Properties:                
      ImageId: !Ref AmazonLinux2AMIId  
      InstanceType: t2.micro        
      Monitoring: false
      SecurityGroupIds: [!Ref BasicSecurityGroup]
      SubnetId: !Ref InstanceSubnetId2
      UserData: 
        Fn::Base64: !Sub |
            #!/bin/bash -xe

            yum install -y httpd aws-cfn-bootstrap

            echo "<h2>Hello2 world from $(hostname -f)</h2>" \
              > /var/www/html/index.html

            systemctl start httpd

            # check if website is working
            curl -s localhost | grep "Hello"

            # Signal the status from cfn-init
            /opt/aws/bin/cfn-signal -e $? \
                --stack ${AWS::StackName} \
                --resource MyInstance2 \
                --region ${AWS::Region}


  MyInstance3:
    Type: AWS::EC2::Instance

    CreationPolicy:
        ResourceSignal:
          Timeout: PT3M
                
    Properties:                
      ImageId: !Ref AmazonLinux2AMIId  
      InstanceType: t2.micro        
      Monitoring: false
      SecurityGroupIds: [!Ref BasicSecurityGroup]
      SubnetId: !Ref InstanceSubnetId2
      UserData: 
        Fn::Base64: !Sub |
            #!/bin/bash -xe

            yum install -y httpd aws-cfn-bootstrap

            echo "<h2>Hello2 world from $(hostname -f)</h2>" \
              > /var/www/html/index.html

            systemctl start httpd

            # check if website is working
            curl -s localhost | grep "Hello"

            # Signal the status from cfn-init
            /opt/aws/bin/cfn-signal -e $? \
                --stack ${AWS::StackName} \
                --resource MyInstance3 \
                --region ${AWS::Region}


  MyInstance4:
    Type: AWS::EC2::Instance

    CreationPolicy:
        ResourceSignal:
          Timeout: PT3M
                
    Properties:                
      ImageId: !Ref AmazonLinux2AMIId  
      InstanceType: t2.micro        
      Monitoring: false
      SecurityGroupIds: [!Ref BasicSecurityGroup]
      SubnetId: !Ref InstanceSubnetId2
      UserData: 
        Fn::Base64: !Sub |
            #!/bin/bash -xe

            yum install -y httpd aws-cfn-bootstrap

            echo "<h2>Hello2 world from $(hostname -f)</h2>" \
              > /var/www/html/index.html

            systemctl start httpd

            # check if website is working
            curl -s localhost | grep "Hello"

            # Signal the status from cfn-init
            /opt/aws/bin/cfn-signal -e $? \
                --stack ${AWS::StackName} \
                --resource MyInstance4 \
                --region ${AWS::Region}                
                
  MyNLB:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties: 
      IpAddressType: ipv4
      LoadBalancerAttributes:  
        - Key: load_balancing.cross_zone.enabled
          Value: !Ref CrossZoneEnabled
      Scheme: internet-facing 
      Subnets: !Ref NLBSubnetsIds
      Type: network
      
  MyListner1:      
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties: 
      DefaultActions: 
        - TargetGroupArn: !Ref MyTargetGroup
          Type: forward 
      LoadBalancerArn: !Ref MyNLB
      Port: 80 
      Protocol: TCP 

  MyTargetGroup: 
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties: 
      HealthCheckEnabled: true
      HealthCheckIntervalSeconds: 10
      HealthCheckPath: /
      HealthCheckProtocol: HTTP 
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 2
      Port: 80
      Protocol: TCP 
      TargetGroupAttributes: 
        - Key: deregistration_delay.timeout_seconds
          Value: 30
      Targets:
        - Id: !Ref MyInstance1
          Port: 80
        - Id: !Ref MyInstance2
          Port: 80
        - Id: !Ref MyInstance3
          Port: 80
        - Id: !Ref MyInstance4
          Port: 80                              
      TargetType: instance 
      VpcId: !Ref VpcId
      
      
Outputs:
    
  DNSName:
    Value: !GetAtt MyNLB.DNSName

    If you use the above template, and repeatedly request the NLB url, you will see that the isolated instance will get about 50% of the traffic without cross-zone balancing. With cross-zone balancing enabled, it will be about 20%. Below are my results based on 100 requests:

    enter image description here