amazon-web-servicesamazon-ec2salt-projectsalt-cloud

getting auth failure on salt-cloud command


I'm using salt stack and I want to try and provision new EC2 instances using the salt-cloud command. But I'm getting an auth failure on salt-cloud command:

[root@salt:~] #salt-cloud -p base_ec2_public ops.example.com
[ERROR   ] AWS Response Status Code and Error: [401 401 Client Error: Unauthorized] {'Errors': {'Error': {'Message': 'AWS was not able to validate the provided access credentials', 'Code': 'AuthFailure'}}, 'RequestID': '3a5e33e2-d1a9-44fa-983c-26691d4f8ee7'}
[ERROR   ] AWS Response Status Code and Error: [401 401 Client Error: Unauthorized] {'Errors': {'Error': {'Message': 'AWS was not able to validate the provided access credentials', 'Code': 'AuthFailure'}}, 'RequestID': '163079c6-2b79-4301-80c8-77ba0d7c896d'}
[ERROR   ] There was a profile error: string indices must be integers, not str

This is my /etc/salt/cloud.providers.d/aws.conf file

----
my-ec2-us-east-public-ips:
  # Set up the location of the salt master
  #
  minion:
    master: salt.example.com

  # Set up grains information, which will be common for all nodes
  # using this provider
  grains:
    node_type: broker
    release: 1.0.1

  # Specify whether to use public or private IP for deploy script.
  #
  # Valid options are:
  #     private_ips - The salt-cloud command is run inside the EC2
  #     public_ips - The salt-cloud command is run outside of EC2
  #
  ssh_interface: public_ips

  # Optionally configure the Windows credential validation number of
  # retries and delay between retries.  This defaults to 10 retries
  # with a one second delay betwee retries
  win_deploy_auth_retries: 10
  win_deploy_auth_retry_delay: 1

  # Set the EC2 access credentials (see below)
  #
  id: "REDACTED"
  key: "REDACTED"

  # Make sure this key is owned by root with permissions 0400.
  #
  private_key: /etc/salt/my_test_key.pem
  keyname: my_test_key
  securitygroup: default

  # Optionally configure default region
  # Use salt-cloud --list-locations <provider> to obtain valid regions
  #
  location: us-east-1
  availability_zone: us-east-1a

  #
  ssh_username: ec2-user

  # Optionally add an IAM profile
  iam_profile: 'arn:aws:iam::REDACTED:user/bluethundr'

  driver: ec2


my-ec2-us-east-private-ips:
  # Set up the location of the salt master
  #
  minion:
   master: salt.example.com
  # Specify whether to use public or private IP for deploy script.
  #
  # Valid options are:
  #     private_ips - The salt-master is also hosted with EC2
  #     public_ips - The salt-master is hosted outside of EC2
  #
  ssh_interface: private_ips

  # Optionally configure the Windows credential validation number of
  # retries and delay between retries.  This defaults to 10 retries
  # with a one second delay betwee retries
  win_deploy_auth_retries: 10
  win_deploy_auth_retry_delay: 1

  # Set the EC2 access credentials (see below)
  #
  id: "REDACTED" 
  key: "REDACTED"

  # Make sure this key is owned by root with permissions 0400.
  #
  private_key: /etc/salt/my_test_key.pem
  keyname: my_test_key

  # This one should NOT be specified if VPC was not configured in AWS to be
  # the default. It might cause an error message which says that network
  # interfaces and an instance-level security groups may not be specified
  # on the same request.
  #
  securitygroup: default

  # Optionally configure default region
  #
  location: us-east-1
  availability_zone: us-east-1a

  # Configure which user to use to run the deploy script. This setting is
  # dependent upon the AMI that is used to deploy. It is usually safer to
  # configure this individually in a profile, than globally. Typical users
  # are:
  #
  # Amazon Linux -> ec2-user
  # RHEL         -> ec2-user
  # CentOS       -> ec2-user
  # Ubuntu       -> ubuntu
  #
  ssh_username: ec2-user

  # Optionally add an IAM profile
  iam_profile: 'arn:aws:iam::REDACTED:user/bluethundr'

  driver: ec2

And this is my /etc/salt/cloud.profiles.d/aws_pofiles.conf

base_ec2:
  provider: my-ec2-us-east-public-ips
  image: ami-869a9cee
  size: t2.micro
  ssh_username: ec2-user

base_ec2_private:/et
  provider: my-ec2-us-east-private-ips
  image: ami-869a9cee
  size: t2.micro
  ssh_username: ec2-user

base_ec2_public:
  provider: my-ec2-us-east-public-ips
  image: ami-e565ba8c
  size: t2.micro
  ssh_username: ec2-user

base_ec2_db:
  provider: my-ec2-us-east-public-ips
  image: ami-e565ba8c
  size: m1.xlarge
  ssh_username: ec2-user
  volumes:
    - { size: 10, device: /dev/sdf }
    - { size: 10, device: /dev/sdg, type: io1, iops: 1000 }
    - { size: 10, device: /dev/sdh, type: io1, iops: 1000 }
    - { size: 10, device: /dev/sdi, tags: {"Environment": "production"} }
  # optionally add tags to profile:
  tag: {'Environment': 'production', 'Role': 'database'}
  # force grains to sync after install
  sync_after_install: grains

base_ec2_vpc:
  provider: my-ec2-us-east-public-ips
  image: ami-a73264ce
  size: m1.xlarge
  ssh_username: ec2-user
  script:  /etc/salt/cloud.deploy.d/user_data.sh
  network_interfaces:
    - DeviceIndex: 0
      PrivateIpAddresses:
        - Primary: True
      #auto assign public ip (not EIP)
      AssociatePublicIpAddress: True
      SubnetId: subnet-813d4bbf
      SecurityGroupId:
        - sg-750af413
  del_root_vol_on_destroy: True
  del_all_vol_on_destroy: True
  volumes:
    - { size: 10, device: /dev/sdf }
    - { size: 10, device: /dev/sdg, type: io1, iops: 1000 }
    - { size: 10, device: /dev/sdh, type: io1, iops: 1000 }
  tag: {'Environment': 'production', 'Role': 'database'}
  sync_after_install: grains

Here's some debug output of the command I'm trying to get working:

[root@salt:~] #salt-cloud -p base_ec2_public ops.example.com  -l debug
[DEBUG   ] Reading configuration from /etc/salt/cloud
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt.example.com
[DEBUG   ] Missing configuration file: /etc/salt/cloud.providers
[DEBUG   ] Including configuration from '/etc/salt/cloud.providers.d/aws.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers.d/aws.conf
[DEBUG   ] Missing configuration file: /etc/salt/cloud.profiles
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/aws_profiles.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/aws_profiles.conf
[DEBUG   ] Configuration file path: /etc/salt/cloud
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] salt-cloud starting
[DEBUG   ] Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.destroy: 'saltify.destroy' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_sizes: 'saltify.avail_sizes' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_images: 'saltify.avail_images' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_locations: 'saltify.avail_locations' is not available.
[DEBUG   ] LazyLoaded rackspace.reboot
[DEBUG   ] LazyLoaded openstack.list_locations
[DEBUG   ] LazyLoaded rackspace.list_locations
[DEBUG   ] Could not LazyLoad parallels.avail_sizes: 'parallels' __virtual__ returned False
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.destroy: 'saltify.destroy' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_sizes: 'saltify.avail_sizes' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_images: 'saltify.avail_images' is not available.
[DEBUG   ] Could not LazyLoad saltify.avail_locations: 'saltify.avail_locations' is not available.
[DEBUG   ] LazyLoaded rackspace.reboot
[DEBUG   ] LazyLoaded openstack.list_locations
[DEBUG   ] LazyLoaded rackspace.list_locations
[DEBUG   ] Using AWS endpoint: ec2.us-east-1.amazonaws.com
[DEBUG   ] AWS Request: https://ec2.us-east-1.amazonaws.com/?Action=DescribeInstances&Version=2014-10-01
[DEBUG   ] AWS Response Status Code: 401
[ERROR   ] AWS Response Status Code and Error: [401 401 Client Error: Unauthorized] {'Errors': {'Error': {'Message': 'AWS was not able to validate the provided acce
ss credentials', 'Code': 'AuthFailure'}}, 'RequestID': '0f483305-6cb2-4c09-ae2f-ec804fd3beea'}
[DEBUG   ] Failed to execute 'ec2.list_nodes()' while querying for running nodes: An error occurred while listing nodes: AWS was not able to validate the provided a
ccess credentials
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 2383, in run_parallel_map_providers_query
    cloud.clouds[data['fun']]()
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3496, in list_nodes
    nodes = list_nodes_full(get_location())
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3346, in list_nodes_full
    return _list_nodes_full(location)
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 3436, in _list_nodes_full
    instances['error']['Errors']['Error']['Message']
SaltCloudSystemExit: An error occurred while listing nodes: AWS was not able to validate the provided access credentials
[DEBUG   ] Generating minion keys for 'ops.jokefire.com'
[DEBUG   ] LazyLoaded cloud.fire_event
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event - data = {'profile': 'base_ec2_public', 'event': 'starting create', '_stamp': '2016-09-13T19:24:13.555913', 'name': 'ops.jokefire.com', 'pr
ovider': 'my-ec2-us-east-public-ips:ec2'}
[INFO    ] Creating Cloud VM ops.jokefire.com in us-east-1
[DEBUG   ] Using AWS endpoint: ec2.us-east-1.amazonaws.com
[DEBUG   ] AWS Request: https://ec2.us-east-1.amazonaws.com/?Action=DescribeAvailabilityZones&Filter.0.Name=region-name&Filter.0.Value.0=us-east-1&Version=2014-10-0
1
[DEBUG   ] AWS Response Status Code: 401
[ERROR   ] AWS Response Status Code and Error: [401 401 Client Error: Unauthorized] {'Errors': {'Error': {'Message': 'AWS was not able to validate the provided acce
ss credentials', 'Code': 'AuthFailure'}}, 'RequestID': 'e9912cf2-2e9b-496f-b607-4b9bae8b8938'}
[ERROR   ] There was a profile error: string indices must be integers, not str
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/cloud/cli.py", line 284, in run
    self.config.get('names')
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 1454, in run_profile
    ret[name] = self.create(vm_)
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 1284, in create
    output = self.clouds[func](vm_)
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 2512, in create
    data, vm_ = request_instance(vm_, location)
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 1742, in request_instance
    az_ = get_availability_zone(vm_)
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 1094, in get_availability_zone
    zones = _list_availability_zones(vm_)
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/ec2.py", line 1242, in _list_availability_zones
    ret[zone['zoneName']] = zone['zoneState']
TypeError: string indices must be integers, not str

Can someone take a stab and let me know why I'm getting auth failures? The redacted AWS keys were taken straight from the AWS interface and copied into the cloud.providers file.


Solution

  • It seems the EC2 credentials are not provided. You may need to check the Key/ID of the EC2 credentials, and their policy.

    For credentials, replace "REDACTED" string with your real key/ID.