How to use an AWX template when each host has a different SSH Key

I am trying to setup AWX. I have created a template and it works fine against an inventory with a single host in it.

Each of my hosts has a different ssh key for the Ansible user, this is to reduce the blast radius if a key was exposed and is a requirement for our security policy.

AWX only allows you to add a single credential of each type to a template, so I can't add a second machine credential to the Template.

edit I believe the way to go is with Custom Credential Types. I have created a custom credential type for each of my hosts, then created a new credential based on each of these credential types. This is an example of one of my credential types

Input Configuration

fields:
  - id: username
    type: string
    label: Username
  - id: my_ssh_key
    type: string
    label: Private Key
    format: ssh_private_key
    secret: true
    multiline: true
required:
  - username
  - my_ssh_key

Injector configuration

file:
  template.my_key: '{{ my_ssh_key }}'
extra_vars:
  userlogin: '{{ username }}'
  my_ssh_key_file: '{{ tower.filename.my_key }}'

In my global Inventory Variables, I have the following

---
ansible_private_key_file: "{{ my_ssh_key_file }}"
ansible_user: "{{ userlogin }}"

In templates I have removed all Credentials entries.

When I then run the Template, I get the following errors

PLAY [all] *********************************************************************
TASK [Gathering Facts] *********************************************************
fatal: [192.168.100.172]: FAILED! => {"msg": "The field 'remote_user' has an invalid value, which includes an undefined variable. The error was: 'userlogin' is undefined. 'userlogin' is undefined. 'userlogin' is undefined. 'userlogin' is undefined"}
fatal: [192.168.100.180]: FAILED! => {"msg": "The field 'remote_user' has an invalid value, which includes an undefined variable. The error was: 'userlogin' is undefined. 'userlogin' is undefined. 'userlogin' is undefined. 'userlogin' is undefined"}
PLAY RECAP *********************************************************************
192.168.100.172            : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   
192.168.100.180            : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

If I hard code the username ansible in my Inventory Variables, then I get the error that the my_ssh_key_file variable is undefined.

Solution

Here's my setup for a custom SSH credential for a host.

Input configuration

    fields:
  - id: username
    type: string
    label: Username
  - id: my_ssh_key
    type: string
    label: Private Key
    format: ssh_private_key
    secret: true
    multiline: true
required:
  - username
  - my_ssh_key

Injector Configuration

extra_vars:
  my_ssh_key_file: '{{tower.filename}}'
  userlogin: '{{username}}'
  userpass: 'null'
file:
  template: '{{my_ssh_key}}'

Nothing in global variables but my inventory looks like this

[server]
server1_ip_server ansible_host=1.2.3.4
[server:vars]
ansible_user=exampleuser
ansible_connection=ssh
ansible_private_key_file="{{ my_ssh_key_file }}"

Playbook is pretty standard, just pick your the group/host and run tasks.