azurehttpsazure-application-gatewaymtlsssl-client-authentication

400 The SSL certificate error from Azure Application Gateway with mTLS setup by Terraform

I try to setup mTLS with an Azure Application Gateway. Unfortunately I always get an error

<html>
<head><title>400 The SSL certificate error</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The SSL certificate error</center>
<hr><center>Microsoft-Azure-Application-Gateway/v2</center>
</body>
</html>

In the gateway logs, I can see the connection attempts, but without any errors. curl or the browsers also don't provide any useful logs. I also checked with openssl verify -CAfile ca.pem client.pem, that the certificate signature matches the CA (result OK).

I set it up via Terraform:

Root CA

resource "tls_private_key" "gateway_mtls_root_ca_private_key" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

resource "tls_self_signed_cert" "gateway_mtls_root_ca" {
  private_key_pem = tls_private_key.gateway_mtls_root_ca_private_key.private_key_pem

  subject {
    common_name  = "root-ca"
    organization = "test"
  }

  validity_period_hours = 24 * 90 # 3 months for testing phase
  is_ca_certificate     = true

  allowed_uses = [
    "cert_signing",
    "crl_signing",
    "digital_signature"
  ]
}

Client Certificate

resource "tls_private_key" "gateway_mtls_client_cert_private_key" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

resource "tls_cert_request" "gateway_mtls_client_cert_request" {
  private_key_pem = tls_private_key.gateway_mtls_client_cert_private_key.private_key_pem

  subject {
    common_name  = "client"
    organization = "test"
  }
}

resource "tls_locally_signed_cert" "gateway_mtls_client_cert" {
  cert_request_pem   = tls_cert_request.gateway_mtls_client_cert_request.cert_request_pem
  ca_private_key_pem = tls_private_key.gateway_mtls_root_ca_private_key.private_key_pem
  ca_cert_pem        = tls_self_signed_cert.gateway_mtls_root_ca.cert_pem

  validity_period_hours = 24 * 30 # 1 month for testing

  allowed_uses = [
    "client_auth",
    "key_encipherment",
    "digital_signature",
  ]
}

Application Gateway

resource "azurerm_application_gateway" "container_gateway" {
  name                = "test-gateway"
  location            = var.resource_group_region
  resource_group_name = var.resource_group_name

  sku {
    name     = "Standard_v2"
    tier     = "Standard_v2"
    capacity = 2
  }

  identity {
    type = "UserAssigned"
    identity_ids = [
      azurerm_user_assigned_identity.gateway_identity.id
    ]
  }

  gateway_ip_configuration {
    name      = local.gateway_ip_config_name
    subnet_id = var.gateway_subnet_ids[0]
  }

  backend_address_pool {
    name         = local.gateway_backend_pool_name
    ip_addresses = var.container_group_ip_addresses
  }

  backend_http_settings {
    name                  = local.gateway_backend_settings
    cookie_based_affinity = "Disabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 20
  }

  frontend_ip_configuration {
    name                 = local.gateway_ip_config_name
    public_ip_address_id = azurerm_public_ip.container_pip.id
  }


  frontend_port {
    name = local.gateway_frontend_https_port_name
    port = 443
  }

  http_listener {
    name                           = local.gateway_https_listener_name
    frontend_ip_configuration_name = local.gateway_ip_config_name
    frontend_port_name             = local.gateway_frontend_https_port_name
    protocol                       = "Https"
    ssl_certificate_name           = local.gateway_ssl_certificate_name
    ssl_profile_name               = local.gateway_ssl_profile_name
  }

  request_routing_rule {
    name               = local.gateway_https_path_based_rules_name
    rule_type          = "PathBasedRouting"
    http_listener_name = local.gateway_https_listener_name
    url_path_map_name  = local.gateway_url_path_map_name

    priority = 1000
  }

  url_path_map {
    name                               = local.gateway_url_path_map_name
    default_backend_address_pool_name  = local.gateway_backend_pool_name
    default_backend_http_settings_name = local.gateway_backend_settings

    path_rule {
      name                       = "test"
      paths                      = ["/*"]
      backend_address_pool_name  = local.gateway_backend_pool_name
      backend_http_settings_name = local.gateway_backend_settings
    }
  }

  ssl_certificate {
    name                = local.gateway_ssl_certificate_name
    key_vault_secret_id = azurerm_key_vault_certificate.gateway_server_certificate.secret_id
  }

  ssl_policy {
    policy_type = "Predefined"
    policy_name = "AppGwSslPolicy20220101"
  }

  trusted_client_certificate {
    name = local.gateway_trusted_client_certificate_name
    data = tls_self_signed_cert.gateway_mtls_root_ca.cert_pem
  }

  ssl_profile {
    name                                 = local.gateway_ssl_profile_name
    trusted_client_certificate_names     = [local.gateway_trusted_client_certificate_name]
    verify_client_certificate_revocation = "OCSP"

    ssl_policy {
      policy_type = "Predefined"
      policy_name = "AppGwSslPolicy20220101"
    }
  }
}

Do you have an idea, what could be wrong in this setup? Or where I can find more useful logs to analyze it? Thank you!

Solution

  • Since we already worked on this issue on another thread, adding the solution below for community:

    As mentioned in the Application Gateway mutual authentication document,

    enter image description here

    Client certificate revocation can be enabled via REST API, ARM, Bicep, CLI, or PowerShell.

    Refer: https://learn.microsoft.com/en-us/azure/application-gateway/mutual-authentication-overview?tabs=powershell#certificate-revocation

    To verify OCSP revocation status has been evaluated for the client request, access logs will contain a property called "sslClientVerify", with the status of the OCSP response.

    enter image description here

    So, I requested you to check the Application gateway access logs and search for the property called sslClientVerify and check its status.

    Refer: https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-diagnostics#access-log

    https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-diagnostics#examples-of-optimizing-access-logs-using-workspace-transformations

    For more detailed troubleshooting, please refer the below document:

    https://learn.microsoft.com/en-us/azure/application-gateway/mutual-authentication-troubleshooting

    You checked the troubleshooting guide, but your case was not listed there, and you also could not find the sslClientVerify in the access logs.

    However, the hint to the OCSP was helpful in finding the root cause of your issue.

    You did configure the verify_client_certificate_revocation setting in Terraform to OCSP as it was the only allowed value, but you missed the part that it's optional and since your generated certificates don't have that configured, it was not working.

    You've now disabled it, and the client certificate is accepted by the Application Gateway.