Global parameters does not appear in the ARMTemplateParametersForFactory.json

I can't figure out how to override global params in my CI/CD pipeline for ADF. I've added TEST_PARAMETER and when I deploy my adf with below task:

- task: AzureResourceManagerTemplateDeployment@3
retryCountOnTaskFailure: ${{ parameters.arm_template_deployment_retry_count }}
continueOnError: false
inputs:
  deploymentScope: 'Resource Group'
  azureResourceManagerConnection: ${{ parameters.deployment_service_connection }}
  subscriptionId: $(global.subscription.id)
  action: 'Create Or Update Resource Group'
  resourceGroupName: $(global.resourceGroup.name)
  location: $(global.location.primary)
  templateLocation: 'Linked artifact'
  csmFile: '$(Build.SourcesDirectory)/adf/ARMTemplateForFactory.json'
  csmParametersFile: '$(Build.SourcesDirectory)/adf/ARMTemplateParametersForFactory.json'
  overrideParameters: >
    -dataFactory_properties_globalParameters_TEST_PARAMETER_value "111"

I get the following errors:

There was an error while overriding 'dataFactory_properties_globalParameters_TEST_PARAMETER_value' parameter because of 'TypeError: Cannot read properties of undefined (reading 'type')', make sure it follows JavaScript Object Notation (JSON)

Starting template validation.
Deployment name is ARMTemplateForFactory-20240708-211103-e1c1
There were errors in your deployment. Error code: InvalidTemplate.
##[error]Deployment template validation failed: 'The following parameters were supplied, but do not correspond to any parameters defined in the template: 'dataFactory_properties_globalParameters_TEST_PARAMETER_value'. The parameters defined in the template are: 'factoryName, ...

but my parameter is not in the "parameters defined in the template"...

I've enabled Include global parameters in ARM template as suggested by Microsoft on Arm Template ADF screen.

I can see that global params are visible in ARMTemplateForFactory.json but... I still can't override them as they don't exist in ARMTemplateParametersForFactory.json. I can see that in the root folder of my adf I have file arm-template-parameters-definition.json and beginnign of this files looks like below

{
    "Microsoft.DataFactory/factories": {
        "properties": {
            "globalParameters": {
                "*": {
                    "value": "="
                }
            }
        },
        "location": "="
    },

I've also did an export from my ADF and below files had those global params included:

ARMTemplateForFactory.json
MyADF_ARMTemplateForFactory.json
MyADF_ARMTemplateParametersForFactory.json

In ARMTTemplateForFactory.json, I can see global param defined:

...
        },
        {
            "name": "[concat(parameters('factoryName'), '/default')]",
            "type": "Microsoft.DataFactory/factories/globalparameters",
            "apiVersion": "2018-06-01",
            "properties": {
                "TEST_PARAMETER": {
                    "type": "string",
                    "value": "\"123\""
                }
...

but tbh, I've expected to see it under parameters and not resources. Why it is being defined as resource instead parameter?

I can see my test global param defined as: "dataFactory_properties_globalParameters_TEST_PARAMETER_value": { "type": "string", "defaultValue": ""123"" },

but below file was still missing those global params:

ARMTemplateParametersForFactory.json

Publish_config.json also seems to reflect that global params should be included:

{"publishBranch":"my-publish-branch","includeGlobalParamsTemplate":true}

I feel lost as I'm not sure what I'm missing here... Is it ``arm-template-parameters-definition.json` configuration or maybe something else?

Update

Based on the further discussions, when adding new ADF global parameters then pushing the changes to GIT publish branch, the parameters node in the template file ARMTemplateForFactory.json will not be populated, which incurred the error during ARM deployment.

I could reproduce the issue if the arm-template-parameters-definition.json in ADF studio lacked the block below.

,
    "Microsoft.DataFactory/factories/globalparameters": {
        "properties": {
            "globalParameters": {
                "*": {
                    "value": "="
                }
            }
        }
    }

Adding those lines by navigating to ADF studio -> Manage -> ARM Template -> Edit parameter configuration and click on Edit can fix the issue.

Before publishing the templates to GIT Repos, we can Export and download the ARM template file to double check if the required parameters/resources are populated.

---

This was the expected error when you were trying to override an ARM template parameter value, but the parameter was not claimed in the parameters node in the ARM template file.

Since the ARM template file ARMTemplateForFactory.json and the parameters file ARMTemplateParametersForFactory.json should be updated by the operations in the ADF studio, here are the steps for your reference to make sure the Test_Global_Parameter is defined first, and then it can be overridden during ARM deployment.

1. To Include global parameters in ARM template and then Publish the change to the ARM template file in the adf_publish branch, will only add the modification in the resources node of the ARM template file but NOT populate any update in the parameters node;

   

2. We can add a new Test_Global_Parameter and then Publish the change again to make sure the parameters node in the ARMTemplateForFactory.json file is updated and the ARMTemplateParametersForFactory.json file is also synchronized with a default value for Test_Global_Parameter;

   

   

3. Then you may proceed to test the ARM deployment with the latest version of this ARM template and to override the Test_Global_Parameter value via a pipeline workflow;

   parameters:
   - name: arm_template_deployment_retry_count
     type: number
     default: 1
   
   - name: deployment_service_connection
     default: ARMSvcCnnSub0
   
   - name: source_adf
     default: azdatafactory-instance0
     values:
     - azdatafactory-instance0
     - azdatafactory-instance1
   
   variables:
     global.subscription.id: $(sub0)
     global.resourceGroup.name: rg-azdatafactory-instance0
     global.location.primary: southeastasia
     adf: ${{ parameters.source_adf }}
   
   trigger:
   - adf_publish
   
   pool:
     vmImage: ubuntu-latest
   
   steps:
   - task: AzureResourceManagerTemplateDeployment@3
     retryCountOnTaskFailure: ${{ parameters.arm_template_deployment_retry_count }}
     continueOnError: false
     inputs:
       deploymentScope: 'Resource Group'
       azureResourceManagerConnection: ${{ parameters.deployment_service_connection }}
       subscriptionId: $(global.subscription.id)
       action: 'Create Or Update Resource Group'
       resourceGroupName: $(global.resourceGroup.name)
       location: $(global.location.primary)
       templateLocation: 'Linked artifact'
       csmFile: '$(Build.SourcesDirectory)/$(adf)/ARMTemplateForFactory.json'
       csmParametersFile: '$(Build.SourcesDirectory)/$(adf)/ARMTemplateParametersForFactory.json'
       overrideParameters: >
         -default_properties_Test_Global_Parameter_value "111"
       # -factoryName "azdatafactory-newinstance"