powershellparallel-processingparallel.foreachforeach-objectpowershell-7.4

Passing variable to foreach-object -parallel which is with in start-job


I developed a PowerShell script which take care of health check related stuffs on all AWS EC2 instance. The one part of this script is powers ON the EC2 instances and wait till EC2 instance comes online. I have multiple AWS accounts and 10000s of EC2 instances against which this has to run. AWS Accounts are SSO integrated.

Main script has start-job script block and inside this block i have foreach-object -parallel block to power on the instances which has specific tag and name value. I have to pass the AWS profile and region to the foreach-object -parallel in order to start the ec2instances. but getting below error

"ForEach-Object: The value of the using variable '$using:profilename' cannot be retrieved because it has not been set in the local session."

foreach ($account in $awsaccounts){

 $AWSprofilename = $account.clientname
 $AWSClientRegion = $account.region

$params = @($AWSprofilename, $AWSClientRegion)
$job = Start-Job -Name $AWSprofilename -ScriptBlock {

param($profilename, $awsregion)

$hostnamefilter ="*ABC*","*XYZ*"

$HostnameFilterblock = [scriptblock]::Create( '$_.Hostname -like "' + ($hostnamefilter -join '" -or $_.Hostname -like "') + '"')

$stoppedinstancesreport = (Get-EC2Instance -ProfileName $profilename -Region $awsregion -Filter @{name = 'tag:Filter'; values = 'filtervalue'}).Instances | Where-Object {($_.State.Name -eq "Stopped")} | Select-Object InstanceID, @{N="Hostname";E={($_ | Select-object -ExpandProperty tags | Where-Object -Property Key -eq Name).Value}}, @{N= "State";E={$_.State.Name}}

$FilteredStoppedServers = $stoppedinstancesreport  | Where-Object $HostnameFilterblock

 $FilteredStoppedServers | ForEach-Object -Parallel {
  
   Start-EC2Instance -ProfileName $using:profilename -Region $using:awsregion -InstanceId $_.InstanceId -Confirm:$false
  
     while (((Get-EC2InstanceStatus -InstanceId $_.InstanceId -ProfileName $profilename -Region $awsregion.Status.Status.Value) -ne "OK")
     {
       Write-Host "wating for $($_.Hostname) to come online completely"
       Start-Sleep -Seconds 5
     }
    }
} -ArgumentList $params
} 

I tried passing variable using "$using:AWSprofilename " and "$using:AWSClientRegion " but still getting same error.


Solution

  • Setting the variable outside start-job worked for me, although its value is ignored (unless -parallel is taken out). You still need the matching param. Maybe someone else can explain it. I would just run everything within foreach-object -parallel.

    $profilename = 'whatever'
    start-job {
      param($profilename)
      foreach-object -parallel { $using:profilename }
    } -argumentlist joe | receive-job -wait -auto
    
    joe
    

    Note that foreach-object has an -asjob parameter.

    $profilename = 'joe'
    foreach-object -parallel { $using:profilename } -asjob | 
      receive-job -wait -auto
    
    joe
    

    Powershell 7 also has start-threadjob, which is faster and works as expected. (and can be installed as the Threadjob module in powershell 5.1)

    start-threadjob {
      param($profilename)
      foreach-object -parallel { $using:profilename }
    } -argumentlist joe | receive-job -wait -auto
    
    joe