Which Spark version is being used by Mapping Data Flows?
I ran into some issues recently which lead me to believe that the Mapping Data Flows used in my Synapse Analytics Workspace are still running Spark 2.4, which has reached EOL. I tried to find out which Spark version is being used and how to change it, without any success.
Is there a way to migrate to a newer version or to at least verify which Spark version is being used under the covers?
The concrete reasons for me caring revolve around the breaking changes regarding date/time handling in Spark 3.x.
You could figure out the version of spark through Monitoring output. Migration to Spark 3.x is already underway. And your subscription should be migrated to 3.x soon (if its not already migrated)
You can find the information in the output details of a data flow activity.
- Azure PowerShell command to list all Azure VM SKU Sizes enabled for Confidential Computing Azure ACC
- Create Azure TimerTrigger Durable Function in python
- Azure File Share: Cannot map network drive
- RBAC with bicep
- ADF expression to convert array to comma separated string
- How to get status of Azure machine learning service pipeline run using Rest Api?
- How do I deploy a Nuxt 3 solution to an Azure Node Web App
- Azure Blob Upload not working in ASP .Net Core Application
- How to clear a Cosmos DB database or delete all items using Azure portal
- Using script commands, how to add multiple scale rules to an Azure Container App?
- User Assigned Managed Identity: No User Assigned or Delegated Managed Identity found for specified ClientId/ResourceId/PrincipalId
- Generate resources dynamically from another group of dynamically generated resources in Terraform
- Column reordering in ADF
- Logic App AuthorizationFailure - vnet integration needed
- Azure blob, controlling filename on download
- Azure VNet peering with Private Link
- Get Private FQDNs, IPs, Names of the Private Endpoints for the Azure resources deployed in an Virtual Network using PowerShell Script
- How to configure appsettings on a auto-generated preview environment
- Using Azure DevOps, how do I migrate a release pipeline to code, similar to build pipeline yml?
- Root login Ubuntu VM on Azure
- Filtering azure devops release build based upon build tag with "Continuous deployment trigger"
- Azure WebApp autoscale
- How to handle long startup times in Azure App Service deployment
- Frontend not available when front and back on same Web App Azure
- Use Salesforce Connectors from Hosted Azure Logic App in VSCode?
- What is considered an ingestion request in Azure Data Explorer?
- Provisioning (SCIM) by Entra/Azure: Why can I not select the "groups" attribute in my attribute mapping?
- Azure function verbose trace logging to Application Insights
- Azure function app GraphServiceClient create list
- Cypress tests fail to run in parallel mode due to mismatched specs between different runs