azurepysparkazure-synapse-analytics

Azure Synapse Workspace error to many cores requested

I am new to Azure Workspace and I am trying to start a basic spark app.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, expr

# Initialize Spark session
spark = SparkSession.builder.appName("Optimized CSV Comparison").getOrCreate()

I get the following error which I dont understand as I am just using the small compute with 3-10 nodes.

enter image description here

this is the error I am getting

    InvalidHttpRequestToLivy: Your Spark job requested 40 vcores. 
However, the workspace has a 12 core limit. Try reducing the numbers of
 vcores requested or increasing your vcore quota. Quota can be increased using Azure Support request https://learn.microsoft.com/en-
us/azure/synapse-analytics/spark/apache-spark-concepts#workspace-level HTTP status code: 400
Solution

  • The VM / node type you've chosen "Small (4 vCores / 32GB)" has 4 cores and since you're selecting autoscale to a possible 10 VMs / nodes this could result in 4 * 10 = 40 cores.

    From the error message you have a 12 core limit.

    Try disabling the autoscale, and just have 1 or 2 nodes OR use autoscale with a minimum of 1 and maximum of 2 nodes.

    (Otherwise you can always ask to increase the quota using the link in the error message, but I expect you don't really need that in this case.)