EMR serverless- Pass jars in console
I'm new with EMR-serverless and I want to know how to pass, in a spark application, jar and packages as for example:
spark-submit --deploy-mode client --jars /usr/lib/hudi/hudi-spark3.3-bundle_2.12-0.11.1-amzn-0.jar,/usr/lib/hudi/hudi-utilities-bundle_2.12-0.11.1-amzn-0.jar ...
I want to set when I submit a job but I cannot find a way about how to do it.
Can someone help me with this, please?
Solution
When submitting a job to EMR Serverless in the console and you want to provide additional options to spark-submit, you can use the "Spark properties" section. Instead of --jars, you can use the spark.jars key and set the value appropriately.
Your Spark application will be a Python script or JAR file on S3 provided as the "Script location" aka entrypoint.
Also note that Hudi is available on the EMR Serverless image and there's some documentation on using Hudi with EMR Serverless.
- AWS CLI Create Default VPC
- Email address is not verified (AWS SES)
- RDS Proxy: PENDING_PROXY_CAPACITY and "DBProxy Target unavailable due to an internal error"
- mTLS on AWS ALBs across multiple regions
- How to debug a aws lambda function?
- AWS SAM retrieve secret value from Secret Manager with dynamic referencing
- Cloudfront redirect when signed cookies not present
- Is there a wildcard character in AWS EventBridge rules?
- Laravel file upload s3 Multipart
- Increase EC2 disk storage without losing any data
- AWS Lambda triggers SES in VPC
- Setting HTTP Headers with Lambda@Edge
- How to find an Amazon EC2 AMI if I only have the id?
- AWS Lambda - Can't set memory larger than 3008 MB
- Which one is the cheapest to use AWS RDS or my own database?
- AWS Glue Please verify role's TrustPolicy
- How to create event bus events with source 'aws.'?
- Terraform and AWS: No Configuration Files Found Error
- How to transfer the packets through NAT gateway instead of public IP?
- Is Aws well architected framework supports API's?
- Can't delete AWS internet Gateway
- Unable to Trigger Lambda function in AWS using Python code from my local setup
- How to login with AWS CLI using credentials profiles
- How to retrieve multiple endpoints using data "aws_vpc_endpoint" resource?
- When pushing to ECS, Docker image errors out with module not found error
- Is there a way to drop file extensions when using AWS CLI with --recursive?
- On-demand self-hosted AWS EC2 runner for GitHub Actions
- How to manage dev and prod environments in a SAM project with CloudFormation Stacks, API Gateway, Lambda, and other AWS services?
- EC2 Instance error telling me multiple IAM Roles are attached when I try to change it...?
- (NoSuchVersion) when calling the GetObject operation: The specified version does not exist