AWS Databricks pricing - should we also pay for EC2 instances seperately, in addition to DBU costs?
am trying to do some cost comparison between AWS Glue and Databricks hosted on an AWS environment. For the comparison, I have chosen m4.xlarge which is equivalent of 1 DPU in AWS Glue (4 vCPUs/16GB memory).
Assuming I have an pyspark job thats expected to run for 1 hour daily for 30 days with 5DPUs. My cost estimator as per AWS is as follows:
glue cost estimator : 5 DPUs x 30.00 hours x 0.44 USD per DPU-Hour = 66.00 USD (Apache Spark ETL job cost)
Databricks cost estimator : This gives an monthly estimate of 74 USD
Am concerned if we have to pay any EC2 cost to AWS for the 6 nodes in addition to this 73 USD. This is due to the note added in the estimate "This Pricing Calculator provides only an estimate of your Databricks cost. Your actual cost depends on your actual usage. Also, the estimated cost doesn't include cost for any required AWS services (e.g. EC2 instances)."
That will be an additional 36 USD approximately for this instance type/count, in addition to databricks cost. Can someone please clarify so we can make a decision to go with AWS Glue or Databricks. I know in databricks we can choose any instance type, but the question is if i pay EC2 cost seperately. Thanks
The answer is yes.
You should pay for all the infrastructure used directly by Databricks.
As mentioned in the footnote you added: This Pricing Calculator provides only an estimate of your Databricks cost. Your actual cost depends on your actual usage. Also, the estimated cost doesn't include the cost for any required AWS services (e.g. EC2 instances).
Think of it as a software license on top of hardware costs that you would pay anyway, whether you use the software or not.
This point was verified with Databricks Solution Architect that accompanies our company while implementing the Databricks solution.
- Terraform version error when deploying to AWS through jenkins?
- How do I add python libraries to an AWS lambda function for Alexa?
- How do I consume json logs inside Fargate using CDK?
- how to view aws log real time (like tail -f)
- AWS Simple AD create Directory User for AWS WorkSpaces via CLI or CloudFormation
- AWS SSO/AWS Opensearch SAML integration
- Amazon S3 - How to fix 'The request signature we calculated does not match the signature' error?
- Huge difference in CDN invocations vs lambda invocations
- BigQuery Transfer Service does not copy rows from S3
- Textract cannot read object from S3 when running on Lambda
- Can't access external MongoDB Atlas database from Elastic Beanstalk's EC2 instance
- How to disable application logs for EKS Cloudwatch Observability addon and other configurations
- Lambda layer's contents don’t match the S3 object after deploying with Terraform
- Why is a web server (i.e Nginx) REQUIRED for FastAPI but not Express API?
- How to disable an AWS region?
- AWS: Make a resource (e.g. a lambda) public in production but private in development
- How can I set permissions to a specific folder with Elastic Beanstalk using a C# ASP.NET Core app?
- Defining a serverless Aurora database in cloudformation
- AWS Launch Configuration error: The requested configuration is currently not supported
- how to refer to resources created with for_each in terraform
- Visual Studio 2022: AWS Toolkit broken, won't let me reinstall
- AWS role vs iam credential in duckdb HTTPFS call
- AWS BATCH - how to run more concurrent jobs
- Does an AWS routing table only affect outbound traffic?
- Weird ("too smart") Route 53 wildcard behavior
- bash script for AWS assume-role
- How do I translate an AWS S3 url into a bucket name for boto?
- Github actions Configure AWS credentials Error: The security token included in the request is invalid
- AWS API Gateway / AWS Network Load Balancer: request ends in 500 InternalServerErrorException, How to specify the port to forward to?
- AWS CLI - Saving the full output as a log file