Execute multiple notebooks in parallel in pyspark databricks
Question is simple:
master_dim.py calls dim_1.py and dim_2.py to execute in parallel. Is this possible in databricks pyspark?
Below image is explaning what am trying to do, it errors for some reason, am i missing something here?
Solution
Just for others in case they are after how it worked:
from multiprocessing.pool import ThreadPool
pool = ThreadPool(5)
notebooks = ['dim_1', 'dim_2']
pool.map(lambda path: dbutils.notebook.run("/Test/Threading/"+path, timeout_seconds= 60, arguments={"input-data": path}),notebooks)
- Get last modified object from S3 using AWS CLI
- "host not allowed" error when deploying a play framework application to Amazon AWS with Boxfuse
- Is there a way to use PyGithub on AWS Lambda
- Neptune throws Bad handshake error while connecting to a IAM Enabled Neptune Instance
- How to extract files from a zip archive in S3
- How do I use the aws cli to set permissions on files in an S3 bucket?
- Nextflow process running in AWS with docker container can't find program in $PATH, although program is there
- How do I setup and use Laravel Scheduling on AWS Elastic Beanstalk?
- Where does docker store it's temp files during extraction?
- Which is the best way on AWS to set up a CI/CD of a Django app from GitHub?
- Make VPC creation optional
- How to point ApiGateway to a specific Lambda alias
- AWS s3 V3 Javascript SDK stream file from bucket (GetObjectCommand)
- Can we setup third Party SSL certificate AWS lightsail loadbalancer
- A proper way to use AWS Redis Cluster with Celery
- How to rollback to previous version in Amazon S3 bucket?
- Why does S3.deleteObject not fail when the specified key doesn't exist?
- I can't delete AWS appconfig configuration and environment
- How to run Anaconda Python on sudo
- `aws ecs execute-command` results in `TargetNotConnectedException` `The execute command failed due to an internal error`
- AWS authorizer returns 500, message: null, with AuthorizerConfigurationException error in response
- Call S3 pre-signed URL with postman
- Terraform error when module has a count on the input that is not yet created
- Is there a way to point my machine's kubectl to a K8s cluster hosted on EC2 Instances?
- MWAA CLI - Stop Dags Execution
- Lambda@Edge not logging on cloudfront request
- User-data scripts is not running on my custom AMI, but working in standard Amazon linux
- Run docker commands in AWS Lambda Function
- Django ALLOWED_HOSTS for Amazon ELB
- CodeDeploy blue/green ECS fargate error: Invalid arn syntax