Persistent and transient EMR equivalent clusters in azure and HDInsight
I wanted to know if I create an HDInsight cluster on Azure is that fully reliable to work on. For example, I don't want to see after a while one of node is wiped out or deleted and I lose my data.
I know we would have two types of cluster on EMR like transient and persistent but I still have doubt even the persistent cluster could at some point lose nodes data.
Does this happen to Azure HDInsight as well? I would like to know opinion from those guys who have experiences on this?
Thanks
Azure HDInsight clusters are similar to Persistent clusters in EMR.
On-demand HDInsight Hadoop clusters are similar to transient clusters in EMR.
AWS to Azure services comparison:
| AWS Service | Azure Service | Description |
|---|---|---|
| EMR | Azure Data Explorer | Fully managed, low latency. distributed big data analytics platform to run complex queries across petabytes of data. |
| EMR | Databricks | Apache Spark-based analytics platform. |
| EMR | HDInsight | Managed Hadoop senhce. Deploy and manage Hadoop clusters in Azure. |
| EMR | Data Lake Storage | Massively scalable, secure data lake functionality built on Azure Blob Storage. |
Azure HDInsight follows a strong separation of compute and storage-- as such the recommendation is to store your data either in Azure Storage blobs and Azure Data Lake Store, or a combination of the two. Both provide an HDFS compatible file system that persists data even if the cluster is deleted.
The benefit of this approach is:
- The data is persistent, even after you delete your HDInsight cluster. This means it will also be available without any data transfer effort should you deploy a new cluster to perform additional processing.
- The costs for storing your data are predominalty driven by the volume of data stored and tranferred, which can be signficantly less than the costs for running a cluster.
- The data is available for multiple clusters to act upon.
For more details, refer Azure Storage overview in HDInsight and Use Azure storage with Azure HDInsight clusters
- Azure PowerShell command to list all Azure VM SKU Sizes enabled for Confidential Computing Azure ACC
- Create Azure TimerTrigger Durable Function in python
- Azure File Share: Cannot map network drive
- RBAC with bicep
- ADF expression to convert array to comma separated string
- How to get status of Azure machine learning service pipeline run using Rest Api?
- How do I deploy a Nuxt 3 solution to an Azure Node Web App
- Azure Blob Upload not working in ASP .Net Core Application
- How to clear a Cosmos DB database or delete all items using Azure portal
- Using script commands, how to add multiple scale rules to an Azure Container App?
- User Assigned Managed Identity: No User Assigned or Delegated Managed Identity found for specified ClientId/ResourceId/PrincipalId
- Generate resources dynamically from another group of dynamically generated resources in Terraform
- Column reordering in ADF
- Logic App AuthorizationFailure - vnet integration needed
- Azure blob, controlling filename on download
- Azure VNet peering with Private Link
- Get Private FQDNs, IPs, Names of the Private Endpoints for the Azure resources deployed in an Virtual Network using PowerShell Script
- How to configure appsettings on a auto-generated preview environment
- Using Azure DevOps, how do I migrate a release pipeline to code, similar to build pipeline yml?
- Root login Ubuntu VM on Azure
- Filtering azure devops release build based upon build tag with "Continuous deployment trigger"
- Azure WebApp autoscale
- How to handle long startup times in Azure App Service deployment
- Frontend not available when front and back on same Web App Azure
- Use Salesforce Connectors from Hosted Azure Logic App in VSCode?
- What is considered an ingestion request in Azure Data Explorer?
- Provisioning (SCIM) by Entra/Azure: Why can I not select the "groups" attribute in my attribute mapping?
- Azure function verbose trace logging to Application Insights
- Azure function app GraphServiceClient create list
- Cypress tests fail to run in parallel mode due to mismatched specs between different runs