azurecloudpartitioningazure-data-exploreradx

Azure Data Explorer: How do Partitioning Policy and Merge Policy work?


In our ADX cluster there is no partitioning policy and no merge policy on a table, but the adx still creates extents. I am confused how this works and what the default settings are. Does anyone know this?

Further, how do a combination of partition keys work? For example I have

{
  "PartitionKeys": [
    {
      "ColumnName": "tenant_id",
      "Kind": "Hash",
      "Properties": {
        "Function": "XxHash64",
        "MaxPartitionCount": 128,
        "Seed": 1,
        "PartitionAssignmentMode": "Uniform"
      }
    },
    {
      "ColumnName": "timestamp",
      "Kind": "UniformRange",
      "Properties": {
        "Reference": "2021-01-01T00:00:00",
        "RangeSize": "7.00:00:00",
        "OverrideCreationTime": false
      }
    }
  ]
}

This will create on every new tenant_id a partition within the next 7 days? But a limit is 128? Or how should I read this?

And what is the benefit of building this small extents based on partition policy when there is a merge policy which merge the small extents to a bigger one? Why not building a bigger one instant?

Thanks


what i did: searching docs and try to goole


Solution

  • In our ADX cluster there is no partitioning policy and no merge policy on a table, but the adx still creates extents

    if you ingest data, extents will be created (either immediately - if you use batch ingestion - or eventually - if you use streaming ingestion).

    a partitioning policy ('null' by default, rarely required to define it) will change how extents are partitioned, and a merge policy (defined by default, rarely required to change it) impacts how extents are merged.

    how do a combination of partition keys work? This will create on every new tenant_id a partition within the next 7 days? But a limit is 128? Or how should I read this?

    given the policy you included, extents in the table will be partitioned as follows:

    what is the benefit of building this small extents based on partition policy when there is a merge policy which merge the small extents to a bigger one? Why not building a bigger one instant?

    I would recommend you go over the following posts/documents:

    1. Data partitioning in Kusto
    2. Extents overview
    3. Partitioning policy