How to create composite indexes in datastore to filter with multiple attributes in entity

We are using Google Datastore for our dashboard. In dashboard we provide filtering option for endusers.

Lets say we have an datastore kind whose structure is:

{
    'attribute1': 'val1',
    'attribute2': 'val2',
    'attribute3': 'val3',
    'timestamp': 123456789,

}

There is a use case where we need to filter data in specific time period, with combinations of different other attributes.

What is the best way to create composite index to achieve this capability in Datastore?

Any thoughts?

Cloud Datastore provides two types of indexes

Built-in indexes -

These are the indexes that Cloud Datastore automatically creates for each property of each entity kind. These built-in indexes are single property indexes and are suitable for simple queries.

Composite indexes -

Composite indexes are manual indexes which are built by the user and not by Cloud Datastore automatically. These are multi property indexes. Composite indexes are needed for complex queries which involves filtering the data using two or more properties. To build a composite index one needs to configure an index.yaml file and then create it by running the following command -

gcloud datastore indexes create ~/path/to/index.yaml/file

Now coming to your use case -

As you want to filter the data based on more than two properties you can not use built-in indexes. So you have to use composite or manual index. For that you need to define an index configuration file and deploy that as mentioned above. You wanted to filter the data based on a specific time period in combination with one or more attributes. So for use cases the following index configuration file should work well.

index.yaml

indexes:
- kind: demo
 properties:
 - name: attribute1
   direction: asc
 - name: attribute2
   direction: asc
 - name: attribute3
   direction: asc
 - name: timestamp
   direction: asc

In the above configuration file the direction property is optional and if you don’t specify it it will take it as ascending(asc) by default. For descending sort order you can specify it as desc. This will work for filtering the data by combination of two or more properties which are mentioned in the configuration file. You may go through this page to know more about indexing in Cloud Datastore.

Please note that index-based query mechanism supports a wide range of queries and is suitable for most of the applications. Still there are some restrictions or limitations on Datastore query. I would suggest you to go through this page to know more about the restrictions while querying in Cloud Datastore.