goetcd3

etcd3 Go Client - How to paginate large sets of keys?

It seems that pagination through a large set of keys involve using WithFromKey() and WithLimit() options to Get(). For example if I want to fetch 2 pages of 10 items I would do something like:

opts := []clientv3.OpOption {
    clientv3.WithPrefix(),
    clientv3.WithSort(clientv3.SortByKey, clientv3.SortAscend),
    clientv3.WithLimit(10),
}

gr, err := kv.Get(ctx, "key", opts...)
if err != nil {
    log.Fatal(err)
}

fmt.Println("--- First page ---")
for _, item := range gr.Kvs {
    fmt.Println(string(item.Key), string(item.Key))
}

lastKey := string(gr.Kvs[len(gr.Kvs)-1].Value)

fmt.Println("--- Second page ---")
opts = append(opts, clientv3.WithFromKey())
gr, _ = kv.Get(ctx, lastKey, opts...)

// Skipping the first item, which the last item from from the previous Get
for _, item := range gr.Kvs[1:] {
    fmt.Println(string(item.Key), string(item.Value))
}

The problem is that the last key is fetched agains as the first item of the second page, which I need to skip and only 9 new items.

Is that the proper way to paginate or am I missing something?

Solution

  • Looking at the following clientv3 code, the next key can be computed by appending 0x00 to the last key.

    https://github.com/coreos/etcd/blob/88acced1cd7ad670001d1280b97de4fe7b647687/clientv3/op.go#L353

    That said, I prefer ignoring the first key from the 2nd and following pages.