javaalgorithmoptimizationjakarta-eefetching-strategy

How does Hibernate's batch-fetching algorithm work?


I found this description of the batch-fetching algorithm in "Manning - Java Persistence with Hibernate":

What is the real batch-fetching algorithm? (...) Imagine a batch size of 20 and a total number of 119 uninitialized proxies that have to be loaded in batches. At startup time, Hibernate reads the mapping metadata and creates 11 batch loaders internally. Each loader knows how many proxies it can initialize: 20, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1. The goal is to minimize the memory consumption for loader creation and to create enough loaders that every possible batch fetch can be produced. Another goal is to minimize the number of SQL SELECTs, obviously. To initialize 119 proxies Hibernate executes seven batches (you probably expected six, because 6 x 20 > 119). The batch loaders that are applied are five times 20, one time 10, and one time 9, automatically selected by Hibernate.

but I still don't understand how it works.

  1. Why 11 batch loaders ?
  2. Why batch loaders can initialize: 20, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1 proxies ?

If anybody could present a step by step algorithm ... :)


Solution

  • I couldn't find any information on the web about how hibernate handles batch loading, but judging from your information, one could guess the following:

    Why 11 batch loaders?

    With a batch size of 20, if you want to minimize the number of loaders required for any combination of proxies, there are basically two options:

    Example: for batch size 40, you would end up with loaders for 40,20,10,9,8,7,6,5,4,3,2,1 loaders.

    1. If you have 33 uninitialized proxies, you can use the following loaders: 20, 10, 3
    2. If you have 119 uninitialized proxies, you can use the following loaders, 40(x2), 20, 10, 9
    3. ...

    Why batch loaders can initialize: 20, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1 proxies ? I think the hibernate team chose this as a balance between the number of loaders required for loading a "common" number N of uninitialized proxies and memory consumption. The could have created a loader for every N between 0 and batch_size, but I suspect that the loaders have a considerable memory footprint so this is a tradeoff. The algorithm can be something like this (educated guess):

    1. n = batch_size; while (n > 10)

      1.1. loader(n); n = n / 2

    2. for n = 0..10 create loader(n)