javagoogle-app-enginecallbackmapreducelow-level-api

Is DatastoreMutationPool flushed to the datastore before the callback?


I'm using mapreduce and I need to persist some entities when they are not in the datastore. I add the new entities to a DatastoreMutationPool so these entities could be persisted with batched calls. When the mapreduce ends a callback function is invoked. The callback function will use some of these entities. My question is, will all they entities be flushed to the datastore before the callback function is invoked or they can still be in the DatastoreMutationPool but not in the datastore.

Thanks.

Example of mapper:

public class MyMapper extends AppEngineMapper<Key, Entity, NullWritable, NullWritable> {
     @Override
     public void map(Key key, Entity value, Context context) {
         ...
         DatastoreMutationPool mutationPool = this.getAppEngineContext(context).getMutationPool();
     mutationPool.put(entity);
     }
}

Example of callback:

@RequestMapping(value="/callback/function",method=RequestMethod.POST)
public void callback(@RequestParam("job_id") String jobIdName){
    JobID jobId = JobID.forName(jobIdName);
    DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
    //search for some entities persisted in the mapper
    ...
}

Solution

  • The mutation pool is per-mapper, while the callback is run outside the mappers once the mapreduce completes. As a result, you can expect that all the mutation pools will have been flushed by the time your callback is run.