javagoogle-app-enginememory-managementmemcachedmahout-recommender

App engine: How does memory work? (using Mahout with the app engine)

I am trying to use Mahout with the app engine. There are a few issues that arose but my main issue right now is: How can I make a recommendation if the model is too large to keep into memory?

I am using the memcache, but the Mahout datamodel is not serializable by default. I am serializing the FastIDSet instead, but then when I have to make a recommendation I have to get the FastIDSet from the memcache, use it as a parameter for the recommender, and then make the recommendation.

This would be fine but keeping both into memory gives me a heap space error, even though my database is not too large (19 MB).

I cannot do this in chunks because Mahout does expect to be initialized with a full data model.

Does anyone have experience with this problem?

Thanks and good day.

Solution

  • I figured it out, had to set my scaling in the appengine-web.xml.

    Still seeing about the speed of the recommender if I have to rebuild the model from the fastIDSET too often.