erlangets

How to reduce the process 's memory usage when I convert a ets table to list in Erlang?

I got a serious problem when I try to convert a ets table to a list by ets:tab2list.

I have 1500 items in a ets table and most of them have a key with a value about 20K text data. The table size of the ets table is about 30 MB.

But when I try to convert this table to a list, it will cost about 400 MB's process memory and seems the process's memory usage is related to the table & item's size. When I try to convert a 200 MB table with 10000 items. It spend almost my VM's memory within one line and make my program terminated by the OS.

Is there other good way to handle my case? It seems ets is not good for my use case? or there are other good way to convert a ets table to a list without so much memory consumption?

Thanks you~~

Eric

ps: I use a table to queue the command list of a redis command and will do a batch insert to redis later to avoid busy network operation. Each item looks like 

{index, {["set", "key"], "text"}

and I need a batch command in the form of

[["set", "key1", "text1"], ["set", "key2", "text2"]]

In my case, the text is document about 20000 words

Solution

  • You can try to implement a batch aproach with: 

    ets:select(Tab, MatchSpec, Limit) -> 
    {[Match], Continuation} | '$end_of_table'

    or 

    ets:match(Tab, Pattern, Limit) ->
     {[Match], Continuation} | '$end_of_table'

    so your batch size its the Limit and you can delete de records after a successfull insert on redis.

    Regards.