kdb+k

Select a table from the inside of external select


I've seen a technique of use an update (mainly for side-effect of adding a new column, I guess) in a way of: update someFun each t from t. Is it good or bad practice to use such technique?

Some experiments:

t1:([]a:1 2);
t2:([]a:1 2;b:30 40);
update s:{(x`a)+x`b} each t2 from t1

Seems we can use different tables to do this, so I guessed we'll have 2x memory over-use.

But:

t:([]a:til 1000000;b:-1*til 1000000);
\ts:10 s0: update s:{(x`a)+x`b} each t from t;
4761 32778560

\ts:10 s1: update s:{(x`a)+x`b} each ([]a;b) from t;
4124 32778976

\ts:10 s2: update s:{x+y}'[a;b] from t;
1908 32778512

gives almost the same result for all cases for memory. I wonder why memory consumptions are the same?


Solution

  • In all examples you're 'eaching' over rows of the table & it seems the memory consumption is a result of building up the vector incrementally (multiple memory block allocations) rather than in one go. Use vector operations whenever possible

    q)n:5000000;t:([]a:til n;b:-1*til n)
    q)
    q)// each row
    q)\ts update s:{(x`a)+x`b} each t from t;
    1709 214218848
    q)v:n#0
    q)\ts {x}each v
    361 214218256
    q)
    q)// vector op
    q)\ts update s:sum a b from t;
    18 67109760
    q)\ts til n
    5 67109040