vectorizationssesimdavx2inner-product

Inner product of two 16bit integer vectors with AVX2 in C++

I am searching for the most efficient way to multiply two aligned int16_t arrays whose length can be divided by 16 with AVX2.

After multiplication into a vector x I started with _mm256_extracti128_si256 and _mm256_castsi256_si128 to have the low and high part of x and added them with _mm_add_epi16.

I copied the result register and applied _mm_move_epi64 to the original register and added both again with _mm_add_epi16. Now, I think that I have:
-, -, -, -, x15+x7+x11+x3, x14+x6+x10+x2, x13+x5+x9+x1, x12+x4+x8+x0 within the 128bit register. But now I am stuck and don't know how to efficiently sum up the remaining four entries and how to extract the 16bit result.

Solution

  • Following the comments and hours of google my working solution:

    // AVX multiply
hash = 1;
start1 = std::chrono::high_resolution_clock::now();
for(int i=0; i<2000000; i++) {
    ZTYPE*  xv = al_entr1.c.data();
    ZTYPE*  yv = al_entr2.c.data();

    __m256i tres = _mm256_setzero_si256();
    for(int ii=0; ii < MAX_SIEVING_DIM; ii = ii+16/*8*/)
    {
        // editor's note: alignment required.  Use loadu for unaligned
        __m256i  xr = _mm256_load_si256((__m256i*)(xv+ii));
        __m256i  yr = _mm256_load_si256((__m256i*)(yv+ii));
        const __m256i tmp = _mm256_madd_epi16 (xr, yr);
        tres =  _mm256_add_epi32(tmp, tres);
    }

    // Reduction
    const __m128i x128 = _mm_add_epi32  ( _mm256_extracti128_si256(tres, 1), _mm256_castsi256_si128(tres));
    const __m128i x128_up = _mm_shuffle_epi32(x128, 78);
    const __m128i x64  = _mm_add_epi32  (x128, x128_up);
    const __m128i _x32 =  _mm_hadd_epi32(x64, x64);

    const int res = _mm_extract_epi32(_x32, 0);
    hash |= res;
}

finish1 = std::chrono::high_resolution_clock::now();
elapsed1 = finish1 - start1;
std::cout << "AVX multiply: " <<elapsed1.count() << " sec. (" << hash << ")" << std::endl;

    It is at least the fastest solution so far: