vectorizationssesimdavx2inner-product

Inner product of two 16bit integer vectors with AVX2 in C++


I am searching for the most efficient way to multiply two aligned int16_t arrays whose length can be divided by 16 with AVX2.

After multiplication into a vector x I started with _mm256_extracti128_si256 and _mm256_castsi256_si128 to have the low and high part of x and added them with _mm_add_epi16.

I copied the result register and applied _mm_move_epi64 to the original register and added both again with _mm_add_epi16. Now, I think that I have:
-, -, -, -, x15+x7+x11+x3, x14+x6+x10+x2, x13+x5+x9+x1, x12+x4+x8+x0 within the 128bit register. But now I am stuck and don't know how to efficiently sum up the remaining four entries and how to extract the 16bit result.


Solution

  • Following the comments and hours of google my working solution:

    // AVX multiply
    hash = 1;
    start1 = std::chrono::high_resolution_clock::now();
    for(int i=0; i<2000000; i++) {
        ZTYPE*  xv = al_entr1.c.data();
        ZTYPE*  yv = al_entr2.c.data();
    
        __m256i tres = _mm256_setzero_si256();
        for(int ii=0; ii < MAX_SIEVING_DIM; ii = ii+16/*8*/)
        {
            // editor's note: alignment required.  Use loadu for unaligned
            __m256i  xr = _mm256_load_si256((__m256i*)(xv+ii));
            __m256i  yr = _mm256_load_si256((__m256i*)(yv+ii));
            const __m256i tmp = _mm256_madd_epi16 (xr, yr);
            tres =  _mm256_add_epi32(tmp, tres);
        }
    
        // Reduction
        const __m128i x128 = _mm_add_epi32  ( _mm256_extracti128_si256(tres, 1), _mm256_castsi256_si128(tres));
        const __m128i x128_up = _mm_shuffle_epi32(x128, 78);
        const __m128i x64  = _mm_add_epi32  (x128, x128_up);
        const __m128i _x32 =  _mm_hadd_epi32(x64, x64);
    
        const int res = _mm_extract_epi32(_x32, 0);
        hash |= res;
    }
    
    finish1 = std::chrono::high_resolution_clock::now();
    elapsed1 = finish1 - start1;
    std::cout << "AVX multiply: " <<elapsed1.count() << " sec. (" << hash << ")" << std::endl;
    

    It is at least the fastest solution so far: