x86x86-6464-bitnasm

REX encoding for instructions with the VEX prefix

[x64]

is there some generic algorithm using which I can determine relevant registers for the rex byte? I'm specifically interested in the calculation of REX for instructions with the VEX prefix (so that I can determine ~R ~X ~B)?

I've tried using the first and last register operand as base and rx, but that only seems to work for stuff like andn, but fails on bextr.

for example -

andn ecx, r15d, edx 
bextr ecx, r15d, edx

respectively encode to: c4 e2 00 f2 ca c4 c2 68 f7 cf I'm confused about the second VEX byte, which is formed like this

~R          [X_______]
~X          [_X______]
~B          [__X_____]
map_select  [___XXXXX]

e2: 1110 0010 (R = 0, X = 0, B = 0) c2: 1100 0010 (R = 0, X = 0, B = 1)

why does bextr encode with B = 1 while andn doesn't?

Solution

  • Look at the field Op/En of instructions ANDN and BEXTR.

    ANDN r32a, r32b, r/m32   Bitwise AND of inverted r32b with r/m32, store result in r32a

    RVM says that the middle operand r32b is encoded as vvvv field of VEX3, while in

    BEXTR r32a, r/m32, r32b   Contiguous bitwise extract from r/m32 using r32b as control; store result in r32a.

    is the middle operand encoded as field r/m of ModRM (RMV).

    If you used one of the low seven registers instead of r15d, your problem would vanish and all bits R,X,B would be 0:

    |C4 E2 78 F2 CA|  andn ecx,eax,edx
|C1 E2 68 F7 C8|  bextr ecx,eax,edx

    There is no general algorithm, you have to consult the manual of instructions individually.