I was following this compiled code (I don't know the compiler nor having the source code).
Sub1:
mov edx,[esp+04h]
and edx,00000300h
or edx,0000007Fh
mov [esp+06h],dx
fldcw word ptr [esp+06h]
retn
My understanding:
Sub1(4byte param1)
edx=param1&0x00000300|0x0000007F
higher 2 bytes of param1 = lower 2 bytes of edx
fldcw ???????
fldcw loads the control word. But what is the control word of a floating-point?
The result is stored into higher 2 bytes of param1. Am I right?
What could be the purpose of this subroutin?
FLDCW is an instruction that loads the 16-bit control word for the x87 FPU. The bit layout of the control word can be found on this Intel web page for example.
The lower eight bits of the control word contain the masks for the IEEE-754 defined exceptions. ORing 0x7F thus masks all floating-point exceptions as bits 6 and 7 are not used.
The upper eight bits of the control word contain the precision control in bits 8 and 9, and the rounding control in bits 10 and 11. By ANDing with 0x300 the precision control PC currently in force is passed through untouched, while the rounding control RC is forced to 0, which corresponds to the IEEE-754 rounding mode "round to nearest or even".
It is impossible to say what exactly the purpose of this function is. It is passed a 4-byte integer on the stack at [esp+4] which is removed by the caller, suggesting C calling conventions. The 4-byte integer passed in is presumably the saved previous value of the FPU control word, stored with FSTCW and zero extended from two to four bytes. The values forced for rounding control and exceptions masks suggest that this function is used to restore some compiler's math library defaults for the x87 control word, but there is no way of knowing this for sure without additional context.