I managed to get alpha working, but i want to change alpha only on one texture. Currently it works only with both. I set depth test and blend factors:
context3D.setDepthTest(false, Context3DCompareMode.LESS);
context3D.setBlendFactors(Context3DBlendFactor.SOURCE_ALPHA, Context3DBlendFactor.ONE_MINUS_SOURCE_ALPHA);
I haven't still figured out, what is exactly source and destionation, any clarification would be nice. Vertex data contains
[x, y, z, u, v, 0|1]
Set vertex registers
context3D.setVertexBufferAt(0, m_vertexBuffer, 0, Context3DVertexBufferFormat.FLOAT_3);
context3D.setVertexBufferAt(1, m_vertexBuffer, 3, Context3DVertexBufferFormat.FLOAT_2);
context3D.setVertexBufferAt(2, m_vertexBuffer, 5, Context3DVertexBufferFormat.FLOAT_1);
Set fragment constant
context3D.setProgramConstantsFromVector(Context3DProgramType.FRAGMENT, 0, Vector.<Number>([textureMul, 0, 0, 0]));
Vertex shader
m44 op, va0, vc0
mov v0, va1
mov v1, va2
Fragment shader
tex ft0, v0, fs0 <2d,clamp,linear>
sub ft0.w, ft0.w, fc0.x
add ft0.w, ft0.w, v1
mov oc, ft0
Where fc0.x is number between 0 and 1. And v1 is either 0 or 1.
Thank god for Windows for PIX program (comes with Microsoft DirectX SDK). I understood how to use properly FLOAT. Since i pass v1 as FLOAT_1, i must use v1.x, so right fragment shader would be
tex ft0, v0, fs0 <2d,clamp,linear>
sub ft0.w, ft0.w, fc0.x
add ft0.w, ft0.w, v1.x
mov oc, ft0