My goal is that I put some objects in a 3D space and call Render(),then this library could tell me the rendering position on 2d screen of each object.
Or like a 3d game engine,but I want to take over the rendering part,I want to render these objects myself.
Can any one help me?
What about XNAMath? That should be good enough to satisfy all your needs. In fact it is now a part of DirectX SDK.
Edit: If you want some practical insight into geometry for computer graphics, here's a comprehensive book that covers all the topics. And no matter what graphics you're designing, a game, cg demo or anything, math is the same everywhere.