I want to know if an executable supports the common security protections such as NX flag, stack cookies or ASLR. It seems ASLR is set at the OS level but how do you know it is enabled? On Windows some executable do not support ASLR so I was wondering how you can determine this on Mac OS X.
First of all ALSR used in OSX 10.6 and below did not randomize all regions of memory. As far as I know ASLR is enabled for all running executables. This is very easy to test for, just fire up a debugger set a break point and record any memory address on the stack. Restart the application and see if that same variable has the same memory address.
I think in OSX 10.7 they started randomizing the dynamic linker. Which linux, bsd, and even windows systems have been doing for a number of years.
For OSX, linked libraries ASLR can be tested for using executing export DYLD_PRINT_SEGMENTS=1 and then running a command. The TEXT memory region is the base address for the library. Run this command twice against any binary. If the base address is different between the two execution then ASLR's dirty work is to blame.
Stack cookies are an entirely different ballgame. This is a compiler level protection and will vary based on the application. Modern versions of GCC should default to stack carnies enabled. Again you should consult your debugger to see if a specific application is using canaries. Just examine the stack frame of any function to see if there is a random value inserted between the locally declared variables and the return address.
As far as the NX flag goes, you should assume any system made after 1999 uses this trivial form of protection. But, this is by far the most simple protection for you to bypass, just ret-to-libc or employ an ROP chain (because of aslr).