I am learning NASM and currently unsure what values should be used for the flags argument for open() as shown in the man page. I'm mainly asking this for clarification so I don't mess something up, and apologies if it's not appropriate for the site.
I read the man page, but couldn't find if the under-the-hood values listed, but since it and this SO answer said they are meant to be bitwise ORed together, I assumed they acted as bit flags (probably should have gathered that from the name). After looking here, I found them in "/usr/include/asm-generic/fcntl.h" on my computer (Ubuntu). This section with the flags looks like this, omitting some lines:
#define O_ACCMODE 00000003
#define O_RDONLY 00000000
#define O_WRONLY 00000001
#define O_RDWR 00000002
#define O_CREAT 00000100 /* not fcntl */
#define O_EXCL 00000200 /* not fcntl */
#define O_NOCTTY 00000400 /* not fcntl */
#define O_TRUNC 00001000 /* not fcntl */
#define O_APPEND 00002000
#define O_NONBLOCK 00004000
#define O_DSYNC 00010000 /* used to be O_SYNC, see below */
#define O_DIRECT 00040000 /* direct disk access hint */
#define O_LARGEFILE 00100000
#define O_DIRECTORY 00200000 /* must be a directory */
#define O_NOFOLLOW 00400000 /* don't follow links */
#define O_NOATIME 01000000
#define O_CLOEXEC 02000000 /* set close_on_exec */
#define __O_SYNC 04000000
#define O_SYNC (__O_SYNC|O_DSYNC)
#define O_PATH 010000000
#define __O_TMPFILE 020000000
#define O_TMPFILE (__O_TMPFILE | O_DIRECTORY)
I saw that a leading zero means the values is in base-8, so I wrote them like this in binary (I'm not sure the standard way of writing in binary), in order of value:
O_RDONLY = 0b 0 /* was 0o0 */
O_WRONLY = 0b 1 /* was 0o1 */
O_RDWR = 0b 10 /* was 0o2 */
O_ACCMODE = 0b 11 /* was 0o3 */
O_CREAT = 0b 100 0000 /* was 0o100 */
O_EXCL = 0b 1000 0000 /* was 0o200 */
O_NOCTTY = 0b 1 0000 0000 /* was 0o400 */
O_TRUNC = 0b 10 0000 0000 /* was 0o1000 */
O_APPEND = 0b 100 0000 0000 /* was 0o2000 */
O_NONBLOCK = 0b 1000 0000 0000 /* was 0o4000 */
O_DSYNC = 0b 1 0000 0000 0000 /* was 0o10000 */
O_DIRECT = 0b 100 0000 0000 0000 /* was 0o40000 */
O_LARGEFILE = 0b 1000 0000 0000 0000 /* was 0o100000 */
O_DIRECTORY = 0b 1 0000 0000 0000 0000 /* was 0o200000 */
O_NOFOLLOW = 0b 10 0000 0000 0000 0000 /* was 0o400000 */
O_NOATIME = 0b 100 0000 0000 0000 0000 /* was 0o1000000 */
O_CLOEXEC = 0b 1000 0000 0000 0000 0000 /* was 0o2000000 */
O_SYNC = 0b 10 0000 0000 0000 0000 0000 /* was 0o10000000 */
O_TMPFILE = 0b 100 0000 0000 0000 0000 0000 /* was 0o20000000 */
I assume some have denoting values on the same bits so they aren't used at the same time. Did I do something wrong in this process, or are these the correct values for using them as flags for the open() syscall?
These are the correct values on your system. There is no guarantee that they'll be the same on another system. That's why you use headers that provide the correct numbers for your system.
As a side-note, this assumption is wrong:
I assume some have denoting values on the same bits so they aren't used at the same time.
They're ensuring that they are passed at the same time, it can't do a thing to ensure they're not passed the same time (e.g. no one actually passes __O_SYNC, they pass O_SYNC, and there is no way for the library to distinguish passing O_SYNC | O_DSYNC from O_SYNC in the first place).
For the specific overlapping flags:
O_ACCMODE is not a flag you pass, it's a mask used to extract the portion of the bits indicating the access mode. It's not relevant to you, but it's helpful for the implementers of open to be able to mask down to those specific bits (or invert the mask and mask down to the other bits) to separate the access mode (read, write, or read-write) from all the other bits indicating non-access-related tweaks to behavior.
O_SYNC is composed of two bits (its own unique bit, and O_DSYNC), because O_SYNC's behavior is a superset of O_DSYNC (O_DSYNC ensures all data is synced after each write, O_SYNC also ensures the metadata is synced), so anything O_DSYNC should be doing should also be done by O_SYNC, and it's convenient to set both bits for the logic in some parts of the code.
O_TMPFILE including O_DIRECTORY is a weird one, because you're not actually opening a directory logically, but any call using O_TMPFILE must pass it the path to an existing directory, not a file, so presumably the implementers found it convenient for their sanity checking to have the checks for "they passed us a directory, not a file name" done the same way for opening a directory as for opening a tempfile in a directory, and the flag secretly including O_DIRECTORY makes those checks easier.