Initialize serial port with x86 assembly

I want to use the serial port COM1 without using the BIOS interrupt 14h, and for this purpose I am following the tutorial at osdev but I have some problems during the initialization. (I am pretty new to asm and to bios related stuff so my code may be very wrong or may be there is a need to initialize other things before I can initialize the serial port)

My current code looks like this, and should be a direct translation of their C code.

%macro outb 2
    mov dx, %1
    mov al, %2
    out dx, al
%endmacro

%macro inb 1
    mov dx, %1
    in al, dx
%endmacro

    bits 16
    org 0x7c00 ;; set the origin at the start of the bootloader address space

    xor ax, ax
    mov dx, ax
    mov es, ax
    mov fs, ax
    mov gs, ax

serial_init:
    outb [com1+1], 0x00 ;; disable all interrupts
    outb [com1+3], 0x80 ;; enable DLAB (set baud rate divisor)
    outb [com1+0], 0x03 ;; Set divisor to 3 (lo byte) 38440 baud
    outb [com1+1], 0x00 ;;                  (hi byte)
    outb [com1+3], 0x03 ;; 8 bits, no parity, one stop bit
    outb [com1+2], 0xc7 ;; enable fifo, clear them, with 14-byte threshold
    outb [com1+4], 0x0b ;; IRQs enabled, RTS/DSR set
    outb [com1+4], 0x1e ;; set in loopback mode, test the serial chip
    outb [com1+0], 0xae ;; test serial chip

hang:
    jmp hang

com1:
    dw 0x3f8

    times 510-($-$$) db 0
    dw 0xaa55

I have some debugging primitives and everything, seems to go smoothly until the line outb [com1+0], 0x03 afterwards, if I read the line control register [com1+5] I get an error but I am not to sure how to interpret it (it seems to be an error related to stop bits, error 3)

Solution

Given the outb macro definition, NASM will expand your outb [com1+1], 0x00 macro invokation into:

mov dx, [com1+1]
mov al, 0x00
out dx, al

Because of the square brackets, the first instruction will load DX from memory, and sadly that does not contain the intended port address (0x03F9)! What you get is 0x0003, composed from the high byte of the word stored at com1 and the next byte in memory that happens to be 0 because of the times zero-padding.

In your defense, the wiki article https://wiki.osdev.org/Serial_Ports does use square brackets when it says that you should send data to e.g. [PORT + 1]. However those brackets have nothing to do with the same brackets used in the assembly programming language.

The C code snippet is clearer. It has define PORT 0x3f8 which in assembly becomes PORT equ 0x03F8. And the outb(PORT + 1, 0x00) instruction becomes outb PORT + 1, 0x00 in assembly. This time NASM will expand your outb macro into these 3 instructions:

mov  dx, PORT + 1    ; Same as `mov dx, 0x03F9`
mov  al, 0x00
out  dx, al

The C code given for reference:

#define PORT 0x3f8          // COM1
 
static int init_serial() {
   outb(PORT + 1, 0x00);    // Disable all interrupts
   outb(PORT + 3, 0x80);    // Enable DLAB (set baud rate divisor)
   outb(PORT + 0, 0x03);    // Set divisor to 3 (lo byte) 38400 baud
   outb(PORT + 1, 0x00);    //                  (hi byte)
   outb(PORT + 3, 0x03);    // 8 bits, no parity, one stop bit
   outb(PORT + 2, 0xC7);    // Enable FIFO, clear them, with 14-byte threshold
   outb(PORT + 4, 0x0B);    // IRQs enabled, RTS/DSR set
   outb(PORT + 4, 0x1E);    // Set in loopback mode, test the serial chip
   outb(PORT + 0, 0xAE);    // Test serial chip (send byte 0xAE and check if serial returns same byte)
 
   // Check if serial is faulty (i.e: not same byte as sent)
   if(inb(PORT + 0) != 0xAE) {
      return 1;
   }
 
   // If serial is not faulty set it in normal operation mode
   // (not-loopback with IRQs enabled and OUT#1 and OUT#2 bits enabled)
   outb(PORT + 4, 0x0F);
   return 0;
}

if I read the line control register [com1+5] ...

Accuracy will be key in programming this hardware. It's even true for programming in general.
The line control register (LCR) is at 03FB (PORT + 3).
The line status register (LSR) is at 03FD (PORT + 5).