The machine restarts when it starts its own OS

I am developing my OS from scratch on C. I have a problem that I can't solve for 2-3 months. When I try to output a pixel, it is not output to the screen. And when I try to call 13h mode for 256 colors (and 320x200 resolution), the machine just reboots.

main.c (kernel):

#include "screen.h"

void kernel_main(){
    init_graphic_mode();
    while (1) {
        set_pixel(5, 5, (uint8_t)1);
    }
}

screen.c:

#include "screen.h"

void set_pixel(int y, int x, uint8_t color)
{
    if (x >= 0 && x < VGA_WIDTH && y >= 0 && y < VGA_HEIGHT) {
        uint8_t* screen = (uint8_t\*)VGA_ADDR;
        uint32_t offset = y * VGA_WIDTH + x;
        screen[offset] = color;
    }
}

void init_graphic_mode() {
    __asm__ __volatile__ (
        "mov $0x0013, %%ax\\n\\t"
        "int $0x10\\n\\t"
        : : : "eax"
    );
}

screen.h:

#ifndef MYSCREEN_H
#define MYSCREEN_H

#include <stdint.h>
#include <stdbool.h>

#define VGA_WIDTH 320
#define VGA_HEIGHT 200
#define SCREEN_SIZE (VGA_WIDTH * VGA_HEIGHT)
#define VGA_ADDR 0xA0000

void set_pixel(int y, int x, uint8_t color);
void init_graphic_mode();

#endif

The first time I had an error "Booting from hard disk... GRUB". I was able to solve it by transferring some files to the "isodir" folder.

When I solved the error, the pixel was not displayed on the screen (then I did not have the init_graphic_mode function and I used 16 colors of 640x480 - standard resolution). I was constantly changing the data types in set_pixel, but it didn't work.

After that I decided to make init_graphic_mode for mode 13h.

I didn't know who to write to, so I did the function via the Internet and via ChatGPT, but it didn't help me. I decided to focus on this version of the function:

void init_graphic_mode() {
    __asm__ __volatile__ (
        "mov $0x0013, %%ax\n\t"
        "int $0x10\n\t"
        : : : "eax"
    );
}

But the pixel was not output without this function (changing VGA_WIDTH and VGA_HEIGHT), and with the function, the machine generally reboots.

Solution

IMHO, you are trying to call the bios in this code

void init_graphic_mode() {
    __asm__ __volatile__ (
        "mov $0x0013, %%ax\\n\\t"
        "int $0x10\\n\\t"
        : : : "eax"
    );
}

BIOS calls need to run in intel 16bit real mode, while you are compiling for 64bit mode (which is not the starting mode of a real 64bit machine) int 0x10 is the bios video mode entry call, if you are still in real mode (because you are booting) this call is ok, but then the compiled code is not. You need to compile to intel 16bit real mode binary code, in order to be able to handle this. You got no errors because the assembly code you uses is also available in 64bit intel assembler (but meaning a different thing).