I would like to create my own database server for educational purposes. My problem is, I cannot really find information about how these systems work. Now, I can create databases and tables and get data from it with basic select statements.
But, when a client connects to the database server, how should I send the result of a query back to the client? What format should be used to get good performance? What other things may need to be taken care of, given the specific scenario?
There are many options, depending on your needs.
JSON would allow almost any kind of client to use your database. It's also very easy to create but the format is somewhat verbose.
If you need a compact format, look at Google's Protocol Buffers which has a text (= readable) and a binary form (= compact). Many languages support protobuf but not as many as JSON (yet).
I suggest not to try to come up with your own format; there are too many already and the task looks more simple than it is. If you make a mistake, it will be hard to fix, especially when others start to use your work.