I’m learning asynchronous programming in Python using asyncio and want to understand how low-level awaitables work, particularly for system events like serial port communication. For example, libraries like pyserial-asyncio allow non-blocking reads from a serial port, but I’m unclear on the underlying mechanics.
To read data asynchronously from a serial port, we need a method that yields control to the event loop until data arrives. Two approaches come to mind:
1. Polling with Sleep:
Use a loop to check for data availability, and if none exists, call await asyncio.sleep(x) to yield control. This feels inefficient. How do we choose the sleep duration ?
2. System Event Notification:
I guess that the OS can notify when data arrives in the serial buffer. But how do we integrate this with asyncio? How does an async method "await" such an OS-level event?
Beyond asyncio.sleep(), what low-level awaitable primitives exist in asyncio to wait for system events (e.g., I/O readiness on serial ports, HTTP requests, file operations)?
Are these primitives provided natively by asyncio, or do libraries like pyserial-asyncio implement custom logic using OS APIs ?
I’ve Tried to look at pyserial-asyncio’s source code but struggled to see how it interfaces with asyncio’s internals. If someone could redirect me toward some ressources and tutorial that do in-depth explanation of those mechanism it would be great.
In brief, asyncio rests on three pillars: coroutines, callback schedulers, and selectors.
When you call socket.read() in synchronous code, you are telling the operating system to "wake up this thread when the operation completes (successfully or unsuccessfully)". Exactly one wait, interrupted only by signal handlers. But in fact, you can also wait for multiple operations at a time using the select module or the selectors module. Just create a selector sel = selectors.DefaultSelector(), register a file object for selection via sel.register(fileobj, events), and call sel.select(). sel.select() is still a blocking system call, but now the thread will wake up when any operation you have registered is ready.
Okay, we can wait for multiple operations at a time, but using selectors directly is inconvenient because we have to manually manage the flow of execution for the entire thread. Especially to avoid being distracted by this feature, we can add an abstraction in the form of coroutines. Coroutines are slightly modified generators, and await is yield from with additional checks (see PEP 0492). With them, we can focus in an asynchronous function (actually a coroutine factory) on solving one specific task, and if we need to wait for something, implicitly register an event for selection and suspend the coroutine (task) via yield.
This begs the question of how to schedule execution. It is actually quite simple. We can create a priority queue (for example, via heapq or sched) to store information about callbacks: when to execute, what function to execute, and with what arguments. And in order to "wake up coroutine" we will just add a callback for coro.send() or coro.throw(). So we get a loop that executes callbacks when the queue is not empty and waits for sel.select() when it is empty). The step of each coroutine (usually between await statements) is simply the execution of the scheduled callback. This is how cooperative multitasking is implemented.
You might ask about how to handle threads, since we cannot interrupt a sel.select() wait. The trick is simple: we can create a non-blocking socket pair and register the first socket for reading. And then any write to the second socket from another thread will wake up the event loop. This is how loop.call_soon_threadsafe() works.
In summary, we have a simple (or complex) system in which the coroutines (tasks) ask the event loop to register a file object (or file descriptor) for selection, then switch to the event loop and thus fall asleep until the event loop is notified that the operation is complete and switches execution back to the coroutine in the order of the callback queue. The loop.call_at(), loop.call_later(), loop.call_soon() (and its closest relative loop.call_soon_threadsafe()) methods are responsible for scheduling callbacks. The loop.add_reader(), loop.add_writer(), loop.remove_reader(), loop.remove_writer() methods are responsible for monitoring. These two groups of methods are the basis for most of the rest of the event loop methods.
At a higher level, you usually work with futures (explicitly or implicitly). This is another abstraction with which, in particular, you can wait for tasks (coroutines themselves do not provide a wait method). The point of them boils down to a very simple thing: to schedule a switch to the coroutine when someone sets a result for the future. This is also a callback, and is added via future.add_done_callback().
Answering the question directly, pyserial-asyncio uses two approaches depending on the operating system used: polling for Windows, monitoring for all others. Both approaches are based on the principles I described above.