I am trying to understand how the underlying process differs if I call the os.listdir() function as compared to running ls in the terminal.
After much searching, I came across these 2 keypoints GNU ls waits until all entries are fetched before returning any of them. Python listing functions do not necessarily invoke GNU ls.
So,I want to find out if the python listing functions also exhibit similar behavior like waiting for all entries before returning any of them or return the entries as received.
os.listdir cannot return entries as it reads them. return doesn't work like that - a function can only return once, and it ends as soon as it returns. os.listdir returns a list of names, and it cannot add more data to that list after it returns.
For os.listdir to not output all its data at once, it would have to be designed differently. For example, it could return an iterator (like os.scandir does), and the iterator could read one directory entry on every iteration. Or it could return a queue.Queue and start a background thread that reads directory entries and puts their names on the queue one by one.
os.listdir was not designed like that. It was designed to read directory entries, put their names in a list, and return that list.