cfile-iosortingexternal-sorting

Reading N integers in a single access from a file in C


Im trying to implement External Sorting in C.

I have to read N integers (fixed depending on main memory) from a file initially so that I can apply quicksort on them and then continue with the merging process.

I can think of these 2 ways:

  1. read N integers one by one from the file and put them in an array then sort them.
  2. read a bulk of data into a big char array and then reading integers from it using sscanf.

1st method is clearly slow and 2nd method is using lot of extra memory (but we have a limited main memory)

Is there any better way?


Solution

  • Don't try to be more clever than your OS, it probably supports some clever memory management functions, which will make your life easier, and your code faster.

    Assuming you are using a POSIX compliant operating system, then you can use mmap(2).

    1. Map your file into memory with mmap
    2. Sort it
    3. Sync it

    This way the OS will handle swapping out data when room is tight, and swap it in when you need it.