File systems provide a mechanism for categorizing (and thus navigating) data on a disk. This makes sense to me. If I want to find some "group" of data, I don't want to have to remember byte offsets myself. I would rather have some look up system that I can dynamically navigate.
However, I don't understand why different file systems must exist. For example, why NTFS, FAT16/32, EXT?
Why should different operating systems (Linux, Windows, etc.) rely on different methods for organizing data on disk?
I think a more appropriate question (and the question you'd like answered) is "Why do multiple file systems exist?". The answer depends on the particular file system, but in many cases it comes down to one (or a mix of) of three reasons:
The original FAT file system was introduced in the late 1970s. In many ways, FAT is great: it has a low memory footprint, and simple design. IIRC, it's still used in embedded systems to this date.
The FAT family of file systems comprises of the original 8-bit FAT, FAT12, FAT16, and FAT32. (There are several other versions, but they're not relevant to this answer.) There were several feature-differences between each version of the FAT file systems, some of which demonstrate the motivation for creating a new version. For example, in moving from 8-bit FAT to FAT12:
None of these features individually were likely the motivation for the creation of FAT12, but together these features are a clear win over 8-bit FAT. Refer to the FAT Wikipedia page for a more complete list of differences.
Before discussing NTFS, we should look at its predecessor: HPFS. The simple design of FAT turned out to be a problem: it constrained what features FAT could offer, and how it performed. HPFS was created to address the shortcomings of FAT. For example, HPFS provide several features FAT could not:
- Support for mixed case file names, in different code pages
- More efficient use of disk space (files are not stored using multiple-sector clusters but on a per-sector basis)
- An internal architecture that keeps related items close to each other on the disk volume
- Separate datestamps for last modification, last access, and creation (as opposed to last-modification-only datestamp in then-times implementations of FAT)
- Root directory located at the midpoint, rather than at the beginning of the disk, for faster average access
That should be compelling enough to demonstrate why HPFS was created, but how does NTFS fit into the picture? HPFS was a joint project by Microsoft and IBM. Due to several differences in opinion, they separated, and Microsoft created NTFS. This is another reason new file systems are created: difference in opinion. There's nothing inherently wrong with this, but it does have the side effect of occasionally fragmenting projects.
As with NTFS, we need to examine the predecessor of ext to understand why it was created. The predecessor of ext is the MINIX file system. MINIX was created for teaching purposes, so it was simple and elided several complex features the UNIX file system offered. The first file system supported by Linux was the MINIX filesystem. The simplicity of the MINIX file system soon became an issue:
MINIX restricted filename lengths to 14 characters (30 in later versions), it limited partitions to 64 megabytes, and the file system was designed for teaching purposes, not performance.
And thus, the extended file system (ie. ext) was created to address the shortcomings of the MINIX file system.
In a similar vain, ext2 was created to address the shortcomings of ext, and so on. For example, ext2 added three separate timestamps (atime, ctime, and mtime), ext3 adding journaling, and ext4 extended storage limits. These were all breaking changes which required a "new" file system. They weren't the only changes between versions, but these changes demonstrate why creating another file system was necessary.
Several file systems are widely used today. Apple File System (APFS) on Apple devices, NTFS on Windows devices, and several different file systems on Linux. Why do different operating systems use different file systems? For Linux, the reason is obvious: Linux needed an open source file system. That's why it initially used the MINIX file system.
For Windows and Apple devices, the difference is more, shall we say, political. Microsoft created NTFS to address the issues it thought were important, and Apple created APFS to address the issues it thought were important. Commercial OS vendors also create their own file systems for product differentiation.
We can kinda see why different OSs use different file systems, but several file systems are actively in use on Linux alone, eg. ext4, Btrfs, ZFS, XFS, and F2FS. What gives?
Linux is a different environment. The Linux kernel source is openly available, and can be modified, booted, and tested by any user. So, if one file system does not support the features you want, or offer the performance you need, you can create a new file system (which is, of course, easier said than done). For example,