I've read a lot of blog posts and SO-questions about Uniform Type Identifiers and how OS X handles file types. However, there are still some things I just don't get:
How are UTIs created by the system for each file? As a developer I passively declare a UTI for my file type but the system is responsible to assign the UTI for each matching file. My current impression is that UTIs are created on-the-fly by the Finder according to the file extension.
Where are UTIs stored on the file system level? I've learned that the UTI can be displayed with the mdls command. Does that imply that the UTI is stored along the Spotlight meta data? What if Spotlight is turned off?
Is it correct that there is no API to manually add or change a UTI for a specific file?
There's actually not that much magic to it. You've asked several different questions, so I'll try to give you each of the answers:
How are UTIs created by the system for each file?
Launch Services maintains a database of all applications (and certain other types of bundles) on your Mac and relevant information declared in their Info.plist files. It updates this information automatically—I think it has a daemon monitor the file system to watch for changes to applications, but I don't know the details. What I do know is that you can ask a tool called lsregister to dump the entire database for you. In Terminal on Mountain Lion:
$ /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/Support/lsregister -dump
The various UTType functions also access this Launch Services database (although I'm not sure if they do it directly or if they communicate with some kind of daemon that does it for them).
Where are UTIs stored on the file system level?
Well, the actual Launch Services database seems to be located somewhere different on each Mac. On mine, it appears to be at /private/var/folders/mf/1xd7vlw90dj5p4z1r5f800bc000101/C/com.apple.LaunchServices-0371025.csstore. (At least, lsregister holds this file open while it's working; I'm not actually sure what's in it, but I assume it's the database.)
This is just a list of the declared UTIs, though. There is no UTI field attached to a given file. When you ask Cocoa for a file's UTI—through, say, -[NSWorkspace typeOfFile:error:] or -[NSURL getResourceValue:forKey:error:]—it actually extracts the path extension from the file name and then calls UTTypeCreatePreferredIdentifierForTag() to fetch the relevant UTI. (It's a little more complicated than that, because it's also looking at things like whether the path leads to a directory or device file or something, but that's the basic idea.)
Does that imply that the UTI is stored along the Spotlight meta data? What if Spotlight is turned off?
Spotlight does keep UTIs of files in its database, but that's just so it can quickly search and filter by type. Like everything else in the Spotlight index, this information is not canonical; it's just used to quickly search data that's actually stored elsewhere. If you turn off Spotlight, that's fine—nothing else depends on it.
Is it correct that there is no API to manually add or change a UTI for a specific file?
Yes, because that UTI is calculated at runtime from other information about the file. Changing a file's UTI makes about as much sense as changing the length of its name—you can't do it without changing the name itself.