Jonathan suggest here: Xcode Includes .xib files that have been deleted! that cleaning all targets and empty the caches will fix the problem with Xcode including deleted .xib files but I cannot find a way to empty the cache in Xcode 4.
How to do that in Xcode 4?
Command-Option-Shift-K to clean out the build folder. Even better, quit Xcode and clean out ~/Library/Developer/Xcode/DerivedData manually. Remove all its contents because there's a bug where Xcode will run an old version of your project that's in there somewhere. (Xcode 4.2 will show you the Derived Data folder: choose Window > Organizer and switch to the Projects tab. Click the right-arrow to the right of the Derived Data folder name.)
In the simulator, choose iOS Simulator > Reset Content and Settings.
Finally, for completeness, you can delete the contents of /var/folders; some caching happens there too.
WARNING: Deleting /var/folders can cause issues, and you may need to repair or reinstall your operating system after doing so.
EDIT: I have just learned that if you are afraid to grapple with /var/folders/ you can use the following command in the Terminal to delete in a more targeted way:
rm -rf "$(getconf DARWIN_USER_CACHE_DIR)/org.llvm.clang/ModuleCache"
EDIT: For certain Swift-related problems I have found it useful to delete ~/Library/Caches/com.apple.dt.Xcode. You lose a lot when you do this, like your spare copies of the downloaded documentation doc sets, but it can be worth it.