What would be the difference between Java 1.4.2's implementation of replace, and Apache 2.3's implementation? Is there a performance gain one over another?
The String.replace() method you linked to takes two char values, so it only ever replaces on character with another (possibly multiple times, 'though).
The StringUtils.replace() method on the other hand takes String values as the search string and replacement, so it can replace longer substrings.
The comparable method in Java would be replaceAll(). replaceAll() is likely to be slower than the StringUtils method, because it supports regular expressions and thus introduces the overhead of compiling the search string first and running a regex search.
Note that Java 5 introduced String.replace(CharSequence, CharSequence) which does the same thing as StringUtils.replace(String,String) (except that it throws a NullPointerException if any of its arguments are null). Note that CharSequence is an interface implemented by String, so you can use plain old String objects here.