I have extensive Java experience and know why finalize() is scheduled for a removal soon. On the other hand, my knowledge about C# is skin-deep - I am more or less aware about what features it offers on the abstract level, but no real experience to speak of. Some time ago I have found a solution looking for a problem and from a search of languages which support C++-like destructors came IDisposable in CLI.
What are the differences which make the latter a continually supported feature?
Well, really, they're not the same.
Java's finalize() has a direct equivalent in C#'s finalizers (declared as ~T(), where T is the class name). Use of finalizers in C# is heavily discouraged now in modern code, due to how tricky they are to get right in practice. Maybe they'll get deprecated too some day. In the meantime, if you really need that kind of behaviour, you really should be using safe handles.
Compared to finalizers, IDisposable seems to provide automatic and deterministic cleanup of resources, like C++'s destructors. That's neat! But I said seems, because actually IDisposable is not magic. On its own, it does nothing. What you really mean is the using statement, which is really just syntactic sugar for a try/finally statement that calls Dispose() in the finally part (with some null checking for good measure).
using var x = new T();
// ...
Java also has that, with the AutoCloseable interface and "try-with-resources" idiom:
try (T x = new T()) {
// ...
}
However IDisposable and using have been introduced much earlier relatively in the language's lifespan compared to "try-with-resources" (C# 2.0 vs Java 7), so their presence is more pervasive across both the standard library and third party libraries.