I have a Drupal application that has a very large recordset, almost half a million nodes (475,181). Even simple joins are becoming too time consuming (3-10s), and we are becoming increasingly dependent on Memcached. I wonder if anyone has used the same technologies that Facebook is using with Drupal.
To answer your question: Not yet, but there is significant work being done on HipHop and a nosql database named MongoDB. More on those below.
As others have mentioned I would first make absolutely sure that your database tables are optimized properly, indexed and that your database has sufficient resources. If your database is choking on under 500k rows it's likely that there is something wrong. (We have over 150k node, about 600k rows in our node_revisions table and because the nodes have multiple images over a million rows in some of our cck tables.) I'm no MySQL expert and was able to get the database query times to under a hundred milleseconsds for most of our queries.) Here are the steps I would take before looking into Switching to a different db engine. (from easiest to hardest)
Here is a great article on how much can be achieved with drupal and 1 single server.
If none of that works here's a few links regarding some of the newer haute technogogies that people are playing with. The most promising is MongoDB, and if you have the development resources to use it on a project I envy you. (It's still a little new new and unpolished for the small shop we are, but I can't wait to sink my teeth into it.)
drupal and Mongodb - there is also a great talk in the sessions from drupalcon SF (last year)