It strikes me as an odd design choice that the current culture information (CurrentCulture and/or CurrentUICulture) is a a property of the running thread. At the very least it seems like the scope of such a thing should be one level up, at the process level.
But these things usually make sense once you hear the rationale. It might be enlightening to find out why the .NET designers decided that Thread was the right place to put this property.
For starters, it wasn't really their choice. That decision was made a long time before they got started, culture is a property of an operating system thread. Review the SDK docs for the Get/SetThreadLocale() API functions for example.
That doesn't completely explain it, they have virtualized other OS features, albeit that this one is very hard to virtualize because so many APIs are culture sensitive, especially COM ones.
The next good reason is because changing the culture process-wide is so extremely difficult to implement. It is an unsolvable race condition. Some other thread might well be in the middle of formatting data that has culture affinity. While a lock would work to prevent it from using culture properties just as it is changing, it can not prevent it from changing in the middle of a chain of formatting calls. It would require them to take some kind of global lock and hold it for the duration of the formatting job. Deadlock is very likely.
There's another aspect to this, one that I think is the real problem. It is related to the Thread.ExecutionContext property. The framework uses this to "flow" thread state from one thread to another. Very obscure but important to imbue things like the security context onto a worker thread. It would have been ideal if that context could also imbue the culture so that you can be sure that any of the workers you start have the same culture you selected and not the operating system default.
It doesn't, I don't really know why. Probably because the very first reason I gave. That does however make it very perilous to change a thread's culture. The kind of bugs that can cause are very subtle. Like creating a SortedDictionary with a string as the key in your main thread with a non-operating system default culture. Then finding out that a worker thread can't occasionally find stuff back anymore because the string sorting rules are different.
EDIT: there's some relief from this problem in .NET 4.5, it supports the new static CultureInfo.DefaultThreadCurrentCulture and DefaultThreadCurrentUICulture properties.
EDIT2: culture now flows as described in the 4th paragraph in .NET 4.6. This should alleviate all concerns.