Why would web crypto SHA-256 encrypt message while SHA-512 fail sometimes?

I encountered this issue when trying to encrypt some message using RSA-OAEP. To my surprise (as a beginner in crypto) that in some condition SHA-256 works while SHA-512 doesn't.

I couldn't find any helpful info from https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/encrypt#rsa-oaep

Below is some modified demo code from https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/generateKey

let message = `{"url":"https://stackoverflow.com/questions/8936984/uint8array-to-string-in-javascript","title":"Uint8Array to string in Javascript - Stack Overflow"}`
let message2 = `just some tree`; // If using SHA-512, this shorter message works but above doesn't

let enc = new TextEncoder()
let dataArray = enc.encode(message)

let keyPair = await window.crypto.subtle.generateKey(
  {
    name: "RSA-OAEP",
    modulusLength: 2048,
    publicExponent: new Uint8Array([1, 0, 1]),
    hash: "SHA-512", // If switched to SHA-256 everything works
  },
  true,
  ["encrypt", "decrypt"]
)
let ciphertext = await window.crypto.subtle.encrypt(
  {
    name: 'RSA-OAEP',
  },
  keyPair.publicKey,
  dataArray
)
let buffer = new Uint8Array(ciphertext, 0, 5);
console.log(`Successful, First 5 bytes: ${buffer}`);

In the above demo code, if the hash function is switched to SHA-256 everything works, but fails to execute if is SHA-512

I have tried a shorter message2. And to my surprised it worked for both SHA-256 and SHA-512

I have tested both async await and .then style. That doesn't seem to be the issue.

I would assume this is related to some kind of length or character limitation? So does that mean I have to use SHA-256? Also curious to learn whether this is a bug or intended feature, if intended what's the official length limitation?

With RSA, only short messages can be encrypted. The maximum message length results from the key size (size of the modulus) minus a value that depends on the padding.
In the case of OAEP, the latter depends on the digest used and is in bytes: 2 * (hLen + 1), where hLen is the output size of the digest in bytes (see here).

So for SHA-256, for a 2048 bits (256 bytes) key, the maximum message length in bytes is: 256 - 2 * (32 + 1) = 190 and for SHA-512: 256 - 2 * (64 + 1) = 126.
In your example, message is UTF-8 encoded 150 bytes, so only SHA-256 can be applied, but not SHA-512. In contrast, message2 has UTF-8 encoded a length of 14 bytes, so both digests can be used, SHA-256 and SHA-512.

Note that longer plaintexts are not encrypted with RSA, but with a hybrid encryption.