Horde Gmail OAuth2 Authentication Failed

Issue:

I am using Horde and googles OAuth2 to login to the users IMAP, when using the below parameters for the Horde_Imap_Client_Socket class.

"username" : "example@gmail.com"
"password" : "XOAUTH2"
"hostspec" : "imap.gmail.com"
"port" : 993
"secure" : "ssl"
"timeout" : 20
"context" :
    "ssl" :
    "verify_peer" : true
    "verify_peer_name" : true
"xoauth2_token" : "{INSERT GOOGLE ACCESS TOKEN HERE}"

I get this error back.

Authentication failed.

Adding "debug" => "php://output" gave the output below.

------------------------------ >> Fri, 19 Feb 2021 19:30:27 +0000 >> Connection to: imap://imap.gmail.com:993/ >> Server connection took 0.1738 seconds. 
S: * OK Gimap ready for requests from 24.231.213.106 t22mb47308959jai 
C: 1 CAPABILITY 
S: * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER AUTH=XOAUTH 
S: 1 OK Thats all she wrote! t22mb47308959jai >> Command 1 took 0.0652 seconds. 
C: 2 AUTHENTICATE XOAUTH2 {REDACTED (TOKEN)} 
C: S: 2 NO [AUTHENTICATIONFAILED] Invalid credentials (Failure) >> Command 2 took 0.2507 seconds. 
C: 3 AUTHENTICATE PLAIN [INITIAL CLIENT RESPONSE (username: {REDACTED})] 
S: 3 NO [AUTHENTICATIONFAILED] Invalid credentials (Failure) >> Command 3 took 0.2358 seconds. 

I assume I am doing something wrong in the code below as I am not that experienced with Horde.

Code:

$credentials = json_decode($provider['credentials'], true);
$params = [
    'username' => $user,
    'password' => "XOAUTH2",
    'hostspec' => $host,
    'port' => $port,
    'secure' => $ssl_mode,
    'timeout' => (int) $this->config->getSystemValue('app.mail.imap.timeout', 20),
    'context' => [
        'ssl' => [
            'verify_peer' => $this->config->getSystemValueBool('app.mail.verify-tls-peer', true),
            'verify_peer_name' => $this->config->getSystemValueBool('app.mail.verify-tls-peer', true),
        ],
    ],
    'xoauth2_token' => new \Horde_Imap_Client_Password_Xoauth2($user, $credentials['access_token'])->getPassword()
];
$this->client = new \Horde_Imap_Client_Socket($params);
try {
    $this->client->login();
} catch (Horde_Imap_Client_Exception $e) {
    throw new ServiceException(
        "Could not connect to IMAP host $host:$port: " . $e->getMessage(),
        (int) $e->getCode(),
        $e
    );
}

This ended up being an issue with having the scopes on the actual authorization url, rather than just in the API Client settings on Google Cloud Console.