Create SCO client profile driver for call control from Bluetooth headset

Overview

Hi everyone, I'm developing a feature for a Softphone (run on windows), that's make answer/end a call from Bluetooth headsets or speakerphones. I have read and do step by step follow Hands-free Profile spec: https://www.bluetooth.com/specifications/specs/hands-free-profile-1-8/

Currently, I have set up "Sevice Level Connection" by send/recv AT commands follow the spec.

Problem

I don't know how to set-up Audio Connection that know as SCO connection. It's seem need to write a custom driver, that I have no experience with. The only tutorial I see from Microsoft is very difficult to understand: https://learn.microsoft.com/en-us/windows-hardware/drivers/bluetooth/creating-a-sco-client-connection-to-a-remote-device
Is there any way to bypass this step without code a driver?
If I can set-up Audio Connection then Can I answer/end call from Bluetooth headset?

This is Initialization Sevice Level Connection diagram: https://i.sstatic.net/9KMlb.png

This is Answer Call diagram: https://i.sstatic.net/M8WZf.png

Solution

Sharing my findings and solution after fighting for it for several days. Probably more adequate for the reverse engineering stackexchange but anyway...

I must say I am more and more "impressed" lately with Microsoft's technical decisions. And the Bluetooth Hands-free is adding points to the above.

The Hands-free profile is available for quite some time/years now and when Microsoft finally got it natively under Windows there's no public API to work with. And like the OP I want to use my BT Handsfree (Plantronics PLT70 if it matters) with all of its functions in my implementation of a SIP Softphone.

The following applies to the Microsoft Bluetooth Stack. Tested under Win10 21H2 (19044.3086);
The device should be normally paired to Windows;
One must have the "Handsfree Telephony" service checked ON in the Device Properties -> Services tab;
One must have the "Bluetooth Audio Gateway Service" / BTAGService stopped (and probably disabled to keep it stopped upon bluetooth radio being plugged in).

Part 1 Open a device driver handle to the Hands-free service instance exposed by the bthenum driver for your handsfree device. I will not go into details on how to find the device path but one could figure it or probably use SetupApi/Registry to find it. The code will look like this (Delphi style, version 5 if it matters):

hBthf := CreateFile('\\?\BTHENUM#{0000111e-0000-1000-8000-00805f9b34fb}_VID&0001000f_PID&0000#8&F070F74&0&48C1ACF774C4_C00000000#{bd41df2d-addd-4fc9-a194-b9881d2a2efa}\service',
    GENERIC_READ or GENERIC_WRITE, FILE_SHARE_READ or FILE_SHARE_WRITE,
    nil, OPEN_EXISTING, FILE_FLAG_OVERLAPPED, 0);

48C1ACF774C4 from the path is my device MAC address. The FILE_FLAG_OVERLAPPED is needed as one has to make simultaneous requests to the driver.

Spin a thread that will continuously send the 0x2A0024 IOCTL (not documented, hence reverse engineering) with an output buffer of 4 bytes to the device driver handle. On this thread the driver will notify you and will make requests whenever one tries to access the audio channel of the Hands-free using the Multimedia API - i.e. any app trying to record/play on it; in my case the playback audio channel is named as "Headset (PLT_M70 Hands-Free AG Audio)". The nofify/request code will be in the 4 byte buffer (a DWORD/INT) once the IOCTL completes.

...
var
  ol: TOverlapped;
  request: Integer;
  br: Cardinal;
...
DeviceIoControl(hBthf, $2A0024, nil, 0, @request, 4, br, @ol)

The "requests" one needs to answer are 4, 5 and 6. I suspect the other codes are notifications (1 is probably for SCO channel up, 2 for SCO channel down, 0 something else?). The reply is sent back to the driver with IOCTL code 0x2A0028 and an eight byte input buffer. The contents of the buffer are: a DWORD with the request code, that one is replying to, and then another byte with 0x01 and another with 0x02 and then two zero bytes. I'm not sure (yet?) what the meaning of these bytes is. I tried with different values with 1-1, 3-3 and it seems as if (doubtedly) they do not matter. Code:

...
var
  b: array of byte;
...
SetLength(b, 8);
...
if (request = 4) or (request = 6) then
begin
  PDWORD(@b[0])^ := request;
  PDWORD(@b[4])^ := $0201;
  DeviceIoControl(hBthf, $2A0028, @b[0], 8, nil, 0, br, @ol);
end;

Part 2 The other part of the solution is to be integrated into the AT command chat on the RFCOMM channel the OP has already worked out. Once the Service Level Connection is established one has to issue these IOCTLs to the driver:

// Service Level Connection status
// -> 0x2A0004, inBuffer = 0x01 0x00 0x00 0x00 0x9B 0x00 0x00 0x00

// -> 0x2A0020, null input, null output

// Set codec
// -> 0x2A0014, inBuffer 0x01 0x00 0x00 0x00 (DWORD Codec ID 1 or 2)

After the above "magic" IOCTLs the Hands-free Audio Gateway device will be "discovered" and made available in Windows for apps to play/record with.

A final note: without the request handler thread in the first part of the solution the device will still appear in Windows, however any app trying to use it will hang. The solution to "unhang" it is to remove the radio (provided it is a removable USB dongle). Not sure whether "disable"/"enable" from Device Manager would work.