The FormatMessage function (FormatMessageA or FormatMessageW) of the Win32 API, when set to the FORMAT_MESSAGE_ALLOCATE_BUFFER flag, allocate a buffer for the output string on the OS side and place its pointer to the address specified by lpBuffer.
That is, the actual type of lpBuffer in this case is &LPWSTR, not LPWSTR as specified by the function argument type.
If you try to call FormatMessageW on a Rust windows crate, you will have to specify &mut PWSTR as a parameter to PWSTR.
In this case, how should I do the type conversion according to Rust conventions?
As far as I have tried, the following works as expected, but I would like to know if there is a better way.
use std::ptr;
use windows::{
core::*,
Win32::{
Foundation::*,
System::Diagnostics::Debug::*,
UI::WindowsAndMessaging::{CreateWindowExW, HMENU, WINDOW_EX_STYLE, WINDOW_STYLE},
},
};
fn get_last_error_msg() -> Result<String> {
unsafe {
let last_error = GetLastError();
let mut lp_allocated_buffer = PWSTR(ptr::null_mut());
let size = FormatMessageW(
FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
None,
last_error.0,
0,
PWSTR(&mut lp_allocated_buffer as *mut PWSTR as _),
0,
None,
);
if size > 0 {
let message_string_result = lp_allocated_buffer.to_string();
let hresult = LocalFree(HLOCAL(lp_allocated_buffer.as_ptr() as _));
if hresult.0 == 0 as _ {
return Ok(message_string_result?);
} else {
return Err(Error::from_win32());
}
} else {
let format_message_err = GetLastError();
eprintln!("{:?}", format_message_err);
return Err(Error::from_win32());
}
}
}
fn main() -> windows::core::Result<()> {
unsafe {
// force error
let a = CreateWindowExW(
WINDOW_EX_STYLE(0),
PWSTR::null(),
PWSTR::null(),
WINDOW_STYLE(0),
0, 0, 0, 0,
HWND(ptr::null_mut()),
HMENU(ptr::null_mut()),
None,
Option::None,
);
match a {
Err(er) => {
let err_msg = get_last_error_msg()?;
eprintln!("{:?}", err_msg);
return Err(er);
}
_ => {}
}
}
return Ok(());
}
The FormatMessageW API is awkward to use. The message text it returns can go to a caller-provided buffer, or an internally allocated buffer, controlled via the FORMAT_MESSAGE_ALLOCATE_BUFFER flag.
Since C doesn't allow function overloading the different modes are supported by "overloading" the lpBuffer parameter instead, potentially leading to a mismatch between the concrete type provided and the literal type accepted by the function signature. This requires an explicit cast in just about any programming language (other than C).
With that background out of the way, the conversion used in the code is correct. The following piece of code does the right thing in taking the address of a pointer, adjusting its type, and wrapping it up in the expected type again:
let mut lp_allocated_buffer = PWSTR(ptr::null_mut());
let size = FormatMessageW(
// ...
PWSTR(&mut lp_allocated_buffer as *mut PWSTR as _),
// ...
);
What makes this well-defined is incredibly subtle, and not reflected in code. Indeed, the majority of tokens (&mut lp_allocated_buffer as *mut PWSTR) are distractful; that's just spelling out an otherwise implicit type coercion from &mut T to *mut T. The part that reads as _1 is what requires attention.
To explain what is going on, let's change the code to read this instead:
let mut buf = PWSTR::null();
let size = FormatMessageW(
// ...
PWSTR(&mut buf as *mut PWSTR as *mut u16),
// ...
);
Most of the changes are cosmetic. The meaningful adjustment is the as *mut u16 cast, making the specific conversion explicit. Going from a *mut PWSTR to a *mut u16 is well-defined due to the way PWSTR is defined2:
#[repr(transparent)]
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub struct PWSTR(pub *mut u16);
The crucial piece is the #[repr(transparent)] attribute that ensures that PWSTR and *mut u16 have identical memory layout, making the as *mut u16 cast well-defined.
1 Much as I like the ergonomics of the _ shorthand, I've found it to make code harder to read by deluding the author's intent, and it is way too flexible, easily crossing the line between sound and unsound code, with no compiler diagnostics issued.
2 As of the time of writing, which correlates to version 0.60.0 of the windows-rs crates family, and version 0.2.0 of the windows-strings crate.