Even if I don't specify borders, how is the high contrast mode displaying a border?
Color contrast mode gets rid of many subtle color differences and tries to make the "foreground" stand out from the background. Text and form fields are considered foreground, regardless of their color and visual settings. Placing a border around these ensures that users will be able to identify them.