How does Java select the default / fallback server certificate when client does not set SNI servername at all?
Background:
I have created PKCS#12 keystore with multiple server certificates. I have tested also with JKS keystore. I use NewSunX509 implementation of X509KeyManager which can select matching server certificate according to requested SNI servername.
I cannot find the rule which JSSE uses to pick the default or "fallback" server certificate, which it returns when client does NOT send SNI servername in TLS handshake. I have not found documentation or figured it out by testing. The implementation (code link) says it sorts the imperfect matches. In practice, the selection seems to be impacted by the order of adding entries in keystore, but it does not simply pick first or last entry, or sort by alias even if the code comment gives that impression.
I don't see any comment about sorting by alias. If the imperfect-match sorting occurs, it uses EntryStatus.compareTo which sorts first by CheckResult (i.e. a measure of how close the cert is to being 'good') then keyIndex which is its position in the list of keyTypes requested by the caller based on the SSL/TLS protocol (i.e. the ciphersuite(s) and/or signature_algorithm values, which may be in preference order). If those are both equal, Collections.sort is stable so it will use the order they were tested and found, see next.
However, sorting occurs only if there is no 'perfect' match; as soon as any 'perfect' match is found it is returned without looking for any others that could exist. So which is returned depends on the order in which keystore entries are looked at and tested. chooseAlias as you linked, and its with-SNI-idalg sibling, first look in 'builder' order if there is more than one, which there usually won't be; and within a 'builder' (i.e. within a keystore) they call getAliases which you can see uses the Enumeration returned by ks.aliases() -- which is determined by the KeyStore instance used.
File-based keystores like PKCS12 and JKS generally use a Map or the older (pre-Collection) Hashtable keyed by alias. In particular PKCS12 uses LinkedHashMap which returns keys/entries in the order inserted, which I think is more or less the order they exist in the stored file (although in PKCS12 certs and privatekeys may be ordered differently and I'm not sure which controls) but that need not be the same order as aliases/names, creation, or anything else. JKS uses Hashtable which enumerates in descending order of the hashcode modulo the map size which depends primarily on the number of entries inserted, and in case of conflict it uses a linked list and returns in reverse order of insertion -- unless maybe rehashing changes that.
For a non-file-based keystore like PKCS11, Windows, Apple it could be determined by the Java interface code, what the underlying facility provides, or a combination.
TLDR: as far as you're concerned, it's unpredictable and for practical purposes might as well be random. If you care which one you get, either write your own KeyManager logic to implement your choice (or use a prewritten one like Apache httpcomponents) or filter the data in the keystore(s) you supply to the default KeyManager.
Also to be clear, this is only for NewSunX509 as you specified. The default KeyManager is (still!) the older SunX509 and that works differently; it has its own HashMap which controls the iteration, rather than using that of the underlying keystore. (Singular, since SunX509 only takes one.)