What does the selector (parameter sel) in the "ip xfrm state add" command achieve?
The source and destination addresses (and additional parameters like ports and protocol) are set in the ID section, but the selector contains a supplementary set of these. Example:
ip xfrm state add src 10.0.0.1 dst 10.0.0.2 proto esp spi 123456 sel src 10.0.0.3 dst 10.0.0.4 enc blowfish 0xaabbccddee
This leads to the following result:
src 10.0.0.1 dst 10.0.0.2
proto esp spi 0x0001e240 reqid 0 mode transport
replay-window 0
enc cbc(blowfish) 0xaabbccddee
sel src 10.0.0.3/32 dst 10.0.0.4/32
Setkey seemingly does not have the opportunity to add such a selector value. It also does not show selectors in the output. The xfrm command from above produces the following "setkey -D" output:
10.0.0.1 10.0.0.2
esp mode=transport spi=123456(0x0001e240) reqid=0(0x00000000)
E: blowfish-cbc aabbccdd ee
seq=0x00000000 replay=0 flags=0x00000000 state=mature
created: Nov 26 01:25:39 2013 current: Nov 26 01:26:07 2013
diff: 28(s) hard: 0(s) soft: 0(s)
last: hard: 0(s) soft: 0(s)
current: 0(bytes) hard: 0(bytes) soft: 0(bytes)
allocated: 0 hard: 0 soft: 0
sadb_seq=0 pid=6959 refcnt=0
So what does the IPsec subsystem eventually do with this selector?
It is probably straight from the IPsec standard (RFC 4301: Security Architecture for the Internet Protocol).
Probably is because the standard describes "nominal model" i.e. implementations do not have to be exactly as described.
In Section 4.4.2. Security Association Database (SAD):
For each of the selectors defined in Section 4.4.1.1, the entry for an inbound SA in the SAD MUST be initially populated with the value or values negotiated at the time the SA was created. ... For a receiver, these values are used to check that the header fields of an inbound packet (after IPsec processing) match the elector values negotiated for the SA. Thus, the SAD acts as a cache for checking the selectors of inbound traffic arriving on SAs. For the receiver, this is part of verifying that a packet arriving on an SA is consistent with the policy for the SA.
In Section 5.2. Processing Inbound IP Traffic (unprotected-to-protected):
IPsec MUST perform the following steps:
...
3a. If the packet is addressed to the IPsec device and AH or ESP is specified as the protocol, the packet is looked up in the SAD. ... If the packet is found in the SAD, process it accordingly (see step 4).
4. Apply AH or ESP processing as specified, using the SAD entry selected in step 3a above. Then match the packet against the inbound selectors identified by the SAD entry to verify that the received packet is appropriate for the SA via which it was received.
5. If an IPsec system receives an inbound packet on an SA and the packet's header fields are not consistent with the selectors for the SA, it MUST discard the packet.
In short for your example
ip xfrm state add src 10.0.0.1 dst 10.0.0.2 proto esp spi 123456 sel src 10.0.0.3 dst 10.0.0.4 enc blowfish 0xaabbccddee
when IPsec (ESP in this case) packet arrives at the server, then probably the following happens:
src 10.0.0.1 dst 10.0.0.2 proto esp spi 123456 is used to find the corresponding SA.sel src 10.0.0.3 dst 10.0.0.4 is the selector from the SP which was used to create our SA.