Look at the comment about "Organization of SPD aka "XFRM rules"" in net/xfrm.h:
- policy rule, struct xfrm_policy (=SPD entry)
- bundle of transformations, struct dst_entry == struct xfrm_dst (=SA bundle)
- instance of a transformer, struct xfrm_state (=SA)
- template to clone xfrm_state, struct xfrm_tmpl
... policy entry has list of up to XFRM_MAX_DEPTH transformations, described by templates xfrm_tmpl. Each template is resolved to a complete xfrm_state (see below) and we pack bundle of transformations to a dst_entry returned to requestor.
Having this template we search through SAD searching for entries with appropriate mode/proto/algo, permitted by selector. If no appropriate entry found, it is requested from key manager.
tmpl defines one transformation for IP packets.
Every transformation means one more layer of encapsulation for IP packets.
Example: to compress traffic and send it over an encrypted tunnel we would specify two
tmpls: the 1st for IPComp, the 2nd for ESP.
tmpl acts as a template for xfrm_state (=SA) creation.
Detailed definitions and explanations for SP and SA can be found in the IPsec standard (RFC 4301: Security Architecture for the Internet Protocol).
In short your example
ip xfrm policy add src $LOCAL dst $REMOTE dir out tmpl src $SRC dst $DST proto esp reqid $ID mode tunnel
creates an SP for the outgoing traffic which matches selector
src $LOCAL dst $REMOTE. Here
$REMOTE is an IP address of packets' final destination.
tmpl src $SRC dst $DST proto esp reqid $ID mode tunnel defines one transformation for the matched traffic — a ESP tunnel: every matched IP packet will be wrapped in a ESP packet and sent to
$DST is an IP address of the remote end of the ESP tunnel.
Of course, I understand we need to specify the $SRC and $DST. But, since those are already specified in the SA, via the ip xfrm state command, why do we need to repeat them in the tmpl?
Strictly speaking SA do not have to exist when we create SP.
A typical case (it is even described in IPsec standard) is when SP exists on its own and creation of the corresponding SA is only triggered by the first IP packet which matches the SP's selector.