During a fully automated deployment on physical hardware (using pxeboot / preseed / kickstart to install and configure a minimal system for bootstrapping) we need to deploy certain secrets - such as our Chef validation key, and a secret key for accessing secure application data.
In our VM environments we can pre-bake images with these already in so when the node is deployed it already has the files locally and they never have to cross the network, however for physical servers where we're building straight from the netboot images we don't have the ability to do this.
I'm trying to come up with both a secure and automated way of deploying these secrets out to servers that are provisioned, in a way that we know they could not end up in the wrong place.
Our provisioning environment has knowledge of all our physical servers and wether they've been provisioned or not - we create node definitions in our Chef server which include the MAC address of the primary interface used for PXEBooting, then set a flag on the node once it's been provisioned. We use this to only offer up PXEBoot files to nodes that have not yet been provisioned, and so we can specify in our node definitions the release / os / roles of the boxes.
Our current working idea is a very simple web service that (over HTTPS) would serve the secrets to nodes which have not yet been provisioned. However this has a number of drawbacks and issues I can see. My primary concern is our only protection is MAC address validation, and MAC addresses can be spoofed (I'm strongly of the opinion you should never consider your internal network secure, though that's probably a different discussion).
Our current provisioning environment is Ubuntu 12.04 and 14.04 with PXEboot / Preseeding on dedicated provisioning servers - though we could adapt other solutions to fit our environment.
I'm mainly concerned about a couple of global secrets that all nodes need access too, our ideal end goal is to not have these global secrets and have per-node secrets, I'm currently investigating a tool to accomplish this but for the moment we have to deal with the current setup.
So how are people securely, automatically distributing global secrets in provisioning environments? Any advise or pointers for ways to do it?