2

I'm using Telmate's Terraform provider for Proxmox and trying to deploy Flatcar linux virtual machines using Cloud Init by passing files via cicustom. Based on their example, I have crafted the following Terraform file:

variable "pve_user" {
}
variable "pve_password" {
}
variable "pve_host" {
}

provider "proxmox" {
  pm_tls_insecure = true
  pm_api_url      = "https://SNIP/api2/json"
  pm_user         = "SNIP"
  pm_password     = "SNIP"
  pm_parallel     = 4
}

resource "null_resource" "cloud_init_config_files" {
  connection {
    type     = "ssh"
    user     = var.pve_user
    password = var.pve_password
    host     = var.pve_host
  }

  provisioner "file" {
    source      = "./templates/cloud-config.yml"
    destination = "/var/lib/vz/snippets/cloud-config.yml"
  }
}

resource "proxmox_vm_qemu" "k8s-masters" {
  depends_on = [
    null_resource.cloud_init_config_files
  ]

  count      = 1
  name       = "VM-${count.index}"
  clone      = "VM-Template"
  full_clone = true

  target_node = "192.168.20.10"
  pool        = "VM"

  cores   = 2
  sockets = 1
  memory  = 10240

  network {
    id     = 0
    model  = "virtio"
    bridge = "vmbr0"
    tag    = 50
  }

  disk {
    id           = 0
    type         = "scsi"
    size         = 30
    storage      = "Pool"
    storage_type = "zfspool"
    backup       = true
    iothread     = true
  }

  onboot = true
  agent  = 1

  os_type         = "cloud-init"
  ssh_user        = "core"
  cicustom  = "user=local:snippets/cloud-config.yml"
  ipconfig0 = "ip=192.168.50.10/24,gw=192.168.50.1"

  sshkeys = "ssh-rsa SNIP"
  ssh_private_key = <<EOF
-----BEGIN RSA PRIVATE KEY-----
SNIP
-----END RSA PRIVATE KEY-----
EOF
}

After running terraform apply, the VM is successfully created and Flatcar is bootstrapped. If you connect to the console via Proxmox, however, the VM shows that the IP Address is one received via DHCP instead of the one I provided. Furthermore, the SSH key does not work so I am unable to connect to the VM to troubleshoot.

If I comment out the cicustom line, and simply rely on ipconfig0 and the other normal options, the VM comes up and my SSH key does in fact work. However, the specified IP address is still not used; the VM just uses one provided by DHCP instead. Despite being able to access the VM, I still want to use a custom Cloud Init config file so I can have access to more powerful configuration options.

I've tried various combinations of my cloud-config.yml file. Including things as simple as:

hostname: test

to things more detailed like:

storage:
  files:
    - path: /opt/file1
      filesystem: root
      contents:
        inline: Hello, world!
      mode: 0644
      user:
        id: 500
      group:
        id: 501
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-rsa SNIP

Despite all of this, my SSH keys never work after I pass something to cicustom and, based on the fact that the hostname is never manipulated, I'm assuming Cloud Init just outright isn't receiving the custom config file at all. I've tried passing in both Ignition and Container Linux Config formatted files.

Is Flatcar Linux broken with Proxmox/cicustom? Searching Google for things like "flatcar" "proxmox", "coreos" "proxmox", "container linux" "proxmox", "flatcar" "cicustom", etc don't seem to turn up much results. I guess there aren't a lot of people out there bridging Cloud Native operating systems with bare metal? hehehe

What does come up, however, is this interesting script. Lines 104-132 discuss creating a Flatcar/CoreOS template for Proxmox. So someone else has done this before, at least? Who knows if they passed in a custom cloud init config file afterwards?

Any ideas?

EDIT: ------------------------------------------------------------

Added logs from user-configdrive systemd unit when not passing cicustom

Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Checking availability of "cloud-drive"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Fetching user-data from datasource of type "cloud-drive"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Attempting to read from "/media/configdrive/openstack/latest/user_data"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 line 6: warning: unrecognized key "chpasswd"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 line 9: warning: incorrect type for "users[0]" (want struct)
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 line 10: warning: unrecognized key "package_upgrade"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Fetching meta-data from datasource of type "cloud-drive"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Attempting to read from "/media/configdrive/openstack/latest/meta_data.json"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Attempting to read from "/media/configdrive/openstack/content/0000"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Parsing user-data as cloud-config
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Merging cloud-config from meta-data and user-data
Aug 20 16:22:17 VM-0 coreos-cloudinit[831]: 2020/08/20 16:22:17 Set hostname to VM-0
Aug 20 16:22:17 VM-0 coreos-cloudinit[831]: 2020/08/20 16:22:17 Authorized SSH keys for core user
Aug 20 16:22:17 VM-0 coreos-cloudinit[831]: 2020/08/20 16:22:17 Failed to apply cloud-config: Invalid option to manage_etc_hosts
Aug 20 16:22:17 VM-0 systemd[1]: user-configdrive.service: Main process exited, code=exited, status=1/FAILURE

EDIT 2: ----------------------------------------------------------

Added the contents of the files referenced by the above Systemd unit.

core@VM-0 ~ $ cat /media/configdrive/openstack/latest/user_data
#cloud-config
hostname: VM-0
manage_etc_hosts: true
ssh_authorized_keys:
  - REDACTED
chpasswd:
  expire: False
users:
  - default
package_upgrade: true

As you pointed out, manage_etc_hosts is set to true here by Proxmox.

core@VM-0 ~ $ cat /media/configdrive/openstack/content/0000
auto lo
iface lo inet loopback

        dns_nameservers 192.168.1.100
        dns_search example.com
auto eth0
iface eth0 inet static
        address 192.168.50.10
        netmask 255.255.255.0
        gateway 192.168.50.1

This section contains all of the correct network configuration, however it is all overridden by DHCP so it's ignored.

core@VM-0 ~ $ cat /media/configdrive/openstack/latest/meta_data.json
{
     "uuid": "e61563c9057e9162c4e14d111fea171379170532",
     "network_config": { "content_path": "/content/0000" }
}

EDIT 3: ----------------------------------------------------------

Added my attempt at a custom Cloud Init file

manage_etc_hosts: false
hostname: "test"
ssh_authorized_keys:
  - REDACTED

EDIT 4: ----------------------------------------------------------

I followed this guide to basically accomplish the same thing directly with qemu instead of using Proxmox. Using:

This cloud-config.ign:

{
  "ignition": { "version": "2.2.0" },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "REDACTED"
        ]
      }
    ]
  },
  "storage": {
    "files": [{
      "filesystem": "root",
      "path": "/etc/hostname",
      "mode": 420,
      "contents": { "source": "data:,test" }
    }]
  }
}

OR this cloud-config.yml:

hostname: test
passwd:
  users:
    - name: tj
      ssh_authorized_keys:
        - REDACTED

both produce the same result. An unconnectable virtual machine that spins up with the hostname localhost.

  • Here are the logs that were produced using cloud-config.ign.
  • Here are the logs that were produced using cloud-config.yml.

EDIT 5: ----------------------------------------------------------

These logs are produced by user-configdrive.service when using Telmate's provider against a Flatcar template.

Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Checking availability of "cloud-drive"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Fetching user-data from datasource of type "cloud-drive"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Attempting to read from "/media/configdrive/openstack/latest/user_data"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Fetching meta-data from datasource of type "cloud-drive"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Attempting to read from "/media/configdrive/openstack/latest/meta_data.json"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Attempting to read from "/media/configdrive/openstack/content/0000"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: Detected an Ignition config. Exiting...
Aug 21 21:41:50 localhost systemd[1]: Started Load cloud-config from /media/configdrive.

EDIT 6: ----------------------------------------------------------

Here is the entire Systemd journal from the first output after using the cloud-config.ign in EDIT 4 with Telmate's provider.

EDIT 7: ----------------------------------------------------------

These qm commands produce a working Flatcar VM using cicustom.

qm create 101 --name test --cores 2 --memory 2048 --net0 "virtio,bridge=vmbr0"

qm set 101 --net0 "virtio,bridge=vmbr0,tag=50"

qm importdisk 101 /mnt/RAIDPool_Templates/template/iso/flatcar_production_qemu_image.img FlashPool

qm set 101 --scsihw virtio-scsi-pci --scsi0 FlashPool:vm-101-disk-0 --ide2 FlashPool:cloudinit --boot c --bootdisk scsi0 --serial0 /dev/tty0 --ipconfig0 ip=dhcp --citype configdrive2

qm set 101 --cicustom user=RAIDPool_Templates:snippets/user-data.yml

qm start 101
ssh -i ~/.ssh/sol.milkyway.kubernetes core@192.168.50.219

user-data.yml:

#cloud-config
hostname: test
manage_etc_hosts: true
ssh_authorized_keys:
  - ssh-rsa REDACTED
chpasswd:
  expire: False
users:
  - default
package_upgrade: true

Passing the same user-data.yml to cicustom within the proxmox_vm_qemu resource doesn't seem to be working though.

10
  • Have you looked at journald for the cloud-init service to see if there are any pertinent complaints and problems when spinning up a host? My experience with cloud-init on on the similar CoreOS saw me use a lot of journalctl /usr/bin/cloud-init, journalctl --unit cloud-* or just plain old journalctl -xe. – Wesley Aug 19 '20 at 16:51
  • @Wesley as I am unable to connect via SSH when providing cicustom I cannot retrieve the logs from this case. If I comment out that line and apply the manifest, then the resources are provisioned successfully and come online. Looks like Flatcar refers to that unit as user-configdrive and the logs I see produced have been added to the original post as an edit at the bottom. – TJ Zimmerman Aug 20 '20 at 16:25
  • In the second to last log line, manage_etc_hosts is being given a key that isn't valid. Check out the local cloud config yml file and see what's being sent to it. Here are the valid keys. Notice that the script that you found also refers to this problem: gist.github.com/chriswayg/… – Wesley Aug 20 '20 at 16:51
  • So my takeaway is that Proxmox wants to pass manage_etc_hosts: true and Flatcar doesn't like that. I confirmed this by looking at /media/configdrive/openstack/latest/user_data referenced in the Systemd Unit. The contents of this file (and the others referenced) have been added in EDIT 2. I have also added EDIT 3 which is a new attempt at passing cicustom with a custom cloud init file containing three entires: manage_etc_hosts: false, hostname: test, and ssh_authorized_keys. After building a new VM, I can't connect with the private key and the VNC console shows the wrong hostname. – TJ Zimmerman Aug 20 '20 at 23:43
  • So it appears that while it is true that Proxmox is passing manage_etc_hosts: true and Flatcar is throwing an error, that is potentially a separate issue from the fact that my cicustom is never processed. Either due to a misconfiguration of my own. Or a bug in Flatcar, Proxmox, or the Terraform provider. Or any combination thereof. – TJ Zimmerman Aug 20 '20 at 23:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.