SaltStack on Packet with Pulumi

In this article, I'm going to walk through using Pulumi to provision a couple of bare metal servers on Packet , using the user data to provision and configure a working SaltStack setup with a single master and single minion node.

NB: Packet is now known as Equinix Metal and Pulumi templates and commands have been updated with the new name. Remember to translate as you work through this.

Why Packet?

I joined Packet on July 27th, 2020 (3 days ago, at the time of writing) and I wanted a quick project that would allow me to try out the platform and see what fun I could have. Packet is a hosting provider that has commoditised bare metal compute over an API. Nice, right? 💯

Why SaltStack?

I've been using and advocating SaltStack since around 2015, it's my goto tool for configuration management and provides some very cool features that aren't available in alternatives, such as beacons , the reactor , and salt-cloud .

When you're running bare metal over an API, there's a few convenience factors that aren't available to you that you'd get for ~~vendor-lockin~~ free on other clouds, such as ASGs; so by providing a SaltStack backbone to our compute, we can begin to put these pieces together for ourselves. The old adage of Cattle vs Pets is still aplicable in a bare metal environment, but with different applications. I'll dive into this in a future post.

Why Pulumi?

Most people reach for Terraform. Hell, I used to. Terraform is fantastic. That being said ... while HashiCorp are making good progress on evolving HCL to support more code-like features with 0.12, why not use a programming language straight up? That's what Pulumi is.

Pulumi is a Infrastructure as Code (IaaC) tool that allows you to describe your infrastructure, much like Terraform . Unlike Terraform, Pulumi doesn't impose a specific DSL, HCL, on you; and instead, you can use your programming language of choice ... provided it's supported.

At the time of writing, Pulumi supports:

C#
F#
Go
JavaScript
Python
TypeScript
VisualBasic

I know, I know. I'm disappinted Rust isn't there either. Maybe one day .

Walkthrough

Creating the Stack

So to create our stack, we need to generate a new Pulumi project. For this example, we'll use the Pulumi local login, which stores the statefile on our local disk. The statefile is very similar to Terraform state. It is needed to build an execution plan for our apply commands. Using local will suffice today, but you should investigate using alternative options for production deployments.

shell

		
				pulumi login --local

We're going to use the TypeScript template to get started. Unfortunately, there isn't a template for all supported languages, but templates do exist for the following:

shell

		
				packet-go           # A minimal Packet.net Go Pulumi program
packet-javascript   # A minimal Packet.net JavaScript Pulumi program
packet-python       # A minimal Packet.net Python Pulumi program
packet-typescript   # A minimal Packet.net TypeScript Pulumi program

If you want to use one of the dotNet languages, you can use a generic template; then add the Packet provider manually. Generic templates for dotNet are called:

shell

		
				csharp              # A minimal C# Pulumi program
fsharp              # A minimal F# Pulumi program
visualbasic         # A minimal VB.NET Pulumi program

To create our project from the TypeScript template, let's run:

shell

		
				pulumi new packet-typescript

You'll be walked through a couple of questions to create your stack, after which you'll have a directory that looks like:

shell

		
				drwxr-xr-x   - rawkode 30 Jul 18:07 node_modules
.rw------- 286 rawkode 30 Jul 18:06 index.ts
.rw-r--r-- 28k rawkode 30 Jul 18:07 package-lock.json
.rw------- 201 rawkode 30 Jul 18:06 package.json
.rw-r--r--  85 rawkode 30 Jul 18:07 Pulumi.dev.yaml
.rw------- 100 rawkode 30 Jul 18:06 Pulumi.yaml
.rw------- 438 rawkode 30 Jul 18:06 tsconfig.json

If we take a look inside of index.ts , we'll see:

typescript

		
				import * as pulumi from "@pulumi/pulumi";
import * as packet from "@pulumi/packet";

// Create a Packet resource (Project)
const project = new packet.Project("my-test-project", {
  name: "My Test Project",
});

// Export the name of the project
export const projectName = project.name;

This TypeScript uses the Pulumi SDKs to provide a nice wrapper around the Packet API. Hopefully it's pretty self-explanatory; you can see that it creates a new project and exports its name.

Exports are similar to Terraform outputs. We use an export when we want to make some attribute from our stack available outside of Pulumi. Pulumi provides the pulumi stack output command, which displays these exports.

We'll use these later.

Cleaning Up the Stack

This step is completely subjective, but I don't like my code just chilling in the top level directory with all the other stuff. What is this, Go? 😏

Fortunately, we can append main: src/index.ts to the Pulumi.yaml file, which tells Pulumi our entry point for this stack lives somewhere else. I'm going to use a src directory.

shell

		
				mkdir src
mv index.ts src/
echo "main: src/index.ts" >> Pulumi.yaml

Creating a "Platform"

I like to create a Platform object / type / class that can be used to pass around the Pulumi configuration and some other common types that my Pulumi projects often need. This saves my function signatures getting too gnarly as we add new components to our stacks.

The Platform object I'm using for this is pretty trivial. It loads the Pulumi configuration and stores our Packet Project, which means we can pass around Platform to other functions and it's a single argument, rather than many.

typescript

		
				// ./src/platform.ts
import { Config } from "@pulumi/pulumi";
import { Project } from "@pulumi/packet";

export type Platform = {
  project: Project;
  config: Config;
};

export const getPlatform = (project: Project): Platform => {
  return {
    project,
    config: new Config(),
  };
};

Now we can update our ./src/index.ts to look like so:

typescript

		
				import * as packet from "@pulumi/packet";
import { getPlatform } from "./platform";

const project = new packet.Project("pulumi-saltstack-example", {
  name: "pulumi-saltstack-example",
});

const platform = getPlatform(project);

export const projectName = platform.project.name;

Creating the SaltMaster

Now we want to create the Salt Master server. For this, I create a new directory with an index.ts that exports a function called createSaltMaster ; which I can consume in our ./src/index.ts , much like we did with our Platform .

Lets start with the complete file, then we'll break it down; sound good? Good 😀

typescript

		
				// ./src/salt-master/index.ts
import {
  Device,
  IpAddressTypes,
  OperatingSystems,
  Plans,
  Facilities,
  BillingCycles,
} from "@pulumi/packet";
import { Platform } from "../platform";
import * as fs from "fs";
import * as path from "path";
import * as mustache from "mustache";

export type SaltMaster = {
  device: Device;
};

export const createSaltMaster = (
  platform: Platform,
  name: string
): SaltMaster => {
  // While we're not interpolating anything in this script atm,
  // might as well leave this code in for the time being; as
  // we probably will shortly.
  const bootstrapString = fs
    .readFileSync(path.join(__dirname, "./user-data.sh"))
    .toString();

  const bootstrapScript = mustache.render(bootstrapString, {});

  const saltMaster = new Device(`master-${name}`, {
    hostname: name,
    plan: Plans.C1LargeARM,
    facilities: [Facilities.AMS1],
    operatingSystem: OperatingSystems.Debian9,
    billingCycle: BillingCycles.Hourly,
    ipAddresses: [
      { type: IpAddressTypes.PrivateIPv4, cidr: 31 },
      {
        type: IpAddressTypes.PublicIPv4,
      },
    ],
    projectId: platform.project.id,
    userData: bootstrapScript,
  });

  return {
    device: saltMaster,
  };
};

SaltMaster Return Type

Because this is TypeScript, we want to be very explicit about the return types within our code. This allows us to catch errors before we ever run our Pulumi stack. As we're using createSaltMaster function to create our SaltMaster, we want that function to return a type with the resources we create.

typescript

		
				export type SaltMaster = {
  device: Device;
};

While our function only returns a Device , it's still nice to encapsulate that in a named type that allows for our function to evolve over time.

This is the function signature for createSaltMaster . You can see our return type is the type we just created, SaltMaster .

typescript

		
				import { Platform } from "../platform";

export const createSaltMaster = (
  platform: Platform,
  name: string
): SaltMaster =>

Our function also takes a couple of parameters, namely platform and name . The platform is our Platform object with our Pulumi configuration and the Packet Project, so we also need to import those too. The name allows us to give our SaltMaster a name when we create the device on Packet. We could hard code this inside the function as salt-master , but then we can't use createSaltMaster more than once for a highly available set up in a later tutorial. Always be thinking ahead, right? 😉

Provisioning with User Data

typescript

		
				import * as fs from "fs";
import * as path from "path";
import * as mustache from "mustache";

const bootstrapString = fs
  .readFileSync(path.join(__dirname, "./user-data.sh"))
  .toString();

const bootstrapScript = mustache.render(bootstrapString, {});

I know ... I know what you're thinking ... but trust me, this will make sense.

As Pulumi allows us to use a feature complete programming language to describe our infrastructure, we also have access to that programming languages ENTIRE eco-system of libraries (npm). As such, if I want to template some user data for a server ... say, to provision and install SaltStack ... I can use a popular template tool, such as mustache, from npm 😉 This is useful because we're not only using a programming language we're familiar with to provision our server, but we can use the same libraries we're familiar with too. It's nice when a plan comes together.

My user data for the Salt Master looks like so: The user data for installing the Salt master is pretty simple and we're not actually using the mustache interpolation yet; however, it's good to bake this in early instead of using TypeScript/JavaScript literals for the user-data. Why? Because we can keep the file saved as user-data.sh , which means we get syntax highlighting in our IDE of choice as we make changes.

shell

		
				# ./src/salt-master/user-data.sh
#!/usr/bin/env sh
apt update
DEBIAN_FRONTEND=noninteractive apt install -y python-zmq python-systemd python-tornado salt-common salt-master

LOCAL_IPv4=$(ip addr | grep -E -o '10\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}')

cat <<EOF >/etc/salt/master.d/listen-interface.conf
interface: ${LOCAL_IPv4}
EOF

systemctl restart salt-master

Creating the Server

Lastly, we need to create the device with the Packet API. We use the Pulumi SDK to do so. I'm explicitly importing the required types that I need to use as much of the type system as I can.

Instead of hard coding strings like c1-large-arm , I can rely on enums provided by the SDK to verify that my string is correct. We do this for BillingCycles , OperatingSystems , and IpAddressTypes too. Funky! 🦩

typescript

		
				import {
  Device,
  IpAddressTypes,
  OperatingSystems,
  Plans,
  Facilities,
  BillingCycles,
} from "@pulumi/packet";

const saltMaster = new Device(`master-${name}`, {
  hostname: name,
  plan: Plans.C1LargeARM,
  facilities: [Facilities.AMS1],
  operatingSystem: OperatingSystems.Debian9,
  billingCycle: BillingCycles.Hourly,
  ipAddresses: [
    { type: IpAddressTypes.PrivateIPv4, cidr: 31 },
    {
      type: IpAddressTypes.PublicIPv4,
    },
  ],
  projectId: platform.project.id,
  userData: bootstrapScript,
});

Consuming our Function

Next up, we can call our createSaltMaster function from ./src/index.ts and we'll have a server with the correct user data for running our salt-master .

typescript

		
				const saltMaster = createSaltMaster(platform, "master-1");
export const saltMasterPublicIp = saltMaster.device.accessPublicIpv4;

We're going to export it's public IPv4 address; so that we can access it easily later and SSH into the machine later.

Creating the SaltMinion

typescript

		
				// ./src/salt-minion/index.ts
import { interpolate } from "@pulumi/pulumi";
import {
  BillingCycles,
  Device,
  Facilities,
  OperatingSystems,
  Plans,
} from "@pulumi/packet";
import { Platform } from "../platform";
import { SaltMaster } from "../salt-master";
import * as fs from "fs";
import * as path from "path";
import * as mustache from "mustache";

type SaltMinion = {
  device: Device;
};

export const createSaltMinion = (
  platform: Platform,
  name: string,
  master: SaltMaster
): SaltMinion => {
  const bootstrapString = fs
    .readFileSync(path.join(__dirname, "./user-data.sh"))
    .toString();

  const saltMinion = new Device(`minion-${name}`, {
    hostname: name,
    plan: Plans.C1LargeARM,
    facilities: [Facilities.AMS1],
    operatingSystem: OperatingSystems.Debian9,
    billingCycle: BillingCycles.Hourly,
    projectId: platform.project.id,
    userData: master.device.accessPrivateIpv4.apply((ipv4) => {
      return mustache.render(bootstrapString, { master_ip: ipv4 });
    }),
  });

  return {
    device: saltMinion,
  };
};

I'm not going to go over the createSaltMinion code like I did the master, as they're rather similar. Lets cover the major difference.

Mustache interpolation

Unlike our Salt master user-data, we need to use the mustache interpolation for the minion. This is because our minions need to know where the master is, and this requires accessing some state from the Pulumi provisioning; namely the IPv4 address of our master. For this, we use mustache template syntax:

shell

		
				cat <<EOF >/etc/salt/minion.d/salt-master.conf
master: {{ master_ip }}
EOF

Instead of passing {} to the mustache render function, we now need to provide some variables. I know this looks weird, so lets explain.

typescript

		
				userData: master.device.accessPrivateIpv4.apply((ipv4) => {
  return mustache.render(bootstrapString, { master_ip: ipv4 });
}),

The device.accessPrivateIpv4 variable is a Pulumi Output . That means that the variable isn't available until the server has been provisioned. So we can't access that variable directly, because its an promise that the value will exist later. As such, we need to use the apply function that's provided which defers the execution of the calling code until the server has been provisioned, thus the IP address is known.

This is where most people, including me, trip up with Pulumi at the beginning. Definitely read the programming model guide, which really helps understand what is going on; as well, it provides some good tips for working with outputs and inputs.

Other than this, our code is pretty much the same as the master. Our user data is a little different, but I'm going to assume that's self-explanatory too. Here's the user data, for completeness.

shell

		
				# ./src/salt-minion/user-data.sh
#!/usr/bin/env sh
apt update
DEBIAN_FRONTEND=noninteractive apt install -y python-zmq python-systemd python-tornado salt-common salt-minion

cat <<EOF >/etc/salt/minion.d/salt-master.conf
master: {{ master_ip }}
EOF

systemctl restart salt-minion

Spinning up the Stack

Before we spin this up, we need to provide our Packet API key as we create the stack.

shell

		
				pulumi stack init production

This will prompt you for a password which is used to encrypt your secrets, namely your Packet API key. This is because we're using a local provider for state. When you do this for your real production deployments, use the Pulumi managed service or Cloud KMS .

shell

		
				pulumi config set --secret packet:authToken
pulumi up

It'll show you a "plan" which you can now confirm and run if you're happy.

shell

		
				Previewing update (production):
     Type                     Name                         Plan
 +   pulumi:pulumi:Stack      pulumi-saltstack-production  create
 +   ├─ packet:index:Project  pulumi-saltstack-example     create
 +   ├─ packet:index:Device   master-master-1             create
 +   └─ packet:index:Device   minion-minion-1              create

Resources:
    + 4 to create

Do you want to perform this update?  [Use arrows to move, enter to select, type to filter]
  yes
  no
  details

That's it! Accept the plan, and you'll have a Salt Master and Salt Minion running on bare metal on Packet cloud.

Confirming the Awesome

Of course, I don't expect you to take my word for it that this "just works" (it does).

Lets use the pulumi stack command to get the public IPv4 address of our Salt master.

shell

		
				pulumi stack output

Which gives us:

shell

		
				Current stack outputs (2):
    OUTPUT              VALUE
    projectName         pulumi-saltstack-example2
    saltMasterPublicIp  147.75.81.106

Now we can SSH into our Salt master and confirm the setup.

shell

		
				ssh -l root 147.75.81.106

We can check that our Salt minion is configured correctly by checking the key approval was requested.

shell

		
				salt-key -L

Accepted Keys:
Denied Keys:
Unaccepted Keys:
minion-1
Rejected Keys:

We have one unaccepted key! Awesome. We can automate the approval of that in another session. Lets accept the key and test out our Salt magic.

shell

		
				salt-key -A minion-1

The following keys are going to be accepted:
Unaccepted Keys:
minion-1
Proceed? [n/Y] y
Key for minion minion-1 accepted.

Lets try a test.ping , a Salt command that makes sure the minions can be contacted by the master.

shell

		
				salt minion-1 test.ping

minion-1:
    True

Finally, lets actually have our minion run a command and show us the output:

shell

		
				salt minion-1 cmd.run "ip addr"

minion-1:
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: enP2p1s0f2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
        link/ether 70:10:6f:b9:eb:49 brd ff:ff:ff:ff:ff:ff
    3: enP2p1s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
        link/ether 70:10:6f:b9:eb:49 brd ff:ff:ff:ff:ff:ff
    4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
        link/ether 70:10:6f:b9:eb:49 brd ff:ff:ff:ff:ff:ff
        inet 147.75.81.34/30 brd 147.75.81.35 scope global bond0
           valid_lft forever preferred_lft forever
        inet 10.80.41.131/31 brd 255.255.255.255 scope global bond0:0
           valid_lft forever preferred_lft forever
        inet6 2604:1380:2000:5500::1/127 scope global
           valid_lft forever preferred_lft forever
        inet6 fe80::7210:6fff:feb9:eb49/64 scope link
           valid_lft forever preferred_lft forever

Until next time 😀