Pack Your Bags: Managing Distributed Applications with CNAB

Pack your bags

Managing Distributed Applications with CNAB

Workshop Setup

It can take a while for things to download and install over the workshop wifi, so please go to the workshop materials directory and follow the setup instructions to get all the materials ready.

Go to https://labs.play-with-docker.com/
Sign in with your Docker Hub account, or create one if you don’t already have an account.
Click Add new instance

Install Porter

curl https://deislabs.blob.core.windows.net/porter/latest/install-linux.sh | bash
export PATH=$PATH:~/.porter

Clone the workshop repository

git clone https://github.com/getporter/porter.git
cd porter/workshop

Agenda

What is CNAB? - 15 minutes
See some bundles - 10 minutes
Build a bundle from scratch - 20 minutes
Walkthrough Porter - 10 minutes
The state of CNAB - 5 minutes
Choose your own adventure - 30 minutes

Introductions

Carolyn Van Slyck

Senior Software Engineer

Microsoft Azure

Jeremy Rickard

Senior Software Engineer

Microsoft Azure

Stop and Say Hi

Move up to the front tables.
Introduce yourself to at least 2 people next to you.
Find out what they enjoy working on.
Share what talks or parts of Velocity they are looking forward to.

First A Quick Demo!

What is an application

What’s a CNAB?

When would you use a bundle?

Install the tools to manage your app: helm, aws/azure/gcloud, terraform
Deploy an app along with its infra: cloud storage, dns entry, load balancer, ssl cert
Get software and its dependencies into airgapped networks
Manage disparate operational tech: such as Helm, Chef, or Terraform, across teams and departments

Let’s Answer That With A Story!

The Cast

You!

Your friend!

Your friend!

Your App!

Your Fans!

Act One!

You Built an App

It Runs Happily In The Cloud

….your cloud

Act Two!

Your Friend Wants To Run It!

How exciting!

So you write extensive docs

You are no longer friends…

.footnote[https://vignette3.wikia.nocookie.net/spongebob/images/f/f0/Spongebob-patrick-crying.jpg/revision/latest?cb=20140713205315]

So you work together…

Then you help a few more people…

Act Three!

Suddenly McGuffin has FANS!

This won’t scale…

So what do we do…

Containers ship our application

… but don’t install everything

This is what CNAB solves

Package All The Logic To Make Your App Happen
Allow Consumer To Verify Everything They Will Install
Distribute Them In Verifiable Way

How it works

Try it out: Install a bundle

$ porter install --reference deislabs/porter-hello-devopsdays:latest

Anatomy of a Bundle

Application Images

CNAB doesn’t change this
Build your application like you do now

The Invocation Image

MSI for the Cloud
It’s a Docker Image
It contains all the tools you need to install your app
It contains configuration, metadata, templates, etc

The Invocation Image

The Bundle Descriptor

JSON!
List of the invocation image(s) with digests
List of the application image(s) with digests
Definitions for inputs and outputs
Can be signed

CNAB Specification

The Bundle format
Defines how things are passed into and out of the invocation image
A required entrypoint in invocation image
Well-defined verbs
- Install
- Upgrade
- Uninstall

Are we done?

We can install (complicated) things
We can verify what we are going to install
But how do we distribute bundles?

Sharing Images With OCI Registries

how docker shares

OCI Regristry ~ Docker Registry

Distributing App and Invocation Images is solved

–

So what about the bundle?

–

It turns out OCI can help here too…

OCI Registries Can Store Lots of Things

CNAB today is working within the OCI Spec (not optimal)
CNAB Spec group working with OCI to improve this

Sharing Bundles With OCI Registries

An Example: Azure MySQL + Wordpress

Let’s make some bundles!

https://porter.sh/pack-your-bags-msp/#setup

]

Tutorial

Whale Hello

porter create

$ porter create --help
Create a bundle. This generates a porter bundle in the current directory.

porter.yaml

The most important file. Edit and check this in. Everything else is optional.

README.md

Explains the other files in detail

Dockerfile.tmpl

Optional template for your bundle’s invocation image

.gitignore

Suggested set of files to ignore in git

.dockerignore

Suggested set of files to not include in your bundle

porter.yaml

mixins:
  - exec

name: HELLO
version: 0.1.0
description: "An example Porter configuration"
tag: getporter/porter-hello

install:
  - exec:
      description: "Install Hello World"
      command: bash
      flags:
        c: echo Hello World

    
  

Try it out: porter create

$ mkdir hello

$ cd hello

$ porter create
creating porter configuration in the current directory

$ ls
Dockerfile.tmpl  README.md  porter.yaml

porter build

$ porter build --help
Builds the bundle in the current directory by generating a Dockerfile 
and a CNAB bundle.json, and then building the invocation image.

Try it out: porter build

$ porter build

Copying dependencies ===>
Copying porter runtime ===>
Copying mixins ===>
Copying mixin exec ===>
Generating Dockerfile =======>
Writing Dockerfile =======>
Starting Invocation Image Build =======>
Generating Bundle File with Invocation Image porter-hello:latest =======>
Generating parameter definition porter-debug ====>

What did Porter do? 🔎

Dockerfile

FROM quay.io/deis/lightweight-docker-go:v0.2.0
FROM debian:stable-slim
COPY --from=0 /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

ARG BUNDLE_DIR

COPY .cnab /cnab
COPY . ${BUNDLE_DIR}
RUN rm -fr ${BUNDLE_DIR}/.cnab
# exec mixin has no buildtime dependencies

WORKDIR ${BUNDLE_DIR}
CMD ["/cnab/app/run"]

    
  

.cnab/

$ tree .cnab/
.cnab
├── app
│  ├── mixins
│  │  └── exec
│  │      ├── exec
│  │      └── exec-runtime
│  ├── porter-runtime
│  └── run
└── bundle.json

.cnab/app/run

#!/usr/bin/env bash
exec /cnab/app/porter-runtime run -f /cnab/app/porter.yaml

.cnab/bundle.json

{
    "description": "An example Porter bundle",
    "invocationImages": [
        {
            "image": "deislabs/porter-hello-devopsdays-installer:latest",
            "imageType": "docker"
        }
    ],
    "name": "porter-hello-devopsdays",
    "parameters": {
        "fields": {
            "porter-debug": {
                "definition": "porter-debug",
                "description": "Print debug information from Porter when executing the bundle",
                "destination": {
                    "env": "PORTER_DEBUG"
                }
            }
        }
    },
    "definitions": { ...

    
  

porter install

$ porter install --help
Install a bundle.

The first argument is the name of the claim to create for the installation. 
The claim name defaults to the name of the bundle.

Flags:
  -c, --credential-set strings         Credential to use when installing the bundle. 
  -f, --file string          Path to the bundle file to install.
      --param strings        Define an individual parameter in the form NAME=VALUE.
      --param-file strings   Path to a parameters definition file for the bundle

CNAB: What Executes Where

cloud picture

Try it out: porter install

$ porter install

installing HELLO...
executing porter install configuration from /cnab/app/porter.yaml
Install Hello World
Hello World
execution completed successfully!

Talk Back

What tools do you use to deploy?
Is it to a cloud? On-premise?
Are you using a mix of tooling and platforms?
What does your deployment lifecycle look like?

A tale of two bundles

You can find these bundles in this repository under and workshop/scratch/azure and workshop/porter-tf/azure

CNAB The Hardway

Tutorial

Hi, My Name is _

Parameters

Variables in your bundle that you can specify when you execute the bundle and are loaded into the bundle either as environment variables or files.

Define a Paramer

parameters:
- name: name
  type: string
  default: llama

Use a Parameter

- "echo Hello, ${ bundle.parameters.name }

Needs double quotes around the yaml entry
Needs double curly braces around the templating
Uses the format bundle.parameters.PARAMETER_NAME

??? Explain defaults and when parameters are required

Try it out: Print Your Name

Modify the hello bundle to print “Hello, YOUR NAME”, for example “Hello, Aarti”, using a parameter.

Edit the porter.yaml to define a parameter named name.
Use the parameter in the install action and echo your name.
Rebuild your bundle with porter build.
Finally run porter install --param name=YOUR_NAME and look for your name in the output.

porter list

$ porter list
NAME          CREATED         MODIFIED        LAST ACTION   LAST STATUS
HELLO_LLAMA   5 seconds ago   3 seconds ago   install       success
HELLO         8 minutes ago   8 minutes ago   install       success

Claims

Claims are records of any actions performed by CNAB compliant tools on a bundle.

$ porter bundle show HELLO
Name: HELLO
Created: 8 minutes ago
Modified: 8 minutes ago
Last Action: install
Last Status: success

Cleanup Hello World

First run porter uninstall without any arguments:

$ porter uninstall
uninstalling HELLO...
executing porter uninstall configuration from /cnab/app/porter.yaml
Uninstall Hello World
Goodbye World
execution completed successfully!

Now run porter uninstall with the name you used for the modified bundle:

$ porter uninstall HELLO_LLAMA
uninstalling HELLO_LLAMA...
executing porter uninstall configuration from /cnab/app/porter.yaml
Uninstall Hello llama
Goodbye llama
execution completed successfully!

Publishing Bundles

Use porter publish to share bundles:

Publishes both the invocation image and the CNAB bundle manifest to an OCI registry.
Uses Docker tags for both

name: HELLO-LLAMA
version: 0.1.0
description: "An example Porter configuration with moar llamas"
tag: "YOURNAME/porter-hello-llama"

Modify the bundle

Let’s make this bundle yours by changing it from the deislabs Docker Hub registry to your own:

Open porter.yaml
Find the invocationImage and the tag fields, and then change deislabs to your docker username.

They should now look like this:
```
invocationImage: "YOURNAME/porter-hello-llama:latest"
tag: "YOURNAME/porter-hello-llama-bundle:latest"
```
Save and close the file.

porter publish

$ porter publish --help
Publishes a bundle by pushing the invocation image and bundle to a registry.


Examples:
  porter publish

Try it out: Publish your bundle

$ porter publish

If you run into trouble here, here are a few things to check:

Make sure that you updated the invocationImage and tag to use your docker hub username
Log into docker hub with docker login

Try it out: Install the bundle

$ porter install --reference YOURNAME/porter-hello-llama:v0.1.0 --param name=YOURNAME

BREAK

bring in the other slides that explain porter meta data, mixins, parameters and credentials

publish, install from our our tag before #4 I may be comforable with terraform and I know what credentials / parameters are needed The person who writes the bundle isn’t the person who runs the bundle

5 goes away

Tutorial

Wordpress

porter credentials generate

$ porter credentials generate --help
Generate a named set of credentials.

The first argument is the name of credential set you wish to generate. If not
provided, this will default to the bundle name. By default, Porter will
generate a credential set for the bundle in the current directory. You may also
specify a bundle with --file.

Bundles define 1 or more credential(s) that are required to interact with a
bundle. The bundle definition defines where the credential should be delivered
to the bundle, i.e. at /home/nonroot/.kube. A credential set, on the other hand,
represents the source data that you wish to use when interacting with the
bundle. These will typically be environment variables or files on your local
file system.

When you wish to install, upgrade or delete a bundle, Porter will use the
credential set to determine where to read the necessary information from and
will then provide it to the bundle in the correct location.

Wordpress Credential Mapping

~/.porter/credentials/wordpress.yaml

name: wordpress
credentials:
- name: kubeconfig
  source:
    path: /Users/carolynvs/.kube/config

    
  

porter.yaml

credentials:
- name: kubeconfig
  path: /home/nonroot/.kube/config

A quick hack

If you are using for Docker for Desktop with Kubernetes

Copy $HOME/.kube/config to $HOME/.kube/internal-config
Edit internal-config and change the server from localhost to host.docker.internal

apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://host.docker.internal:6443
  name: docker-for-desktop-cluster

Specify this config file for porter credentials generate on the next slide for the kubeconfig.

Try it out: porter credentials generate

Generate a set of credentials for the wordpress bundle in this repository.

Change to the wordpress directory under the workshop materials
Run porter build
Run porter credentials generate and follow the interactive prompts to create a set of credentials for the wordpress bundle.

??? we all do this together

Try it out: porter install –credential-set

Install the wordpress bundle and pass it the named set of credentials that you generated.

$ porter install --credential-set wordpress

Cleanup Wordpress

$ porter uninstall --credential-set wordpress

??? Explain why –credential-set is required again for uninstall

Authoring Bundles

Porter Manifest In-Depth

Metadata

name: porter-workshop-tf 
version: 0.1.0
description: "An example using Porter to build the from scratch bundle"
tag: getporter/porter-workshop-tf

Mixins

Declare any mixins that you are going to use

mixins:
  - azure
  - terraform

Parameters

Define parameters that the bundle requires.

parameters:
  - name: location
    type: string
    default: "EastUS"

  - name: server-name
    type: string

  - name: database-name
    type: string

    
  

Credentials

Define credentials that the bundle requires and where they should be placed in the bundle when it is executing:

environment variables (env)
files (path)

credentials:
  - name: subscription_id
    env: AZURE_SUBSCRIPTION_ID

  - name: tenant_id
    env: AZURE_TENANT_ID

  - name: client_id
    env: AZURE_CLIENT_ID

  - name: client_secret
    env: AZURE_CLIENT_SECRET

    
  

Credentials

Variables that can be specified when the bundle is executed that are associated with the identity of the user executing the bundle, and are loaded into the bundle either as environment variables or files.

They are mapped from the local system using named credential sets, instead of specified on the command-line.

Credentials vs. Parameters

Parameters

Application Configuration
Stored in the claim
🚨 Available in plaintext on the local filesystem

Credentials

Identity of the user executing the bundle
Is not stored in the claim
Has to be presented every time you perform an action

Credentials, Passwords and Sensitive Data

Credentials are for data identifying data associated with a user. They are re-specified every time you run a bundle, and are not stored in the claim.
Parameters can store sensitive data using the sensitive flag. This prevents the value from being printed to the console.
We (porter) and the CNAB spec are working on more robust storage mechanisms for claims with sensitive data, and better ways to pull data from secret stores so that they don’t end up on the file system unencrypted.

In all honesty this area is a work in progress. I would shove anything sensitive in a credential for now but be aware of the distinction and where the CNAB spec is moving.

Bundle Outputs

Often, a bundle will produce something. This might be a new certificate, a hostname, or some other piece of information. Bundle Outputs are used to provide this information back to the CNAB runtime.

outputs:
  - name: STORAGE_ACCOUNT_KEY
    type: string

View Bundle Outputs with Porter

$ porter bundle show


Outputs:
-------------------------------------------------------------------------------------------
  Name                 Type    Value
-------------------------------------------------------------------------------------------
  STORAGE_ACCOUNT_KEY  string  JKb9C+J+nFtGrDyBW4Y0zaIK5hzIvi2gW3SfnmnkcunyXSYV3HucQGNIo...

Custom Dockerfile

Specify a custom Dockerfile for the invocation image

Use a different base image
Add users, tweak the environment and configuration
Install tools and applications

⚠️ You are responsible for copying files into the bundle into ${BUNDLE_DIR}

porter.yaml

dockerfile: Dockerfile.tmpl

Dockerfile.tmpl

FROM debian:stable-slim

ARG BUNDLE_DIR

RUN apt-get install -y curl

COPY myscript.sh ${BUNDLE_DIR}

Try it out: Custom Dockerfile

Make a bundle that uses a custom dockerfile template and uses mcr.microsoft.com/azure-cli as its base image

Create a porter bundle in a new directory with porter create.
Modify the porter.yaml to uncomment out #dockerfile: Dockerfile.tmpl.
Edit Dockerfile.tmpl to use mcr.microsoft.com/azure-cli for the base image.
Edit the install action to run az help.
Build the bundle.
Install the bundle.

Porter’s Default Dockerfile

Uses Debian for the base image
Installs root ssl certificates

🎩✨ Automatically copies everything into the bundle for you

Actions and Steps

Actions

Actions map to the verbs you use when you use Porter.

porter install
porter upgrade
porter uninstall

These are defined in the CNAB specification. You can also define your own actions and run them

$ porter invoke --action myaction

Steps

Within an action you can define a series of ordered steps.

A step must complete successfully before the next step is executed.
A step is defined using a mixin. Each step can use only one mixin. So far we have used the exec mixin that lets you run commands.
Mixins must be declared ahead of time in the mixins section of the manifest.

install: # action
  - exec: # step
      description: "Install my application"
      command: bash
      arguments:
        - install-myapp.sh

    
  

Referencing files from the Manifest

We recommend referencing files using relative paths

$ tree
├── Dockerfile
├── README.md
├── porter.yaml
└── scripts
*    └── do-things.sh

porter.yaml

install:
  - exec:
      description: Do Things
      command: bash
      arguments:
*        - scripts/do-things.sh

    
  

Step Outputs

You tell the mixin what data to extract and put into an ouput
Each mixin output will look different, but they all require a name
Porter stores the output value, making it available to later steps
Not all mixins support outputs

Helm Mixin Output

install:
- helm3:
    ...
    outputs:
    - name: mysql-password
      secret: mydb
      key: mysql-password

    
  

Kubernetes Mixin Output

install:
  - kubernetes:
      description: "Create NGINX Deployment"
      manifests:
        - manifests/nginx
      wait: true
      outputs:
        - name: "IP_ADDRESS"
          resourceType: service
          resourceName: nginx-deployment
          jsonPath: "{.spec.clusterIP}"

    
  

Hot Wiring the Manifest

These variables are available for you to use in the manifest:

bundle.name
bundle.parameters.PARAMETER_NAME
bundle.credentials.CREDENTIAL_NAME
bundle.outputs.OUTPUT_NAME

Templating

Porter uses a template engine to substitute values into the manifest.

Needs double quotes around the yaml entry
Use double curly braces around the templated value

Example

connectionString: ${bundle.outputs.host}:${bundle.outputs.port}

Mixins

“Mixins are the lifeblood of Porter.

They adapt between CNAB and existing tools. Porter is just glue.”

Mixins Available Today

exec
kubernetes
helm
azure
aws
gcloud
terraform

Try it out: Install a Helm Chart

Make a new bundle and install the Helm chart for etcd-operator

Create a porter bundle in a new directory with porter create.
Modify the porter.yaml to use the helm mixin and define credentials for kubeconfig.
Using the helm mixin, install the latest stable/etcd-operator chart with the default values.
Build the bundle.
Generate credentials for your bundle.
Install your bundle.

Installing Mixins

Anyone can make a mixin and have people install it using Porter

porter mixin install terraform

What if I need a mixin that doesn’t exist? 😰

You can always use a custom dockerfile to install the tool you need and then execute the commands with the exec mixin.

Custom mixins are just easier to use, but aren’t necessary.

Make Your Own Mixin

What makes a mixin?

Executable written in any language
Communicates to Porter on stdin and stdout
Supports a few commands: build, schema, install, upgrade, uninstall
Translates the steps from the porter manifest to commands against any external tool or service

What mixin would you make?

Docker
Google Cloud
CloudFormation
AWS
Artifactory
Vault
Dominos 🍕
What else?

Now run your bundle

Run porter install --reference [your new tag]

Example tag of cnabaholic/hello-people:latest:

Run porter install --reference cnabaholic/hello-people:latest

How does it do that?

Updates the bundle with the new images and digests
Stores the bundle information as a combination:
- OCI manifest list annotations to the main manifest list for the tag
- A new manifest list for the bundle credentials and parameters.
When pulling a bundle, it reconstructs it from the parts mentioned above

See OCI Bundle Format for an example.

CNAB Tooling and SDKs

Porter
Docker App
Duffle
cnab-go

https://cnab.io/community-projects

State of CNAB

CNAB Core Spec is release 1.0 very soon
CNAB Signing Spec is underway
Tooling is finishing support for CNAB Core

Questions and Feedback

??? What should someone do if they are interested in CNAB for their work or personal projects? What is the timeline for the project and how should they be thinking about beginning to incorporate it?

Choose your own adventure!

ASCII Art Gophers - workshop/asciiart
Create a MySQL on Azure - workshop/porter-tf-aci
Manage VMs with gcloud - workshop/gcloud-compute
Manage Buckets with aws - workshop/aws-bucket