Mojo: Juju Service Orchestration distilled part 3

This is the third in a series of three articles about Mojo. In the first article we introduced Mojo, and in the second article we took a closer look at specifications and manifests. Today we’re going to walk through an example service deployment so you can see Mojo in action.

The only things you’ll need to be able to follow along with this example are a computer running Ubuntu Trusty or Utopic (to be able to install Mojo from our PPA), and access to a Juju environment. If you don’t already have Juju set up, this page should get you started.

The example service we’re going to deploy is the documentation website for Mojo itself, which can be found at mojo.canonical.com. It’s a relatively trivial service consisting of static content served by Apache but there’s enough detail to give you a good idea of how you could deploy more complex services for yourself, and how to structure a Mojo specification branch to allow for close collaboration between developers and operations.

So first of all, we need to install Mojo. We can do that as follows:

sudo add-apt-repository ppa:mojo-maintainers/ppa
sudo apt-get install mojo

Now let’s create a “project” for the example service we’re going to deploy:

sudo mojo project-new --series trusty mojo-how-to

What’s happening here is that Mojo is creating a top level project directory (/srv/mojo/mojo-how-to) as well as an LXC for running build commands. It runs the build commands in LXC with no network access to ensure that you have a repeatable build process. Everything needed for the build process to work should be already downloaded in the collect step, and this ensures you have no dependencies in your build process that you’re not aware of.

project-new is the only command you need to run with sudo. All other commands are run as a regular user, and Mojo will prompt you for your sudo password if appropriate. We’re planning to use unprivileged containers for Mojo, but until we do, there’s some manual fixes needed to permissions for things to work as expected:

sudo chmod 755 /var/lib/lxc/mojo-how-to.trusty
sudo chmod 755 /var/lib/lxc
# Please be aware of
# https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1244635 - Mojo
# will move to non-root LXC for trusty and later, which will provide a
# longer term fix for this.

Now we need to create a “workspace”, which is just a place where Mojo will assemble all the deployment artifacts and initiate the actual deployment from:

mojo workspace-new --project mojo-how-to --stage=mojo-how-to/devel --series trusty lp:~mojo-maintainers/mojo/mojo-specs mojo-how-to

At this point we’re ready to begin the deployment. Make sure you’ve bootstrapped the Juju environment that you want to deploy this example service into. Then we can actually kick off the deployment as follows:

mojo run --project mojo-how-to --series trusty --stage mojo-how-to/devel lp:~mojo-maintainers/mojo/mojo-specs mojo-how-to

So let’s take a detailed look at what’s happening at each stage from that one command above. To remind ourselves of what we’re doing, let’s take a look at the manifest file:

# We need the markdown package to be able to generate the docs for Mojo
builddeps packages=make,markdown
# Run the collect step
collect
# Run the build step
build
# Create our charm repository
repo
# Pull in any secrets - this is only used in the production stage
secrets
# Deploy services only
deploy config=services local=landscape
# Copy our built resources to the instances
script config=upload-built-content
# And now deploy relations as well
deploy config=relations
# Run verify steps
include config=manifest-verify

First of all, we’re confirming we have “make” and “markdown” installed in the LXC we’ll use to build this service. This is because we’re going to generate the documentation using a Makefile target that converts markdown files to html.

$ mojo run --project mojo-how-to --series trusty --stage mojo-how-to/devel /home/mthaddon/repos/mojo/mojo-specs mojo-how-to
[sudo] password for mthaddon:
2015-01-26 10:39:18 [INFO] All changes applied successfully.
2015-01-26 10:39:18 [INFO] Retrieve the spec's manifest
2015-01-26 10:39:18 [INFO] Manifest comment:

#############################################################################
We need the markdown package to be able to generate the docs for Mojo
#############################################################################

2015-01-26 10:39:18 [INFO] Installing apt repos and packages
2015-01-26 10:39:18 [INFO] Running command in container 'mojo-how-to.trusty': sudo -E -u root env DEBIAN_FRONTEND=noninteractive apt-get update
Ign http://archive.ubuntu.com trusty InRelease
Ign http://archive.ubuntu.com trusty-updates InRelease
Hit http://archive.ubuntu.com trusty Release.gpg
Get:1 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]
Hit http://archive.ubuntu.com trusty Release
Get:2 http://archive.ubuntu.com trusty-updates Release [62.0 kB]
Ign http://security.ubuntu.com trusty-security InRelease
Hit http://archive.ubuntu.com trusty/main i386 Packages
Hit http://archive.ubuntu.com trusty/restricted i386 Packages
Hit http://archive.ubuntu.com trusty/universe i386 Packages
Get:3 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Hit http://archive.ubuntu.com trusty/multiverse i386 Packages
Hit http://archive.ubuntu.com trusty/main Translation-en
Hit http://archive.ubuntu.com trusty/main Translation-en_GB
Hit http://archive.ubuntu.com trusty/multiverse Translation-en
Hit http://archive.ubuntu.com trusty/multiverse Translation-en_GB
Get:4 http://security.ubuntu.com trusty-security Release [62.0 kB]
Hit http://archive.ubuntu.com trusty/restricted Translation-en
Hit http://archive.ubuntu.com trusty/restricted Translation-en_GB
Hit http://archive.ubuntu.com trusty/universe Translation-en
Hit http://archive.ubuntu.com trusty/universe Translation-en_GB
Get:5 http://archive.ubuntu.com trusty-updates/main i386 Packages [397 kB]
Get:6 http://security.ubuntu.com trusty-security/main i386 Packages [190 kB]
Get:7 http://archive.ubuntu.com trusty-updates/restricted i386 Packages [8846 B]
Get:8 http://archive.ubuntu.com trusty-updates/universe i386 Packages [241 kB]
Get:9 http://security.ubuntu.com trusty-security/restricted i386 Packages [8846 B]
Get:10 http://archive.ubuntu.com trusty-updates/multiverse i386 Packages [9558 B]
Hit http://archive.ubuntu.com trusty-updates/main Translation-en
Hit http://archive.ubuntu.com trusty-updates/multiverse Translation-en
Hit http://archive.ubuntu.com trusty-updates/restricted Translation-en
Hit http://archive.ubuntu.com trusty-updates/universe Translation-en
Get:11 http://security.ubuntu.com trusty-security/universe i386 Packages [84.9 kB]
Get:12 http://security.ubuntu.com trusty-security/multiverse i386 Packages [1412 B]
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/multiverse Translation-en
Hit http://security.ubuntu.com trusty-security/restricted Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Fetched 1067 kB in 3s (303 kB/s)
Reading package lists... Done
2015-01-26 10:39:24 [INFO] Running command in container 'mojo-how-to.trusty': sudo -E -u root env DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2015-01-26 10:39:25 [INFO] Running command in container 'mojo-how-to.trusty': sudo -E -u root apt-get -y install make markdown
Reading package lists... Done
Building dependency tree
Reading state information... Done
make is already the newest version.
markdown is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Next, we’re running the collect phase to pull down charms as well as the Mojo codebase itself, which we’ll generate the html documentation from:

2015-01-26 10:39:25 [INFO] Manifest comment:

#############################################################################
Run the collect step
#############################################################################

2015-01-26 10:39:25 [INFO] Building resource tree
Updating mojo from parent (bzr+ssh://bazaar.launchpad.net/~mthaddon/mojo/self-hosting-docs)
Updating apache2 from parent (bzr+ssh://bazaar.launchpad.net/+branch/charms/trusty/apache2)
Updating content-fetcher from parent (bzr+ssh://bazaar.launchpad.net/~gnuoy/charms/precise/content-fetcher/trunk)
Updating nrpe-external-master from parent (bzr+ssh://bazaar.launchpad.net/+branch/charms/precise/nrpe-external-master)

Next we run the build step, which generates a tar file with the rendered html, css and images from the Mojo codebase:

2015-01-26 10:39:46 [INFO] Manifest comment:

#############################################################################
Run the build step
#############################################################################

2015-01-26 10:39:46 [INFO] Running script build
2015-01-26 10:39:47 [INFO] Running command in container 'mojo-how-to.trusty': sudo -E -u root getent passwd mthaddon
mthaddon:x:1000:1000::/home/ubuntu:/bin/bash
2015-01-26 10:39:47 [INFO] Running command in container 'mojo-how-to.trusty': env MOJO_WORKSPACE_DIR=/srv/mojo/mojo-how-to/trusty/mojo-how-to MOJO_STAGE=mojo-how-to/devel MOJO_BUILD_DIR=/srv/mojo/mojo-how-to/trusty/mojo-how-to/build MOJO_REPO_DIR=/srv/mojo/mojo-how-to/trusty/mojo-how-to/charms MOJO_LOCAL_DIR=/srv/mojo/mojo-how-to/trusty/mojo-how-to/local MOJO_SPEC_DIR=/srv/mojo/mojo-how-to/trusty/mojo-how-to/spec MOJO_PROJECT=mojo-how-to MOJO_SERIES=trusty MOJO_LOG_DIR=/srv/mojo/mojo-how-to/trusty/mojo-how-to/log MOJO_WORKSPACE=mojo-how-to sudo -E -u mthaddon /srv/mojo/mojo-how-to/trusty/mojo-how-to/spec/mojo-how-to/devel/build
Cleaning documentation directory...
cd docs/www && rm -f *.html
Generatiing html documentation in docs/www...
for mdfile in docs/mojo/*.md; do \
htmlfile=${mdfile%.*}.html ; \
htmlfile=${htmlfile/mojo/www} ; \
cp docs/templates/header.html ${htmlfile} ; \
markdown $mdfile >> ${htmlfile} ; \
cat docs/templates/footer.html >> ${htmlfile} ; \
done
./
./images/
./images/mojo-brand.png
./images/mojo-jenkins.png
./hacking.html
./mojo-insights-article-introduction.html
./css/
./css/mojo-scratch.css
./readme.html
./index.html
./mojo-insights-article-deeper-dive.html

So we’ve generated our tar file with the rendered html documentation, but now we also need to create a local charm repository to deploy from based on the Charms we pulled down in the collect phase:

2015-01-26 10:39:47 [INFO] Manifest comment:

#############################################################################
Create our charm repository
#############################################################################

2015-01-26 10:39:47 [INFO] Build a charm repository
2015-01-26 10:39:47 [INFO] /srv/mojo/mojo-how-to/trusty/mojo-how-to/build/apache2 => /srv/mojo/mojo-how-to/trusty/mojo-how-to/charms/trusty/apache2 (copy)
2015-01-26 10:39:47 [INFO] /srv/mojo/mojo-how-to/trusty/mojo-how-to/build/content-fetcher => /srv/mojo/mojo-how-to/trusty/mojo-how-to/charms/trusty/content-fetcher (copy)
2015-01-26 10:39:47 [INFO] /srv/mojo/mojo-how-to/trusty/mojo-how-to/build/nrpe-external-master => /srv/mojo/mojo-how-to/trusty/mojo-how-to/charms/trusty/nrpe-external-master (copy)

Next we pull in any secrets needed. As you can see from the comment we only need to do this as part of the production “stage”. The reason for this is that when we’re deploying the production “stage” we want the service to be served on https as well as http. This would mean that for the production “stage” to work, we’d need to already generated the SSL certificates for the service, and put them in the right directory so that Mojo can pick them up. We don’t want the SSL certificates (or other secrets) to be included in the Mojo specification branch because we want to be able to share the specification branch with as many people as possible, but we obviously need to restrict access to the SSL certificates themselves. Since we’re running with a “stage” of mojo-how-to/devel, the secrets phase is a no-op for us:

2015-01-26 10:39:47 [INFO] Manifest comment:

#############################################################################
Pull in any secrets - this is only used in the production stage
#############################################################################

2015-01-26 10:39:47 [INFO] Pulling secrets from /srv/mojo/LOCAL/mojo-how-to/mojo-how-to/devel to /srv/mojo/mojo-how-to/trusty/mojo-how-to/local

And now we get to actually deploying the service. Initially we’re just deploying the services (not the relations). We’re doing this because we want to copy the built resources (html documentation) to our apache2 unit before adding the relation to the content-fetcher subordinate charm, as it will depend on those resources already being there.

This highlights a nice pattern of how you can break your deployments into component parts if you want to. Another example of why you might need to do this is if some services require one instance of a service to be deployed first, and then subsequent units to be added later for “leader” status to be established.

In any case, here’s our services deployment:

2015-01-26 10:39:47 [INFO] Manifest comment:

#############################################################################
Deploy services only
#############################################################################

2015-01-26 10:39:47 [INFO] Running juju-deployer
Running: python /usr/local/bin/juju-deployer -c /srv/mojo/mojo-how-to/trusty/mojo-how-to/charms/mojo-how-to/devel/services -c /srv/mojo/mojo-how-to/trusty/mojo-how-to/charms/local.cfg -s 60 -d mojo-how-to -W -u
2015-01-26 10:39:49 [DEBUG] deployer.cli: Using runtime GoEnvironment on canonistack
2015-01-26 10:39:49 [INFO] deployer.cli: Starting deployment of mojo-how-to
2015-01-26 10:39:51 [DEBUG] deployer.env: Connected to environment
2015-01-26 10:39:51 [DEBUG] deployer.import: Getting charms...
2015-01-26 10:39:51 [DEBUG] deployer.deploy: Resolving configuration
2015-01-26 10:39:51 [INFO] deployer.import: Deploying services...
2015-01-26 10:39:51 [DEBUG] deployer.import: <deployer.env.go.GoEnvironment object at 0xb67d944c>
2015-01-26 10:39:52 [INFO] deployer.import:  Deploying service apache2 using local:trusty/apache2
2015-01-26 10:40:07 [DEBUG] deployer.import:  Waiting for deploy delay
2015-01-26 10:41:07 [INFO] deployer.import:  Deploying service content-fetcher using local:trusty/content-fetcher
2015-01-26 10:41:14 [DEBUG] deployer.import:  Waiting for deploy delay
2015-01-26 10:42:14 [INFO] deployer.import:  Deploying service nrpe using local:trusty/nrpe-external-master
2015-01-26 10:42:19 [DEBUG] deployer.import:  Waiting for deploy delay
2015-01-26 10:43:24 [DEBUG] deployer.import: Adding units...
2015-01-26 10:43:25 [DEBUG] deployer.import:  Service 'apache2' does not need any more units added.
2015-01-26 10:43:25 [WARNING] deployer.import: Config specifies num units for subordinate: content-fetcher
2015-01-26 10:43:25 [WARNING] deployer.import: Config specifies num units for subordinate: nrpe
2015-01-26 10:43:25 [DEBUG] deployer.import: Waiting for units before adding relations
2015-01-26 10:49:27 [DEBUG] deployer.env:  Delta unit: apache2/0 change:installed
2015-01-26 10:50:17 [DEBUG] deployer.env:  Delta unit: apache2/0 change:installed
2015-01-26 10:50:32 [DEBUG] deployer.env:  Delta unit: apache2/0 change:started
2015-01-26 10:50:32 [INFO] deployer.import: Adding relations...
2015-01-26 10:50:33 [INFO] deployer.import:  Exposing service 'apache2'
2015-01-26 10:50:34 [INFO] deployer.cli: Deployment complete in 646.23 seconds

As mentioned above, we now want to copy our built resources to the apache2 unit:

2015-01-26 10:50:34 [INFO] Manifest comment:

#############################################################################
Copy our built resources to the instances
#############################################################################

2015-01-26 10:50:34 [INFO] Running script upload-built-content

Having done that, we can create the relations between apache2 and the content-fetcher charm, as well as nrpe for our service checks:

2015-01-26 10:50:58 [INFO] Manifest comment:

#############################################################################
And now deploy relations as well
#############################################################################

2015-01-26 10:50:58 [INFO] Running juju-deployer
Running: python /usr/local/bin/juju-deployer -c /srv/mojo/mojo-how-to/trusty/mojo-how-to/charms/mojo-how-to/devel/relations -s 60 -d mojo-how-to -W -u
2015-01-26 10:51:00 [DEBUG] deployer.cli: Using runtime GoEnvironment on canonistack
2015-01-26 10:51:00 [INFO] deployer.cli: Starting deployment of mojo-how-to
2015-01-26 10:51:05 [DEBUG] deployer.env: Connected to environment
2015-01-26 10:51:05 [DEBUG] deployer.import: Getting charms...
2015-01-26 10:51:05 [DEBUG] deployer.deploy: Resolving configuration
2015-01-26 10:51:05 [INFO] deployer.import: Deploying services...
2015-01-26 10:51:05 [DEBUG] deployer.import: <deployer.env.go.GoEnvironment object at 0xb684c44c>
2015-01-26 10:51:06 [DEBUG] deployer.import:  Service 'apache2' already deployed. Skipping
2015-01-26 10:51:06 [DEBUG] deployer.import:  Service 'content-fetcher' already deployed. Skipping
2015-01-26 10:51:06 [DEBUG] deployer.import:  Service 'nrpe' already deployed. Skipping
2015-01-26 10:51:11 [DEBUG] deployer.import: Adding units...
2015-01-26 10:51:13 [DEBUG] deployer.import:  Service 'apache2' does not need any more units added.
2015-01-26 10:51:13 [WARNING] deployer.import: Config specifies num units for subordinate: content-fetcher
2015-01-26 10:51:13 [WARNING] deployer.import: Config specifies num units for subordinate: nrpe
2015-01-26 10:51:13 [DEBUG] deployer.import: Waiting for units before adding relations
2015-01-26 10:51:14 [INFO] deployer.import: Adding relations...
2015-01-26 10:51:15 [INFO] deployer.import:  Adding relation apache2 <-> content-fetcher
2015-01-26 10:51:17 [INFO] deployer.import:  Adding relation apache2 <-> nrpe
2015-01-26 10:51:19 [DEBUG] deployer.import: Waiting for relation convergence 60s
2015-01-26 10:56:42 [DEBUG] deployer.env:  Delta unit: content-fetcher/0 change:started
2015-01-26 11:01:42 [DEBUG] deployer.env:  Delta unit: nrpe/0 change:started
2015-01-26 11:01:42 [INFO] deployer.cli: Deployment complete in 642.69 seconds

Finally, having deployed the entire service, we want to run checks that it’s configured as we expect. In this case, we’re going to run a script that connects to the apache2 instance and runs all of the nagios checks that have been configured as a result of the relation between apache2 and nrpe-external-master:

2015-01-26 11:01:42 [INFO] Manifest comment:

#############################################################################
Run verify steps
#############################################################################

2015-01-26 11:01:42 [INFO] Manifest comment:

#############################################################################
The service is up and running, let's verify it
#############################################################################

2015-01-26 11:01:42 [INFO] Running script verify
10.55.32.177: + /usr/lib/nagios/plugins/check_disk -u GB -w 25% -c 20% -K 5% -p /
10.55.32.177: DISK OK - free space: / 8 GB (90% inode=90%);| /=0GB;6;7;0;9
10.55.32.177: + /usr/lib/nagios/plugins/check_load -w 8,8,8 -c 15,15,15
10.55.32.177: OK - load average: 1.82, 1.83, 1.37|load1=1.820;8.000;15.000;0; load5=1.830;8.000;15.000;0; load15=1.370;8.000;15.000;0;
10.55.32.177: + /usr/lib/nagios/plugins/check_swap -w 90% -c 75%
10.55.32.177: SWAP OK - 100% free (0 MB out of 0 MB) |swap=0MB;0;0;0;0
10.55.32.177: + /usr/lib/nagios/plugins/check_procs -w 125 -c 150
10.55.32.177: PROCS OK: 83 processes | procs=83;125;150;0;
10.55.32.177: + /usr/lib/nagios/plugins/check_users -w 20 -c 25
10.55.32.177: USERS OK - 0 users currently logged in |users=0;20;25;0
10.55.32.177: + /usr/lib/nagios/plugins/check_procs -w 3 -c 6 -s Z
10.55.32.177: PROCS OK: 0 processes with STATE = Z | procs=0;3;6;0;
########################
# Nagios Checks Passed #
########################
########################
# Successfully verified #
########################

So now we’ve deployed our entire service, and checked that it works as expected.

A few closing points worth mentioning.

Firstly, because our manifest file ends with include config=manifest-verify we see that we could run mojo run --manifest-file manifest-verify to just run the verification steps defined in that file. This gives us a repeatable verification step for this service, but you can see how you could use a similar pattern to perform any kind of repeatable operation for your service, such as a content update, Charm updates or scaling out. Just create a “sub-manifest” file with the steps needed, including verification of the changes you’ve made, and then it can be run with the --manifest-file option against which environment you’re operating on, whether that’s development or production.

And secondly, we ran this with the --stage of mojo-how-to/devel. To deploy the live service itself we also did mojo run, but used the --stage of mojo-how-to/production. The differences are easy to see in the specification, so it’s clear to developer and the operations team what’s happening where:

• 2 units vs. 1 for redundancy
• SSL certificates and an apache vhost config that’s listening on https
• Using the landscape-client Charm to register our instances in a Landscape server which can manage package upgrades for us on an ongoing basis

Thanks for reading these posts about Mojo. If you want to get involved, please see our hacking page, and help us improve Mojo!

About the author

Tom Haddon is a squad lead within Canonical’s IS department. He manages a globally distributed team of senior systems administrators rotating between three functions: Projects, Operations and Webops (devops). He has a strong focus on cloud technologies including OpenStack, Juju and MAAS.