Introduction to Continuous Integration/Deployment using CircleCI

At Enki, a major component of our Continuous Integration and Continuous Deployment culture is CircleCI.

Each time a push is made to one of our repositories, CircleCI will start running a series of steps to reproduce the production environment inside of a fresh, and isolated container, build necessary Docker images, run the tests, linting etc., and let us know if anything goes wrong.

If the branch being pushed to is one of the deploy-specific branches, CircleCI will also deploy to the appropriate cloud service.

Continuous Integration/Deployment Process

This allows us to have an automated self-documenting Continuous Integration/Deployment pipeline, enabling us to safely ship code at a fast pace.

Overview

Every CircleCI config at Enki is built using Workflows.

Note: CircleCI config file is written in YAML and stored at the root level of your project at the your-project/.circleci/config.yml path) .

A workflow is a set of rules for defining a collection of jobs and their run order.

Workflows enable us to split each distinct devops process into a single atomic unit (called job), allowing us to modularize our deploy pipeline and increase development speed.

Since each job is an independent process, we can easily construct workflows where independent jobs run in parallel making the process more efficient and failed jobs can be restarted without needing to restart the whole build.

Each CircleCI config file is split into two main sections, jobs and workflows.

Note: A CircleCI config file starts with the version step which helps CircleCI issue warnings about deprecations and breaking changes.

Jobs

The first top-level section in a CircleCI config is called jobs.

Within the jobs section, each individual job starts with a job name and is usually split into steps that represent chunks of the job process. A job can also contain items such as the current working directory (by default ~/project), executors (environment in which the job runs, for example MacOS), environment variables, etc.

Note: the machine property in the config below denotes an executor that allows us to have a dedicated, ephemeral VM environment with full access to OS (Linux) resources. That’s how we have access to the echo command.

Here’s an example of the above config running.

Workflows

The second top-level section in a CircleCI config is called workflows.

This section decides in which order the jobs will run and on which branches or tags.

Note: The workflows section starts with the version step for the same reason as the entire config (warnings about breaking changes and deprecations).

The top-level item in workflows is the name of a workflow (our common convention is to use a concatenation of all job names within that workflow with an _ in between. For example build_test_deploy).

Note: we can have multiple workflows in the workflows section.

Each workflow has a jobs field that specifies the order in which jobs will run and on which branch or tag filters.

Note: workflows can also have triggers. By default, the trigger is pushing to a branch.

Each job within a workflow, among other things, can have a:

• name.
• requires field specifying which jobs it depends on.
• type field that can be specified as approval and make this job require manual approval before proceeding.
• filters field allowing us to specify for which branches or tags this job will run.

Here’s an example:

This config will generate a pipeline where install runs first, then build and lint run in parallel, then deploy runs (if we are executing on either the develop or master branch).

Here’s an example of a config like the one above running on a deploy-branch (i.e. master)

Example CircleCI Workflow for a deploy branch (in our case “master” or “develop”)

and on a non-deploy branch (for this pull request) which skips the deploy step.

Example CircleCI Workflow for a non-deploy branch

In terms of JavaScript, the above workflow can be thought of as:

With above in mind, here’s how a real-word Continuous Integration/Deployment Pipeline could look like:

CircleCI Continuous Integration/Deployment Pipeline that automatically ships code to AWS

Caveats

Sharing data between jobs

Any workflow job that is creating data that should be shared with other jobs has to use the persist-to-workspace step.

One example could be the install job that runs npm install and has to persist the node_modules directory so it can be shared with other jobs:

This would allow other jobs, for example lint, to use the eslint installed in the install job by attaching to the same workspace into which we persisted the node_modules (by using the attach_workspace step):

Here’s a simplified workflow example (for this config), containing an install step that installs cowsay, and a use step that uses it to display the following:

Another example could be a build step that persist the dist folder, created when building a frontend app, to the deploy step that then uploads it to S3.

Let’s consider the install_build_lint_deploy process from above:

Here’s how the corresponding config would look like for a pipeline that ships a website to an S3 bucket called example.com:

All jobs in the config above share data via the workspace.

• The install job persist the node_modules
• The lint job uses eslint from node_modules
• The build job persist the dist directory (that is built together with the node_modules)
• The deploy job sends the dist directory to S3

Note: it is common practice to actually cache the node_modules so that we do not need to re-install dependencies if nothing has changed.

Summary

Hopefully, this was a useful introduction to CircleCI. Demonstrated above are only some of the capabilities but CircleCI has many more.

So far we’ve learned that CircleCI Config is comprised of two things:

1. jobs, which are standalone definitions of devops processes (installing dependencies, testing, etc)
2. workflows, which define how, when and in what order those devops processes run

As a next step, it might be useful to checkout public projects like React Native or Electron that are using CircleCI for their continuous integration pipelines.

Happy devopsing from the Enki team 😊