Jenkins Workflow: Scripting Out Complex Builds

In this second and final part of the series, we’ll be taking a look at the Jenkins Workflow plugin as a solution for setting up more complex Jenkins pipelines.

We’ll be picking up where part one of the series left off. In part one, Jeff Reifman guided you through setting up a Jenkins instance on Digital Ocean and creating your first Jenkins build. If you haven’t already read it yet, I suggest doing so before continuing. It’s okay—I’ll wait. I can be very patient…

… All caught up? Great! Let’s do this.

What Is Jenkins Workflow?

Jenkins Workflow is a plugin for Jenkins. Once installed, a new item type becomes available: a “Workflow”. Workflow projects can be used for the same purposes as regular “Freestyle” Jenkins projects, but they also have the ability to orchestrate much larger tasks that can span multiple projects, and even create and manage multiple workspaces in a single Workflow. What’s more, all of this management can be organized into a single script, rather than spread out across a collection of configurations, projects, and steps.

The Pre-Workflow

Before we can start building Workflows, we have to install the Jenkins Workflow plugin. From the Jenkins Dashboard, click Manage Jenkins, then Manage Plugins. Switch to the Available tab and search for “Workflow”.

Check the box for Workflow Plugin, then Install without restart.

Now, here’s the catch. There are multiple plugins named “Workflow Plugin”, so you’ll need to repeat the above steps several times. Alternatively, you could also click the multiple checkboxes or install the Workflow Aggregator plugin.

Once “Workflow Plugin” no longer shows up in the list of available plugins, then go ahead and restart Jenkins by navigating to /restart and clicking Yes.

A Simple Flow

Let’s get our Workflow-feet wet. We’ll take the project Jeff set up in part one and build it as a Workflow.

To get started, head to the Jenkins Dashboard and click New Item. Name the new item “Shell Test (Workflow)”, and select the Workflow type.

Click OK to create the new project. You’ll land on the project’s configuration page.

You’ll notice that the configuration options differ from standard Jenkins projects. There are no longer any options to add a GitHub repo, build steps, or post-build actions. Instead, there’s a new section called Workflow.

Within the Workflow section is a text box labeled Script. That is where you’ll be defining the Workflow script Jenkins will run when it initializes a build of the project.

“What type of script?” you ask. Excellent question. The Jenkins Workflow plugin uses a language called Groovy for its scripts. Groovy is a versatile scripting language for the JVM. Don’t worry, you don’t really need to know Groovy or Java to get things working—the Jenkins Workflow plugin uses a small DSL on top of Groovy, and it is very easy to combine commands to build out your project workflow.

Go ahead and add the following to the script box:

node {
  git 'https://github.com/redhotvengeance/hello-jenkins.git'
  sh 'uptime'
}

So what’s happening here?

First off, we open a node block. Nodes are where Workflow actions happen. When you allocate a node, a new workspace (a context/folder) is created. All of the code within the node block executes within that workspace. This helps ensure that build steps don’t pollute each other.

Next, we run a Git command with git 'https://github.com/redhotvengeance/hello-jenkins.git'. This command clones the Git repo into our workspace.

Lastly, we tell Workflow to execute the uptime shell command with sh 'uptime'.

Click Save, and you’ll be taken to the project landing page. In the left-hand menu is a button labeled Build Now. Click it to kick off a build.

Once the build is complete, click on build #1 found in the Build History section. Then click Console Output in the left-hand menu.

Here we can see everything logged while the build was executed. It started by allocating a node in the “Shell Test (Workflow)” workspace. Then it fetched the Git repo. Lastly, it executed the uptime shell script, which printed the server uptime stats.

And that’s it! We’ve now recreated the same steps as the normal Jenkins project setup in part one, except this time as a Workflow. Now let’s use these same concepts to do something a bit more complex.

Fake It Until You Make It

Before we can create our complex Workflow, we need the work that will flow through it. Fake project(s) to the rescue!

Since we’re already using Groovy to script our Workflow, let’s use Gradle for our fake projects. Gradle is a build system that uses Groovy (surprising, I know!). To use Gradle, we’ll need to install it on our server. SSH into your server (check out Jeff’s part one if you need to refresh your memory) and run:

sudo apt-get install gradle

There—we’re good to go.

We’ll be using two repos in our new Workflow. The first is the builder. Our builder project is very simple—it has a Gradle build script in it with the following code:

task createBuild << {
  new File("built.txt").write("You cannot pass.\n")
}

So what’s happening here? Gradle works by executing “tasks”, and the Gradle build script defines these tasks. We’ve defined a task called createBuild, and what it does is create a text file called built.txt with the content:

You cannot pass.

That’s it. (Well, I did say it was simple!)

The second Git repo is our packager. The packager also has a Gradle build script, but it is a tad more complex:

task createPackage << {
  String packageText = "I am a servant of the Secret Fire, wielder of the flame of Anor. You cannot pass. The dark fire will not avail you, flame of Udûn. Go back to the Shadow!"
  String builtText = new File('built.txt').text
  new File("package.txt").write(packageText + "\n\n" + builtText)
}

The createPackage task also creates a text file (called package.txt), but it expects to use content from built.txt, which doesn’t exist in the packager repo. If built.txt existed in the repo, the generated package.txt would contain:

I am a servant of the Secret Fire, wielder of the flame of Anor. You cannot pass. The dark fire will not avail you, flame of Udûn. Go back to the Shadow!

You cannot pass.

If built.txt is missing, our createPackage task will throw an error.

So every time we build our packager, we need to first run our builder and make the resulting built.txt available to the packager so it can create package.txt.

And that’s exactly what we’ll be setting up a Jenkins Workflow to do!

Let the Work Flow Through You

Head to the Jenkins Dashboard, click New Item, name it “Assembler”, select Workflow, and click OK.

Let’s start scripting. First, we’ll open up a node block, just as before:

node {

}

Next, let’s clone our builder repo:

node {
  git 'https://github.com/redhotvengeance/jenkins-workflow-build.git'
}

Now we need to run our Gradle build script to generate the built.txt file:

node {
  git 'https://github.com/redhotvengeance/jenkins-workflow-build.git'
  sh 'gradle createBuild'
}

Finally, let’s make sure everything is working as we expect it to. We’ll add a cat to print out the contents of the built.txt file:

node {
  git 'https://github.com/redhotvengeance/jenkins-workflow-build.git'
  sh 'gradle createBuild'
  sh 'cat ./built.txt'
}

Click Save, and then start a build. Once it is done, take a look at the Console Output.

Excellent! We are successfully cloning the repo, running the createBuild task, and have confirmed that the content of built.txt is You cannot pass.. Now on to the packager.

Just like with the builder, we need to clone our packager repo. Let’s add the packager code:

node {
  git 'https://github.com/redhotvengeance/jenkins-workflow-build.git'
  sh 'gradle createBuild'
  sh 'cat ./built.txt'
}

node {
  git 'https://github.com/redhotvengeance/jenkins-workflow-package.git'
  sh 'gradle createPackage'
  sh 'cat ./package.txt'
}

Since we didn’t explicitly create a new workspace, the createPackage task will run in the same workspace as the createBuild task, meaning that the built.txt file that the packager is expecting will be available to it.

Go ahead and Save and Build Now, and then view the Console Output.

Everything ran as expected—our builder was cloned and ran, and our packager was cloned and ran. And if we look at the output—there it is! The Balrog of Morgoth has been fully Gandalf’d.

Cool, but Isn’t This a Bit…

Contrived? Most definitely.

But a complex concept is really just a bunch of simple concepts mashed together. On the surface, we assembled Gandalf’s speech on the bridge of Khazad-dûm. But really, we took the build output of one project and injected it into the build output of another project.

What if instead of Gandalf’s dialogue, the build outputs were executables from separate codebases that all need to be assembled together for the software that you ship? You’d use the same Workflow we set up here: cloning, building, copying, and packaging. With the Jenkins Workflow plugin, all it took was a few lines of Groovy. And as a bonus, everything is contained in a single script!

There are also other tools available to help manage and visualize a Jenkins Workflow. CloudBees offers a Workflow Stage View feature on their enterprise Jenkins Platform.

This only scratches the surface of what can be done with the Jenkins Workflow plugin. Be sure to check out the related links below to learn more.

Good luck getting your work flowing!

Related Links

• Jenkins Website


• Jenkins Workflow Plugin Page


• Jenkins Workflow Plugin GitHub Page


• Jenkins Workflow Plugin Tutorial


• CloudBees Jenkins Workflow Plugin Docs


• Jenkins Workflow Plugin Walkthrough Video