Importing Builds (Jenkins, Drop Folder, etc.)
  • 18 Feb 2022
  • 10 Minutes to read
  • Dark
    Light
  • PDF

Importing Builds (Jenkins, Drop Folder, etc.)

  • Dark
    Light
  • PDF

BuildMaster is designed to continuously deliver your applications and components from source code to production, but unlike monolith CI/CD platforms like Azure DevOps or GitLab, you can work the tools and processes you're already using, including dedicated Continuous Integration servers like Jenkins and TeamCity.

While BuildMaster can certainly perform continuous integration and build automation, there are a lot of good reasons to continue use your existing continuous integration servers:

  • Save time and resources on retraining: if your developers are already familiar with the CI tools and processes you've setup, they won't have to spend time learning something new
  • Reuse complex automations: processes like automated testing are complex to develop and are often fragile. Moving these processes to a new system like BuildMaster will require some degree of effort that might be better spent elsewhere
  • Keep your existing infrastructure: you can use the CI server infrastructure that you've already built and is providing you with value

A large percentage of customers use BuildMaster in exactly this manner.

If you want to keep using the CI tool you already love and use, BuildMaster lets you do it. But BuildMaster is as solid a Continuous Integration server as a Continuous Delivery tool. Whether you build inside the program or continue using an outside tool, you can do it with BuildMaster.

CI Server "Builds" vs BuildMaster "Builds"

While both continuous integration servers and BuildMaster use the term "build," the concepts are a bit different.

A "build" in a continuous integration server is a collection of logs, test results, and files that is created when a developer commits new code to a source control repository

  • A "job" (Jenkins) or "build configuration" (TeamCity) defines which repository these will be created from, how they'll be created, who will be notified, etc.
  • These are intended for engineers and may not have any relation to a business application or component, and may not be understandable by testing or operations departments

A BuildMaster build is a broader concept. It represents a new version of an application or component that may be released to production using a pipeline with various stages that represent your software delivery process

  • Builds have a lifecycle that is not only visible to business stakeholders, but can be approved at different stages in the process
  • Organized under releases and applications that testers, operations, and managers understand

Connecting to your CI Server Using Resource Credentials

Resource credentials are used to connect BuildMaster to your Continuous Integration (CI) server, whether that's Azure DevOps in the cloud or a self-managed TeamCity Server. Each supported CI server has its own type of Resource Credential, and that type is defined in the corresponding extension.

For example, installing the Jenkins extension will allow you to create a Jenkins Resource Credential. When creating a resource credential for a CI server, you'll enter two types of information:

  • Server connection information, such as your Jenkins Server's URL or a TeamCity API endpoint
  • Credential information, such as username and password or an authorization token

You can create as many resource credentials as you need, at both the system- and application-level, which means you can connect to multiple CI servers and share those connections across applications. This allows your team to continue working with whatever variety of tools they already use and love.

Importing Artifacts from CI Server Builds

A continuous integration can be configured to not only compile your application's source code and run automated tests against it, but to capture "artifacts" of that process. These artifacts typically include whatever files the compiler output, i.e. your compiled application.

BuildMaster can automatically import these artifacts so you can deploy them as needed. This ensures that BuildMaster will be able to deploy it to future stages whether the CI server is accessible or not; in addition, the relationship doesn't require active access from both.

Different types of operations are used to get artifacts from different CI servers and projects in those systems. For example, Jenkins::Import-Artifact is used to get artifacts from a build job on a Jenkins server, and TeamCity::Import-Artifact is used for TeamCity.

Example: Simple Import from a TeamCity Server

For example, here's a very simple import from a TeamCity server using a credential named KramericaTeamCity.

 TeamCity::Import-Artifact
 (
     Credentials: KramericaTeamCity,
     Project: ProfitCalc,
     BuildConfiguration: v8builds,
     Artifact: profit-calc-web.zip
 ); 

HOWTO: Use Import Artifact Operations

The specifics of these operations are documented in the corresponding Tool & Service Integrations pages, but they are all used similarly. Essentially, you specify various option using parameters:

  1. Specify a Resource Credential, which tells the operation which server to connect to. These can be specified and overridden on the operation itself, which means you technically don't need to use a resource credential at all. There are only a few cases where this would be appropriate, such as retrieving code from a public repository using its URL.
  2. Specify a Job/Project Configuration, which tells the operation which set of builds to import from the CI server. This is server specific, as Jenkins uses Jobs, TeamCity uses Projects and Build Configurations, and many have different options (such as a Branch) to further clarify.
  3. Specify Build Number, whether that's a variable like $TeamCityBuildNumber that comes from a prompt (see Prompting for Build Numbers) or a provider-specific label like "lastSuccessful" or "lastFinished"
  4. Specify Artifact Name(s), which are the files to download and associate with the BuildMaster build

Like all operations in OtterScript, you specify variables for any of these parameters

Example: Import from a Team City Server

This operation configuration has a lot of properties defined for illustrative purposes, and will get source from the accounts repository under the kramerica-industries organization (no resource credential needed because it's a public).

TeamCity::Import-Artifact
(
    Credentials: KramericaTeamCity,
    Project: ProfitCalc,
    BuildConfiguration: v8builds,
    Artifact: profit-calc-web.zip
);

Automatically Queueing/Starting Builds on your CI Server

The easiest type of integration is to simply import that last successful build from your CI server, as discussed above. But you may wish to have BuildMaster create a build in your CI server instead.

There a few reasons you may want to do this:

  • Create different builds then are automatically triggered by check-in
    • A "Release Configuration" build that instructs the compiler to optimize for speed and does not include debugging symbols
    • A "Release Branch" build that uses a different source control branch to create the build
  • Inject contextual information from BuildMaster (release number, etc.) into the application code after it's checked-out but prior to being built by your CI server; you can then display this information once the application is running in an environment
  • Link from your CI server back to the application or build in BuildMaster, so you can have cross-links in both system

Fortunately, this is all easy with OtterScript. Different types of operations are used to queue or start builds on different CI servers. For example, Jenkins::Queue-Build is used to start a build job on a Jenkins server, and TeamCity::Queue-Build is used to create a build on TeamCity.

HOWTO: Use the Queue/Start Operations

The specifics of these operations are documented in the corresponding Tool & Service Integrations pages, but they are all used similarly. Essentially, you specify various option using parameters.

  1. Specify a Resource Credential, which tells the operation which server to connect to. These can be specified and overridden on the operation itself, which means you technically don't need to use a resource credential at all. There are only a few cases where this would be appropriate, such as retrieving code from a public repository using its URL.
  2. Specify a Job/Project Configuration, which tells the operation which job or configuration to create a build from on the CI server. This is server specific, as Jenkins uses Jobs, TeamCity uses Projects and Build Configurations, and many have different options (such as a Branch) to further clarify.
  3. Specify Arguments that will be used by the underlying configuration; these are typically user-defined keys or variables that will, in turn, be passed to the tools the CI server uses, such as MSBuild or Maven
  4. Specify Whether to Wait which determines whether BuildMaster will wait for the build to complete, or just queue it and continue executing your OtterScript; generally speaking, you should wait for builds so that it's clear in the execution logs whether it was successful or not

Like all operations in OtterScript, you specify variables for any of these parameters.

Example: Queuing a Jenkins Build then Importing Created Artifacts

For example, here's a set of operations that will queue a build in Jenkins with a "release" configuration, then import the created artifacts after completion:

Jenkins::Queue-Build
(
    Credentials: KramericaJenkins,
    Job: ProfitCalc,
    AdditionalParameters: bldconfig=release&ext_trg=yes,
    WaitForCompletion: true,
    JenkinsBuildNumber => $JenkinsBuildNumber
);
Jenkins::Import-Artifact
(
    Credentials: KramericaJenkins,
    Job: ProfitCalc,
    BuildNumber: $JenkinsBuildNumber,
    Artifact: archive.zip
);

Capturing CI Server Build Numbers

Whether you prompt for a build number or ask the CI server for the lastSucessfulBuild, BuildMaster lets you capture and link this back to your CI server's build number.

Capturing this reference at build time is very important so you can see exactly what code was used to create the build artifacts:

  • From a debugging perspective, this saves you wasted time trying to figure out which code was deployed and saves frustration when the deployed application doesn't seem to match the code
  • From an auditing perspective, this lets you see exactly who made which changes to a deployed application and when those changes were made

BuildMaster gives you a lot of flexibility in capturing this information. Although you could always manually dig through the execution logs to find this information, a simpler approach is as follows.

  1. Specify an _output parameter_ on the appropriate operation (such as [import artifact] or [queue build]); the operation will them set the value of runtime variable to be the CI server's build number;
  2. Set a build variable from the runtime variable; because runtime variables only exist during build time, capturing this as a build variable will create a permanent record on the build

Example: Capturing Jenkins Build Number

By simply specifying the TeamCityBuildNumber output parameter and the $tcBuildNumber runtime variable, the Set-BuildVariable operation will set the value of $tcBuildNumber to be the Jenkins build number. You can then use this to identify everything about the code that went into the build.

TeamCity::Queue-Build
(
    Credentials: KramericaTeamCity,
    Project: ProfitCalc,
    BuildConfiguration: v8builds,
    Artifact: profit-calc-web.zip,
    TeamCityBuildNumber => $tcBuildNumber
); 

Set-BuildVariable TeamCityBuildNumber
(
    Value: $tcBuildNumber
);

The second operation will then set $TeamCityBuildNumber as a build variable, which means it will be not only visible on the build page, but you can also use it in all future deployments on that build.

Tip: Use a Variable Renderer to Display a Hyperlink
You can do this in BuildMaster with Variable Value Renderers. These are essentially instructions for how to render variables in the UI that have certain names (such $JenkinsBuildNumber). Because you can specify HTML in a value renderer, you link directly to your CI server to help with easy navigation for users.

Prompt for Build Number at Build Time

Using distinct build numbers for each build will help keep documentation clear, avoid silos and confusion, and will make tracking a specific build much simpler. Assigning a distinct build number at the time of the build, therefore, distinguishes the build from its inception.

When you create a new build from the web UI, a dropdown box appears. From here, you can select the build numbers that were generated by your CI Server.

Variable prompts are configured in an application's Release Templates. If you're not familiar with the feature, that's okay-- all applications have a "Default" release template that's used behind-the-scenes to determine which pipeline to use, and which variables to prompt for at release, build, and deploy time.

After you configure a branch name variable prompt (e.g. $JenkinsBuildNumber), BuildMaster will create a build-scoped variable based on whatever the user selects. You can use that variable in your OtterScript plans to import artifacts from that build.

Example: Creating a Drop-down for Jenkins Builds

The following steps will configure a $JenkinsBuildNumber variable prompt when new builds are created that defaults to master.

  • Navigate to the Default release template (Releases > Release Template > "Default")
  • Add a Build Template Variable
    • Type: Select "Dynamic List"
      • This will open a window where you can select the type of list
      • Select "Jenkins Builds" as the type
      • Select the Credential you've already configured
    • Name: JenkinsBuildNumber
    • Initial value: lastSuccessfulBuild
    • Options: value is required

Automatically Importing into BuildMaster after CI Build

At the end of a build process in your CI server, you may wish to automatically import those artifacts into BuildMaster so you can smoothly move these builds into the pipeline for testing and release. This can be programmatically done using the Release & Build Deployment API.

When you want to specify the build number to import from, simply specify that as a variable (e.g. $JenkinsBuildNumber) when making the API request.

That will automatically create a build-scoped variable, which can then be used in your OtterScript plans to import artifacts from that build.

Example: Triggering an Importing Artifact Build Using PowerShell

Invoke-WebRequest http://{buildmaster-server}/api/releases/builds/create `
-ContentType "application/json" `
-Method POST `
-Body '{"applicationName": "hdars", "releaseNumber": "2.14.0", "$JenkinsBuildNumber": "88512" }' `
-Headers @{"X-ApiKey" = "<api-key>"}