Terraform Modules

Terraform Modules are used to package and reuse resource configurations with Terraform.

A Terraform feed in ProGet acts as a private module registry that allows you to store your own modules. You can also create connectors to other Terraform Module Registries such as the Hashicorp registry to use third-party modules through a Terraform feed and create a curated list of approved Terraform modules.

Terraform feeds are available in ProGet version 2024.20+.

Terraform Module Packages

Terraform Modules are not distributed using self-describing package files; instead, a Terraform module registry (e.g. registry.terraform.io) is simply a list of versions and a "pointer" to a GitHub repository where the files can be downloaded.

While this makes things simpler for publishers to the free and open-source Hashicorp registry, it's not a good fit for organizations that rely on reliable and repeatable processes. For example, the GitHub repository could be deleted or changed, meaning the module will suddenly no longer be available.

Terraform Feeds in ProGet store modules in self-describing Universal Packages. This not only makes the modules portable and independent from external resources, but cryptographically seals them.

Terraform Module Package Specifications

A Terraform Module Package is a specially-formatted ZIP file:

1. the file has a .upack file extension (not .zip)
2. there is a manifest file in the root directory named upack.json with the following properties
   1. group is used for the module's namespace
   2. name is formatted with 2-parts («module-name».«provider»)
   3. version is a valid, 3-part semantic version number
3. The /package directory in the zip file contains the module's content

Automatically-Created Module Packages

When a module is downloaded or pulled into a Terraform Feed, a universal package will be automatically created from the contents. The upack.json manifest files in these packages will contain some additional metadata.

"modulePackaging": {
   "packagedDate": "2024-11-06T22:37:41.2253568Z",
   "using": "ProGet/24.0.0.0",
   "registry": "https://registry.terraform.io/",
   "registryMetadata": {
      "id": "terraform-aws-modules/vpc/aws/3.14.1",
      ...
   }
}

This data is only provided for auditing purposes and as a "snapshot" of what was used to create the package.

Creating and Uploading Terraform Module Packages to ProGet

To create Terraform module packages for your private ProGet registry, package your module content into a ZIP file with a .upack extension, including a upack.json manifest that defines the group (namespace), name (formatted as module-name.provider), and version (a semantic version like 4.20.0).

You can upload these packages to your ProGet instance using the ProGet UI or the pgutil command line tool, or let ProGet automatically generate packages with additional metadata when modules are pulled into a Terraform Feed. Once uploaded, you'll need to reference the module in your .tf files using the format «proget-host-name»/«feed-name»__«namespace»/«module-name»/«provider» with the relevant version.

This setup allows you to use your ProGet registry as a private module source instead of the public Hashicorp registry, ensuring controlled and consistent module usage. For more about creating and installing packages, see HOWTO: Create and Upload Terraform Modules to ProGet.

Connectors for Terraform Feeds

You can connect a Terraform feed to another Terraform Registry, such as registry.terraform.io. Modules that are downloaded or pulled through the feed will be automatically converted into a package and cached/installed in the feed.

Connector Listing & Search Limitations

Unless the remote registry implements a special search/listing APIs like registry.terraform.io, you will not see a listing of modules. However, you can find the modules if you type in their 3-part name: «namespace»/«module-name»/«provider». ProGet feeds do not currently implement this search API, which means self-connectors will not list packages from another feed.

Connector Download Limitations

Because a Terraform Module Registry simply provides the Terraform CLI with a "pointer" to download content, ProGet needs to decode and follow those pointers to download and package the module's content.

ProGet supports two types of download pointers:

• HTTP Urls to .zip files, which most private repositories appear to use
• Generic Git Repositories at GitHub.com, which the Hashicorp registry seems to exclusively use

Unlike the Terraform CLI, ProGet does not use the Git client to download the module files from GitHub. Instead, ProGet uses a rudimentary string replacement to download the repository from GitHub's archive endpoint.

For example, the Hashicorp registry responds with this pointer when requesting a module:

X-Terraform-Get: git::https://github.com/sourcegraph/terraform-aws-executors?ref=d9d6b1db18013b7dbd01edce457980c58e7a1c90

ProGet will transform this pointer to a usable GitHub URL:

https://github.com/sourcegraph/terraform-aws-executors/archive/d9d6b1db18013b7dbd01edce457980c58e7a1c90.zip

The contents will then be downloaded from that URL and packaged. If you run into any issues with specific modules from the Hashicorp registry or another one, please let us know.

Authenticating to Terraform Feeds

If your feed is not configured for anonymous access, then you'll need to configure the Terraform CLI to authenticate.

This can be done be either setting an environment variable with your API Key (TF_TOKEN_{hostName}=«api-key») or by adding the following to your CLI Configuration File (.terraformrc or terraform.rc):

credentials "«proget-hostname»" {
   token = "«api-key»"
}

Note that the Terraform CLI limits authentication by host name, which means that only one API Key per host name is possible. See the Terraform's CLI Configuration documentation and/or Creating and Managing API Keys in ProGet to learn more.

Support for Terraform Providers

ProGet's Terraform Feeds do not currently host cloud provider plugins such as aws, azurerm, etc.

In our research, users reported no interest in creating their own Terraform provider plugins (let alone hosting them). They also saw no value in proxying these plugins from Hashicorp's public registry, as internet access was inherently required to use Terraform.

However, we're open to implementing support for provider plugins if there's a need; just let us know.

Support for Terraform Backends

A Terraform Backend stores state files that Terraform uses to keep track of the resources it manages.

ProGet Asset Directories can be used as an HTTP backend by simply setting address as the desired folder in your asset directory.

For example:

terraform {
  backend "http" {
    address = "https://proget.corp/endpoints/my-assets/terraform/backends"'
    username = "api"
    password ="abcd12345"
  }
}

Note that the username/password is only required if your asset directory is authenticated, and it can alternatively be configured using the TF_HTTP_USERNAME and TF_HTTP_PASSWORD environment variables.