Terraform Modules

Terraform Modules are used to package and reuse resource configurations with Terraform.

A Terraform feed in ProGet acts as a private module registry that allows you to store your own modules. You can also create connectors to other Terraform Module Registries such as the Hashicorp registry to use third-party modules through a Terraform feed and create a curated list of approved Terraform modules.

Terraform feeds are available in ProGet ProGet 2024.20+.

Using Terraform Feeds as a Private Registry

Before using a Terraform Feed as a Private Module Registry, you'll need to package your modules. This is as simple as zipping the content and uploading it to the ProGet UI or using the upack CLI; see Terraform Module Packages to learn more.

Once you've uploaded a package to your feed, you can add the module to your Terraform configuration using the following format:

module "«module-name»" {
  source  = "«proget-host-name»/«feed-name»__«namespace»/«module-name»/«provider»"
  version = "«version»"
}

For example the my-company/my-module module that uses aws might look like this:

module "example_module" {
  source = "proget.corp/internal-terraform__my-company/my-module/aws"
  version = "4.20.0"
}

Once you add the module to your configuration, you can add in additional variables and then run terraform init.

Connecting to another Terraform Module Registry

You can connect a Terraform feed to another Terraform Registry, such as registry.terraform.io. Modules that are downloaded or pulled through the feed will be automatically converted into a package and cached/installed in the feed. See Terraform Module Packages to learn more.

Connector Listing & Search Limitations

Unless the remote registry implements a special search/listing APIs like registry.terraform.io, you will not see a listing of modules. However, you can find the modules if you type in their 3-part name: «namespace»/«module-name»/«provider». ProGet feeds do not currently implement this search API, which means self-connectors will not list packages from another feed.

Connector Download Limitations

Because a Terraform Module Registry simply provides the Terraform CLI with a "pointer" to download content, ProGet needs to decode and follow those pointers to download and package the module's content.

ProGet supports two types of download pointers:

• HTTP Urls to .zip files, which most private repositories appear to use
• Generic Git Repositories at GitHub.com, which the Hashicorp registry seems to exclusively use

Unlike the Terraform CLI, ProGet does not use the Git client to download the module files from GitHub. Instead, ProGet uses a rudimentary string replacement to download the repository from GitHub's archive endpoint.

For example, the Hashicorp registry responds with this pointer when requesting a module:

X-Terraform-Get: git::https://github.com/sourcegraph/terraform-aws-executors?ref=d9d6b1db18013b7dbd01edce457980c58e7a1c90

ProGet will transform this pointer to a usable GitHub URL:

https://github.com/sourcegraph/terraform-aws-executors/archive/d9d6b1db18013b7dbd01edce457980c58e7a1c90.zip

The contents will then be downloaded from that URL and packaged. If you run into any issues with specific modules from the Hashicorp registry or another one, please let us know.

Authenticated Feeds

If your feed is not configured for anonymous access, then you'll need to configure the Terraform CLI to authenticate.

This can be done be either setting an environment variable with your API Key (TF_TOKEN_{hostName}=«api-key») or by adding the following to your CLI Configuration File (.terraformrc or terraform.rc):

credentials "«proget-hostname»" {
   token = "«api-key»"
}

Note that the Terraform CLI limits authentication by host name, which means that only one API Key per host name is possible. See the Terraform's CLI Configuration documentation and/or Creating and Managing API Keys in ProGet to learn more.

Terraform Module Packages

Terraform Modules are not distributed using self-describing package files; instead, a Terraform module registry (e.g. registry.terraform.io) is simply a list of versions and a "pointer" to a GitHub repository where the files can be downloaded.

While this makes things simpler for publishers to the free and open-source Hashicorp registry, it's not a good fit for organizations that rely on reliable and repeatable processes. For example, the GitHub repository could be deleted or changed, meaning the module will suddenly no longer be available.

Terraform Feeds in ProGet store modules in self-describing Universal Packages. This not only makes the modules portable and independent from external resources, but cryptographically seals them.

Creating Terraform Module Packages

A Terraform Module Package is a specially-formatted ZIP file:

1. the file has a .upack file extension (not .zip)
2. there is a manifest file in the root directory named upack.json with the following properties
   1. group is used for the module's namespace
   2. name is formatted with 2-parts («module-name».«provider»)
   3. version is a valid, 3-part semantic version number
3. The /package directory in the zip file contains the module's content

You can create a Terraform Module Packages from the ProGet UI or using the upack commandline client.

Example Package Manifest

For example, a upack.json for the my-company/my-module/aws module might look like this:

{
  "group": "my-company",
  "name": "my-module.aws",
  "version": "4.20.0"
}

You can also use additional Universal Package properties like summary and description to provide additional metadata for users in ProGet.

Automatically-Created Module Packages

When a module is downloaded or pulled into a Terraform Feed, a universal package will be automatically created from the contents. The upack.json manifest files in these packages will contain some additional metadata.

"modulePackaging": {
   "packagedDate": "2024-11-06T22:37:41.2253568Z",
   "using": "ProGet/24.0.0.0",
   "registry": "https://registry.terraform.io/",
   "registryMetadata": {
      "id": "terraform-aws-modules/vpc/aws/3.14.1",
      ...
   }
}

This data is only provided for auditing purposes and as a "snapshot" of what was used to create the package.

Support for Terraform Providers

ProGet's Terraform Feeds do not currently host cloud provider plugins such as aws, azurerm, etc.

In our research, users reported no interest in creating their own Terraform provider plugins (let alone hosting them). They also saw no value in proxying these plugins from Hashicorp's public registry, as internet access access was inherently required to use Terraform.

However, we're open to implementing support for provider plugins if there's a need; just let us know.

Support for Terraform Backends

A Terraform Backend stores state files that Terraform uses to keep track of the resources it manages.

ProGet Asset Directories can be used as an HTTP backend by simply setting address as the desired folder in your asset directory.

For example:

terraform {
  backend "http" {
    address = "https://proget.corp/endpoints/my-assets/terraform/backends"'
    username = "api"
    password ="abcd12345"
  }
}

Note that the username/password is only required if your asset directory is authenticated, and it can alternatively be configured using the TF_HTTP_USERNAME and TF_HTTP_PASSWORD environment variables.