Feed Replication & Mirroring Overview
  • 23 Mar 2022
  • 4 Minutes to read
  • Dark
    Light
  • PDF

Feed Replication & Mirroring Overview

  • Dark
    Light
  • PDF

ProGet's replication features lets you synchronize packages, containers, and assets across multiple ProGet instances. This can help implement a variety of scenarios and complex network architectures, including:

Configuration

A "replication" requires at least two instances of ProGet Enterprise Edition, and at least one feed of the same type across those instances. However, you can configure a replication with multiple feeds, and you can also configure multiple replications on a single instance. This allows you to design complex content distribution schemes, such as multi-hub private content delivery network

Communication Mode

At least one instance is configured with "outgoing" communication (similar to how a web browser accesses a web server), and the other with "incoming" communication (like how a web server responds to a web browser).

Prior to ProGet v6, incoming replication configurations are called replication servers and outgoing replication configurations are called replication clients.

Behind the scenes, incoming communication is handled by the Replication API endpoint that's hosted on the ProGet Web Application, just like all other API endpoints. This means that it's accessed using HTTP/S in the same manner your browser and other client tools interact with ProGet.

Outgoing communication is handled by the ProGet Service application, which communicates with the other instance's Replication API endpoint. The "outgoing" instance is where most of the work is done, and is where you'll find detailed replication logs.

Replication Mode

Regardless of the communication mode you choose, you can configure how content (packages, containers, and assets) are synchronized across ProGet instances.

Each instance must be configured in one of these way:

  • Mirror Content; Push and pull packages so that the feed is in sync. The other instance should be configured to Mirror Content
  • Pull Content from other Instance ; Download missing content from the other instance and delete content that was deleted from that instance. The other instance should be configured to Push Content.
  • Push Content to other Instance; Upload missing content to the other instance and delete content on that instance that was deleted on this instance. The other instance should be configured to Pull Content.

You can also select "Custom/Advanced", which allows for more control over what gets replicated.

Multiple Replications

Instead of selecting multiple feeds in a single replication, you can configure multiple replications on a single instance. This is common to do when distributing content for edge computing, but there are other reasons you may wish to do this.

  • Use different sync tokens, so that you don’t have to share them for all feeds
  • Replicate content from multiple other instances in outgoing mode
  • Different replication modes on different feeds, like having some be Pull and others be Push

Whatever the reason, be cautious when configuring different replications for the same feed. For example, if you pull packages from and delete content from two different instances, you may find unexpected results.

Best-Practices

Based on usage and support tickets we have encountered in the past, we recommend the following guidelines:

  • DO: ensure all ProGet installations involved in replication are running the exact same version of ProGet, or at the very least, the same minor version (e.g. 5.2.x)
  • DO NOT: configure replication to another feed in the same instance of ProGet
  • DO NOT: configure two instances to be both incoming and outgoing replication feeds pointing to each other
  • DO NOT: use replication for availability purposes if you are already using a package store that provides its own replication, for example if you are using the AWS package store with cross-region replication
  • DO NOT: configure an incoming replication feed to pull changes and then distribute the sync token to untrusted parties as that will effectively grant full access to the contents of the feed, which could include poisoned packages, malware, etc.

Troubleshooting

Perhaps the most common issue related to replication is that ProGet Enterprise (or trial) is required for each server cluster involved in the replication, otherwise an error will be generated and replication will not occur.

Replication Status & Overview

To view the list of feeds that have replication configured, visit the Replication Overview page.

Logging

Outgoing Replication Feeds

The logs for outgoing replication feeds can be found on the Executions page. To view them, visit the Administration > Executions page, then filter by "Feed Replication".

Incoming Replication Feeds

The logs for incoming replication feeds will be found in the Diagnostic Center. Visit Administration > Diagnostic Center > View Log Messages and filter the category by "Feed Replication" to view any error messages.

Note that by default, only incoming replication errors are logged. To enable logs for successful connections, visit the Advanced Settings page and set Diagnostics.MinimumLogLevel to 0 to enable debug messages. Changing this setting requires the web application to be restarted. It is recommended that this setting only remain at 0 temporarily, then reset back to the default value of 20.

Outgoing & Incoming Options Missing

The outgoing/incoming (client/server) replication distinction was added in ProGet v5.2, and previous versions only allowed one or the other to be configured, and not both. The options "replicate from external feed" and "allow other feeds to replicate with this one" are equivalent to client and server respectively.

Docker

Docker feed replication is supported starting in ProGet 5.2.5, and has the following limitations:

  • Publish date and download count are not synchronized.
  • All blobs and manifests are synchronized, so retention rules need to be set up on both ends of synchronization.
  • Deletion records ("tombstones") are not recorded in Docker feeds, so images can only be added by feed replication, never removed; a retention rule must be configured to delete untagged images.

Was this article helpful?