HOWTO: Multisite Development / Federated Architecture

Modified on July 26, 2024

A "federated architecture" often refers to software developed at multiple sites with multiple teams. ProGet supports multi-site development by defining how each site interacts with packages through replication and mirroring. To illustrate how ProGet can support federated development, we will use the fictional company "Kramerica".

In this article, we will describe three different scenarios for federated development and provide step-by-step instructions for configuring feed replication and mirroring. Since this can be configured on a per-feed basis, enterprises can use all three scenarios for different packages.

Scenario: Centrally-Approved Packages

In this scenario Kramerica's package approval workflow takes place at its headquarters in New York. London and Tokyo need to have the packages on-site so that they can be used by the development team, but Kramerica only wants them to use the approved packages published by New York.

Since the London and Tokyo sites are to initiate communication with New York for their packages, the New York feed is set up as the "incoming" replication feed and the branch sites as the "outgoing" feed.

Scenario: Coequal Development

In this scenario, Kramerica teams work on similar projects and develop their own content (packages, container images, feeds) to share with each other.

In this example, developers in London and Tokyo are both working on Kramerica libraries, but want to use their own ProGet instance for bandwidth and security reasons. To achieve this, the feeds are configured to allow both sites to publish packages, which are then automatically replicated to/from the other site.

Also, if we want Tokyo and London to mirror each other, one must be configured as the "incoming" replication feed and the other as the "outgoing". In this scenario, either feed can be configured as an incoming feed, but we will use Tokyo.

Scenario: Global Package Distribution

In this scenario, one of Kramerica's sites (Los Angeles) develops packages that all other sites must access. Instead of connecting all sites to Los Angeles, the New York branch can distribute packages from Los Angeles to other sites in Tokyo, London, and Berlin. In this way, all sites remain aligned to one central source.

This scenario is unique in that the New York location must be set up as both an incoming and outgoing feed. It goes to Los Angeles to replicate packets from there, and goes to the branch offices so they can replicate from New York. This means that you need to configure two separate replications in the New York feed. One replication is set up as incoming ( pushing to the branch offices) and the other is set up as outgoing ( pulling from the Los Angeles feed).

How to Configure Feed Replication and Mirroring

In all three scenarios, incoming and outgoing replication feeds must be configured to work. For each scenario, repeat the following steps, changing the appropriate configuration options in ProGet.

Replication and mirroring take place at the feed level. Different feeds can use different scenarios, or an entire site can work a certain way by configuring every feed at that site the same way. For example, in the centralized publishing scenario, each feed in New York must be set up for incoming replication, and each feed in London and Tokyo must be set up for outgoing replication.

Step 1: Setup Incoming Replication Feed

First, your incoming replication feed must be set up in ProGet. In most cases, the feed you use to publish packages will be your replication feed.

Navigate to Replication> Configure New Replication.

Choose the feeds you wish to setup for incoming replication.

Communication type:
Select Incoming to allow other ProGet instances to connect to this instance.

Authentication mode:
Requires a specific sync token or allows any valid API key

• Sync token
  Click the "Generate" button to generate a sync token and copy it, as you will need to paste it into the outgoing feeds you want to connect.
• API key
  Navigate to Settings > Integration & Extensibility > API Key > and create an API key for the feed layer. Copy it as you will need to paste it into the outgoing feeds you want to connect.

Replication Mode Options:
Mirror content: Push and pull packets so that the feed is synchronized. The other instance should be configured to mirror content.
Push content from other instance: Upload missing content to the other instance and delete content that has been deleted on this instance.
Retrieve content from other instance: Download missing content from the other instance and delete content that has been deleted on this instance.
Custom / Advanced: Mix and match push and pull content properties.

Select the desired replication mode and proceed to the summary.

Step 2: Setup Outgoing Replication Feed

Once you have set up an incoming feed, you need to configure the outgoing feed that will be associated with your incoming feed. The outgoing feeds are usually the sites that do not publish their own packages but replicate from an approved package promoting feed.

Navigate to Replication > Configure New Replication. Under Feeds, select the feeds you want to set up for outgoing replication feed and proceed to the Communication Mode tab.

Communication type:
Select Outgoing to allow ProGet to routinely connect to another instance.

Other Instance URL:
Copy the base URL of the instance from the incoming feed you want to connect to.

Authorization
Paste here the sync token or API key you generated in step 1.

Other Feed Names: Leave the box checked if your feed names are the same on both instances. Otherwise, uncheck the box and enter the name of the incoming feed you are connecting to next to Name for "Feed Name".

Select the desired replication mode and proceed to the summary (see Step 1 for a detailed summary of options).

Step 3: Replicate Packages

After your incoming and outgoing feeds are connected, packets are automatically replicated at regular intervals. However, you can also manually replicate packages immediately.

Navigate to Replication > Replication Overview in your outgoing feed and select Run.

After execution, the packages from the incoming feed are replicated to the outgoing feeds based on the configuration you set up.

Replication can also be leveraged to set up your Disaster Recovery plan or Edge Computing.

ProGet v6 And Earlier

Setting up replication in ProGet v6 and earlier is the same in theory, but with slightly different steps due to a change in UI.

To configure feed replication, navigate to Manage Feed > Connectors & Replication > Feed Replication > Configure.

Incoming Replication Configuration:

• Outgoing Option:
  Leave this option disabled as you configure an incoming feed.
• Incoming Options:
  Allow external feeds to replicate from this feed (read only)
  Allow local feeds to be modified by external feeds (read/write)

Outgoing Replication Configuration:

• Outgoing Options:
  Apply only external changes to the local feed (pull-only)
  Apply local changes to external feed only (push-only).
  Mirror external feed

• Incoming Options:
  Leave this set to Disabled, as you are configuring an outgoing feed.