Skip to main content

Running Replicator (Legacy)

Only available in Sonatype Nexus Repository Pro. Interested in a free trial? Start here.

Warning

In release 3.48.0, we introduced Content Replication, which removes Replicator and makes replication easier to set up and deploy. This documentation remains available for current implementations only. We do not recommend implementing this version of replication as we will be completely removing it once the new replication design is available.

Note

Replication is not appropriate for disaster recovery. If you need a disaster recovery solution, please see our resilient deployment options and backup and restore procedures.

Repository replication requires an administrator to manually run the Replicator. The Replicator is the tool that executes the copying of the binaries from the source blob store to the target blob store. It uses a YAML file so that it knows so that it knows to which target blob store to copy the binary files.

Prerequisites

First, ensure you've met the prerequisites outlined in Replication Prerequisites (Legacy). These include but are not limited to the following:

  • Source and target instances must be on the same Nexus Repository Pro version.

  • Replicator requires a *nix, or Windows 10 Linux subsystem.

  • For file-based blob stores, the source and target repository blob stores must be readable/writable on the machine where the Replicator tool (the replication CLI jar) runs.

  • The Replicator must be able to see and communicate with all repository instances. The Replicator directory in the Nexus Repository bundle is portable; this means you can deploy the Replicator on the source instance, target instance, or even in a separate location as long as it can see and communicate with all repositories.

  • You must have appropriate permissions to run the Replicator app, rsync, and/or AWS CLI as well as permission to read/write to/from the source and target blob stores.

Creating the Configuration YAML

In order to configure the Replicator, you will need to create a YAML file like the following example.

As a best practice, name this file config.yml and place it in the $install-dir/replicator folder. (This location is not a requirement.)

As mentioned in the Prerequisites above, this YAML can be on the source instance, target instance, or a separate machine.

sources:
  - path: /data/newyork/sonatype-work/nexus3/blobs/default
    type: file
    targets:
      - path: /data/singapore/sonatype-work/nexus3/blobs/default
        type: file
        repositoryName: rawtarget
        connectionName: ny-to-singapore-replication
      - path: /data/london/sonatype-work/nexus3/blobs/default
        type: file
        repositoryName: rawtarget
        connectionName: ny-to-london-replication
  - path: s3://mysourceblobstore/
    type: s3
    targets:
      - path: s3://mytargetblobstore/
        type: s3
        repositoryName: maventarget
        connectionName: ny-to-singapore-maven-replication
      - path: /data/london/sonatype-work/nexus3/blobs/blobstore2
        type: file
        repositoryName: maventarget
        connectionName: ny-to-london-maven-replication

This YAML file provides Replicator with the following information:

sources - a list of source blobstores from which content will be replicated

Each source must then provide the following:

  • path - the root of the blobstore to replicate

  • type - the type of blob store (file or s3)

  • targets- list of target blob stores (1 or more) to which to replicate content

    • Each target must then provide the following:

      • path- the path to the root of the blob store that will receive content

      • type- the type of blob store (file or s3)

      • repositoryName- name of the repository on the target instance

      • connectionName- name of the replication connection that is configured in Nexus Repository between this source and target.

Note that the source & target S3 paths should be terminated with a forward slash

StartingReplication

To start replication, you will need access to a terminal. By default, the Replicator is located in the$install-dir/replicatorfolder.

You also need the appropriate replicator permissions to run the Replicator app, rsync, and/or AWS CLI as well as permission to read/write to/from the source and target blob stores.

Use the following steps to start replication:

  1. Upload a component to your source repository (e.g., a Maven-Releases repository).

  2. Open the terminal and navigate to the$install-dir/replicatorfolder.

  3. Start the Replicator using the following command where replicationconfigfile is a path to the configuration YAML as described above.

    1. By default, the Replicator looks for a config.yml file in the same directory where the nexus-replicator-cli-<version>.jar is located. If your YAML and jar are in the same directory, you do not need to provide a path.

    2. Use the -debug flag to see the output of the Replicator in the terminal as it runs. By default, the Replicator will run continuously.

java -jar nexus-replicator-cli-<version>.jar -replicationconfigfile config.yml [-debug]

In the target instance, browse to the target repository. You should now see the component replicated from the source repository you uploaded in step 1.

When you first configure a replication connection, you may need to wait for a few seconds before you will see your content in the target repository. This is because queues are managed so that the first file listing for the Replicator to parse may only contain a single item. As the Replicator runs repeatedly, it will work its way through all of the items in the queue (with a configurable maximum of 100 items per execution) and will copy the files from the source blob store to the target blob store. You can modify the number of items per execution in the $data-dir/etc/nexus.properties file by configuring the nexus.replication.outputqueue.batchsize property.

Note

Other potential advanced arguments to use in your command include the following:

  • rsync - you can use this if you have an rsync executable with a non-standard name (e.g., -rsync myrsync)

  • aws - you can use this if you have an aws executable with a non-standard name (e.g., -aws myawscli)

  • syncbatchsize - you can use this to control the batch size for aws replication (e.g., -syncbatchsize 20)

  • interval - you can use this to configure the number of seconds that Replicator will wait between its checks for new file listings to replicate; by default, replicator runs every 1 second (e.g., -interval 5)

  • mode - by default, this is set to RUN_ALWAYS; however, you can use this to run Replicator only once rather than having it run continuously by using -mode RUN_ONCE

Replicator Locking

It is not possible for two Replicators to run on the same file blob store source. If a file blob store is already configured for replication, there will be a replicator-running.lock file in the $data-dir/blobs/(blobstorename) directory that will prevent another Replicator from running on that same source. Should someone attempt to run another Replicator on this same blob store, they will receive an error message and not be able to proceed. Nexus Repository should automatically remove the file when the Replicator is no longer running; however, should this fail to occur, you can also manually remove the lock file in order to configure a new Replicator against that file blob store.

The replicator lock file will have content similar to the following example:

Lock acquired at Wed Jul 14 14:33:58 MDT 2021 by jvm 88202@hostname

You can use this file to check when a lock was acquired and by what jvm. This can help you identify when Replicator has been running for a long time so that you can terminate the process if necessary and delete the lock file.

Run the Replicator on Separate Hardware (Optional)

The Replicator can be run in a different hardware or host to minimize the resource utilization on the Nexus Repository instance. The benefit of running the Replicator from a separate environment is scaling by using additional hardware to run the Replicator, shifting the resource utilization from the Nexus Repository instance.

If you want to test this set up, ensure you have an environment that satisfies the requirements outlined in therepository replication prerequisitessection.

Note

A maximum of one Replicator can run for one source blob store.

  1. Copy the nexus-replicator-(version).jar file from the Nexus Repository $install-dir/replicator folder to the new environment.

  2. Ensure the locations of the blob store paths as defined in the configuration YAML are accessible to this environment. These might be mapped network paths.

  3. Upload a component to your source repository (e.g., the Maven-Releases repository).

  4. From the terminal, run the Replicator using the following command (after configuring your config.yml file):

java -jar nexus-replicator-cli-<version>.jar -replicationconfigfile config.yml

You should see the output of the Replicator as it runs.

In the target instance, browse to the target repository. You should now see the component replicated from the source repository in step 3.