Rolling Upgrades in High Availability
High availability (HA) environments may upgrade Nexus Repository without downtime (i.e., rolling upgrade, zero-downtime upgrade).
Zero-Downtime
Zero-downtime upgrades in Nexus Repository mean the system remains available during updates, especially in HA clusters, ensuring uninterrupted artifact access. This is achieved through rolling upgrades, allowing users and builds to continue without service disruption.
Rolling Upgrades
Updating a Nexus Repository by sequentially upgrading nodes in a high availability cluster, minimizing downtime for artifact access.
Rolling upgrades involve stopping and upgrading each node before then upgrading the database schema. While an individual node is stopped, the other active nodes continue to serve traffic, resulting in no downtime for consumers of the service.
Mixed Mode
Having different versions of Nexus Repository installed on the nodes of a cluster. While running in mixed mode, new features are not available until all the nodes on the cluster as well as the database schema are upgraded.
While upgrading nodes, administrators may see a warning indicating that the cluster is running in mixed mode. Running in mixed mode outside of an active upgrade is not supported.
The following section details some limitations:
Review the supported HA rolling upgrade paths before upgrading
You may need to upgrade to an intermediate version before going to the latest version. Upgrading Nexus Repository from 3.71.0 to 3.72.0 is the first version where zero-downtime upgrades are available.
The HA cluster nodes should only run on different versions temporarily.
Running in mixed mode outside of an active upgrade is not supported. Nexus Repository may become unstable and behave in unexpected ways. Complete the upgrade in as short amount of time as possible to minimize the amount of time Nexus Repository nodes are on different versions.
Enable rolling upgrades on all nodes of the cluster.
All nodes must have rolling upgrades enabled or they return an error when Nexus Repository detects mixed mode.
New features are not available until the database schema upgrade is complete.
New nodes cannot join the cluster while the database schema migration is active
When adding nodes during schema migration, these nodes stop with an error code and log the errors.
Upgrading the database schema is a one-way process.
You cannot roll back once you proceed with finalizing the upgrade by updating the database schema. You will need to restore from a backup to roll back.
You cannot configure sticky sessions from within the helm chart
Without sticky sessions, users may see UI inconsistencies during the upgrade process due to changes made between versions. Sticky sessions need to be configured on your load balancer or ingress controller.
Prerequisites for Rolling Upgrades
To use rolling upgrades with zero downtime, you must meet the following prerequisite requirements:
Requires Nexus Repository Pro 3.71.0 or greater
Nexus Repository Pro in an HA environment with a PostgreSQL database using the supported HA Helm chart version 71.0.0 or above.
Enable zero-downtime upgrades on each node
Restart each instance after enabling the feature on each node before proceeding with the upgrade.
Configure sticky sessions on your load balancer
NGINX provides documentation on enabling session persistence .
Enable Zero-Downtime Upgrades
Zero-downtime upgrades must be enabled and restarted on every node before upgrading them. The process to enable is different depending on the environment you are running your cluster. Select the appropriate method below:
Helm Chart
Update the
values.yamlwith the zero downtime configuration.zeroDowntimeEnabled: true
OpenShift
Update the Nexus Repository custom resource definition (NexusRepo)
spec.statefulset.container.env.zeroDowntimeEnabled = true
Docker and Kubernetes
Update the
statefulset.yamlwith the zero downtime configuration.NEXUS_ZERO_DOWNTIME_ENABLED: true
Manual Environment Variable
Set the environment variable for every node.
NEXUS_ZERO_DOWNTIME_ENABLED=true
Manual through nexus.properties
modify the
$data-dir/sonatype-work/nexus3/etc/nexus.propertiesfor each node.nexus.zero.downtime.enabled = true
Rolling Back on Zero-Downtime Upgrades
Zero downtime upgrades each node to the latest Nexus Repository version before updating the database. At any point before finalizing the changes to the database, the cluster may be rolled back to the version of Nexus Repository running previously.
Mixed Mode
If the upgrade has not been finalized, the cluster is in mixed mode. Bring down the instance(s) with the higher version to move the nodes back to the older versions.
Finalize not selected
The finalize button appears when all instances have the upgraded version, but the database is not yet upgraded. At this stage, you can start instances with the previous version to put the cluster back into mixed mode.
Finalize in progress
Once finalized, rolling back is possible by restoring from a backup.
Phase 1 - Upgrade Each Node
One node at a time; take the node offline, upgrade the node, and start the node using the new Nexus Repository version. Upgrade the nodes by replacing the binaries or follow the instructions for Kubernetes environments.
Nodes are upgrades one at a time.
The load balancer distributes requests to active nodes.
PostgreSQL database remains on the old schema.
Phase 2 - Finalize Upgrade
Finalize the upgrade by migrating your database schema if applicable; not all releases will require a database schema migration.
When the upgrade requires a database schema migration, administrators with the nexus:* permission see a banner at the bottom of the screen. There is a delay of up to 10 minutes before the banner appears.
 |
Select Finalize Upgrade in the Upgrade Pending banner.
This opens a dialogue to proceed with the database schema migration.
Select Acknowledge that you have backed up your database.
Select Continue to proceed with database schema migration.
The banner shows that an upgrade is in progress; when complete, it displays an Upgrade Complete banner.
Any errors are captured in the audit log.
Reload the user interface to use new features.
Finalize Migration with REST APIs
REST API users can finalize their upgrade via the REST APIs listed in the table below.
API | Description |
|---|---|
GET /service/rest/v1/clustered/upgrade-database-schema | Returns an object that describes the current state:
|
POST /service/rest/v1/clustered/upgrade-database-schema | When called, this starts the database migration process asynchronously. Unsuccessful response codes indicate that a precondition failed and administrators should view the error message. |
DELETE /service/rest/v1/clustered/upgrade-database-schema | If an error occurs and a retry is possible, use this endpoint to clear the error state. |
The process for finalizing your upgrade and migrating your database schema is as follows:
Use a GET request to verify that the state is MIGRATION_REQUIRED
Use a POST request to start a migration
Poll the GET endpoint; it will return one of the following statuses:
MIGRATION_SUCCEED - indicates successful database schema migration. Send a DELETE request to clear the status.
MIGRATION_FAILED_RETRYABLE - indicates that a precondition for migration failed; consult the accompanying error message to identify and remediate the cause. After remediation, use a POST to try again.
MIGRATION_FAILED - indicates that an error occurred, and we do not believe it can be rectified without Support team assistance.
After successful database schema migration, your upgrade is complete.
Supported Rolling Upgrade Paths
We may occasionally have limitations on eligible version upgrade ranges. Attempting to exceed the upgrade windows will result in new nodes failing to start. Follow this section to stay up to date on any required intermediate versions.
Starting Version | Can Upgrade with Zero Downtime to These Versions |
|---|---|
3.71.0 | 3.72.0+ |