Cleanup Policies
- Storage costs will increase
- Performance is impacted
- Artifact discovery will take longer
- Consuming all available storage will results in server failure
You can create cleanup policies and assign them to one or more proxy or hosted repositories so that a scheduled cleanup task (Admin - Cleanup repositories using their associated policies) will automatically soft delete artifacts from the repository. A soft delete means that artifacts are only marked for removal and not yet deleted from the disk. Disk space is not reclaimed until the Admin - Compact blob store task runs.
The Admin - Compact blob store task does not apply to S3 blob stores, which are cleaned up using AWS Lifecycle.
See here for more general information on tasks.
Cleanup Policy Security Requirements
Only users with the admin (i.e.,
nexus:*) privilege can use the Cleanup Policy left navigation item. Any user with privilege to edit a repository (such as nexus:repository-admin:maven2:maven-central:edit for the default maven-central repository) can adjust the policy a repository uses. Permission to edit the cleanup task is covered by the same permissions as other tasks (nx-tasks-update or nx-tasks-all). There is no individual privilege for just the Cleanup task.
Cleanup Policies
You can create and assign one or more cleanup policies to a repository under Administration → Repositories. You can assign cleanup policies to both proxy and hosted repositories.
Each policy's criteria will be ANDed together, removing only components that meet all of the specified conditions.
Name and Format are required fields and the name can not be edited after policy creation. You can also enter cleanup criteria to define the policy and can edit these fields later if desired as well.
Before you save the policy, you can preview a sample of the components that the policy would remove. Select a repository from the Preview Repository drop-down menu; then, select the Preview button to return a sample of the components that the cleanup policy would delete if applied to that repository at that point in time. You can also use the preview feature after saving the cleanup policy.
Criteria
The table below lists the available cleanup criteria and the formats to which they can apply:
| Criteria | Maven | Docker¹ | Helm | Yum | npm | NuGet | RubyGems | p2 | PyPI | Go | R | Raw | GitLFS | Bower | Apt | CocoaPods | Conda | Conan |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Component Age |  | | | | | | | | | | | | | ² | | | | |
| Component Usage | | | | | | | | | | | | | | ² | | | | |
| Release Type | | N/A | | | ||||||||||||||
Asset Name Matcher | | | | * | | | | | | | | | N/A | | | | | |
¹ - The Docker cleanup policy only checks against the tagged components. Cleanup is only complete when the Docker - Delete unused manifests and images task has run.
² - Bower functionality is for proxy repositories only.
"N/A" in the above table means Sonatype does not believe that the criteria applies to the format as built but acknowledges that it is possible that some customers may have their own schemes. If you believe this to be untrue, feel free to contact us at nexus-feedback@sonatype.com and let us know.
Component Age (Days)
This criteria sets how long to keep content based on component age, which is calculated based off of when the component was first downloaded from the public repository. If the criteria is set to 30 days, then the cleanup policy will soft delete components not modified (i.e., no one has re-deployed content to the same path) within the last 30 days.
Component Usage (Days)
This criteria sets how long to keep content based on when a component was last downloaded.
Release Type
Use this criteria to set the cleanup policy to either prereleases or releases.
Prereleases are identified differently by format:
| Format | What is considered prerelease |
|---|---|
| Maven | The versions contains -SNAPSHOT |
| npm | npm uses semantic versioning, so a version is considered prerelease if it contains the "-" character. |
| Yum | The non-case-sensitive "release" property in the RPM header contains one of the following: alpha, beta, rc, pre, prerelease, snapshot |
Asset Name Matcher
Components comprise one or more assets (individual files) in a blobstore. You can view asset names after selecting an asset in the Browse or Search views.
The specified RegEx will be evaluated against asset names. If there is a match, Nexus Repository deletes the associated component and all of its contained assets.
Expression Syntax
The expression engine uses an Elastic Search Regexp query syntax, from Apache Lucene.
The expressions are not Perl (PCRE) or Java util.regex.Pattern compatible regular expressions and use a limited set of operators.
Expression Examples
| Components in Target Repository | Policy Expression | Cleanup Case | Remaining Components After Cleanup Execution |
|---|---|---|---|
hello/-/hello-0.0.1.tgz hello/-/hello-0.0.2.tgz | hello/-/hello-0.0.[2-9].tgz
| All hello components with point versions 2-9. | hello/-/hello-0.0.1.tgz
|
org/example/test.jar
com/example/test.jar
test/example/test.jar
| (org|com)/.*
| Everything from the org and com groups. | test/example/test.jar
|
org/sonatype/team1/ui/5.0/ui-5.0.jar org/sonatype/team2/format/1.0/format-1.0.jar | org/sonatype/team[2-3].*
| Everything from the org.sonatype group on teams 2-3. | org/sonatype/team1/ui/5.0/ui-5.0.jar
|
pool/main/libc/libcap2/libcap2_2.25-1.2_amd64.deb | pool/main/z/zsh/zsh-common_5.4.2-3ubuntu3_all.deb
| A specific single Apt component. | pool/main/libc/libcap2/libcap2_2.25-1.2_amd64.deb |
Cleanup preview will show the component name but may be analyzing by the path. For example, in the example above, the preview might show hello-0.0.2.tgz and hello-0.0.3.tgz despite the fact the component contains hello/-/.
Cleanup Task
When you start a server with cleanup abilities enabled, Nexus Repository automatically creates a task named Cleanup service with the type Admin - Cleanup repositories using their associated policies. By default, this task is scheduled to run daily at 1AM server time. Similar to other tasks, you can edit, disable, and manually execute this task if desired. If you do delete this task, Nexus Repository will automatically recreate it on server restart. For more on tasks in general, see Configuring and Executing Tasks.
When run, this task executes cleanup of all repositories that have a policy other than None set. There is no partial execution. This task cannot be manually created and either runs or does not.
Hard Deleting Cleaned Up Components
Cleanup tasks only soft delete components; they mark them for deletion, but don't actually fully remove them. No blob store space is reclaimed until you execute an Admin - Compact blob store task.
For Azure blob stores, the Admin - Compact blob store task asks the Azure blob store to perform a hard delete by calling the delete function on the Azure client. This function then marks the specified blob for deletion, and it is deleted during garbage collection on the Azure side. This may vary depending on whether or not you have the soft delete feature enabled as described in Azure's documentation.
This task does not apply to S3 blob stores, which are cleaned up using AWS Lifecycle.
See here for more general information on configuring tasks.
Docker Cleanup Strategies
Docker has a unique way of managing components and assets (See Components and Assets in Docker) and therefore requires some thought when designing a cleanup strategy.
The following table outlines what tasks will soft-delete blobs in blob stores for Docker repositories:
| Optimal Run Order | Task Type | Feature | What it deletes? |
|---|---|---|---|
| 1 | Docker - Delete incomplete uploads | Tasks | soft-delete dangling uploads in temporary blob store storage which have not been resumed |
| 2 | Admin - Cleanup repositories using their associated policies | Cleanup Policies | soft-delete old published or downloaded docker components i.e. tags, not layers or manifests |
| 3 | Docker - Delete unused manifests and images | Tasks | soft-delete orphaned layers and manifests no longer referenced by tags, possibly orphaned by cleanup policies |
The following table outlines what features actually perform a hard-delete (i.e. free storage space) of soft-deleted blobs:
| Feature | Blob Store Type |
|---|---|
| Admin - Compact blob store task | File |
| S3 blob store configuration Expiration Days | AWS S3 |
Additional Information
Determining which Repositories Use the Most Space
Refer to the Support article: Investigating Blobstore and Repository Size and Space Usage, and the nx-blob-repo-space-report.groovy script that is provided.
For purposes of repository size, you'll want to look at totalBytes within the output.
The above script may have performance issues with large blobstores and is not applicable for S3 or PostgreSQL; it will only work on file-based blob stores and OrientDB.
Cleanup Policies' Impact to Other "Delete" Tasks
The implementation documented on this page should replace the need for any Maven - Delete usused SNAPSHOT and Repository - Delete unused components tasks by using the Last Downloaded Before criteria.
Maven - Delete SNAPSHOT tasks are not yet completely replaced .
Docker - Delete unused manifests and images is not replaced. In fact, it is necessary to run after your cleanup policies to remove orphaned layers and manifests.
Docker - Delete Incomplete Uploads , Admin - Cleanup tags and any other task specific to delete or cleanup not mentioned here is not covered by policy cleanup at all and need to continue to be used as they are.
If you are using an older version of Nexus Repository 3, you will need to use these existing delete tasks rather than cleanup policies.
For more on tasks in general, see Configuring and Executing Tasks.
Upgrading Existing Tasks
There is no migration in place, so creation of similar policies then assignment to repositories as well as deletion or disabling of existing tasks must all be done manually. Because cleanup is also implemented as a scheduled task, there is no collision if both remain running however it is a resource drain. We recommend rollover to this feature once configured and comfortable.
Related Content
See Keeping Disk Usage Low for further tips.