Data Retention and Purging

Lifecycle scans produce data that persists in IQ Server's working directory. IQ Server can purge this data automatically based on data retention rules that you define.

Purging is turned on by default in new installations of IQ Server. It's key to note that this automatic purging does not delete your data. It merely moves it to a trash directory for easy manual deletion.

Viewing

To see data retention rules, click Orgs and Policies from the left toolbar, then select a Root Org, Org, or Application. Scroll down to the section labeled Data Retention. An example is below.

Understanding and Editing

Editing

To edit Data Retention Rules, click Orgs and Policies from the left toolbar, then select the Root Organization or an Organization. Scroll down to the section labeled Data Retention and click the Edit button at the right. You'll be presented with the screen shown below. In this example, we're editing at the Organization level.

Build Stages

Application Reports are seperated by build stage. An Application Report from a scan at the Source stage is saved and handled seperately from a Report from a scan at the Release stage. When editing Data Retention rules, you'll edit each build stage seperately.

In the context of Data Retention, Continuous Monitoring reports are treated as their own build stage.

Hierarchy & Inheritance

Like other features in IQ Server, Data Retention rules follow an internal hierarchy with Root Org at the top, Organizations below, and Applications at the bottom. For Data Retention rules specifically, Organizations can inheret rules from the Root Org, but Applications always inheret rules from their Organization.

Rules

Application Reports are purged based on two possible rules:

  • Age, which is the time since the report was generated
  • The number of reports stored in total for that application.

.Reports are purged when either of those rules are satisfied.

Success Metrics

Success Metrics are generated by reviewing policy violation data. Because of this, policy violation data is saved even after violations are resolved and no longer affect your application. For Success Metrics, the only possible rule is age.

Note that this purging is limited to those policy violations which do not affect the current state of an application, i.e. policy violations that were previously resolved. Unresolved policy violations, including those which have merely been waived or grandfathered, are not purged regardless of how many years ago those violations were first discovered.

Trash Directory

In order to provide a safety net and ensure that you have ultimate control over your data, the purging action taken by IQ Server does not delete data. Instead, it compress the files and places them into a trash directory inside the working directory. By default, this is sonatype-work/clm-server/trash. Because the files are compressed, this purging action saves disk space even if you don't delete the data.

The trash directory contains subdirectories with names in a YYYY-MM-DD/XX format, with XX being the first two hex characters of the internal application ID.

Purged reports are placed Inside these dated subdirectories and named in the format app-{internalApplicationId}-report-{reportId}.zip.

To restore a purged report, simply unzip it into the report sub directory of IQ Server's working directory, i.e. sonatype-work/clm-server/report by default.

Avoid deleting purged reports inside the dated directory matching the current day in case IQ server is in process of writing a trashed report to that location.

Default Values

For new IQ Server installations, the Root Organization uses the following default data retention values. Organizations and Apps are set to inheret these values by default.

StageMaximum AgeMax # of Reports
develop3 monthsN/A
source3 monthsN/A
build3 monthsN/A
stage-release3 monthsN/A
release10 yearsN/A
operate10 yearsN/A
continuous-monitoring3 monthsN/A

Timing

Purging occurs once a day at midnight, local server time.

Best Practices

Because you may have hundreds of apps with daily scans, IQ Server can consume a significant amount of space. It's usually best-practice to leave data purging on.

Application Reports for internal dev builds typically become irrelevant after a few months, but by contrast, reports for a release/production version of your app should be kept for longer. By default, IQ Server purges after 3 months for early scans and 10 years for scans that occur later in an app's lifetime.

Socialize your IQ Server purging strategy and establish a way for users to request that some data not be purged.

The contents of your trash directory will likely contain sensitive data. Compressed reports still contain things like application names, component names, file extensions, and vulnerability data.

Regardless of your business size, review the trash directory when convenient. Remember that the only reports you'll find in the trash directory are reports that have already been purged by IQ Server.

Application Reports can be purged manually with a POST request to the path /tasks/purgeObsoleteReports on the administrative port of IQ Server, e.g.

curl -X POST http://localhost:8071/tasks/purgeObsoleteReports