Data Retention and Purging
Lifecycle scans produce data that persists in IQ Server's working directory. IQ Server can purge this data automatically based on data retention rules that you define.
Purging is turned on by default in new installations of IQ Server. It's key to note that this automatic purging does not delete your data. It merely moves it to a trash directory for easy manual deletion.
To see data retention rules, click Orgs and Policies from the left toolbar, then select a Root Org, Org, or Application. Scroll down to the section labeled Data Retention. An example is below.
Understanding and Editing
To edit Data Retention Rules, click Orgs and Policies from the left toolbar, then select the Root Organization or an Organization. Scroll down to the section labeled Data Retention and click the Edit button at the right. You'll be presented with the screen shown below. In this example, we're editing at the Organization level.
Application Reports are separated by stage. An Application Report from a scan at the Source stage is saved and handled separately from a Report from a scan at the Release stage. When editing Data Retention rules, you'll edit each stage separately.
In the context of Data Retention, Continuous Monitoring reports are treated as their own stage. Rules set at the Continuous Monitoring stage will only remove reports created by the Continuous Monitoring feature.
Hierarchy & Inheritance
Like other features in IQ Server, Data Retention rules follow an internal hierarchy with Root Org at the top, Organizations below, and Applications at the bottom. For Data Retention rules specifically, Organizations can inherit rules from the Root Org, but Applications always inherent rules from their Organization.
Application Reports are purged based on two possible rules:
- Age, which is the time since the report was generated
- The number of reports stored in total for that application.
Reports are purged when either of those rules are satisfied.
Success Metrics are generated by reviewing policy violation data. Because of this, policy violation data is saved even after violations are resolved and no longer affect your application. For Success Metrics, the only possible rule is age.
Note that this purging is limited to those policy violations which do not affect the current state of an application, i.e. policy violations that were previously resolved. Unresolved policy violations, including those which have merely been waived or grandfathered, are not purged regardless of how many years ago those violations were first discovered.
In order to provide a safety net and ensure that you have ultimate control over your data, the purging action taken by IQ Server does not delete data. Instead, it compress the files and places them into a trash directory inside the working directory. By default, this is
sonatype-work/clm-server/trash. Because the files are compressed, this purging action saves disk space even if you don't delete the data.
trash directory contains subdirectories with names in a
YYYY-MM-DD/XX format, with XX being the first two hex characters of the internal application ID.
Purged reports are placed Inside these dated subdirectories and named in the format
To restore a purged report, simply unzip it into the
report sub directory of IQ Server's working directory, i.e.
sonatype-work/clm-server/report by default.
Avoid deleting purged reports inside the dated directory matching the current day in case IQ server is in process of writing a trashed report to that location.
For new IQ Server installations, the Root Organization uses the following default data retention values. Organizations and Apps are set to inherit these values by default.
|Stage||Maximum Age||Max # of Reports|
Purging occurs once a day at midnight, local server time.
Because you may have hundreds of apps with daily scans, IQ Server can consume a significant amount of space. It's usually best-practice to leave data purging on.
Application Reports for internal dev builds typically become irrelevant after a few months, but by contrast, reports for a release/production version of your app should be kept for longer. By default, IQ Server purges after 3 months for early scans and 10 years for scans that occur later in an app's lifetime.
Socialize your IQ Server purging strategy and establish a way for users to request that some data not be purged.
The contents of your trash directory will likely contain sensitive data. Compressed reports still contain things like application names, component names, file extensions, and vulnerability data.
Regardless of your business size, review the trash directory when convenient. Remember that the only reports you'll find in the trash directory are reports that have already been purged by IQ Server.
Application Reports can be purged manually with a POST request to the path
/tasks/purgeObsoleteReports on the administrative port of IQ Server, e.g.
curl -X POST http://localhost:8071/tasks/purgeObsoleteReports