Data Retention and Purging
Lifecycle scans produce data that persists in IQ Server's work directory. IQ Server can purge this data automatically based on data retention rules that you define.
Purging is turned on by default in new installations of IQ Server. It's key to note that this automatic purging does not delete your data. It merely moves it to a trash directory for easy manual deletion.
Viewing
To see data retention rules, click Orgs and Policies from the left toolbar, then select the Root Organization or an Organization. Scroll down to the section labeled Data Retention. An example is below.
Understanding and Editing
Editing
To edit Data Retention Rules, click Orgs and Policies from the left toolbar, then select the Root Organization or an Organization. Scroll down to the section labeled Data Retention and click the Edit button at the right. You'll be presented with the screen shown below. In this example, we're editing at the Organization level.
Stages
Application Reports are separated by stage. An Application Report from a scan at the Source stage is saved and handled separately from a Report from a scan at the Release stage. When editing Data Retention rules, you'll edit each stage separately.
In the context of Data Retention, Continuous Monitoring reports are treated as their own stage. Rules set at the Continuous Monitoring stage will only remove reports created by the Continuous Monitoring feature.
Hierarchy & Inheritance
Like other features in IQ Server, Data Retention rules follow an internal hierarchy with Root Org at the top, Organizations below, and Applications at the bottom. For Data Retention rules specifically, Organizations can inherit rules from the Root Org, but Applications always inherent rules from their Organization.
Rules
Application Reports are purged based on two possible rules:
- Age, which is the time since the report was generated
- The number of reports stored in total for that application.
Reports are purged when either of those rules are satisfied.
Success Metrics
Success Metrics are generated by reviewing policy violation data. Because of this, policy violation data is saved even after violations are resolved and no longer affect your application. For Success Metrics, the only possible rule is age.
Note that this purging is limited to those policy violations which do not affect the current state of an application, i.e. policy violations that were previously resolved. Unresolved policy violations, including those which have merely been waived or grandfathered, are not purged regardless of how many years ago those violations were first discovered.
Trash Directory
In order to provide a safety net and ensure that you have ultimate control over your data, the purging action taken by IQ Server does not delete data. Instead, it compress the files and places them into a trash directory inside the working directory. By default, this is sonatype-work/clm-server/trash. Because the files are compressed, this purging action saves disk space even if you don't delete the data.
The trash directory contains subdirectories with names in a YYYY-MM-DD/XX format, with XX being the first two hex characters of the internal application ID.
Purged reports are placed Inside these dated subdirectories and named in the format app-{internalApplicationId}-report-{reportId}.zip.
To restore a purged report, simply unzip it into the report sub directory of IQ Server's working directory, i.e. sonatype-work/clm-server/report by default.
Avoid deleting purged reports inside the dated directory matching the current day in case IQ server is in process of writing a trashed report to that location.
Default Values
For new IQ Server installations, the Root Organization uses the following default data retention values. Organizations and Apps are set to inherit these values by default.
| Stage | Maximum Age | Max # of Reports |
|---|---|---|
| develop | 3 months | N/A |
| source | 3 months | N/A |
| build | 3 months | N/A |
| stage-release | 3 months | N/A |
| release | 10 years | N/A |
| operate | 10 years | N/A |
| continuous-monitoring | 3 months | N/A |
Timing
Purging occurs once a day at midnight, local server time.
Post Request
Application Reports can be purged manually with a POST request to the path /tasks/purgeObsoleteReports on the administrative port of IQ Server, e.g.
curl -X POST http://localhost:8071/tasks/purgeObsoleteReports