Data Retention Policy REST API - v2

NEW IN RELEASE 63

Data retention policies help to limit the disk space consumption of IQ Server by defining what data is obsolete and hence can be removed from the server. The REST API described here allows you to inspect and change the retention policies that are in effect.

Query Current Retention Policies

By making a HTTP GET request to the following URL relative to IQ Server's base URL the data retention policies for a given organization can be retrieved:

GET /api/v2/dataRetentionPolicies/organizations/{organizationId}

You can use the Organization REST API to locate the organizationId for an organization.

Assuming a local installation of IQ Server with its default configuration, the following example using the cURL tool queries the retention policies for the root organization:

curl -u admin:admin123 http://localhost:8070/api/v2/dataRetentionPolicies/organizations/ROOT_ORGANIZATION_ID

Again, assuming the default configuration, the above request would yield this JSON response payload (formatted here for readability):

{
  "applicationReports": {
    "stages": {
      "develop": {
        "inheritPolicy": false,
        "enablePurging": true,
        "maxAge": "3 months"
      },
      "build": {
        "inheritPolicy": false,
        "enablePurging": true,
        "maxAge": "3 months"
      },
      "stage-release": {
        "inheritPolicy": false,
        "enablePurging": true,
        "maxAge": "3 months"
      },
      "release": {
        "inheritPolicy": false,
        "enablePurging": true,
        "maxAge": "10 years"
      },
      "operate": {
        "inheritPolicy": false,
        "enablePurging": true,
        "maxAge": "10 years"
      },
      "continuous-monitoring": {
        "inheritPolicy": false,
        "enablePurging": true,
        "maxAge": "3 months"
      }
    }
  },
  "successMetrics": {
    "inheritPolicy": false,
    "enablePurging": true,
    "maxAge": "1 year"
  }
}

For application reports, each stage in the application lifecycle (develop, build, etc.) has its own retention policy. A notable exception are application reports that were created by continuous monitoring: They are not subject to the retention policy of whatever stage is being monitored but have their own retention policy under the key continuous-monitoring.

Note that the response lists only those stages that the installed product license enables. So you might see fewer entries with your IQ Server than shown in the above example.

A single retention policy for the application reports consists of these fields:

FieldDescription
inheritPolicyA boolean flag denoting whether the retention policy is being inherited from the parent organization (true) or it is locally customized (false). If the configuration is inherited, the remaining fields reflect that inherited configuration.
enablePurgingA boolean switch that enables (true) or disables (false) automatic purging.
maxAge(Optional) The maximum age that a report is allowed to reach before it is purged from the server. This criteria is omitted from the response if not set and otherwise carries values of the form {positive-number} day|days|week|weeks|month|months|year|years. Note that the latest application report in any given stage is always kept, regardless of its age.
maxCount(Optional) The maximum number of reports to retain, e.g. 100. When this number is exceeded the oldest reports are deleted first. This criteria is omitted from the response if not set for a retention policy. Normally, it is sufficient to configure a maxAge for automatic purging but where applications have a high frequency of evaluations and reports, the maxCount criteria can be used to enforce an upper limit and purge the excess reports even if they haven't reached the maximum age yet.

There is no priority between the maxAge and maxCount criteria. If both are set, then you may see the oldest reports that are below maxAge being removed because maxCount is exceeded, or old reports being deleted because maxAge is exceeded even if the number of reports is below maxCount.

The retention policy for Success Metrics consists mostly of the same fields as those for application reports:

FieldDescription
inheritPolicyA boolean flag denoting whether the retention policy is being inherited from the parent organization (true) or it is locally customized (false). If the configuration is inherited, the remaining fields reflect that inherited configuration.
enablePurgingA boolean switch that enables (true) or disables (false) automatic purging of data for Success Metrics.
maxAge(Optional) The maximum period of past time for which Success Metrics should be provided. This criteria is omitted from the response if not set and otherwise carries values of the form {positive-number} year|years.  For instance, the setting 1 year means that data which does not contribute to the metrics of the last 12 months is subject to purging.

Configure Retention Policies

A HTTP PUT request to the same URL when reading the current configuration is used to change the retention policies:

PUT /api/v2/dataRetentionPolicies/organizations/{organizationId}

The request needs to be accompanied by a JSON payload that has generally the same format as described above. It is however allowed to omit all but one stage from the stages map to selectively update only the retention policy for a few stages or to only specify successMetrics to update the retention policy for Success Metrics and not application reports.

The next example would update the retention policy of the root organization regarding application reports for the build stage:

curl -u admin:admin123 -H "Content-Type: application/json" -X PUT -d '{"applicationReports": {"stages": {"build": {"enablePurging": true, "maxAge": "2 weeks", "maxCount": 100}}}}' http://localhost:8070/api/v2/dataRetentionPolicies/organizations/ROOT_ORGANIZATION_ID

Note that when enablePurging is true, at least one purge criteria, maxAge or maxCount, needs to be specified.

If the request body is malformed, no changes to the retention policies are carried out and the status code 400 ("Bad Request") is sent back.

Purging of obsolete data is done automatically by a background job that runs once day. So enabling purging or setting stricter retention policies does typically not take effect until the next day.