Success Metrics

Introduction to Success Metrics

Sonatype Lifecycle automatically tracks statistics about its usage called Success Metrics. These statistics reflect policy evaluation, violation, and remediation, which are aggregated monthly or weekly. By following the trends in policy evaluation data, you can monitor progress toward your organization’s goals with Lifecycle. For example, an organization using Lifecycle to select safer open-source components can measure its success with the newly discovered violations' statistics. Success Metrics are available in the Lifecycle UI and through a REST API. The data in the UI provides visualizations out of the box while the API allows you to use your own tools to analyze data or automate data collection.

The Success Metrics data is designed to demonstrate progress toward your organization’s goal with Lifecycle. This is not designed to audit the health of an application or organization. As a result, it only includes aggregated counts of violations and general risk categories for violation counts.

Calculating Success Metrics

Success Metrics are aggregated from historical data across all stages. Sonatype Lifecycle can tell if a violation was previously identified in an earlier scan, even if that scan was in a different stage. For example, if a violation appears in both the Build stage and Release stage, Success Metrics will only count that violation one time.

The REST API returns these aggregated counts for each application over a specified time period. Sonatype Lifecycle tracks exactly when each violation is first found and when it is resolved. This is used to calculate how long the violation was opened. An average of all violation resolution times is returned as the Mean Time To Resolution (MTTR). Like with other tallies, this statistic is aggregated across all stages.

The Success Metrics UI can display data for multiple applications. To handle this, Lifecycle will total or average statistics across all applications as appropriate.

Risk Categories

Success Metrics groups violations into categories corresponding to their Threat Level, a subjective value placed on the perceived risk of a policy violation.

CategoryLevel

Low

1
Moderate2, 3
Severe4, 5, 6, 7
Critical8, 9, 10

 

Additionally, the API groups violations into several categories. These categories are defined below.

Violation Type

Definition

SecurityThese violations are related to the Security Vulnerability Severity Score.
LicensePolicy violations based on the License or License threat group for a component.
QualityQuality violations are based on the component's age or popularity.
OtherThese are any policy violations that do not fit into any of the other categories.

Success Metrics in the UI

Sonatype Lifecycle can generate Success Metrics reports through the user interface. These reports are standardized and provide visualizations and derived metrics based on the Success Metrics Data. The data charts and graphs in these reports are designed to show trends in application policy violations over time allowing you to monitor progress toward your goals. The reports created through the UI can be customized to show specific applications or all organizations.

Notes on Success Metrics Reports

  • Success Metrics reports are generated on a per-user basis. A report created by one user is not visible to another.
  • Large data sets can take significant time to load the first time a report is accessed. If the report includes the most recent evaluations, every load might take time.
  • Administrators can turn off Success Metrics in the System Preferences menu.

Generating the Report

  1. Navigate to the Lifecycle UI.
  2. Select Success Metrics from the navigation.
  3. Click + Add Report in the upper right corner of the screen.
  4. Name the report and select options. The “Aggregation data” section allows you to specify whether to include the most recent evaluations in the report. By default, only full calendar weeks and months are included. The “Scope” section allows you to specify which applications and organizations to include in your report. By default, all applications to which you have access are included. Alternatively, you can specify a custom set of organizations and applications. Clicking on Custom will display a list of organizations and applications to which you have access.
  5. View the report from the Success Metrics page.

Reviewing the Report

Success Metrics Reports are defined on a per-user basis. A report created by one user will not be visible to others.

The resulting report has six sections, each presenting a visualization for application data. Please note that Success Metrics reports are defined on a per-user basis. A report created by one user will not be visible to others.

  • 12-Week Policy Violation Activity

The 12-Week Policy Violation section shows a collection of graphs that represent all violation discovery and remediation information for the past 12 weeks. Each bar in the graph represents activity for a week. If a violation is discovered and fixed in the same week, the violation will appear in the totals for both discovered and remediated violations.

Screenshot of 12-Week Policy Violation Activity

  • 12-Week Open Violation Totals
     The 12-Week Open Violation Totals panel charts the number of unresolved violations for each policy type.

Screenshot of 12-Week Open Violation Totals

  • Averages
    The Averages panel displays the average number of evaluations performed each month and the average number of violations per application. The chart separates critical violations and violations by policy type. The information in this panel covers the past year.

Screenshot of Average Number of Violations Discovered Per Month, Per Application

  • Mean Time To Resolution (MTTR) by Month
    Mean Time To Resolution by Month plots the average time to resolve a violation for the last year. Like the other panels, this information is broken down by policy type.

Screenshot of Mean Time to Resolution by Month

  • Applications
    The Applications panel provides statistics about the average number of violations present in each application and how many of those violations are critical. Again, this data is separated by policy type.

Screenshot of Applications with Violations by Policy Type

  • Components
    The Components panel provides statistics about the components used in each application. The chart highlights the top five most common components across all applications and the five components with the most policy violations.

Screenshot of Components view from report

Success Metrics via API

Sonatype Lifecycle also provides data through a REST API. The Success Metrics API lets you retrieve data about your applications. Using the returned data, you can create custom reports or data visualizations to better measure your organization’s use of Lifecycle. The API allows you to automate data collection and quickly receive Lifecycle data. This is the same raw data used in the UI and can be fetched as either a JSON or CSV format.

Making an API Request

The Success Metrics Data API can be accessed using a POST request. The API can return data as either a JSON or CSV. The returned format is specified in the request headers along with authentication information and time period. The start and stop dates for the request accept the  ISO 8601 format. In this format, the first week of the year is the week with the first Thursday in January. The annotated API request below has information about the specific values the Success Metrics API accepts. Visit the documentation page for more information.

Annotated API Request

The following is an annotated example API request using the cURL tool. An explanation of the options and values for the request is in the comment lines, indicated with a // .

curl \
    // username and password
    -u admin:admin123 \
    // Success Metrics API uses the POST method
    -X POST \
    // The content type can either be application/json or text/csv
    -H "Content-Type: application/json" \
    -d '{
    // Time period can be either "MONTH" or "WEEK"
    "timePeriod":"MONTH",
    // For MONTH requests firstTimePeriod accepts an ISO 8601 year and month without a timezone
    // For WEEK requests firstTimePeriod accepts an ISO 8601 week year and week (eg "2009-W01")
    "firstTimePeriod": "2021-01",
    // lastTimePeriod requires a date in the same format as firstTimePeriod. 
    // lastTimePeriod cannot be before first time period
    // Omitting lastTimePeriod will return all successive data including partial data for current time period
    "lastTimePeriod": "2021-03",
    // applicationIds accepts an array of internal, Lifecycleassigned, IDs
    // leaving these blank will return data for all applications available to the user
    "applicationIds": [],
    "organizationIds": []
}' \
'http://localhost:8070/api/v2/reports/metrics'

API Value Reference

The tables below define the individual values returned by the Success Metrics API. The values are grouped into categories based on the kind of data returned.

Application Data

The Success Metrics API returns data for each application the user has permission to view. The following fields provide information to identify each application and its parent organization.

FieldData DescriptionData Type
applicationId The unique ID for each application. This is assigned by Lifecycle.string 
applicationPublicId The customer-assigned application ID.string 
applicationName The customer-assigned application name.string 
organizationId The unique ID for the application's parent organization. This is assigned by Lifecycle.string 
organizationName The customer-assigned name of the organization.string 
timePeriodStart This date is the beginning of the data-collection period. The returned data is aggregated weekly or monthly as specified in the API request.ISO 8601 date 
evaluationCount The total number of scans (evaluations) for the application during the returned timeframe. integer 

Time to Resolution Data

The mean time to resolution statistics (MTTR) tracks the average time it took to resolve a violation after it was detected These statistics track the average time from a violation being identified to remediation. The time is listed in milliseconds.

FieldData DescriptionData Type
mttrLowThreat Mean time to resolution for level 1 threats in milliseconds.integer 
mttrModerateThreat Mean time to resolution for levels 2 and 3 threats in milliseconds.integer 
mttrSevereThreat Mean time to resolution for levels 4-7 threats in milliseconds.integer  
mttrCriticalThreat Mean time to resolution for levels 8-10 threats in milliseconds.integer

Threat Discovery Data

FieldData DescriptionData Type
discoveredCountSecurityLow The number of newly discovered low-level security threats in the time period.integer
discoveredCountSecurityModerate The number of newly discovered moderate-level security threats in the time period.integer
discoveredCountSecuritySevere The number of newly discovered severe-level security threats in the time period.integer
discoveredCountSecurityCritical The number of newly discovered critical-level security threats in the time period.integer
discoveredCountLicenseLow The number of newly discovered low-level license threats in the time period.integer
discoveredCountLicenseModerate The number of newly discovered moderate-level license threats in the time period.integer
discoveredCountLicenseSevere The number of newly discovered severe-level license threats in the time period.integer
discoveredCountLicenseCritical The number of newly discovered critical-level license threats in the time period.integer
discoveredCountQualityLow The number of newly discovered low-level quality threats in the time period.integer
discoveredCountQualityModerate The number of newly discovered moderate-level quality threats in the time period.integer
discoveredCountQualitySevere The number of newly discovered severe-level quality threats in the time period.integer
discoveredCountQualityCritical The number of newly discovered critical-level quality threats in the time period.integer
discoveredCountOtherLow The number of newly discovered low-level other threats in the time period.integer
discoveredCountOtherModerate The number of newly discovered moderate-level other threats in the time period.integer
discoveredCountOtherSevere The number of newly discovered severe-level other threats in the time period.integer
discoveredCountOtherCritical The number of newly discovered critical-level other threats in the time period.integer

Policy Remediation Data

FieldData DescriptionData Type
fixedCountSecurityLow The number of low-level security violations fixed in the time period.integer 
fixedCountSecurityModerate The number of moderate-level security violations fixed in the time period.integer 
fixedCountSecuritySevere The number of severe-level security violations fixed in the time period.integer 
fixedCountSecurityCritical The number of critical-level security violations fixed in the time period.integer 
fixedCountLicenseLow The number of low-level license violations fixed in the time period.integer 
fixedCountLicenseModerate The number of moderate-level license violations fixed in the time period.integer 
fixedCountLicenseSevere The number of severe-level license violations fixed in the time period.integer 
fixedCountLicenseCritical The number of critical-level license violations fixed in the time period.integer 
fixedCountQualityLow The number of low-level quality violations fixed in the time period.integer 
fixedCountQualityModerate The number of moderate-level quality violations fixed in the time period.integer 
fixedCountQualitySevere The number of severe-level quality violations fixed in the time period.integer 
fixedCountQualityCritical The number of critical-level quality violations fixed in the time period.integer 
fixedCountOtherLow The number of low-level other violations fixed in the time period.integer 
fixedCountOtherModerate The number of moderate-level other violations fixed in the time period.integer 
fixedCountOtherSevere The number of severe-level other violations fixed in the time period.integer 
fixedCountOtherCritical The number of critical-level other violations fixed in the time period.integer 

Waived Violation Data

FieldData DescriptionData Type
waivedCountSecurityLow The number of low-level security violations waived in the specified time period.integer 
waivedCountSecurityModerate The number of moderate-level security violations waived in the specified time period.integer 
waivedCountSecuritySevere The number of severe-level security violations waived in the specified time period.integer 
waivedCountSecurityCritical The number of critical-level security violations waived in the specified time period.integer 
waivedCountLicenseLow The number of low-level license violations waived in the specified time period.integer 
waivedCountLicenseModerate The number of moderate-level license violations waived in the specified time period.integer 
waivedCountLicenseSevere The number of severe-level license violations waived in the specified time period.integer 
waivedCountLicenseCritical The number of critical-level license violations waived in the specified time period.integer 
waivedCountQualityLow The number of low-level quality violations waived in the specified time period.integer 
waivedCountQualityModerate The number of moderate-level quality violations waived in the specified time period.integer 
waivedCountQualitySevere The number of severe-level quality violations waived in the specified time period.integer 
waivedCountQualityCritical The number of critical-level quality violations waived in the specified time period.integer 
waivedCountOtherLow The number of low-level other violations waived in the specified time period.integer 
waivedCountOtherModerate The number of moderate-level other violations waived in the specified time period.integer 
waivedCountOtherSevere The number of severe-level other violations waived in the specified time period.integer 
waivedCountOtherCritical The number of critical-level other violations waived in the specified time period.integer 

Open Data

Note: Violations that are Legacy Violations or have been waived are not considered open violations and will not appear as the results of these API calls.

FieldData DescriptionData Type
openCountAtTimePeriodEndSecurityLow The number of unresolved low-level security threats at the end of the time period.integer 
openCountAtTimePeriodEndSecurityModerate The number of unresolved moderate-level security threats at the end of the time period.integer 
openCountAtTimePeriodEndSecuritySevereThe number of unresolved severe-level security threats at the end of the time period.integer 
openCountAtTimePeriodEndSecurityCriticalThe number of unresolved critical-level security threats at the end of the time period.integer 
openCountAtTimePeriodEndLicenseLowThe number of unresolved low-level license threats at the end of the time period.integer 
openCountAtTimePeriodEndLicenseModerateThe number of unresolved moderate-level license threats at the end of the time period.integer 
openCountAtTimePeriodEndLicenseSevereThe number of unresolved severe-level license threats at the end of the time period.integer 
openCountAtTimePeriodEndLicenseCriticalThe number of unresolved critical-level license threats at the end of the time period.integer 
openCountAtTimePeriodEndQualityLowThe number of unresolved low-level quality threats at the end of the time period.integer 
openCountAtTimePeriodEndQualityModerateThe number of unresolved moderate-level quality threats at the end of the time period.integer 
openCountAtTimePeriodEndQualitySevereThe number of unresolved severe-level quality threats at the end of the time period.integer 
openCountAtTimePeriodEndQualityCriticalThe number of unresolved critical-level quality threats at the end of the time period.integer 
openCountAtTimePeriodEndOtherLowThe number of unresolved low-level other threats at the end of the time period.integer 
openCountAtTimePeriodEndOtherModerateThe number of unresolved moderate other-level threats at the end of the time period.integer 
openCountAtTimePeriodEndOtherSevereThe number of unresolved severe-level other threats at the end of the time period.integer 

openCountAtTimePeriodEndOtherCritical

The number of unresolved critical-level other threats at the end of the time period.integer 

Disabling Success Metrics

Success Metrics can be disabled by an administrator in the Lifecycle UI:

  1. Click on the System Preferences cog in the upper right part of the screen.
  2. Select Success Metrics from the dropdown menu and toggle to either enable or disable them.
  3. Click Update to save your changes.

Screenshot of window to enable success metrics

Additional Resources

Success Metrics is a powerful tool to track progress toward goals with Sonatype Lifecycle. The UI provides visualizations and reports while the API provides customizable data and allows you to retrieve data for use with your own tools.

Additionally, the Success Metrics Community Project provides a simple web application for viewing Success Metrics and creating PDF reports. This tool is not supported by Sonatype. The following are Sonatype resources for Success Metrics: