CI/CD Pipelines - Sonatype for GitLab CI
Project Configuration
Any project that wants to leverage GitLab's CI/CD pipeline must include a .gitlab-ci.yml pipeline definition file in the project's root. Documentation on the various options for pipeline and job definitions is available here. A GitLab pipeline job that performs policy evaluation consists of the following elements and may look like this:
iq_policy_eval:
stage: test
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
- /sonatype/evaluate -i WebAppX target/web-app-x.war
artifacts:
name: "policy-eval-$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
paths:
- WebAppX-policy-eval-report.htmlLet's explore some of these elements in more detail:
iq_policy_eval - the name of the job and can be anything you like
stage: test - the pipeline stage in which to run the policy evaluation; can be any one of the built-in stages or any custom stage defined earlier in the pipeline definition file
image: sonatype/gitlab-nexus-iq-pipeline:latest - the Docker image that will be used during the execution of this pipeline job. Available versions can be found in the GitLab Sonatype IQ docker image repository.
script - the script statements that will be run in the context of the given Docker image; in this case, the following minimal elements are required:
/sonatype/evaluate is the shell script inside the docker container that executes the policy evaluation
-i WebAppX is the unique ID of the application being evaluated, as defined in the IQ server
target/web-app-x.war is a list of one or more artifacts from the build environment to perform the evaluation against
See the documentation provided with the image on Docker Hub for details about other options available during policy evaluation
artifacts - list of directories or files that will be zipped and archived with the executed pipeline; see the GitLab documentation for more information on how long these artifacts live and how to access them
name - the name to give the artifact zip file (note: pipeline variables can be used to create the name, as shown in the above example)
paths - references to one or more files or directories in the build environment to include in the archive
WebAppX-policy-eval-report.html - this is the name of the report generated by the policy evaluation and is comprised of '<app id>-policy-eval-report.html'
In order to make GitLab CI/CD pipeline job artifacts available across jobs you should add them to the pipeline cache, which could be defined a little further up in the file, as follows:
cache:
paths:
- target/
- <submodule1>/target/
- <submodule2>/target/
Here is a typical output for a policy evaluation job configured as above (some lines were omitted for brevity):
Commencing IQ policy evaluation... [INFO] Validating IQ Server version http://192.168.1.201:8070... [INFO] Validating application ID repo-a with the IQ Server http://192.168.1.201:8070... [INFO] Discovered commit hash 'c857286a3c4bc332e9b4c279fa7fab8dcb9377e3' via environment variable CI_COMMIT_SHA [INFO] Starting scan... [INFO] Scan target: /builds/root/repo-a/target/dependency/commons-fileupload-1.4.jar ... [INFO] 2021-04-09T15:34:33.869Z Finished scanning target: /builds/root/repo-a/target/sonatype-clm/module.xml [INFO] 2021-04-09T15:34:33.869Z Scanned 1455 total files [INFO] Fingerprinting completed in 1 seconds for 8 archives, 1296 total files [INFO] Discovered repository url 'http://172.100.0.2/root/repo-a' via environment variable CI_PROJECT_URL [INFO] Waiting for policy evaluation to complete... [INFO] Assigned scan ID 361b686717f94080a73ccb9c0a2d99cb [INFO] Policy evaluation completed in 10 seconds. [INFO] [ERROR] The IQ Server reports policy failing due to Policy(Sev-Critical) [ Component(displayName=org.apache.logging.log4j : log4j-core : 2.3, hash=58a3e964db5307e30650) [ Constraint(Severity) [Security Vulnerability Severity >= 8 because: Found security vulnerability CVE-2017-5645 with severity >= 8 (severity = 9.8), on condition 0] ]] [ERROR] The IQ Server reports policy failing due to Policy(Sev-High) [ Component(displayName=Junit : junit : 4.12, hash=2973d150c0dc1fefe998) [ Constraint(Severity) [Security Vulnerability Severity >= 4 because: Found security vulnerability CVE-2020-15250 with severity >= 4 (severity = 5.5), on condition 0] ]] ... [ERROR] The IQ Server reports policy failing due to Policy(Sev-High) [ Component(displayName=commons-io : commons-io : 2.2, hash=83b5b8a7ba1c08f9e8c8) [ Constraint(Severity) [Security Vulnerability Severity < 8 because: Found security vulnerability sonatype-2018-0705 with severity < 8 (severity = 7.8), on condition 0] ]] [INFO] [INFO] [INFO] ********************************************************************************************* [INFO] Policy Action: Failure [INFO] Stage: build [INFO] Number of components affected: 1 critical, 3 severe, 0 moderate [INFO] Number of open policy violations: 1 critical, 12 severe, 0 moderate [INFO] Number of grandfathered policy violations: 0 [INFO] Number of components: 8 [INFO] The detailed report can be viewed online at http://192.168.1.201:8070/ui/links/application/repo-a/report/361b686717f94080a73ccb9c0a2d99cb [INFO] ********************************************************************************************* [INFO] Processing policy evaluation results... ...IQ policy evaluation complete
The policy evaluation job's output lists all the files that are considered for evaluation. If there are policy violations, they are listed as errors. Each policy violation states its severity, the component that triggered the violation, and the failed policy details.
The policy evaluation summary can be found at the end of the job's output. It also contains a link to a detailed report in Lifecycle.
IQ Server Connection Information
The required connection information for the IQ server can be specified at either the group or project level using the CI/CD settings, as follows:
The following three environment variables must be defined if not set explicitly in the job definition via script parameters:
NEXUS_IQ_URL
NEXUS_IQ_USERNAME
NEXUS_IQ_PASSWORD
An optional environment variable, named NEXUS_IQ_REPORT_FORMAT, can be set to control the content of the generated evaluation report. If the variable is set to summary, the evaluation report will contain brief summary information and a link to the detailed report on the IQ server. Otherwise, if set to enhanced, the evaluation report will contain a summary header, which includes a link to the detailed report on the IQ server, and details about the components that trigger policy actions, accompanied by policy violation details.
Specific to container scanning, for connecting to remote registries, the following two variables must be defined as well. These are the credentials for your Docker registry:
NEXUS_CONTAINER_IMAGE_REGISTRY_USER
NEXUS_CONTAINER_IMAGE_REGISTRY_PASSWORD
Problems with the IQ server connection will appear in the pipeline job output and can be diagnosed as follows:
Self-signed Certificates
If the IQ Server uses a self-signed certificate, additional configuration is needed to make Sonatype for GitLab CI work.
Two approaches could be taken to make that happen:
Add the self-signed certificate to the Docker container created from the GitLab Sonatype IQ Docker image on the fly, or
Create a custom Docker image from the GitLab Sonatype IQ Docker image, add the self-signed certificate to the new image, and use this newly created image to run policy evaluations against IQ Server.
Add the self-signed certificate on the fly
In order to achieve that the self-signed certificate must be available during the pipeline execution. It could be checked into the repository or pulled at runtime from a common location (e.g. using wget or similar tools).
Below you can see an example of a stripped-down pipeline that adds the self-signed certificate to the Docker container on the fly, assuming the certificate (i.e. iq-server.cert) is checked into the root folder of the repository. If you're implementing this in your environment, make sure the paths are updated accordingly:
iq_policy_eval:
stage: test
image: sonatype/gitlab-nexus-iq-pipeline:latest
before_script:
- keytool -keystore /opt/java/openjdk/lib/security/cacerts -storepass changeit -noprompt -trustcacerts -importcert -alias iq-server -file iq-server.cert
script:
- /sonatype/evaluate -i WebAppX target/web-app-x.war
artifacts:
name: "policy-eval-$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
paths:
- WebAppX-policy-eval-report.htmlCreate a custom Docker image
Here's a working Dockerfile that can be used to build a custom image, which contains the self-signed certificate:
FROM sonatype/gitlab-nexus-iq-pipeline COPY iq-server.cert /certs/ RUN keytool -keystore /opt/java/openjdk/lib/security/cacerts -storepass changeit -noprompt -trustcacerts -importcert -alias iq-server -file /certs/iq-server.cert
The image can be built (replace your-org and the image version with meaningful values) e.g.
docker build -t your-org/gitlab-nexus-iq-pipeline:1.0.0 .
Then it can be published and used for policy evaluations e.g.
iq_policy_eval:
stage: test
image: your-org/gitlab-nexus-iq-pipeline:1.0.0
script:
- /sonatype/evaluate -i WebAppX target/web-app-x.war
artifacts:
name: "policy-eval-$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
paths:
- WebAppX-policy-eval-report.htmlSonatype IQ Policy Evaluation
Note
Available for all GitLab tiers
The /sonatype/evaluate action allows you to perform policy evaluations against one or more build artifacts as part of a GitLab CI/CD pipeline.
Usage
myJob:
stage: <GitLab pipeline stage>
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
/sonatype/evaluate [options] <Archives or directories to scan>Example
policyEval:
stage: test
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
/sonatype/evaluate -i SomeWebApp target/our-web-app.war Available Options
Option | Description |
|---|---|
| IQ Server URL. If not provided, the NEXUS_IQ_URL environment variable must be set. Required |
| IQ Server credentials. If not provided, the NEXUS_IQ_USERNAME and NEXUS_IQ_PASSWORD environment variables must be set. Required |
| The ID of the application on the IQ Server. Required |
| The IQ Server stage to run analysis against. Accepted values: |
| The ID of the organization on the IQ Server. This determines the organization under which the application will be created in case the application doesn't exist and the automatic application creation configuration is enabled. Default: |
| Controls the verbosity of policy evaluation HTML reports. Accepted values: |
| Name of the policy evaluation HTML report. Default: |
| Path to a JSON file where the results of the policy evaluation will be stored in a machine-readable format. Default: |
| Fail on policy evaluation warnings. Default: |
| Ignore system errors (IO, network, server, etc.). Default: |
| Ignore scanning errors (Corrupt files or malformed files, etc). Default: |
| Proxy to use; format host[:port]. Default: |
| Credentials for the proxy. Format: |
| Enable callflow analysis. Default: |
| Runs callflow analysis only for the given namespaces. Can be specified more than once, e.g: - |
| Enable debug logs. WARNING: This may expose sensitive information in the logs. Default: |
| Shows the help screen. |
| Sets Java system properties as key-value pairs. Can be specified more than once, e.g: |
Evaluation Report
The evaluate action generates an HTML report that includes a brief summary and a link to the detailed report on the IQ server. Additionally, it can provide information about the components that trigger policy actions, including specific policy violation details.
The evaluation report is stored in the build directory and is named -policy-eval-report.html by default. You can change the name using --report-name.
If you want to save the report as a pipeline artifact, you can add it in the artifacts section of the evaluation step in your .gitlab-ci.yml file, as follows:
artifacts:
paths:
- <application-id>-policy-eval-report.htmlAdditional Examples
In this case you can specify multiple scan targets (directories or files), separated by spaces, as part of the same evaluation:
policyEval:
stage: test
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
/sonatype/evaluate -i SomeSystem web-module/target/our-web-app.war lib-module/target/our-shared-lib.jar In this case you can perform separate evaluations against each artifact. You can either do that within the same job or define separate jobs for each IQ application.
This example uses the same job:
policyEval:
stage: test
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
/sonatype/evaluate -i SomeWebApp web-module/target/our-web-app.war
/sonatype/evaluate -i SomeSharedLib lib-module/target/our-shared-lib.jar This example uses multiple jobs:
webPolicyEval:
stage: test
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
/sonatype/evaluate -i SomeWebApp web-module/target/our-web-app.war
libPolicyEval:
stage: test
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
/sonatype/evaluate -i SomeSharedLib lib-module/target/our-shared-lib.jar This is a more complete example that illustrates one way of setting up a more realistic or typical pipeline with multiple stages and multiple policy evaluations.
A couple of important points:
It is recommended that each policy evaluation in the pipeline be configured to run at the appropriate IQ stage, as will be shown below.
It is important that policy evaluations for artifacts built from development/feature branches be run at the 'develop' IQ stage so as to not interfere with or create churn for the policy evaluation history of the default/main branch.
In this example we're going to use three pipeline stages for a maven project: build, test and release with a Nexus IQ policy evaluation for test and release. Since we're doing multiple evaluations we're going to setup a hidden template job that will consume an IQ_STAGE variable, as follows:
.nexus_iq_policy_eval: &nexus_iq_policy_eval
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
- /sonatype/evaluate -i SomeWebApp -t $IQ_STAGE target/*.war
artifacts:
paths:
- ./SomeWebApp-policy-eval-report.htmlThe period before the job name marks the job as hidden so it will not show up or be executed directly as part of the pipeline.
The ampersand-prefixed name after the job name is an alias for the job name that can be referenced in other job definitions.
Next, we'll setup the three policy evaluations using the template created above.
There are two policy evaluations configured at the test stage, one for the default/main branch and one for all other branches.
The GitLab pipeline rules 'only' and 'except' are used to specify which job runs against which branch.
iq_policy_main:
<<: *nexus_iq_policy_eval
stage: test
variables:
IQ_STAGE : "build"
only:
- main
iq_policy_branch:
<<: *nexus_iq_policy_eval
stage: test
variables:
IQ_STAGE : "develop"
except:
- mainThose two jobs are mutually exclusive based on the name of the branch supplied to the pipeline rules.
Note
If your default branch is named something other than 'main' simply replace 'main' with the actual name of your default branch. Consult the GitLab documentation for more information.
Note
Newer versions of GitLab, starting around version 12.3, are phasing in a new way of defining pipeline rules and may at some point deprecate the 'only' and 'except' rules shown above. So, pay particular attention to the documentation for the version you are using to know what's supported.
The release stage policy evaluation below uses the GitLab rule 'when' to specify that the given job will be manually started by a user. If all of the stages prior to release, in this example, have completed successfully GitLab will add a play button to the pipeline visualization that can be used to manually start the release policy evaluation job.
iq_policy_release:
<<: *nexus_iq_policy_eval
stage: release
variables:
IQ_STAGE : "release"
only:
- main
when: manualHere's a complete example showing all the jobs in proper context:
image: maven:latest
variables:
MAVEN_CLI_OPTS: "--batch-mode"
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
cache:
paths:
- .m2/repository/
- target/
stages:
- build
- test
- release
build:
stage: build
script:
- mvn $MAVEN_CLI_OPTS clean install
artifacts:
name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
when: on_success
paths:
- target/*.war
.nexus_iq_policy_eval: &nexus_iq_policy_eval
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
- /sonatype/evaluate -i SomeWebApp -t $IQ_STAGE target/*.war
artifacts:
paths:
- ./SomeWebApp-policy-eval-report.html
iq_policy_main:
<<: *nexus_iq_policy_eval
stage: test
variables:
IQ_STAGE : "build"
only:
- main
iq_policy_branch:
<<: *nexus_iq_policy_eval
stage: test
variables:
IQ_STAGE : "develop"
except:
- main
iq_policy_release:
<<: *nexus_iq_policy_eval
stage: release
variables:
IQ_STAGE : "release"
only:
- main
when: manual
Container Scanning via Sonatype Container Security
In order to scan containers in GitLab CI/CD, the Docker in Docker image must be used. Further details on Docker in Docker can be found on the GitLab Docs.
The environmental variables defined in the settings must be passed to the script since they are needed in the Docker container that will run within the Docker they were passed in to. In addition to explicitly passing in the parameters, two additional runtime parameters are needed:
-v /tmp:/tmp -v /var/run/docker.sock:/var/run/docker.sock
A starter example can be seen below where webgoat-8.0 is scanned as a part of the build:
image: docker:docker:24.0.5
services:
- docker:24.0.5-dind
build:
stage: build
script:
- docker run -v /tmp:/tmp -v $CI_PROJECT_DIR:/sonatype/reports -v /var/run/docker.sock:/var/run/docker.sock -e NEXUS_IQ_URL -e NEXUS_IQ_USERNAME -e NEXUS_IQ_PASSWORD -e NEXUS_CONTAINER_IMAGE_REGISTRY_USER -e NEXUS_CONTAINER_IMAGE_REGISTRY_PASSWORD sonatype/gitlab-nexus-iq-pipeline:latest /sonatype/evaluate -i SomeContainerApp container:http://registry.hub.docker.com/webgoat/webgoat-8.0
artifacts:
paths:
- $CI_PROJECT_DIR/$CI_PROJECT_NAME-policy-eval-report.htmlIn the example above, NEXUS_CONTAINER_IMAGE_REGISTRY_USER and NEXUS_CONTAINER_IMAGE_REGISTRY_PASSWORD are only needed if the image being scanned is private. If the image being scanned is publicly accessible, credentials do not need to be passed in.
Vulnerability Report Update
Note
Available for GitLab Ultimate
The /sonatype/create-vulnerability-report action can populate the Vulnerability Report under the Secure section in GitLab based on data provided by a previous policy evaluation step. For more information about the Vulnerability Report, refer to GitLab docs.
This action reads an evaluate action result file and generates a dependency scanning report, which GitLab automatically ingests. To connect the two steps, the --result-file parameter must be used in the evaluate step, and the result file must be exposed as a pipeline artifact in that step to make it available for subsequent steps.
GitLab will ingest the generated vulnerability report only if it is set as a pipeline artifact of type: dependency_scanning.
Note
This step will connect to the IQ Server to retrieve vulnerability details. The credentials set for the evaluate step will be used.
Usage
dependency_scanning:
needs: [ <evaluate-step-name> ]
stage: <GitLab pipeline stage>
when: always
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
- /sonatype/create-vulnerability-report [options]
artifacts:
reports:
dependency_scanning: <vulnerability-report.json>Example
iq_policy_evaluation:
stage: test
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
- /sonatype/evaluate -i testapp -r scan-result.json *.jar
artifacts:
when: always
paths:
- scan-result.json
dependency_scanning:
needs: ["iq_policy_evaluation"]
stage: test
when: always
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
- /sonatype/create-vulnerability-report -r scan-result.json --report-file vulnerability-report.json
artifacts:
reports:
dependency_scanning: vulnerability-report.json Available Options
Options | Description |
|---|---|
| The name of the JSON file where the results of the policy evaluation were stored. Required |
| The name of generated dependency scanning report, which is used to populate the Vulnerability Report section in GitLab. Required |
Fetch and Store SBOM Files
Note
Available for all GitLab tiers
The /sonatype/fetch-sbom action can download an SBOM file associated with a previous policy evaluation step from Sonatype IQ Server, more precisely an SBOM file associated with a specific scan ID. The SBOM file is also stored as a pipeline artifact.
The evaluate stores the scan ID in the NEXUS_IQ_SCAN_ID environment variable that can be later used by the fecth-sbom action.
GitLab Ultimate customers can use the SBOM file to populate the Dependency List security section, as long as the following conditions are met:
The SBOM is in JSON format, and the content conforms to the CycloneDX standard version 1.4 or 1.5.
The
--update-dependency-listflag is present.The SBOM file is set as a pipeline artifact of type:
cyclonedx.
For more information about the dependency list, refer to GitLab docs.
Usage
download_sbom:
needs: [ <evaluate-step-name> ]
when: always
stage: <GitLab pipeline stage>
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
- /sonatype/fetch-sbom [options]Example
iq_policy_evaluation:
stage: test
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
- /sonatype/evaluate -i testapp -r scan-result.json *.jar
artifacts:
when: always
reports:
dotenv: evaluate.env
paths:
- scan-result.json
download_sbom:
needs: ["iq_policy_evaluation"]
when: always
stage: deploy
image: sonatype/gitlab-nexus-iq-pipeline:latest
script:
- /sonatype/fetch-sbom -i testapp -si ${NEXUS_IQ_SCAN_ID} -ss cycloneDx -sv 1.5 -sf json -n sbom.cdx.json -udl
artifacts:
reports:
cyclonedx: sbom.cdx.jsonAvailable Options
Options | Description |
|---|---|
| IQ Server URL. If not provided, the |
| IQ Server credentials. If not provided, the |
| The ID of the application on the IQ Server. Required |
| The scan ID of the SBOM file to be downloaded. The |
| SBOM standard. Accepted values: |
| SBOM version. Accepted values for cycloneDx: |
| SBOM format. Accepted values: |
| The name of the SBOM file stored as a pipeline artifact. Default: |
| GitLab Ultimate: Enhance the SBOM with metadata needed for the Dependency List feature. Default: |