Automated Pull Requests
NEW IN IQ SERVER RELEASE 79
Overview
Sonatype IQ for SCM automatically creates pull requests for policy violations on components with suggested remediation.
After any policy evaluation that generates new violations with available remediations, the IQ Server will clone the repository and attempt to create a pull request to bump the component to the new version. The pull request will contain links to the full IQ policy report that triggered it as well as direct links to any security vulnerability details.
Currently supported platforms: Bitbucket Cloud, Bitbucket Server, GitHub, GitLab
Currently supported ecosystems: Maven, NPM, Gradle, Go
Breaking Changes
Breaking changes information will inform the developer about the amount of work required to perform the version upgrade.
Advanced Remediation Strategies
Remediations that fix policy violations for the target component and its dependencies are preferred, if available, over remediations that only fix policy violations for the component. Otherwise, remediations that fix policy violations for the target component only are used, if such remediations exist.
Requirements
Private repositories
Automated pull requests will only work on repositories that cannot be accessed publicly.
Azure DevOps
Repositories need to be private.
Bitbucket Cloud
Repositories need to be private.
Bitbucket Server
Repositories need to be private.
GitHub
On github.com this means that the repository will need to be private. On GitHub Enterprise all repositories will work.
GitLab
NEW IN IQ SERVER RELEASE 97
Repositories need to be private or internal , regardless whether you are using GitLab SaaS or GitLab self-managed. Public projects will not work.
Note: While the term 'pull request' is used throughout this documentation, the equivalent terminology in GitLab is 'merge request'.
Nexus IQ Stage
Automated pull requests are supported on all stages except for the 'develop' stage.
Token permissions
See the Required Token Permissions section under Source Control Configuration for further details.
Configuration
To enable pull requests, ensure that your application is configured with:
- a repository URL
- a token for your SCM system (see Token Permissions in Requirements). Some SCM systems also require a username
- the 'Pull Requests' option is enabled
- a default branch to use as the 'base branch' for the pull request
The repository URL must be at the application level, but the other attributes can be at the application, organization, or root organization level. See Source Control Configuration for more details.
Repository URL
The repository URL must be entered as an https
URL. For example https://github.com/sampleaccount/samplerepo. The trailing .git
is optional. If an ssh://
URL is entered it will attempt to automatically convert it to https://
style. Entering a
git://
style URL is not supported. The same repository URL may be used for multiple IQ applications.
Default branch
This option lets you configure which git branch should be the base branch for pull requests created for this IQ application. The default value is the master
branch.
Common usage scenarios:
- All development happens off a primary single branch (normally
master
). This is the most common approach you would see on most OSS components on Github. Pull requests are created off ofmaster
and merged right in. Here you would just use the default value ofmaster
for the default branch. - The
master
branch is reserved for production and kept in a deployable state. Commonly known as Git Flow , development occurs on another branch (e.g.develop
) and pull requests are created off this branch. Once the development branch is ready for release it is merged to master. Here you would use the value of the development branch (e.g.develop
) for the default branch. - Some git repositories have multiple stable branches (for example jgit). Here each branch would be a separate IQ application and each would be configured for the appropriate branch.
IQ Server Clone Directory Configuration
For each configured application, the repository will be cloned on the host that the IQ Server runs on. The default location is:
<sonatypeWork>/source-control
Note that the default for sonatypeWork
is ./sonatype-work/clm-server.
The clone directory can be customized as follows, note that if the cloneDirectory
does not have a leading "/", then the path will be relative to <sonatypeWork>
.
In IQ Server Release 140 and later
NEW IN RELEASE 140
Use the source control configuration REST API.
In IQ Server Release 139 and prior
Use the IQ Server configuration file e.g.
sourceControl: cloneDirectory: /your/path/here
See the main documentation for the config.yml file for more details on how to modify it.
Version Bumping
When the feature checks out your source and attempts to find the component to remediate, it must be able to be found within the manifest. If the component and version cannot be found in the repository source, a pull request cannot be created.
Maven
Any pom.xml
that exists in the repository will be searched. Any pom.xml
that is in a src/test
directory is ignored.
Component defined inside a
<dependencies>
element with an inline<version>
element.<dependencies> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.9.9.3</version> </dependency> </dependencies>
Component defined inside a
<dependencyManagement>
element with an inline<version>
element.<dependencyManagement> <dependencies> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.9.9.3</version> </dependency> </dependencies> </dependencyManagement>
Component defined inside either a
<dependencies>
element, or a<dependencyManagement>
element, with the version defined via a variable in a<properties>
section.<properties> <jackson.version>2.9.9.3</jackson.version> </properties> <dependencies> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>${jackson.version}</version> </dependency> </dependencies>
NPM
NEW IN IQ SERVER RELEASE 84
Nexus IQ for SCM supports updates to your package.json
file. Upon policy violations for Javascript components, we will scan your package.json
for matching components and create a Pull Request for every match that has a remediation available.
Any package.json
that exists in the repository will be examined. Any
package.json
that contains in its path a directory named
test
,
spec
,
node_modules
, or
node
will be ignored. These are considered test files or build artifacts and not relevant project manifest files.
Dependencies
Nexus IQ for SCM supports
Components defined inside the
"dependencies"
section"dependencies": { "growl": "=1.9.0" }
Components defined inside the
"devDependencies"
section"devDependencies": { "eslint": "4.11.0" }
Note: Nexus IQ for SCM can only bump dependencies that are part of a scan and that have a policy violation with a remediation. Most often
devDependencies
are not included as part of a final build.Components defined inside the
"peerDependencies"
section"peerDependencies": { "react-dom": "~16.0.0" }
Components defined inside the
"optionalDependencies"
section"optionalDependencies": { "fsevents": "^1.2.5" }
Semantic Version Ranges
Nexus IQ for SCM supports the same semantic version range syntax from NodeJS Semver itself.
So for example the following is valid:
^6.1.0 <6.5.2
This will match version 6.1.0 all the way up, but not including, 6.5.2
Version Pinning and Best Practices
It is important to note that Nexus IQ for SCM will always bump to a specific version, therefore pinning it in your manifest. In the above range syntax example, this means that a package.json containing:
"dependencies": { "package": "^6.1.0 <6.5.2" }
Will result in a pull request to replace the contents with:
"dependencies": { "package": "6.4.1" }
Important: Pinning your dependency version is a Sonatype recommended best practice. Only a pinned version allows you to control remediation through security and quality policies. For example, version 1.3.7 might pass policy, but version 1.3.8 might not. A version range of ~1.3.0 does not allow this type of control.
Updating Your Lock Files
Nexus IQ for SCM will not update your lock files (NPM package.lock
or Yarn yarn.lock
). Lock files are highly dependent on your particular development workflow which likely includes a tailored build command and, of course, the latest security best practices such as authentication and procuring packages from a good repository manager. Why turn Nexus IQ into a Continuous Integration server or build system when you already have one?
In order to include proper lock file changes as part of the pull request, Sonatype recommends leveraging your CI system to perform those changes and run appropriate tests. From a high level this looks like:
- Create a CI job to scan for Nexus IQ for SCM created branches.
- Run your tailored build command which installs packages and run tests and updates lock files appropriately. For NPM this should include an
npm install
which will initiate any required updates to thepackage-lock.json
. - Automatically commit those lock file changes and push them to your SCM to be included in the pull request.
Now your lock file changes are guaranteed to be correct , and hopefully don't look like this
.
Gradle
NEW IN IQ SERVER RELEASE 103
Nexus IQ for SCM supports updates to build.gradle
files. Upon policy evaluations, for Gradle based projects we will scan your build.gradle
files for matching components and create a Pull Request for every match that has a remediation available.
All build.gradle
files that exist in the repository will be examined, except build.gradle
files in src/test
directories, which are ignored. These are considered test files or build artifacts and not relevant project manifest files.
Only Groovy based build files are supported.
NEW IN IQ SERVER RELEASE 120
Additionally, if a gradle.properties
file is found in the root directory, it will be considered as well.
Dependencies
Nexus IQ for SCM supports:
Component defined directly inside a
dependencies
block with a literal version.dependencies { // string dependency notation implementation "org.apache.struts:struts2-core:2.2.1" // map dependency notation smokeTest group: 'org.springframework', name: 'spring-api', version: '2.5' }
Component defined inside a
constraints
block of adependencies
block with a literal version.dependencies { constraints { // string dependency notation implementation "org.apache.struts:struts2-core:2.2.1" // map dependency notation smokeTest group: 'org.springframework', name: 'spring-api', version: '2.5' } }
Component defined inside a
dependencies
block with a property or variable version. Component versions are also looked up in thegradle.properties
file, if one exists.// Project property ext { slf4jSimpleVersion = '1.7.30' } // Groovy variable - dynamic typed def jerseyVersion = '2.29.1' // Groovy variable - static typed String jerseyVersionStr = '2.29.1' dependencies { implementation group: 'org.glassfish.jersey.core', name: 'jersey-core', version: jerseyVersion implementation group: 'org.glassfish.jersey.core', name: 'jersey-common', version: jerseyVersionStr implementation "org.slf4j:slf4j-simple:$slf4jSimpleVersion" }
Component defined inside a
dependencies
block with version constraints.// project-level inline declaration ext.slf4jVersion = '1.7.30' dependencies { implementation('org.slf4j:slf4j-api') { version { strictly '[1.7, 1.8[' prefer "$slf4jVersion" } } }
Gardle version ranges are supported for all the above mentioned cases.
ext { asmVersion = '[7.1,[' } dependencies { implementation "org.apache.struts:struts2-core:[2.2.1 , 2.2.10)" implementation group: 'org.slf4j', name: 'slf4j-core', version: ']1.20,1.30[' implementation('org.ow2.asm:asm') constraints { implementation("org.ow2.asm:asm:$asmVersion") { because 'we require a JDK 9 compatible bytecode generator' } } implementation('org.slf4j:slf4j-simple') { version { require '[1.7, 1.8[' prefer '1.7.25' reject '1.7.20', '1.7.21' } } }
Version Pinning and Best Practices
It is important to note that Nexus IQ for SCM will always bump to a specific version, therefore pinning it in your manifest. In the below range syntax example, this means that a build.gradle
containing:
dependencies { implementation "org.apache.struts:struts2-core:[2.2.1 , 2.2.10)" }
Will result in a pull request to replace the contents with (assuming that version 2.2.8 passes the policy check):
dependencies { implementation "org.apache.struts:struts2-core:2.2.8" }
Important: Pinning your dependency version is a Sonatype recommended best practice. Only a pinned version allows you to control remediation through security and quality policies. For the above example, version 2.2.8 might pass policy check, but version 2.2.9 might not. A version range does not allow this type of control.
Go
NEW IN IQ SERVER RELEASE 104
Any go.mod
file that exists in the repository will be searched for dependencies. Remediation PRs are created for the following scenarios:
Required modules declared in single or block
require
directivesmodule example.com/my/thing go 1.12 require golang.org/x/net v1.2.3 require golang.org/x/net/v2 v2.3.4 require ( golang.org/x/crypto v1.4.5 // indirect golang.org/x/text/v3 v3.6.7 )
Replaced modules declared in single or block
replace
directives, having a full version specified on the right side of thereplace
directivemodule example.com/my/thing go 1.12 replace golang.org/x/net v1.2.3 => golang.org/x/net v1.3.0 // a remediation PR may be created replace ( golang.org/x/crypto => golang.org/x/crypto v1.4.5 // a remediation PR may be created golang.org/x/text v1.2.3 => ../x/text // no remediation PR will be created golang.org/x/rest => ../x/rest // no remediation PR will be created )
Automated Pull Request Daily Activity
NEW IN IQ SERVER RELEASE 97
On the Source Control Configuration screen, we display any recent activity related to automated pull request creation.
Summary information for each attempt to create a pull request will be shown in a table. This information is purged on a daily rotating basis determined by when the server was last started, so at any time a maximum of 24 hours of results is shown. Note that the most common reason a pull request is not created is that the policy violation occurred in a transitive dependency (ie: where that dependency is not explicitly present in any project dependency configuration files). Remediating transitive dependencies is not currently supported.