Automated Pull Requests

NEW IN IQ SERVER RELEASE 79

Overview

Sonatype IQ for SCM will automatically create pull requests for policy violations on components that have an available version which remediates those violations.

Pull Request Commenting



After any policy evaluation that generates new violations with available remediations, the IQ Server will clone the repository and attempt to create a pull request to bump the component to the new version. The pull request will contain links to the full IQ policy report that triggered it as well as direct links to any security vulnerability details.

Currently supported platforms: GitHub, Bitbucket Server, Bitbucket Cloud

Currently supported ecosystems: Maven, NPM

Requirements

Private repositories

Automated pull requests will only work on repositories that cannot be accessed publicly.

GitHub

On github.com this means that the repository will need to be private. On GitHub Enterprise all repositories will work.

Bitbucket Server

Repositories need to be private.

Bitbucket Cloud

Repositories need to be private.

Nexus IQ Stage

Automated pull requests are supported on all stages except for the 'develop' stage.

Minimum Git Client Version

If the native git client is installed and available on the path it is preferred. The minimum git version that has been tested is version 2.16.0.

Token permissions

See the Required Token Permissions section under Source Control Configuration for further details.

Configuration

To enable pull requests, ensure that your application is configured with:

  • a repository URL
  • a token for your SCM system (see Token Permissions in Requirements). Some SCM systems also require a username
  • the 'Pull Requests' option is enabled
  • a default branch to use as the 'base branch' for the pull request

The repository URL must be at the application level, but the other attributes can be at the application, organization, or root organization level. See Source Control Configuration for more details.

Repository URL

The repository URL must be entered as an https  URL. For example https://github.com/sampleaccount/samplerepo. The trailing .git is optional. If an ssh:// URL is entered it will will attempt to automatically convert it to https:// style. Entering a git:// style URL is not supported. The same repository URL may be used for multiple IQ applications.

Default branch

This option lets you configure which git branch should be the base branch for pull requests created for this IQ application. The default value is the master  branch.

Common usage scenarios:

  1. All development happens off a primary single branch (normally master). This is the most common approach you would see on most OSS components on Github. Pull requests are created off of master and merged right in. Here you would just use the default value of master for the default branch.
  2. The master branch is reserved for production and kept in a deployable state. Commonly known as Git Flow, development occurs on another branch (e.g. develop) and pull requests are created off this branch. Once the development branch is ready for release it is merged to master. Here you would use the value of the development branch (e.g. develop) for the default branch.
  3. Some git repositories have multiple stable branches (for example jgit). Here each branch would be a separate IQ application and each would be configured for the appropriate branch.

IQ Configuration file (config.yml) Options

These configuration options can be set in the Nexus IQ Server configuration file. See the main documentation for the config.yml for details on how to modify this file.

Location on disk of cloned repositories

For each configured application, the repository will be cloned on the host the IQ Server runs on. The default location is:

<sonatypeWork>/source-control

Note that the default for sonatypeWork  is ./sonatype-work/clm-server

This can be customized in the config.yml with:

sourceControl:
  cloneDirectory: /your/path/here

If a leading "/" is not specified, then the path will be relative to <sonatypeWork>.

Git Client

Automated pull requests will work out of the box with no additional software installations necessary using the bundled JGit application library. JGit is a Java implementation of git that supports all git functionality necessary for the pull requests feature. However, JGit does not support two git features that can improve performance: shallow clone and sparse checkout. Shallow clones allow us to only clone the minimal amount of git history to enable the feature, and sparse checkouts allow us to only checkout the files necessary to bumping versions. These two features provide large disk-space savings, increased time performance and reduced network traffic.

If the native git client is installed and available on the system path, then it will be preferred over JGit. If you are having problems with native git, this behaviour can be overridden in the config.yml by forcing the usage of java git (JGit).

sourceControl:
  gitImplementation: java

Git Executable

If a native git client is installed but is not on the system path, the fully qualified path to the executable can be specified.

sourceControl:
  gitExecutable: /usr/bin/git


Version Bumping

When the feature checks out your source and attempts to find the component to remediate, it must be able to be found within the manifest. If the component and version cannot be found in the repository source, a pull request cannot be created.

Maven

Any pom.xml that exists in the repository will be searched. Any pom.xml that is in a src/test directory is ignored.

  • Component defined inside a <dependencies> element with an inline <version> element.

    <dependencies>
  <dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.9.9.3</version>
  </dependency>
</dependencies>

  • Component defined inside a <dependencyManagement> element with an inline <version> element.

    <dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-databind</artifactId>
      <version>2.9.9.3</version>
    </dependency>
  </dependencies>
</dependencyManagement>

  • Component defined inside either a <dependencies> element, or a <dependencyManagement> element, with the version defined via a variable in a <properties> section.

    <properties>                                                                                                                                                                                                 
  <jackson.version>2.9.9.3</jackson.version>
</properties>                                                                                                                                                                                                 

<dependencies>
  <dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>${jackson.version}</version>
  </dependency>
</dependencies>

NPM

NEW IN IQ SERVER RELEASE 84

Nexus IQ for SCM supports updates to your package.json  file. Upon policy violations for Javascript components we will scan your package.json for matching components and create a Pull Request for every match that has a remediation available.

Any package.json that exists in the repository will be examined. Any package.json that contains in its path a directory named test, spec, node_modules, or node will be ignored. These are considered test files or build artifacts and not relevant project manifest files.

Dependencies

Nexus IQ for SCM supports

  • Components defined inside the "dependencies" section

    "dependencies": {
  "growl": "=1.9.0"
}

  • Components defined inside the "devDependencies" section

    "devDependencies": {
  "eslint": "4.11.0"
}

    Note: Nexus IQ for SCM can only bump dependencies that are part of a scan and that have a policy violation with a remediation. Most often devDependencies are not included as part of a final build.

  • Components defined inside the "peerDependencies" section

    "peerDependencies": {
  "react-dom": "~16.0.0"
}

  • Components defined inside the "optionalDependencies" section

    "optionalDependencies": {
  "fsevents": "^1.2.5"
}

Semantic Version Ranges

Nexus IQ for SCM supports the same semantic version range syntax from NodeJS Semver itself.

So for example the following is valid:

^6.1.0 <6.5.2

This will match version 6.1.0 all the way up, but not including, 6.5.2

Version Pinning and Best Practices

It is important to note that Nexus IQ for SCM will always bump to a specific version, therefore pinning it in your manifest. In the above range syntax example, this means that a package.json containing:

"dependencies": {
  "package": "^6.1.0 <6.5.2"
}


Will result in a pull request to replace the contents with:

"dependencies": {
  "package": "6.4.1"
}

Important: Pinning your dependency version is a Sonatype recommended best practice. Only a pinned version allows you to control remediation through security and quality policies. For example, version 1.3.7 might pass policy, but version 1.3.8 might not. A version range of ~1.3.0 does not allow this type of control.

Updating Your Lock Files

Nexus IQ for SCM will not update your lock files (NPM package.lock or Yarn yarn.lock). Lock files are highly dependendant on your particular development work flow which likely includes a tailored build command and, of course, the latest security best practices such as authentication and procurring packages from a good repository manager. Why turn Nexus IQ into a Continuous Integration server or build system when you already have one?

In order to include proper lock file changes as part of the pull request, Sonatype recommends to leverage your CI system to perform those changes and run appropriate tests. From a high level this looks like:

  1. Create a CI job to scan for Nexus IQ for SCM created branches.
  2. Run your tailored build command which installs packages and run tests and updates lock files appropriately. For NPM this should include an npm install which will initiate any required updates to the package-lock.json.
  3. Automatically commit those lock file changes and push them to your SCM to be included in the pull request.

Now your lock file changes are guaranteed to be correct (big grin), and hopefully don't look like this (sad).