NEW IN IQ SERVER RELEASE 79

Overview

Sonatype IQ for SCM automatically creates pull requests for policy violations on components with suggested remediation.

After any policy evaluation that generates new violations with available remediations, the IQ Server will clone the repository and attempt to create a pull request to bump the component to the new version. The pull request will contain links to the full IQ policy report that triggered it as well as direct links to any security vulnerability details.

Currently supported platforms: Bitbucket Cloud, Bitbucket Server, GitHub, GitLab

Currently supported ecosystems: Maven, NPM, Gradle, Go

Breaking Changes

NEW IN IQ SERVER RELEASE 108

Breaking changes information is made available through the Advanced Development Pack add-on feature for Lifecycle. If available, the breaking changes information will inform the developer about the amount of work required to perform the version upgrade.

Advanced Remediation Strategies

NEW IN IQ SERVER RELEASE 111

If the Advanced Development Pack add-on feature is enabled, remediations that fix policy violations for the target component and its dependencies are preferred, if available, over remediations that only fix policy violations for the component. Otherwise, remediations that fix policy violations for the target component only are used, if such remediations exist.

Requirements

Private repositories

Automated pull requests will only work on repositories that cannot be accessed publicly.

Azure DevOps

Repositories need to be private.

Bitbucket Cloud

Repositories need to be private.

Bitbucket Server

Repositories need to be private.

GitHub

On github.com this means that the repository will need to be private. On GitHub Enterprise all repositories will work.

GitLab

NEW IN IQ SERVER RELEASE 97

Repositories need to be private or internal, regardless whether you are using GitLab SaaS or GitLab self-managed. Public projects will not work.

Note: While the term 'pull request' is used throughout this documentation, the equivalent terminology in GitLab is 'merge request'.

Nexus IQ Stage

Automated pull requests are supported on all stages except for the 'develop' stage.

Minimum Git Client Version

If the native git client is installed and available on the path it is preferred. The minimum git version that has been tested is version 2.16.0.

Token permissions

See the Required Token Permissions section under Source Control Configuration for further details.

Configuration

To enable pull requests, ensure that your application is configured with:

a repository URL
a token for your SCM system (see Token Permissions in Requirements). Some SCM systems also require a username
the 'Pull Requests' option is enabled
a default branch to use as the 'base branch' for the pull request

The repository URL must be at the application level, but the other attributes can be at the application, organization, or root organization level. See Source Control Configuration for more details.

Repository URL

The repository URL must be entered as an https URL. For example https://github.com/sampleaccount/samplerepo. The trailing .git is optional. If an ssh:// URL is entered it will attempt to automatically convert it to https:// style. Entering a git:// style URL is not supported. The same repository URL may be used for multiple IQ applications.

Default branch

This option lets you configure which git branch should be the base branch for pull requests created for this IQ application. The default value is the master branch.

Common usage scenarios:

All development happens off a primary single branch (normally master). This is the most common approach you would see on most OSS components on Github. Pull requests are created off of master and merged right in. Here you would just use the default value of master for the default branch.
The master branch is reserved for production and kept in a deployable state. Commonly known as Git Flow, development occurs on another branch (e.g. develop ) and pull requests are created off this branch. Once the development branch is ready for release it is merged to master. Here you would use the value of the development branch (e.g. develop) for the default branch.
Some git repositories have multiple stable branches (for example jgit). Here each branch would be a separate IQ application and each would be configured for the appropriate branch.

IQ Configuration file (config.yml) Options

These configuration options can be set in the Nexus IQ Server configuration file. See the main documentation for the config.yml for details on how to modify this file.

Location on disk of cloned repositories

For each configured application, the repository will be cloned on the host the IQ Server runs on. The default location is:

<sonatypeWork>/source-control

Note that the default for sonatypeWork is ./sonatype-work/clm-server

This can be customized in the config.yml with:

sourceControl:
  cloneDirectory: /your/path/here

If a leading "/" is not specified, then the path will be relative to <sonatypeWork>.

Git Client

Automated pull requests will work out of the box with no additional software installations necessary using the bundled JGit application library. JGit is a Java implementation of git that supports all git functionality necessary for the pull requests feature. However, JGit does not support two git features that can improve performance: shallow clone and sparse checkout. Shallow clones allow us to only clone the minimal amount of git history to enable the feature, and sparse checkouts allow us to only checkout the files necessary to bumping versions. These two features provide large disk-space savings, increased time performance, and reduced network traffic.

If the native git client is installed and available on the system path, then it will be preferred over JGit. If you are having problems with native git, this behavior can be overridden in the config.yml by forcing the usage of java git (JGit).

sourceControl:
  gitImplementation: java

Git Executable

If a native git client is installed but is not on the system path, the fully qualified path to the executable can be specified.

sourceControl:
  gitExecutable: /usr/bin/git

Version Bumping

When the feature checks out your source and attempts to find the component to remediate, it must be able to be found within the manifest. If the component and version cannot be found in the repository source, a pull request cannot be created.

Maven

Any pom.xml that exists in the repository will be searched. Any pom.xml that is in a src/test directory is ignored.

Component defined inside a <dependencies> element with an inline <version> element.

<dependencies>
  <dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.9.9.3</version>
  </dependency>
</dependencies>

Component defined inside a <dependencyManagement> element with an inline <version> element.

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-databind</artifactId>
      <version>2.9.9.3</version>
    </dependency>
  </dependencies>
</dependencyManagement>

Component defined inside either a <dependencies> element, or a <dependencyManagement> element, with the version defined via a variable in a <properties> section.

<properties>                                                                                                                                                                                                 
  <jackson.version>2.9.9.3</jackson.version>
</properties>                                                                                                                                                                                                 

<dependencies>
  <dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>${jackson.version}</version>
  </dependency>
</dependencies>

NPM

NEW IN IQ SERVER RELEASE 84

Nexus IQ for SCM supports updates to your package.json file. Upon policy violations for Javascript components, we will scan your package.json for matching components and create a Pull Request for every match that has a remediation available.

Any package.json that exists in the repository will be examined. Any package.json that contains in its path a directory named test , spec , node_modules , or node will be ignored. These are considered test files or build artifacts and not relevant project manifest files.

Dependencies

Nexus IQ for SCM supports

Components defined inside the "dependencies" section
```
"dependencies": {
  "growl": "=1.9.0"
}
```
Components defined inside the "devDependencies" section
```
"devDependencies": {
  "eslint": "4.11.0"
}
```
Note: Nexus IQ for SCM can only bump dependencies that are part of a scan and that have a policy violation with a remediation. Most often devDependencies are not included as part of a final build.
Components defined inside the "peerDependencies" section
```
"peerDependencies": {
  "react-dom": "~16.0.0"
}
```
Components defined inside the "optionalDependencies" section
```
"optionalDependencies": {
  "fsevents": "^1.2.5"
}
```

Semantic Version Ranges

Nexus IQ for SCM supports the same semantic version range syntax from NodeJS Semver itself.

So for example the following is valid:

^6.1.0 <6.5.2

This will match version 6.1.0 all the way up, but not including, 6.5.2

Version Pinning and Best Practices

It is important to note that Nexus IQ for SCM will always bump to a specific version, therefore pinning it in your manifest. In the above range syntax example, this means that a package.json containing:

"dependencies": {
  "package": "^6.1.0 <6.5.2"
}

Will result in a pull request to replace the contents with:

"dependencies": {
  "package": "6.4.1"
}

Important: Pinning your dependency version is a Sonatype recommended best practice. Only a pinned version allows you to control remediation through security and quality policies. For example, version 1.3.7 might pass policy, but version 1.3.8 might not. A version range of ~1.3.0 does not allow this type of control.

Updating Your Lock Files

Nexus IQ for SCM will not update your lock files (NPM package.lock or Yarn yarn.lock). Lock files are highly dependent on your particular development workflow which likely includes a tailored build command and, of course, the latest security best practices such as authentication and procuring packages from a good repository manager. Why turn Nexus IQ into a Continuous Integration server or build system when you already have one?

In order to include proper lock file changes as part of the pull request, Sonatype recommends leveraging your CI system to perform those changes and run appropriate tests. From a high level this looks like:

Create a CI job to scan for Nexus IQ for SCM created branches.
Run your tailored build command which installs packages and run tests and updates lock files appropriately. For NPM this should include an npm install which will initiate any required updates to the package-lock.json.
Automatically commit those lock file changes and push them to your SCM to be included in the pull request.

Now your lock file changes are guaranteed to be correct , and hopefully don't look like this .

Gradle

NEW IN IQ SERVER RELEASE 103

Nexus IQ for SCM supports updates to build.gradle files. Upon policy evaluations, for Gradle based projects we will scan your build.gradle files for matching components and create a Pull Request for every match that has a remediation available.

All build.gradle files that exist in the repository will be examined, except build.gradle files in src/test directories, which are ignored. These are considered test files or build artifacts and not relevant project manifest files.

Only Groovy based build files are supported.

NEW IN IQ SERVER RELEASE 120

Additionally, if a gradle.properties file is found in the root directory, it will be considered as well.

Dependencies

Nexus IQ for SCM supports:

Component defined directly inside a dependencies block with a literal version.

dependencies {

  // string dependency notation
  implementation "org.apache.struts:struts2-core:2.2.1"

  // map dependency notation
  smokeTest group: 'org.springframework', name: 'spring-api', version: '2.5'
}

Component defined inside a constraints block of a dependencies block with a literal version.

dependencies {

  constraints {
    // string dependency notation
    implementation "org.apache.struts:struts2-core:2.2.1"

    // map dependency notation
    smokeTest group: 'org.springframework', name: 'spring-api', version: '2.5'
  }
}

Component defined inside a dependencies block with a property or variable version. Component versions are also looked up in the gradle.properties file, if one exists.

// Project property
ext {
  slf4jSimpleVersion = '1.7.30'
}

// Groovy variable - dynamic typed
def jerseyVersion = '2.29.1'

// Groovy variable - static typed
String jerseyVersionStr = '2.29.1'

dependencies {

  implementation group: 'org.glassfish.jersey.core', name: 'jersey-core', version: jerseyVersion

  implementation group: 'org.glassfish.jersey.core', name: 'jersey-common', version: jerseyVersionStr

  implementation "org.slf4j:slf4j-simple:$slf4jSimpleVersion"
}

Component defined inside a dependencies block with version constraints.

// project-level inline declaration
ext.slf4jVersion = '1.7.30'

dependencies {

  implementation('org.slf4j:slf4j-api') {
    version {
      strictly '[1.7, 1.8['
      prefer "$slf4jVersion"
    }
  }
}

Gardle version ranges are supported for all the above mentioned cases.

ext {
  asmVersion = '[7.1,['
}

dependencies {

  implementation "org.apache.struts:struts2-core:[2.2.1 , 2.2.10)"

  implementation group: 'org.slf4j', name: 'slf4j-core', version: ']1.20,1.30['

  implementation('org.ow2.asm:asm')

  constraints {
    implementation("org.ow2.asm:asm:$asmVersion") {
      because 'we require a JDK 9 compatible bytecode generator'
    }
  }

  implementation('org.slf4j:slf4j-simple') {
    version {
      require '[1.7, 1.8['
      prefer '1.7.25'
      reject '1.7.20', '1.7.21'
    }
  }
}

Version Pinning and Best Practices

It is important to note that Nexus IQ for SCM will always bump to a specific version, therefore pinning it in your manifest. In the below range syntax example, this means that a build.gradle containing:

dependencies {
  implementation "org.apache.struts:struts2-core:[2.2.1 , 2.2.10)"
}

Will result in a pull request to replace the contents with (assuming that version 2.2.8 passes the policy check):

dependencies {
  implementation "org.apache.struts:struts2-core:2.2.8"
}

Important: Pinning your dependency version is a Sonatype recommended best practice. Only a pinned version allows you to control remediation through security and quality policies. For the above example, version 2.2.8 might pass policy check, but version 2.2.9 might not. A version range does not allow this type of control.

Go

NEW IN IQ SERVER RELEASE 104

Any go.mod file that exists in the repository will be searched for dependencies. Remediation PRs are created for the following scenarios:

Required modules declared in single or block require directives

module example.com/my/thing

go 1.12

require golang.org/x/net v1.2.3
require golang.org/x/net/v2 v2.3.4

require (
  golang.org/x/crypto v1.4.5 // indirect
  golang.org/x/text/v3 v3.6.7
)

Replaced modules declared in single or block replace directives, having a full version specified on the right side of the replace directive

module example.com/my/thing

go 1.12

replace golang.org/x/net v1.2.3 => golang.org/x/net v1.3.0 // a remediation PR may be created

replace ( 
  golang.org/x/crypto => golang.org/x/crypto v1.4.5        // a remediation PR may be created
  golang.org/x/text v1.2.3 => ../x/text                    // no remediation PR will be created
  golang.org/x/rest => ../x/rest                           // no remediation PR will be created
)

Automated Pull Request Daily Activity

NEW IN IQ SERVER RELEASE 97

On the Source Control Configuration screen, we display any recent activity related to automated pull request creation.

Summary information for each attempt to create a pull request will be shown in a table. This information is purged on a daily rotating basis determined by when the server was last started, so at any time a maximum of 24 hours of results is shown. Note that the most common reason a pull request is not created is that the policy violation occurred in a transitive dependency (ie: where that dependency is not explicitly present in any project dependency configuration files). Remediating transitive dependencies is not currently supported.

Automated Pull Requests

Overview

Breaking Changes

Advanced Remediation Strategies

Requirements

Private repositories

Azure DevOps

Bitbucket Cloud

Bitbucket Server

GitHub

GitLab

Nexus IQ Stage

Minimum Git Client Version

Token permissions

Configuration

Repository URL

Default branch

IQ Configuration file (config.yml) Options

Location on disk of cloned repositories

Git Client

Git Executable

Version Bumping

Maven

NPM

Dependencies

Semantic Version Ranges

Version Pinning and Best Practices

Updating Your Lock Files

Gradle

Dependencies

Version Pinning and Best Practices

Go

Automated Pull Request Daily Activity