Python Application Analysis

Python scanning supports binary packaged archives (.whl/tar.gz) and coordinates in the requirements.txt manifests from the Python Package Index (PyPI). For the best results, we recommend first creating a Python virtual environment and resolving the dependencies using a pip install against the requirements file. This will bring only the dependencies needed by the project into the build while resolving any included dependency ranges in the requirements file.

Starting with Sonatype IQ CLI 2.0, Python Pipfile.lock files are supported, enabling accurate and efficient dependency identification directly from Pipenv projects. This feature eliminates the need for manual workarounds or complex CI configurations when analyzing Python projects with Pipenv. Note that if you are using the Jenkins plugin, you will need to upgrade it to benefit from this functionality, as the changes were made within the scanner.

How to get the best results

To produce the best outcome the following is suggested:

Create a requirements.txt file by using "pip freeze > requirements.txt".
Using requirements.txt, download the binaries by executing "pip download -r requirements.txt -d <output_dir>" or "pip wheel -r requirements.txt -w <output_dir>".
Run a scan on <output_dir>.

Example workflow. A video demonstration is also available here.

cd project_folder
mkdir ~/py-envs
python3 -m venv ~/py-envs/project_folder
source ~/py-envs/project_folder/bin/activate
pip install -r requirements.txt

requirements.txt

Use pip freeze to create the requirements file.
Lifecycle scanners will only use the manifest named requirements.txt while ignoring other variants.
Avoid including variable ranges for dependences as these will be ignored by the Lifecycle scanner.
The requirements.txt must use the == operator and version without wildcards.
Additional flags should be added to requirements.txt files to scope to the target os/arch as found in the environment markers of the Python documentation.

.whl files: may be matched to multiple environmental Python packages. These will show as duplicates in the Lifecycle scan report.

Evaluation: Source code and manifest analysis

The Python coordinate-based matching feature provides the ability to scan and evaluate Python dependencies found in Python manifest files without running the pip install first. Files named requirements.txt (generated using a pip command) and poetry.lock (Poetry lock files) will be analyzed.

Converting from other formats

setup.py

A setup.py file can be used to generate a requirements.txt file by first installing its packages (e.g. via pip install . ), ideally to a virtual environment, and then running:

pip freeze > requirements.txt for Python 2 or
pip3 freeze > requirements.txt for Python 3

What do we parse from the file?

requirements.txt

Requirements using the "==" operator and version without wildcards will be considered. One requirement could be matched to multiple distributions of the same Python package. Using the sys-platform marker makes the dependency more specific. For example:

altgraph==0.10.2

pywin32 ==1.0 ; sys_platform == 'win32'

poetry.lock

Refer to the sample poetry.lock excerpt below:

Dependencies with name and exact version in the [[package]] section are required and evaluated. For example:

Name: six
Version: 1.16.0

Dependencies with extension and qualifier in the [metadata.files] are required and evaluated. For example:

Extension: whl
Qualifier: py2.py3-none-any

Package dependencies with name and exact version in the [package.dependencies] section are evaluated. For example:

Name: colorama
Version: 0.4.4

Sample poetry.lock

[[package]]
name = "six"
version = "1.16.0"
description = "Python 2 and 3 compatibility utilities"
category = "main"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*"

[package.dependencies]
colorama = "0.4.4"

[metadata]
lock-version = "1.1"
python-versions = "^3.8"
content-hash = "7ae52da2736b4294be7a184e040cc78412add14e92b816077ede183f9e1c636c"

[metadata.files]
six = [
    {file = "six-1.16.0-py2.py3-none-any.whl", hash = "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"},
    {file = "six-1.16.0.tar.gz", hash = "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926"},
]
colorama = [
    {file = "colorama-0.4.4-py2.py3-none-any.whl", hash = "sha256:9f47eda37229f68eee03b24b9748937c7dc3868f906e8ba69fbcbdd3bc5dc3e2"},
    {file = "colorama-0.4.4.tar.gz", hash = "sha256:5941b2b48a20143d2267e95b1c2a7603ce057ee39fd88e7329b0c292aa16869b"},
]

Exclude devDependencies

IQ Server automatically excludes scanning devDependencies for projects using poetry versions < 1.5.1

Due to removal of category in the format of poetry.lock file from version 1.5.1 onwards, older versions of IQ Server will not exclude the devDependencies while scanning projects using the new poetry versions.

IQ Server will automatically exclude devDependencies for poetry versions 1.5.1 or higher, if pyproject.toml exists and is discoverable.

Pipfile.lock

From IQ CLI 2.0 onwards, Pipfile.lock files from Pipenv projects are also analyzed. This ensures direct and accurate identification of dependencies from Pipenv, eliminating the need for manual workarounds or CI configurations for Python projects.
If you use the Jenkins plugin, note that you must upgrade the plugin version to take advantage of the new scanner capabilities introduced in IQ CLI 2.0.
You can find an example of a Pipfile.lock in the PyPA docs. Note that we only parse the dependencies under the default key, as those are the ones intended for production.

Steps to analyze using the Sonatype IQ CLI

Create requirements

Run pip freeze

pip freeze > requirements.txt

The requirements.txt encoding is UTF-8. Special note for Microsoft Windows users, the cmd.exe encoding may need to be changed to UTF-8. Please refer to Microsoft documentation on how to do this.

Example file content

altgraph==0.10.2

backports-abc==0.5

backports.ssl-match-hostname==3.5.0.1

bdist-mpkg==0.5.0

certifi==2018.1.18

chardet==3.0.4

click==6.7

confire==0.2.0

Django==1.6

django-countries==3.3

django-make-app==0.1.3

docopt==0.6.2

enum34==1.1.6

Add environment markers (optional)

Adding environment markers can simplify the results by filtering out components that are not relevant to your deployment platform. Only the sys_platform environment marker is supported at the moment.

Add the environment marker next to the component(s) in the requirements.txt.

e.g.

Django==1.6; sys_platform == 'win32'

Run a scan

Invoke a Sonatype IQ CLI scan of the directory containing requirements.txt. Instructions on how to do this can be found here Sonatype IQ CLI

Note

If your project uses Pipenv, you can instead rely on a Pipfile.lock being present. Starting with IQ CLI 2.0, the scanner automatically detects and analyzes dependencies from Pipfile.lock.