Skip to main content

Repository Import

Only available in Sonatype Nexus RepositoryTM Pro. Interested in a free trial? Start here.

The Repository - Import external files task takes content from a directory on the server and imports it into a repository in Nexus Repository. This allows you to bring external content into a repository from any source.

Planning and Requirements

Before using this task, review the following considerations:

  • Store content in a location available to any node in highly available (HA) deployments

    Any available node is used to run the steps of the import task so the import content needs to be accessible to all nodes in the cluster. This is commonly the same store location for the cluster.

  • Migrated Content

    Import is limited to artifacts and will not include any server configurations or metadata such as tags. Metadata and checksums are generated automatically on import and do not maintain exported attributes. The Blob created and Blob updated attributes are set to the time generated during import.

    Import does not include hidden files and directories (e.g., those with names that start with a period).

  • Space Requirements

    Copying all components will double the assets and space on disk; hard linking does not do this but will take additional space to store all of the metadata.

  • Location Requirements

    The source directory must be writable by the user running Nexus Repository 3. The import history is stored in the source directory to enable incremental imports.

  • Timing

    This task can take some time to run, and multiple imports are not done in parallel. Only one import/export task will run at a time. Each task needs to be completed before the next one will run. Other import tasks are queued to run after the initial one finishes.

  • Frequency

    You may configure the task to run manually or on a schedule depending on your needs.

  • Performance

    We do not recommend that you import directly to object storage such as S3 as doing so can result in poor performance.

  • Deployment Policy Impacts for Hosted Repositories

    If your hosted repository's deployment policy is set to Disable Redeploy, this will prevent overwriting and updating existing assets during import.

  • Hard Link Requirements

    • To use hard links, the target repository blob store must be on the same file system as the source.

    • Hard linking is not supported for object storage blob stores such as S3.

Configuration

Use the following steps to create a new import task.

  1. From the Settings panel, navigate to Tasks and select the Create task button

  2. Select the task, Repository - Import external files from the task list

  3. Fill out the import configuration from the options below:

  • Task name

    A unique name to reference the task.

  • Notification email

    The email address where notifications on the status of the import task are sent.

  • Send notification on

    A notification may be sent only if the task fails or on either a success or failure.

  • Target repository

    The repository into which to import content. This list will only be populated with Hosted repositories of the supported formats.

  • Source directory

    The directory from which to import content. The files must be in the directory structure that supports the desired format.

  • Batch Size

    Memory optimization feature for troubleshooting performance issues. We recommend leaving this at its default value of 1000. The minimum value is 1.

  • Enable Hard Links

    For some formats, enabling this option will create hard links for the blobs in the source directory to be used in the target directory. A physical copy of the blobs will not be created in the target directory. This can save time if the source directory will remain accessible by the current instance.

  • Task frequency

    Configure the job to run automatically at a specific time or manually as a run-once task. Often import jobs are manually run. Scheduled jobs will only import files when the artifacts are newly added or missed from a previous import.

Ensure you have read the hard link requirements under Planning and Requirements.

nx3-tasks-import-repository.png

Running the Task

Only one import/export task can run at a time. If you attempt to run more than one at a time, the others will be queued to run after the initial one completes.

When the task runs, it generates a .nexus/nxrm-import-export-task/ directory inside of the source directory. This .nexus/nxrm-import-export-task/ directory contains metadata that Nexus Repository uses for future imports so that it will not import content multiple times if the task is run more than once. This will allow you to keep placing content into this directory while only importing newer content if you run the task again.

As Nexus Repository processes the import, it will log every single file it imports to the task log; a new task log is created each time a task runs. If an error occurs with any particular file, Nexus Repository will simply log that fact, and then continue importing the rest of the content.

Resetting Data (.dat) Files

If you need to reset the import data files, run the following command:

rm -r .nexus/nxrm-import-export-task

Deleting Import Task Source Cache Files Inside Nexus Repository 2 Repository Storage Directory

Should you need to reset an import, you will need to delete the cache.

The following steps provide a workaround for deleting .dat files inside a Nexus Repository 2 storage directory where the .dat files are stored in a .nexus directory instead of .nexus/nxrm-import-export-task/:

  1. Locate the source .nexus directory of the import

    1. In Nexus Repository 2, the default location would be sonatype-work/nexus/storage/<repoid>/.nexus

  2. Use a command like the following to find and delete all the import cache files:

find sonatype-work/nexus/storage/releases/.nexus -type f -name 'imported-files*.dat' -delete

Supported Formats

You can import content into repositories of any format listed below. Some formats require that content be structured in a certain manner:

Format

Since

Hard Link

Directory Layout

Raw

3.23.0

3.31.0

No specific layout is required, content will simply be imported exactly as laid out in the source directory

Maven

3.23.0

3.31.0

Files need to be laid out in the usual maven GAV structure, /(groupId)/(artifactId)/(version)/(artifactId-version.extension)

For example:

{rootdir}/com/mycompany/awesome-artifact/1.0/awesome-artifact-1.0.jar

maven-metadata files at each level (G, A, or V) will also be imported.

npm

3.25.0

3.34.0

No specific layout is required, will simply import all tgz packages contained in the source directory

NuGet

3.25.0

3.35.0

No specific layout is required, will simply import all Nuget packages contained in the source directory

Docker

3.26.0

3.35.0

V1

  • Image metadata must be stored in the file v1/images/{layerId}/json

  • Image content must be stored in the file v1/images/{layerId}/layer

  • Tags must be stored in the file v1/repositories/{name}/tags/{tag}

  • Image index must be stored in the file v1/repositories/{name}/images/{image}

V2

  • Blobs must be stored in path /v2/-/blobs/{digest}

  • Manifests must be stored in file /v2/{name}/manifests/{tag}

Examples:

(Note the &#58 in the following path, this is simply an ASCII encoded colon ':' as Windows file systems do not support this character in the filename)

{rootdir}/v2/-/blobs/sha256&#5803f4658f8b782e12230c1783426bd3bacce651ce582a4ffb6fbbfa2079428ecb

{rootdir}/v2/helloworld/manifests/latest

{rootdir}/v2/helloworld/manifests/1.0

PyPI

3.26.0

3.35.0

No specific layout is required, will simply import all archives contained in the source directory (along with any .asc signature files alongside)

RubyGems

3.26.0

3.35.0

No specific layout is required, will simply import all gem files contained in the source directory

Yum

3.26.0

3.35.0

No specific layout is required, will simply import all of the rpm (or drpm) files, as well as any comps.xml files in the source directory