Go Repositories

Introduction

The Go programming language (golang) introduced Go modules starting in Go version 1.11. This feature is intended to make dependency version information explicit and easier to manage.

Nexus Repository supports Go modules dependency resolution using proxy and group repositories. Go packages can be cached and served from Nexus Repository using a single endpoint, without builds incurring repeated downloads of packages from the public internet.

The Go repository format is disabled by default in HA-C environments.

Shifting from the “Monorepo”

The Go programming language used a centralized monolithic git repositories to source Golang code as a means to simplify dependency management.  This was done using a version control system where the Go command line would perform a full clone of all required dependencies. The Go community realised that this has a number of negative impacts for any typical CI process as cloning repositories can be a time consuming process and was a barrier for developers working on isolated sub-project that did not require every dependency from the repository. 

  • No standards for dependency management.
  • Without a native package manager, Go developers relayed on tools like dep.
  • Only vendoring and GOPATH were available to bring in outside dependencies.

Fetching sources from any number of version control systems such as Github meant that centralized proxies such as Nexus Repository could not be used. To solve this problem, Go introduced the $GOPROXY environment variable.

About Go Modules

Go modules is the standard package management system for Golang. It offers a common toolset to build proxy servers to versioned Go dependencies, where you’re no longer dependent on a monolithic platform. This providies for more stable packages in your CI environment.

go.mod

Versions 1.11 and later introduced features to access Go modules: the Module Download API. This allows you to send GET requests from the client with the command go get. In return, the argument passes packages downstream to a custom URL in Nexus Repository. Go modules allows you to maintain multiple versions of related Go packages. Code is shared between Go projects similar to the way formats like maven, pypi, and npm.

The go.mod file is defined by a hierarchy of Go source files. This should be added to can be found in the project’s root directory. This file is characterized as such:

  • generated after initialization (go mod init [module])
  • specifies builds with all dependencies and exact versions
  • imports and tags groups of Go packages versioned together as one

In a shift from the constraints of the GOPATH model, older techniques like vendoring are now almost entirely redundant. It has the ability to store all the requirements needed to ensure availability. Providing a go.mod file with the correct list of requirements is the recommended way to achieve reproducible builds in a scenario where our modules are being used.

Consider this example go.mod:

module github.com/any-go-repo/come-together

require (
    some-repo.com/example/paul v0.0.0-xxx.xxx
    some-repo.com/example/ringo v1.5.4
    some-repo.com/example/john v0.0.0-xxx-xxx
    some-repo.com/example/george v1.2.0
    some-repo.com/example/billy v1.2.0
)

When you run go get, the command downloads new dependencies to the module. In the example, the go.mod you can see the namespaces of all newly retrieved content pulled downstream and made available to your team (some-repo.com).

Another file, go.sum, is maintained along with go.mod. go.sum keeps a record of secret hashes with specific project versions. You and your team must ensure both files are committed together when maintaining versions. If not, any conflicts between the contents of go.mod and go.sum could lead to errors.

GOPROXY

Go modules introduced an environment variable $GOPROXY.  When set, all modules are fetched using a Download API using the base url as the proxying endpoint. Nexus Repository requires the setting of the $GOPROXY environment variable pointing to a Go repository.

export GOPROXY=http://localhost:8081/repository/go-all


After the export, the variable with the repository URL string is displayed in thr .profile (or .bash_profile, .bashrc). Any requested Go projects are cached in Nexus Repository. When a Go command determines that it needs a module, it looks to the endpoint provided in the path from the remote location and downloads them to the repository.

Configuring a Go Project in Nexus Repository

In versions Nexus Repository 3.17 and higher you can serve and cache modules, remotely, from resources such as GitHub. Currently hosted repositories aren’t available. However you can address two use cases:

  • Configure the repository manager to cache projects
  • Create a simple project and download dependencies to the project

Use Case 1: Proxy Configuration

Start up your local instance of Nexus Repository and work through the steps below. In this procedure (and in Use Case 2) you’ll build and run your projects in a global temporary directory (/tmp).

  1. In Nexus Repository, click Create repository to start anew then Save.
  2. Copy the URL from the new repository: http://localhost:8081/repository/go-demo-proxy/
  3. Paste the URL to $GOPROXY with the new endpoint:

    export GOPROXY=http://localhost:8081/repository/go-demo-proxy
  4. In the terminal, clone a project from GitHub in a directory nested in /tmp:

    mkdir -p /tmp/go-demo
    cd /tmp/go-demo
    git clone https://github.com/gobuffalo/buffalo
    ls # to list the contents
  5. Go to the directory: cd buffalo.
  6. Build packages from the Git repository: go build.
  7. Browse the repository and locate the following assets for each component proxied visible in tree view:

    .zip
    .mod
    .info

In your terminal you can compare the module structure. For example, check out this cloud.google.com project root. Take a look at this view with the tree command and compare with the Browse UI in the repository manager:

$ cd cloud.google.com/
$ tree
.
└── go
    └── @v
        ├── list
        ├── list.lock
        ├── v0.26.0.info
        ├── v0.26.0.mod
        ├── v0.31.0.info
        ├── v0.31.0.mod
        ├── v0.36.0.info
        └── v0.36.0.mod

Use Case 2: Create a main.go and Add Dependencies

In this scenario, we’re going to create a new project in a new repository and add dependencies. We’ll call this one another-go. Now, the first step is nearly identical, except you’ll initialize this new project with its own repository ID separate from the previous use case. This also means you need to update the $GOPROXY variable so you can browse the project’s dependencies in the new repository.

  1. Create a new directory in /tmp:
    mkdir -p go-demo2
  2. In Nexus Repository, click Create repository to configure a new directory, then Save.
  3. Copy the URL from your new repository:
    http://localhost:8081/repository/another-go/
  4. Update $GOPROXY with the new endpoint:
    GOPROXY=http://localhost:8081/repository/another-go/

Alternatively, you can open up your .bash_profile (or .bashrc) and add the URL as the $GOPROXY. If you go this route save your updates with source ~/.bash_profile (or source ~/.bashrc)

Now that your caching proxy is configured to receive modules, create a sample module. In this example, let’s use github.com/foo/emoji-message as the module path. You’re going to create a go.mod project then iterate with a new dependency. Additionally, you’ll create a main.go, your first executable program.

Go to your terminal and follow the steps below.

  1. Create a temporary “workspace” directory:
    mkdir -p /tmp/go-again
  2. Change your directory to the target project location:
    cd /tmp/go-again
  3. Run go mod init [module path] in your terminal to start the project
    1. go mod init github.com/foo/emoji-message
      It returns this message:
      go: creating new go.mod: module github.com/foo/emoji-message.
  4. Enter cat go.mod to view the file. It shows a minimal record of the module name and version.
    module github.com/foo/emoji-message go 1.12
  5. Create a main.go:
    touch main.go
  6. We’re going to import an emoji project from Github (github.com/kyokomi/emoji) and add a function.
    go get github.com/kyokomi/emoji
  7. Paste the sample code in the go.mod file (below).

    package main 
    import ( 
        "fmt"
        "github.com/kyokomi/emoji" 
    ) 
    func main() { 
        fmt.Println("Greetings Emoji!")
        emoji.Println(":speech_balloon: May I take your order?") 
        burgerMessage := emoji.Sprint("I'd like a :hamburger: and :fries:!!")
            fmt.Println(burgerMessage)
    }
  8. Use the go command to run the app: go run main.go. This produces the output from the code. Be sure to check out another-go in the Browse UI to see the module and dependencies in tree view.
  9. View the new requirement documented in the go.mod file:

    $ cat go.mod
    module github.com/foo/emoji-message
    go 1.12
    require github.com/kyokomi/emoji v2.1.0+incompatible
  10. Locate the go.sum from the project root: cat go.sum. You’ll see the nested versions of the project, each with an associated cryptographic hash.

Non-Anonymous

At the time of implementation of the Go repository in Nexus Repository, the Go client does not have a means of authentication. This means Nexus Repository instances with anonymous disabled will not be able to reach the Go endpoints, rendering proxy useless. See here for details on anonymous security.

Grouping Go Repositories

A repository group is the recommended way to expose all your Go repositories. A repository group allows you to expose the aggregated content of multiple Go repositories with one URL; it is this url that you should set the $GOPROXY environment variable url to.

To create a Go group repository, create a new repository using the recipe go (group) as documented in Repository Management.

Minimal configuration steps are:

  • Define Name

  • Select Blob store for Storage

  • Add Go repositories to the Members list in the desired order

Go configuration examples

If you need to access dependencies from authenticated sources, you'll need to run Athens internally; this locally hosted server is able to be configured with authentication credentials to access private repositories. You may still decide to proxy an external server to avoid unnecessary duplication of caching and therefore storage space.

The following diagram depicts an example set up: