Go Repositories

Introduction

The Go programming language (golang) introduced Go modules starting in Go version 1.11. This feature is intended to make dependency version information explicit and easier to manage.

Sonatype Sonatype Nexus Repository supports Go modules dependency resolution using proxy and group repositories. Go packages can be cached and served from Sonatype Nexus Repository using a single endpoint, without builds incurring repeated downloads of packages from the public internet.

Shifting from the “Monorepo”

The Go programming language uses centralized monolithic git repositories to source Golang code as a means to simplify dependency management. This was done using a version control system where the Go command line would perform a full clone of all required dependencies. The Go community realised that this has a number of negative impacts for any typical CI process as cloning repositories can be a time consuming process and was a barrier for developers working on isolated sub-project that did not require every dependency from the repository.

No standards for dependency management.
Without a native package manager, Go developers relayed on tools like dep.
Only vendoring and GOPATH were available to bring in outside dependencies.

Fetching sources from any number of version control systems such as Github meant that centralized proxies such as Sonatype Nexus Repository could not be used. To solve this problem, Go introduced the $GOPROXY environment variable.

About Go Modules

Go modules is the standard package management system for Golang. It offers a common toolset to build proxy servers to versioned Go dependencies, where you’re no longer dependent on a monolithic platform. This provides for more stable packages in your CI environment.

go.mod

Versions 1.11 and later introduced features to access Go modules: the Module Download API.This allows you to send GET requests from the client with the command go get.In return, the argument passes packages downstream to a custom URL in Sonatype Nexus Repository.Go modules allows you to maintain multiple versions of related Go packages. Code is shared between Go projects similar to the way formats like maven, pypi, and npm.

The go.mod file is defined by a hierarchy of Go source files. This should be added to can be found in the project’s root directory. This file is characterized as such:

generated after initialization (go mod init [module])
specifies builds with all dependencies and exact versions
imports and tags groups of Go packages versioned together as one

In a shift from the constraints of the GOPATH model, older techniques like vendoring are now almost entirely redundant. It has the ability to store all the requirements needed to ensure availability. Providing a go.mod file with the correct list of requirements is the recommended way to achieve reproducible builds in a scenario where our modules are being used.

Consider this example go.mod:

module github.com/any-go-repo/come-together

require (
    some-repo.com/example/paul v0.0.0-xxx.xxx
    some-repo.com/example/ringo v1.5.4
    some-repo.com/example/john v0.0.0-xxx-xxx
    some-repo.com/example/george v1.2.0
    some-repo.com/example/billy v1.2.0
)

When you run go get, the command downloads new dependencies to the module. In the example, the go.mod you can see the namespaces of all newly retrieved content pulled downstream and made available to your team (some-repo.com).

Another file, go.sum, is maintained along with go.mod. go.sum keeps a record of secret hashes with specific project versions. You and your team must ensure both files are committed together when maintaining versions. If not, any conflicts between the contents of go.mod and go.sum could lead to errors.

GOPROXY

Go modules introduced an environment variable $GOPROXY. When set, all modules are fetched using a Download API using the base url as the proxying endpoint. Sonatype Nexus Repository requires the setting of the $GOPROXY environment variable pointing to a Go repository.

export GOPROXY=http://localhost:8081/repository/go-all

After the export, the variable with the repository URL string is displayed in thr .profile (or .bash_profile, .bashrc). Any requested Go projects are cached in Sonatype Nexus Repository. When a Go command determines that it needs a module, it looks to the endpoint provided in the path from the remote location and downloads them to the repository.

sumdb

Use the method detailed below to allow Sonatype Nexus Repository to proxy the Go module ecosystem checksum database so that Go clients behind firewalls can verify Go module checksums by only accessing Sonatype Nexus Repository directly.

In Sonatype Nexus Repository, create a proxy Go repository pointing tohttps://proxy.golang.org (identified as golang-proxy below).
In Sonatype Nexus Repository, create a proxy raw repository pointing to https://sum.golang.org (identified as golang-sum-proxy below).
Uncheck the Strict Content Type Validation option; if you don't turn off content validation, Sonatype Nexus Repository will return '404 Not Found' errors for sum file requests.
On the client workstation, set the following environment variables:

go env -w GOPROXY=https://nexus.example.com/repository/golang-proxy
go env -w 'GOSUMDB=sum.golang.org https://nexus.example.com/repository/golang-sum-proxy'

SeeGo's documentation about sumdbas an additional reference.

Configuring a Go Project in Sonatype Nexus Repository

You can serve and cache modules remotely from resources such as GitHub. Currently, hosted repositories aren’t available. However, you can address two use cases:

Configure the repository manager to cache projects
Create a simple project and download dependencies to the project

Use Case 1: Proxy Configuration

Start up your local instance of Sonatype Nexus Repository and work through the steps below. In this procedure (and in Use Case 2) you’ll build and run your projects in a global temporary directory (/tmp).

In Sonatype Nexus Repository, click Create repository to start anew then Save.
- Name: go-demo-proxy
- Remote storage: https://proxy.golang.org/
- Blob store, e.g. default
Copy the URL from the new repository: http://localhost:8081/repository/go-demo-proxy/

Paste the URL to $GOPROXY with the new endpoint:

export GOPROXY=http://localhost:8081/repository/go-demo-proxy

In the terminal, clone a project from GitHub in a directory nested in /tmp:

mkdir -p /tmp/go-demo
cd /tmp/go-demo
git clone https://github.com/gobuffalo/buffalo
ls # to list the contents

Go to the directory: cd buffalo.
Build packages from the Git repository: go build.
Browse the repository and locate the following assets for each component proxied visible in tree view:
```
.zip
.mod
.info
```

In your terminal you can compare the module structure. For example, check out thiscloud.google.comproject root. Take a look at this view with the tree command and compare with the Browse UI in the repository manager:

$ cd cloud.google.com/
$ tree
.
└── go
    └── @v
        ├── list
        ├── list.lock
        ├── v0.26.0.info
        ├── v0.26.0.mod
        ├── v0.31.0.info
        ├── v0.31.0.mod
        ├── v0.36.0.info
        └── v0.36.0.mod

Use Case 2: Create a main.go and Add Dependencies

In this scenario, we’re going to create a new project in a new repository and add dependencies. We’ll call this one another-go. Now, the first step is nearly identical, except you’ll initialize this new project with its own repository ID separate from the previous use case. This also means you need to update the $GOPROXY variable so you can browse the project’s dependencies in the new repository.

Create a new directory in /tmp:
mkdir -p go-demo2
In Sonatype Nexus Repository, click Create repository to configure a new directory, then Save.
- Name: another-go
- Remote storage: https://proxy.golang.org/
- Blob store, e.g. default
Copy the URL from your new repository:
http://localhost:8081/repository/another-go/
Update $GOPROXY with the new endpoint:
GOPROXY=http://localhost:8081/repository/another-go/

Alternatively, you can open up your .bash_profile (or .bashrc) and add the URL as the $GOPROXY. If you go this route save your updates with source ~/.bash_profile (or source ~/.bashrc)

Now that your caching proxy is configured to receive modules, create a sample module. In this example, let’s use github.com/foo/emoji-message as the module path. You’re going to create a go.mod project then iterate with a new dependency. Additionally, you’ll create a main.go, your first executable program.

Go to your terminal and follow the steps below.

Create a temporary “workspace” directory:
mkdir -p /tmp/go-again
Change your directory to the target project location:
cd /tmp/go-again
Run go mod init [module path] in your terminal to start the project
- go mod init github.com/foo/emoji-message
  It returns this message:
  go: creating new go.mod: module github.com/foo/emoji-message.
Enter cat go.mod to view the file. It shows a minimal record of the module name and version.
module github.com/foo/emoji-messagego 1.12
Create a main.go:
touch main.go
We’re going to import an emoji project from Github (github.com/kyokomi/emoji) and add a function.
go get github.com/kyokomi/emoji

Paste the sample code in the main.go file (below).

package main 
import ( 
    "fmt"
    "github.com/kyokomi/emoji" 
) 
func main() { 
    fmt.Println("Greetings Emoji!")
    emoji.Println(":speech_balloon: May I take your order?") 
    burgerMessage := emoji.Sprint("I'd like a :hamburger: and :fries:!!")
        fmt.Println(burgerMessage)
}

Use the go command to run the app: go run main.go. This produces the output from the code. Be sure to check out another-go in the Browse UI to see the module and dependencies in tree view.

View the new requirement documented in the go.mod file:

$ cat go.mod
module github.com/foo/emoji-message
go 1.12
require github.com/kyokomi/emoji v2.1.0+incompatible

Locate the go.sum from the project root: cat go.sum. You’ll see the nested versions of the project, each with an associated cryptographic hash.

Warning

At the time of implementation of the Go repository in Sonatype Nexus Repository, the Go client does not have a means of authentication. This means Sonatype Nexus Repository instances with anonymous disabled will not be able to reach the Go endpoints, rendering proxy useless. See details on Anonymous Access.

Grouping Go Repositories

A repository group is the recommended way to expose all your Go repositories. A repository group allows you to expose the aggregated content of multiple Go repositories with one URL; it is this url that you should set the $GOPROXY environment variable url to.

To create a Go group repository, create a new repository using the recipe g o (group) as documented in Repository Management.

Minimal configuration steps are:

Define Name
Select Blob store for Storage
Add Go repositories to the Members list in the desired order

Go Configuration Examples

If you need to access dependencies from authenticated sources, you'll need to run Athens internally; this locally hosted server is able to be configured with authentication credentials to access private repositories. You may still decide to proxy an external server to avoid unnecessary duplication of caching and therefore storage space.

The following diagram depicts an example set up: