Adding middleware to Go HTTP client requests

The Go standard library has fantastic support for making HTTP requests. However, sometimes, a request needs to be modified or an action needs to be taken upon response (eg. logging a response).

In many cases, adding these logs to the every request would involve a lot of duplicate code. In other cases accessing the client might not be possible because of restricted access in a different package or third party library. Thus, I introduce RoundTrippers.

A RoundTripper can be added to an http client and executes a http transaction allowing the request to be inspected and modified and the response to be inspected and modified.

A RoundTripper can be set on a http client Transport. The interface looks like:

type RoundTripper interface {
	RoundTrip(*http.Request) (*http.Response, error)
}

and a RoundTripper is set on a client like:

client := &http.Client{
	Transport: myRoundTripper,
}

The http.Client will resolve to the DefaultTransport if the client transport is nil. It's fairly easy to chain RoundTrippers as middleware, the DefaultTransport round tripper should be considered as a base since it comes with handy defaults:

var DefaultTransport RoundTripper = &Transport{
	Proxy: ProxyFromEnvironment,
	DialContext: defaultTransportDialContext(&net.Dialer{
		Timeout:   30 * time.Second,
		KeepAlive: 30 * time.Second,
	}),
	ForceAttemptHTTP2:     true,
	MaxIdleConns:          100,
	IdleConnTimeout:       90 * time.Second,
	TLSHandshakeTimeout:   10 * time.Second,
	ExpectContinueTimeout: 1 * time.Second,
}

Now that the basics have been covered we can use a single chaining example which I originally saw by Philipp from GoWebExamples.com applied to web server middleware. It's a simple little pattern that's easy to understand.

// internalRoundTripper is a holder function to make the process of
// creating middleware a bit easier without requiring the consumer to
// implement the RoundTripper interface.
type internalRoundTripper func(*http.Request) (*http.Response, error)

func (rt internalRoundTripper) RoundTrip(req *http.Request)
                                            (*http.Response, error) {
	return rt(req)
}

// Middleware is our middleware creation functionality.
type Middleware func(http.RoundTripper) http.RoundTripper

// Chain is a handy function to wrap a base RoundTripper (optional)
// with the middlewares.
func Chain(rt http.RoundTripper, middlewares ...Middleware)
                                                    http.RoundTripper {
	if rt == nil {
		rt = http.DefaultTransport
	}

	for _, m := range middlewares {
		rt = m(rt)
	}

	return rt
}

From here writing custom middlewares for your needs is straightforward. Two examples below:

• CustomTimer executes a function after the RoundTrip has finished and shows the time since. This demonstrates how operates can occur before the request is made, in this case we are capturing the start time and then executing code after the request has been made using a defer.
• DumpResponse prints the response body to stdout. Similarly to CustomTimer, a defer is used to run the dump after the request has been made. The difference here is named response are used to access the response data.

// CustomTimer takes a writer and will output a request duration.
func CustomTimer(w io.Writer) Middleware {
	return func(rt http.RoundTripper) http.RoundTripper {
		return internalRoundTripper(func(req *http.Request)
                                            (*http.Response, error) {
			startTime := time.Now()
			defer func() {
				fmt.Fprintf(w, ">>> request duration: %s",
                    time.Since(startTime))
			}()

			return rt.RoundTrip(req)
		})
	}
}

// DumpResponse uses dumps the response body to console.
func DumpResponse(includeBody bool) Middleware {
	return func(rt http.RoundTripper) http.RoundTripper {
		return internalRoundTripper(func(req *http.Request)
                                    (resp *http.Response, err error) {
			defer func() {
				if err == nil {
					o, err := httputil.DumpResponse(resp, includeBody)
					if err != nil {
						panic(err)
					}

					fmt.Println(string(o))
				}
			}()

			return rt.RoundTrip(req)
		})
	}
}

Once you've written middleware that fits your needs, adding them to your HTTP client is as easy as creating an http.Client and setting the Transport.

client := http.Client{
    // Using the Chain function, we can add middlewares to our base
    // RoundTripper. These middlewares will run on every http request. 
    Transport: Chain(nil, CustomTimer(os.Stdout), DumpResponse(false)),
}

// Start making requests
_, err := client.Get("http://jonfriesen.ca")
if err != nil {
    panic(err)
}

Another great use for RoundTripper middleware is within third party clients. For example, the GitHub Go library does not have native support for adding headers to requests it makes. It's common for third party clients to support setting the http.Client, for example:

// AddHeader adds a header to the request.
func AddHeader(key, value string) Middleware {
	return func(rt http.RoundTripper) http.RoundTripper {
		return internalRoundTripper(func(req *http.Request)
                                            (*http.Response, error) {
			header := req.Header
			if header == nil {
				header = make(http.Header)
			}

			header.Set(key, value)

			return rt.RoundTrip(req)
		})
	}
}

func main() {
    userClient := github.NewClient(&http.Client{
        Transport: Chain(nil, AddHeader("key", "value")),
    })
}