This article explains how to extract net.Conn from Go’s net/http and go-sql-driver/mysql to directly read and write HTTP and MySQL connections. While rarely needed in typical library usage, this can be useful when you want to rewrite payloads at the transport layer level.

The Japanese version of this article is available here.

net/http

We will cover both Client and Server.

Client

The http.Client struct has a Transport field. This field determines the transport layer used for communication. By default, DefaultTransport is used, which is an instance of the Transport struct.

We use the DialContext (or DialTLSContext) field of this Transport struct to access net.Conn.

DialContext is a function that determines how TCP connections are created. By default, it uses the DialContext method of net.Dialer. DialTLSContext works the same way but targets encrypted connections using TLS.

Create a wrapper function around DialContext (or DialTLSContext) as shown below, and access net.Conn within it. For example, you can store the necessary values in a Context and use them to write to net.Conn.

type valueKeyType int
var valueKey contextKeyType = iota

func main(){
	client := http.Client{
		Transport: wrapTransport(nil)
	}
	// Store an arbitrary value in the Context and attach it to the request
	ctx := context.WithValue(context.Background(), valueKey, "test")
	req, err := http.NewRequestWithContext(ctx, "GET", "http://~", nil)
	if err != nil {
		log.Fatal(err)
	}
	resp, err := client.Do(req)
	...
}

func wrapTransport(base *http.Transport) *http.Transport {
	if base == nil {
		base = http.DefaultTransport.(*http.Transport)
	}

	t.DialContext = wrapDialContext(base.DialContext)
	t.DialTLSContext = wrapDialContext(base.DialTLSContext)
	return t
}

func wrapDialContext(dc func(ctx context.Context, network, addr string) (net.Conn, error)) func(ctx context.Context, network, addr string) (net.Conn, error) {
	if dc == nil {
		return nil
	}
	return func(ctx context.Context, network, addr string) (net.Conn, error) {
		conn, err := dc(ctx, network, addr)
		if err != nil {
			return nil, err
		}

		// Extract the value from the Context
		value := ctx.Value(valueKey)
		// Perform operations on net.Conn
		conn.Write([]byte(value))

		return conn, err
	}
}

Note that you need to be careful about Keep-Alive. HTTP reuses connections as much as possible, sending multiple requests and responses over a single connection. This means the extracted net.Conn may be shared across multiple requests. To prevent this, set DisableKeepAlives to true in the Transport struct.

func wrapTransport(base *http.Transport) *http.Transport {
	...
	t.DisableKeepAlives = true
	...
}

Server

By using the ConnContext field of the http.Server struct to include net.Conn in the Context, you can access net.Conn through the Context when handling requests.

type connKeyType int
var connKey connKeyType = iota

func main() {
	http.HandleFunc("/", handler)

	server := &http.Server{
		Addr: ":8080",
		ConnContext: ConnContext,
	}

	log.Fatal(server.ListenAndServe())
}

func handler(w http.ResponseWriter, r *http.Request) {
	ctx := r.Context()
	conn := ctx.Value(connKey)
	conn.Write([]byte("test"))
	...
}

func ConnContext(ctx context.Context, conn net.Conn) context.Context {
	return context.WithValue(ctx, connKey, conn)
}

Note that how you extract the Context from http.Request in the handler differs depending on the framework (e.g., Echo).

go-sql-driver/mysql

go-sql-driver/mysql is a client for communicating with MySQL servers in Go. It is implemented as a driver for database/sql, which provides a common interface for SQL-related operations.

Similar to net/http.Client, this library also provides a DialContext function, which we can use.

type valueKeyType int
var valueKey contextKeyType = iota

func main() {
	// Use "mytcp" as the network in the Data Source Name when opening the DB (you can also use "tcp")
	mysql.RegisterDialContext("mytcp", DialContext("tcp"))

	db, err := sql.Open("mysql", "user:password@mytcp(localhost:3306)/database")
	if err != [
		log.Fatal(err)
	]
	defer db.Close()

	// Store an arbitrary value in the Context and attach it to the request
	ctx := context.WithValue(context.Background(), valueKey, "test")

	_, err := db.ExecContext(ctx, "INSERT INTO ...")
	...
}

func DialContext(netP string) mysql.DialContextFunc {
	return func(ctx context.Context, addr string) (net.Conn, error) {
		// The default TCP connection creation is hardcoded, so we replicate it here
		// ref: https://github.com/go-sql-driver/mysql/blob/bcc459a906419e2890a50fc2c99ea6dd927a88f2/connector.go#L48-L49
		nd := net.Dialer{}
		conn, err := nd.DialContext(ctx, netP, addr)
		if err != nil {
			return nil, err
		}

		// Extract the value from the Context
		value := ctx.Value(valueKey)
		// Perform operations on net.Conn
		conn.Write([]byte(value))

		return conn, err
	}
}

Similar to Keep-Alive in net/http.Client, MySQL also reuses connections by default. To prevent this, set MaxIdleConns to 0.

func main() {
	...
	defer.db.Close()
	db.SetMaxIdleConns(0)
	...
}

Note that transport layer implementation depends on each driver, so you cannot achieve the same thing with database/sql alone (not limited to MySQL).

Conclusion

This article covered net/http and go-sql-driver/mysql. If other libraries also provide functions like DialContext or ConnContext, you may be able to do the same thing with them. I would like to investigate and write about those if I get the chance.

I needed this for my research, but there were no reference articles available, so I figured it out by reading the library source code. This is admittedly a niche topic, but precisely because of that, I hope it proves useful when someone needs it.

2023-12-02 Update: I gave a talk about this topic at Go Conference mini 2023 Winter IN KYOTO, so please check out the slides as well.

References