containerd's gogo/protobuf migration

containerd has migrated from github.com/gogo/protobuf to google.golang.org/protobuf in April 2022. This document describes the effort.

Background

containerd has been using gogo/protobuf since the very beginning (TODO: when?). While I wasn't involving the original decision, using gogo/protobuf seemed popular at that time. For example, the following project are/were using gogo/protobuf;

• Vitess (migrating off in 2021)
• Istio (migrating off in 2022)
• etcd
• CockroachDB

Compared to Google's original Go Protocol Buffers package, gogo/protobuf was much performant and allowed more customization regarding code generation.

However, the maintainers of gogo/protobuf decided to step down and the project is looking for new maintainers since May 2020.

Unfortunately, the personal circumstances of the maintainers have changed and we are no longer able to keep up with the issues and feature requests that naturally crop up for a popular open source project.

In particular, the recent golang/protobuf release 1.4.x (AKA APIv2: https://blog.golang.org/protobuf-apiv2), has created a big chunk of work required to be compatible with the new world of Go protobufs.

While taking the ownership was technically possible, maintaining the RPC package was not really containerd's focus. So I started the migration effort in 2021 (TODO: links).

containerd/protobuild

containerd is using Protobuild that wraps protoc-gen-go and/or other code generators that take .proto files. Previously protoc-gen-go was supporting gRPC through its "plugin" mechanism, but the design has been changed since (TODO: version). Now protoc-gen-go is only responsible for Protocol Buffers, and gRPC needs protoc-gen-go-grpc.

Supporting multiple generators was lacking in Protobuild. I extended Protobuild to support protoc-gen-go and protoc-gen-go-grpc.

• Support multiple generators
• Remove plugins and import_path

In addition to that, protoc-gen-go doesn't allow much customization compared to gogo/protobuf. I had to write a small utility program that rewrites Go's AST.

• Add go-fix-acronym

containerd/ttrpc

ttrpc is containerd's own RPC protocol, which is basically "gRPC without HTTP".

I have removed gogo/protobuf from ttrpc, and written a new code generator that could be used with protoc-gen-go.

• Use google.golang.org/protobuf instead of github.com/gogo/protobuf
• Add protoc-gen-go-ttrpc

containerd/containerd

containerd has been using a lot of gogo/protobuf extensions. I have removed all of them in the following PRs. Luckily containerd was converting these auto-generated structs to internal domain models right after getting the messages. The blast radius of the changes was much smaller than I initially expected.

• Remove gogoproto.customtype
• Remove enumvalue_customname, goproto_enum_prefix and enum_customname
• Remove gogoproto.stdtime
• Remove all gogoproto extensions
• Consolidate gogo/protobuf dependencies under our own protobuf package

Because of the PRs above, the size of the final migration PR was manageable (TODO: size).

• Migrate off from github.com/gogo/protobuf

containerd/imgcrypt

containerd is depending on containerd/imgcrypt and imgcrypt is depending on containerd. In order to workaround the cyclic dependencies, I had to use Go's reflect package.

• Use reflect to support diff.ApplyConfig with/without gogo's types.Any

Retrospective

Went Well

• Submitted small incremental PRs instead of having a big one.

Needs Improvements

• It wasn't really estimated and I haven't discussed much about the work inside Amazon.
• Possibly due to that, I wasn't asking much help except for code reviews, inside/outside Amazon.