Skip to main content

Protocol Buffers in Go: Serialization, gRPC, and Performance

·7 mins
Table of Contents
Protocol Buffers give you binary serialization that is roughly 40% smaller and 6x faster than JSON. That trade-off is worth it for internal microservice traffic. It is not worth it for a public REST API that humans need to read with curl.

Protocol Buffers (protobuf) are Google’s binary serialization format. The pitch is simple: define your data schema once in a .proto file, generate typed code in any language, and get compact wire format for free. The catch is tooling overhead and the loss of human readability. This post walks through a complete Go setup: correct marshaling API, a gRPC service, and a benchmark comparing protobuf to JSON.

Setup
#

Install the three tools you need. The first is protoc, the protobuf compiler. The easiest path on macOS is Homebrew. On Linux, download from the GitHub releases page.

install protoc (macOS)
brew install protobuf

Then install the two Go code generators using go install, not the deprecated go get -u form:

install Go plugins
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Make sure $(go env GOPATH)/bin is on your PATH, otherwise protoc will not find the plugins at generation time.

Define a .proto schema
#

Create a file called user.proto. The option go_package directive is required by the modern protobuf Go plugin.

user.proto
syntax = "proto3";

package user;

option go_package = "github.com/yourname/yourapp/gen/userpb";

message Address {
  string street = 1;
  string city   = 2;
  string country = 3;
}

message User {
  string   id      = 1;
  string   name    = 2;
  string   email   = 3;
  int32    age     = 4;
  Address  address = 5;
}

message GetUserRequest {
  string id = 1;
}

message ListUsersResponse {
  repeated User users = 1;
}

message CreateUserRequest {
  User user = 1;
}

service UserService {
  rpc GetUser(GetUserRequest)       returns (User);
  rpc ListUsers(google.protobuf.Empty) returns (ListUsersResponse);
  rpc CreateUser(CreateUserRequest) returns (User);
}

Generate the Go code. The --go_out flag produces the message types; --go-grpc_out produces the service interfaces:

generate Go code
protoc \
  --go_out=gen/userpb --go_opt=paths=source_relative \
  --go-grpc_out=gen/userpb --go-grpc_opt=paths=source_relative \
  user.proto
Important

Regenerate every time you change user.proto. The generated .pb.go files are checked in to version control so that consumers do not need the protoc toolchain to build your project. Treat them as generated artifacts, not hand-edited code.

Correct marshaling: proto.Marshal and proto.Unmarshal
#

The generated struct types do not have Marshal() or Unmarshal() methods. The correct API lives in the google.golang.org/protobuf/proto package:

main.go
package main

import (
	"fmt"
	"log"

	"google.golang.org/protobuf/proto"

	userpb "github.com/yourname/yourapp/gen/userpb"
)

func main() {
	original := &userpb.User{
		Id:    "usr-001",
		Name:  "Alice",
		Email: "alice@example.com",
		Age:   31,
		Address: &userpb.Address{
			Street:  "10 Downing Street",
			City:    "London",
			Country: "UK",
		},
	}

	// Serialize to wire bytes.
	data, err := proto.Marshal(original)
	if err != nil {
		log.Fatalf("marshal: %v", err)
	}
	fmt.Printf("wire size: %d bytes\n", len(data))

	// Deserialize back into a new message.
	decoded := &userpb.User{}
	if err := proto.Unmarshal(data, decoded); err != nil {
		log.Fatalf("unmarshal: %v", err)
	}

	fmt.Printf("name: %s, city: %s\n", decoded.Name, decoded.Address.City)
}
Warning

Do not import github.com/golang/protobuf/proto. That is the old v1 API. Always use google.golang.org/protobuf/proto (no version suffix in the import path despite being the v2 runtime).

gRPC server implementation
#

The protoc-gen-go-grpc plugin generates an interface you implement:

server/server.go
package server

import (
	"context"
	"sync"

	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"
	"google.golang.org/protobuf/types/known/emptypb"

	userpb "github.com/yourname/yourapp/gen/userpb"
)

type UserServer struct {
	userpb.UnimplementedUserServiceServer
	mu    sync.RWMutex
	store map[string]*userpb.User
}

func NewUserServer() *UserServer {
	return &UserServer{store: make(map[string]*userpb.User)}
}

func (s *UserServer) GetUser(ctx context.Context, req *userpb.GetUserRequest) (*userpb.User, error) {
	s.mu.RLock()
	defer s.mu.RUnlock()
	u, ok := s.store[req.Id]
	if !ok {
		return nil, status.Errorf(codes.NotFound, "user %q not found", req.Id)
	}
	return u, nil
}

func (s *UserServer) ListUsers(ctx context.Context, _ *emptypb.Empty) (*userpb.ListUsersResponse, error) {
	s.mu.RLock()
	defer s.mu.RUnlock()
	users := make([]*userpb.User, 0, len(s.store))
	for _, u := range s.store {
		users = append(users, u)
	}
	return &userpb.ListUsersResponse{Users: users}, nil
}

func (s *UserServer) CreateUser(ctx context.Context, req *userpb.CreateUserRequest) (*userpb.User, error) {
	s.mu.Lock()
	defer s.mu.Unlock()
	s.store[req.User.Id] = req.User
	return req.User, nil
}

Register and start the server:

cmd/server/main.go
package main

import (
	"log"
	"net"

	"google.golang.org/grpc"

	userpb "github.com/yourname/yourapp/gen/userpb"
	"github.com/yourname/yourapp/server"
)

func main() {
	lis, err := net.Listen("tcp", ":50051")
	if err != nil {
		log.Fatalf("listen: %v", err)
	}
	s := grpc.NewServer()
	userpb.RegisterUserServiceServer(s, server.NewUserServer())
	log.Println("gRPC server listening on :50051")
	log.Fatal(s.Serve(lis))
}

gRPC client with deadline context
#

Always set a deadline on outgoing gRPC calls. A missing deadline means a slow server can hold your goroutine indefinitely.

cmd/client/main.go
package main

import (
	"context"
	"log"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	userpb "github.com/yourname/yourapp/gen/userpb"
)

func main() {
	conn, err := grpc.NewClient(
		"localhost:50051",
		grpc.WithTransportCredentials(insecure.NewCredentials()),
	)
	if err != nil {
		log.Fatalf("dial: %v", err)
	}
	defer conn.Close()

	client := userpb.NewUserServiceClient(conn)

	// Every call gets its own deadline-bounded context.
	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
	defer cancel()

	user, err := client.CreateUser(ctx, &userpb.CreateUserRequest{
		User: &userpb.User{
			Id:    "usr-001",
			Name:  "Alice",
			Email: "alice@example.com",
			Age:   31,
		},
	})
	if err != nil {
		log.Fatalf("CreateUser: %v", err)
	}
	log.Printf("created: %s", user.Id)
}

Benchmark: protobuf vs JSON
#

The following benchmark serializes and deserializes the same User message with an Address sub-message using both encodings.

bench_test.go
package bench_test

import (
	"encoding/json"
	"testing"

	"google.golang.org/protobuf/proto"

	userpb "github.com/yourname/yourapp/gen/userpb"
)

var pbUser = &userpb.User{
	Id:    "usr-001",
	Name:  "Alice Wonderland",
	Email: "alice@example.com",
	Age:   31,
	Address: &userpb.Address{
		Street:  "10 Downing Street",
		City:    "London",
		Country: "UK",
	},
}

type JSONUser struct {
	ID      string     `json:"id"`
	Name    string     `json:"name"`
	Email   string     `json:"email"`
	Age     int32      `json:"age"`
	Address JSONAddress `json:"address"`
}
type JSONAddress struct {
	Street  string `json:"street"`
	City    string `json:"city"`
	Country string `json:"country"`
}

var jsonUser = JSONUser{
	ID: "usr-001", Name: "Alice Wonderland", Email: "alice@example.com", Age: 31,
	Address: JSONAddress{Street: "10 Downing Street", City: "London", Country: "UK"},
}

func BenchmarkProtoMarshal(b *testing.B) {
	for i := 0; i < b.N; i++ {
		_, _ = proto.Marshal(pbUser)
	}
}

func BenchmarkJSONMarshal(b *testing.B) {
	for i := 0; i < b.N; i++ {
		_, _ = json.Marshal(jsonUser)
	}
}

func BenchmarkProtoUnmarshal(b *testing.B) {
	data, _ := proto.Marshal(pbUser)
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		out := &userpb.User{}
		_ = proto.Unmarshal(data, out)
	}
}

func BenchmarkJSONUnmarshal(b *testing.B) {
	data, _ := json.Marshal(jsonUser)
	b.ResetTimer()
	for i := 0; i < b.N; i++ {
		var out JSONUser
		_ = json.Unmarshal(data, &out)
	}
}

Running go test -bench=. -benchmem on an Apple M2:

OperationFormatns/opbytes/opallocs/opWire size
Marshalprotobuf19596174 bytes
MarshalJSON11802242127 bytes
Unmarshalprotobuf3104327-
UnmarshalJSON195049612-

Protobuf is roughly 40% smaller on the wire and 5-6x faster to serialize. The allocation advantage is also meaningful at high throughput.

When to use protobuf vs JSON
#

ScenarioRecommendation
Internal microservice-to-microservice callsProtobuf + gRPC. You control both sides; take the performance.
Public REST APIJSON. Curl-ability and human readability matter more than wire efficiency.
Browser clientsJSON or gRPC-Web (add extra complexity only if you have a real bottleneck).
Streaming large datasets (server-streaming RPC)Protobuf. The framing and multiplexing in gRPC HTTP/2 are a genuine win.
Debugging production trafficJSON. You can read it without a schema.
flowchart TD
    Q1{Who are the clients?}
    Q1 -->|Internal services| Q2{High throughput?}
    Q1 -->|External / browser| JSON[Use JSON REST]
    Q2 -->|Yes| PROTO[Use Protobuf + gRPC]
    Q2 -->|No| EITHER[Either works, pick JSON for debuggability]
**Using the old import path.** `github.com/golang/protobuf/proto` is the v1 compatibility shim. It works but the generated code from `protoc-gen-go` v2 uses `google.golang.org/protobuf/proto`. Mix them and you get subtle incompatibilities. Use the new path everywhere. **Forgetting to regenerate after .proto changes.** The generated code is a snapshot. Change a field name in the `.proto` and the old generated file will silently disagree. Wire up `go generate` in your Makefile so `make generate` always produces fresh code before a build. **No deadline on gRPC client calls.** A gRPC call without a context deadline can hang forever if the server is slow or unresponsive. Always use `context.WithTimeout` or `context.WithDeadline`. **Treating generated code as read-only.** The generated `.pb.go` files are checked in and should not be edited by hand, but the surrounding service code (your server struct, your client wrappers) absolutely needs custom logic. Do not put business logic in the generated file. Do not avoid adding helper functions in the same package because it "feels wrong" to sit next to generated code. **Importing `google.golang.org/protobuf/v2`.** This path does not exist. The v2 runtime lives at `google.golang.org/protobuf` (no `/v2` suffix). The confusion comes from the fact that the module semantically replaced v1, but the module path does not follow standard Go major-version conventions.

If you want to go deeper on any of this, I offer 1:1 coaching sessions for engineers working on AI integration, cloud architecture, and platform engineering. Book a session (50 EUR / 60 min) or reach out at manuel.fedele+website@gmail.com.

Related