Unless you’ve been living under a rock (or a monolith, badum tish!) for the past couple of years, you will have heard the term microservices at some point. Chances are you, or someone you know, are developing microservices.

It can be deceptively simple to get started, choose a bounded context, knock up a REST API and away you go, right? Well, no.

Is JSON over HTTP really the most efficient way to communicate? Is it really that scalable? And furthermore now you have to write (or generate) a shed tonne of documentation and API contracts to make sure people are using your API correctly. Not to mention maintaining a consistent request/response data structure across all your microservices.

Enter protocol buffers and gRPC.

Protocol Buffers

Protocol buffers are “a language-neutral, platform-neutral, extensible mechanism for serializing structured data” developed by Google. They let us define the interface for our microservice; think of it as an interface in a object oriented programming language. As well as the methods in the microservice interface, protocol buffers allow us to specify the types of messages the server needs in requests and gives back in response.

Ok, example time. Lets say we run an eCommerce website and have a microservice for managing customers. Our protocol buffer interface might look something like this:

// The Customer service definition.
service Customer {
  // Get all Customers with a filter - A server-to-client streaming RPC.
  rpc GetCustomers (CustomerFilter) returns (stream CustomerRequest) {}

  // Create a new Customer - A simple RPC.
  rpc CreateCustomer (CustomerRequest) returns (CustomerResponse) {}
}

As you can see we’ve got 2 methods here, one to get a list of customers and one to create a new customer. The syntax looks a little bit like golang (it’s not though!), so that’ll help a bit if you’re familiar with it. So each of our methods has a parameter and a return type (CustomerFilter, CustomerRequest etc). You’ll note that the GetCustomers function has a stream keyword before the return type, this is because protocol buffers support data streams to and from the client/server. The parameters and return types are defined as messages:

// Request message for creating a new customer.
message CustomerRequest {
  int32 id = 1; // Unique ID number for a Customer.
  string name = 2;
  string email = 3;
  string phone = 4;

  message Address {
    string street = 1;
    string city = 2;
    string state = 3;
    string zip = 4;
    bool isShippingAddress = 5;
  }

  repeated Address addresses = 5;
}

// Response message for creating a new customer.
message CustomerResponse {
  int32 id = 1;
  bool success = 2;
}

// Request message for filtering Customers.
message CustomerFilter {
  string keyword = 1;
}

Again the syntax is quite straight forward, just define a message and the fields within that message. Protocol buffers are strongly typed so you’ll need to define those field types too. Finally each field has a unique integer identifier applied to it, this is used during the binary encoding.

For more information on protocol buffers, message types and streams see the official documentation.

gRPC

Right I know what your’re saying, RPC is not a new concept, they’ve been in use since the 80’s. gRPC is a “a high performance, open-source universal RPC framework”, again developed by Google. It’s one of those awesome projects thats been in use at Google for many years but has recently been contributed back to the Open Source community. What gRPC does for us is allow client and server applications to communicate simply and transparently. A developer building a client can call procedures from the server as if it was a normal local function call, no messy serialisation and HTTP calls. And it does this via…you guessed it, protocol buffers. gRPC bases it’s transport mechanism on HTTP/2 which supports normal request/response but also bi-directional streaming, which depending on your use case, can be a huge efficiency boost.

Check our the gRPC website for more info.

Get to the code!

Ok, ok, thats enough theory. So the bread and butter of the protocol buffers and gRPC paradigm is the usage of the protocol compiler, protoc. What protoc does for us is take your protcol buffer file and compile it into a client/server library which you can use in your application. You can compile this library into 10 different programming languages, from C++ to Java to Node JS, how cool is that! See the gRPC docs for a full language list. This has already solved a huge problem in microservice development, we can now be sure that our client/server developers will be creating apps that work together perfectly no matter what language they are built in.

So as an example, to compile a protocol buffer for a Golang application, the command looks like this:

$ protoc -I customer/ customer/customer.proto --go_out=plugins=grpc:customer

So this command tells protoc which protocol buffer file (.proto) to load and then where to generate the client library. Check out the gRPC docs for details on the command for each different language.

I’ve been experimenting with this myself, check out the repo, and have compiled the protocol buffer above into Golang, NodeJS, PHP and Python.

Currently PHP can only be used as a gRPC client, not a server. Sad times.

Once these client libraries were generated, I was able to quickly and easily develop wrapper client and server apps in each of these languages. Full disclosure, I’m a PHP and Golang guy, so my NodeJS and Python skills are not great. But that being said I was still able to build all of these apps in a single evening, that gives you an idea how awesome gRPC and protocol buffers are for this microservice scenario.

Each of the client applications creates 2 customers and then lists them back out again. Some examples of running the apps in various languages

Golang

Server: 

$ go run server/main.go

Client: 

$ go run client/main.go
2017/04/01 16:12:43 A new Customer has been added with id: 101
2017/04/01 16:12:43 A new Customer has been added with id: 102
2017/04/01 16:12:43 Customer: id:101 name:"Shiju Varghese" email:"[email protected]" phone:"732-757-2923" addresses:<street:"1 Mission Street" city:"San Francisco" state:"CA" zip:"94105" > addresses:<street:"Greenfield" city:"Kochi" state:"KL" zip:"68356" isShippingAddress:true >
2017/04/01 16:12:43 Customer: id:102 name:"Irene Rose" email:"[email protected]" phone:"732-757-2924" addresses:<street:"1 Mission Street" city:"San Francisco" state:"CA" zip:"94105" isShippingAddress:true >

Python

Server: 

$ python server.py

Client: 

$ python client.py
-------------- CreateCustomer --------------
A new Customer has been added with id: 101
A new Customer has been added with id: 102
-------------- GetCustomers --------------
Customer: id = 101, name = Shiju Varghese, email = [email protected], phone = 732-757-2923, addresses = [(street = 1 Mission Street, city = San Francisco, state = CA, zip = 94105), (street = Greenfield, city = Kochi, state = KL, zip = 68356)]
Customer: id = 102, name = Irene Rose, email = [email protected], phone = 732-757-2924, addresses = [(street = 1 Mission Street, city = San Francisco, state = CA, zip = 94105)]

The cool part

So I can run the client and server in the same language, but why? As I’ve been saying, gRPC and protocol buffers give us a consistent interface for any language. So I can easily run the server in say NodeJS and the client in Go and it’ll work exactly the same:

Server: 

$ node server/server.js

Client: 

$ go run client/main.go
2017/04/01 16:18:30 A new Customer has been added with id: 101
2017/04/01 16:18:30 A new Customer has been added with id: 102
2017/04/01 16:18:30 Customer: id:101 name:"Shiju Varghese" email:"[email protected]" phone:"732-757-2923" addresses:<street:"1 Mission Street" city:"San Francisco" state:"CA" zip:"94105" > addresses:<street:"Greenfield" city:"Kochi" state:"KL" zip:"68356" isShippingAddress:true >
2017/04/01 16:18:30 Customer: id:102 name:"Irene Rose" email:"[email protected]" phone:"732-757-2924" addresses:<street:"1 Mission Street" city:"San Francisco" state:"CA" zip:"94105" isShippingAddress:true >

And there you have it. Please check out the repo and play with this yourself.

Acknowledgements