Thursday, 18 August 2016

Using Protocol Buffers 3.0.0 With Eclipse and Maven

A long time ago, programmers preferred sharing structured data in the form of XML files. However, in recent years, JSON has managed to replace XML as the most preferred data sharing format. Why? Well, compared to XML, JSON is not only easier to parse and generate, but also far more compact.

But what if you want a format that is more compact than JSON? And what if you want it to be more easily readable? Then, you should consider using Google's Protocol Buffers. Protocol buffers are extremely compact, and can very efficiently handle large amounts of data. Using them is as easy as working with objects in your preferred programming language.

In this tutorial, I'm going to show you how to work with Protocol buffers 3.0.0 while using Eclipse Luna and Maven. Therefore, to be able follow this tutorial, you must have:
  • The latest version of Eclipse configured to work with Maven
  • A general understanding of the Java programming language
1. Create and Configure a New Project

Fire up Eclipse and create a new Maven Project. Make sure you specify an appropriate Group Id and Artifact Id.
Next, open the pom.xml file and, in the dependencies tab, add a new dependency for protobuf-java.
Then, add the exec-maven-plugin to it using the following code:
As you can see in the above code, we'll be using a class called Main that will be executed when we run this project. Therefore, add a new Java class called Main inside the src/main/java directory. I am using protobuftutorial as the package name.

Next, open the Run Configurations dialog and set the Goals field to clean verify exec:java.
At this point, our project is configured and ready.

2. Create a Schema for the Protocol Buffers

Protocol buffers need schemas, which are nothing but simple text files with the .proto extension. You can create these schemas inside the src/main/resources. For now, create just one file called country.proto.

This file will represent a country, and will have the following fields:
  • Name, which will be of type string
  • Capital, which will also be of type string
  • Cities, which will be a list of strings
  • Population, which will be an integer
  • HDI (Human Development Index, which can be one of very high, high, medium, low or very low)
Accordingly, add the following code to the proto file:
syntax = "proto3";

option java_package = "protobuftutorial";
option java_outer_classname = "CountryProto";

message Country {
        string name = 1;
        string capital = 2;
        int32 population = 3;
        repeated string city = 4;
        enum HDI {
                VERY_HIGH = 0;
                HIGH = 1;
                MEDIUM = 2;
                LOW = 3;
                VERY_LOW = 4;
        HDI hdi = 5;
As you can see, in the first line we specify the syntax we will be using in the file. It can either be proto2 or proto3.

Next, we specify the java package of our project. And then, the name of the Java class that should be generated.

I am sure, the rest of the file looks intuitive to you. Nevertheless, note that to represent a list, you must prefix the data type with the repeated keyword.

Every field in the message should have a unique tag associated with it. The number after the = sign represents the tag.

3. Getting the Protocol Buffer Code Generator

Now that the proto file is ready, we need to compile it. In order to do that, you must download the protocol buffers compiler. The easiest way to get it is to download the binary from GitHub. Make sure you download the file appropriate for your operating system. For example, if you are using Ubuntu 64-bit, you would download

After the download is complete, extract the ZIP file. You'll see that it contains a file called protoc, which is nothing but the compiler.

You can, if you want to, add protoc to your PATH variable.

All you need to do now is compile country.proto using protoc. To do so, open a new terminal and run protoc. As arguments, you must specify the absolute path of the directory in which your Maven project exists, the directory inside which you want to keep the generated files(it will be the same directory in which is present), and the absolute path of country.proto. Here's a sample call:
./protoc -I=/home/whycouch/protobuftutorial/src/main/ \
--java_out=/home/whycouch/protobuftutorial/src/main/java/ \
After the compilation, if you refresh your Maven project, you should see a new file called in it.

4. Working With Protocol Buffers

The CountryProto class allows you to create new Country objects. But first, you must create a Builder for the Country object. Using the Builder, you can set the values of all the fields in the Country object.
Builder countryBuilder = CountryProto.Country.newBuilder();
countryBuilder.setName("United States of America");
countryBuilder.setCapital("Washington, D.C.");

countryBuilder.addCity("Los Angeles");
You can write all the above code inside the Main class.

Once you have set all the fields, call the build() method to generate the object.
Country usa =;
The Country object can now be stored and shared in a compact manner. For example, here's how you can store it in a file. All you need to do is pass a FileOutputStream to its writeTo() method.
try {
 FileOutputStream output = new FileOutputStream("/tmp/");
} catch (IOException e) {
Reading from a protocol buffers file is just as easy. All you need to do is parse it using the appropriate parseFrom() method. You can then call various getters to fetch the values of the fields.
try {
 Country usa = CountryProto.Country.parseFrom(
                    new FileInputStream("/tmp/"));
 for(int i=0;i < usa.getCityCount();i++) {
  System.out.println("- " + usa.getCity(i));
} catch (FileNotFoundException e) {
} catch (IOException e) {
And that's all there is to it. You now know how to use protocol buffers.

I've created a GitHub repository for this tutorial. You can take a look at the full project there.

If you found this tutorial useful, please do like it and share it.

No comments:

Post a Comment