A Tale of Too Many Protocols

Accrescent is a secure, private, and easy-to-use Android app store. We’ve been laser-focused on making that vision a reality. In particular, we’ve been developing features and refactoring the Accrescent server to make it more secure, laying the groundwork for app developer-facing features like an app publishing API (which makes it easy to automatically publish app updates), and preparing our code and infrastructure for opening up app submissions to anyone who wants to sign up.

While we initially planned on immediately developing console features for app developers, we faced some challenges which required us to make some more fundamental changes first; but that doesn’t make them any less exciting! This post is the first in a short series chronicling the challenges we encountered, how we solved them, and how close we are to opening up app submissions at the end of it all.

Out of sync

Some of the more frustrating problems we’ve encountered are related to how the Accrescent console and the Accrescent server speak to each other. For context, the Accrescent console is the web application app developers use to submit and manage their apps and which Accrescent reviewers use to review app submissions. The Accrescent server is the server application which accepts and processes requests made from the Accrescent console.

For these two applications to communicate, they need to speak the same language — in this case, JSON. To do this, they each define message structures for the operations they require in the programming language they’re written in — TypeScript for the console, Kotlin for the server — and make sure they’re compatible. For example, a request for an Accrescent reviewer to review a new app submission might contain:

• The ID of the new app to review
• Whether the review is an approval (the app may continue to be published) or a rejection (changes need to be made before publication)

In TypeScript, that request structure might look like this:

interface Review {
    appDraftId: string;
    result: ReviewResult;
}

enum ReviewResult {
    Approved = "approved",
    Rejected = "rejected",
}

while in Kotlin, a compatible structure might look like this:

@Serializable
data class ReviewRequest(
    val appDraftId: String,
    val result: ReviewResult,
)

@Serializable
enum class ReviewResult {
    @SerialName("approved")
    APPROVED,
    @SerialName("rejected")
    REJECTED,
}

Even without understanding either of these languages, you can already begin to see a problem: we have to define each message twice! While this approach wasn’t an issue when there were only a few message types, it quickly became difficult to manage once we added more types and more complex invariants — Accrescent has almost 100 such message types and counting. Remember, each definition has to be compatible, otherwise the console and the server won’t understand each other! If we add a field, we have to add it in both places. If we remove a field, we have to remove it from the applications in the correct order, or one of them breaks. If we say, “review requests must have at least one rejection reason if the review is a rejection, but zero if it’s an approval,” we have to implement that in both codebases. Besides being tedious, this approach also made it very easy to make mistakes, causing bugs in the software our users rely on.

Complexity was growing, and we needed a solution. So what could we do?

Protobuf to the rescue!

Well, what if we could define our message structures in only one place? It turns out we can! Protocol buffers provide a schema language we can use to define our message structures in one location, and from our schemas, we can generate the TypeScript¹ and Kotlin code we need in our applications. We can then use gRPC to generate the code we need to send and receive these messages across applications. Here’s what a protobuf definition for ReviewRequest could look like:

syntax = "proto3";

message ReviewRequest {
  string app_draft_id = 1;
  ReviewResult review_result = 2;
}

enum ReviewResult {
  REVIEW_RESULT_UNSPECIFIED = 0;
  REVIEW_RESULT_APPROVED = 1;
  REVIEW_RESULT_REJECTED = 2;
}

If we need to add more complex invariants, we can use protovalidate to get consistent validation across languages. For example:

syntax = "proto3";

import "buf/validate/validate.proto";

message ReviewRequest {
  string app_draft_id = 1;
  ReviewResult review_result = 2;
  repeated string rejection_reasons = 3;

  option (buf.validate.message).cel = {
    id: "rejection_reasons_match_result"
    message: "must have >= 1 rejection reasons if review is a rejection"
    expression:
      "this.review_result == ReviewResult.REVIEW_RESULT_REJECTED "
      "? size(this.rejection_reasons) > 0"
      ": size(this.rejection_reasons) == 0"
  };

}

enum ReviewResult {
  REVIEW_RESULT_UNSPECIFIED = 0;
  REVIEW_RESULT_APPROVED = 1;
  REVIEW_RESULT_REJECTED = 2;
}

At Accrescent, we’re big fans of protocol buffers, and we already use them for our app store API, the protocol the Accrescent app uses to fetch app information from the store. So why not use them here too?

Unfortunately, there’s a catch. gRPC fundamentally doesn’t work in the browser, meaning we can’t use it in the Accrescent console. Thankfully, there are some alternatives which can be used in a browser and which allow us to keep most of our gRPC/protobuf tooling.

gRPC alternatives

There are three main gRPC alternatives we evaluated while attempting to solve this problem: gRPC-Web, Connect, and gRPC Transcoding. The first was gRPC-Web.

gRPC-Web

gRPC-Web is an official gRPC project that had its first stable release in 2018. Naturally, it was designed to make gRPC available on the web. It accomplishes this by defining a gRPC-like protocol which is compatible with web browsers, then using a proxy server to translate gRPC-Web to gRPC so that application servers can understand it.

gRPC-Web has two significant problems for us, however. First, the official client generator appears to be soft deprecated. From their roadmap:

NOTE: Due to the status of gRPC-Web’s core dependencies — Google Closure, which has been archived, and Protobuf JavaScript, which is receiving only minimal updates — the gRPC-Web project is no longer able to deliver new, modern solutions for the open source community. As a result, we do not plan to be adding new features going forward.

We recommend you to use gRPC-Gateway as an alternative.

Second, the translation proxy gRPC-Web uses is Envoy. While Envoy is a high-quality, well-maintained project, it is a very large, complex codebase written in a memory-unsafe programming language, and we don’t need most of its features. For security, we would rather not run it on our network.

Connect

Connect was a promising candidate. Created by Buf in 2022, Connect makes it possible to serve a regular gRPC API, a gRPC-Web API, and a similar, browser-friendly Connect API all from the same server, no proxy required. It also supports generating client libraries using connect-es and protobuf-es, so hooking the Accrescent console up to it would be very straightforward. With Connect, we could simplify our architecture like so:

That sounds great! — but we’re currently unable to use it. Connect has no Java implementation, and the Accrescent server is written in Kotlin (using Java libraries). We need to use a Java-compatible language because we strictly depend on apksig, Google’s library for verifying APK signatures, so Connect is off the table until they have a Java library. We’ll come back to Connect later.

gRPC Transcoding

Finally, there is gRPC Transcoding. Similar to gRPC-Web, gRPC Transcoding defines how browser-friendly HTTP/JSON requests and responses can be translated to and from a regular gRPC API. Unlike gRPC-Web, it isn’t its own protocol, but it defines a way to build one by adding annotations to a regular protobuf schema. For example, here is a simple gRPC API definition with gRPC Transcoding annotations:

syntax = "proto3";

import "google/api/annotations.proto";

service HelloService {
  rpc SayHello(SayHelloRequest) returns (SayHelloResponse) {
    // Maps GET requests to /v1/hello/{name} to this RPC, treating {name} as the
    // name passed in SayHelloRequest. We can use the API like so:
    //
    // $ curl https://api.example.com/v1/hello/Alice
    // {"message": "Hello, Alice!"}
    option (google.api.http) = {
      get: "/v1/hello/{name}"
    };
  }
}

message SayHelloRequest {
  string name = 1;
}

message SayHelloResponse {
  string message = 1;
}

Easy enough. And remember the gRPC-Gateway that gRPC-Web mentioned? We can use it to generate a small, memory-safe translation proxy, so we don’t need to use Envoy! Our architecture then ends up looking like this:

At this point, we were convinced that gRPC Transcoding was the right option for making our gRPC API available to the Accrescent console. All that was left was to teach the console how to communicate using the new protocol.

Bridging the gap

With regular gRPC, we can generate client code directly from the API definition and use that in our application with code as simple as apiClient.callMethod(). Unfortunately, things aren’t so simple for gRPC Transcoding. There are multiple ways to generate clients for gRPC Transcoding APIs, but first, we started with the conventional route: OpenAPI generation.

OpenAPI generation

The OpenAPI generation route looks like this:

1. Define a regular gRPC/protobuf API schema in .proto files.
2. Add gRPC Transcoding annotations to the schema.
3. Using a tool like protoc-gen-openapiv2 and the protobuf schema as input, generate an OpenAPI schema which describes the HTTP/JSON API served via gRPC Transcoding.
4. Use an OpenAPI generator like OpenAPI Generator to generate client code for your chosen language and/or framework from the generated OpenAPI schema.

As complicated as this workflow is, it seemed to check all of our boxes. OpenAPI Generator even directly supports Angular, which we use for the Accrescent console. However, this solution quickly came back to bite us with multiple severe issues as we tried integrating it into the Accrescent console:

• Custom decoding logic: the generated OpenAPI schemas were based on the ProtoJSON encoding of protocol buffers. As a result, certain common types like timestamps were encoded in a special way which would require implementing custom decoding logic in the Accrescent console to read them.
• Incorrect definitions: OpenAPI Generator is the best supported OpenAPI client generator we could find by far. Yet the generated Angular client was often incorrect, omitting valid states from type definitions and not representing more subtle semantics of protocol buffer messages, causing confusing runtime errors that were supposed to be structurally impossible.
• No protovalidate support: while expected, we were unable to use protovalidate to check the validity of server responses, requiring us to do so manually.
• Insecure code generation: OpenAPI Generator openly documents that it is vulnerable to code injection attacks from untrusted schemas. While this isn’t a problem for us since we trust our own schemas, we don’t want to require third-party developers using our API to open themselves up to this risk by trusting our schemas.

It was clear that we needed a better solution. But what?

The wide, wide world of protobuf plugins

We knew we wanted to keep the API definition strictly in the protobuf schema so that it stays in one place, and OpenAPI wasn’t working for us. So the natural next thought for us was: what if we skipped the OpenAPI step entirely? Instead of generating

protobuf schema ➝ OpenAPI schema ➝ client code

what if we had a pipeline that generated

protobuf schema ➝ client code

directly?

Protobuf plugins allow us to do exactly that. By writing a protobuf plugin, we could hook into the protobuf code generation process to generate exactly what we need. Buf’s protoplugin library provides a simple way to write protobuf plugins which solve all of the aforementioned issues with the OpenAPI generation approach:

• No custom decoding logic is needed since protoplugin uses idiomatic TypeScript types generated using protobuf-es.
• protobuf-es is fully compliant with the protobuf conformance tests, guaranteeing correct generated types and behavior.
• protobuf-es has built-in protovalidate support, enabling us to check message validity with a single function call.
• protoplugin offers utility functions which make it easy to sanitize inputs and avoid code injection attacks from untrusted schemas.

Furthermore, since we fully control the generated client, we can make it use Angular’s HttpClient class for idiomatic Angular integration.

Thus, we wrote protoc-gen-grpc-angular: a protobuf plugin for generating Angular clients for gRPC Transcoding-based APIs. We then packaged our API client for the Accrescent console using Angular’s package format, and at long last, our problems were solved!

Filling the gap

What we had so far — a custom gRPC-Gateway proxy, protoc-gen-grpc-angular, and the generated console client — worked well. We started using this system in our development version of the Accrescent console and didn’t encounter any major hiccups. Yet we couldn’t help but think that there had to be a simpler alternative. Maintaining an additional proxy and a custom protobuf plugin is a lot of work for a single API. We had to find out: was there any way to remove one or both of these components?

Removing the proxy

The translation proxy was necessary because the Accrescent server understood gRPC while the Accrescent console could speak only gRPC Transcoding. But if we could make the Accrescent server understand gRPC Transcoding itself, there would be no need for a translation proxy.

Some web server frameworks such as ASP.NET support using gRPC Transcoding like this. But the framework we used, Quarkus ², does not yet because it uses Vert.x 4 for gRPC support, which doesn’t support gRPC Transcoding. However, Vert.x 5 supports both gRPC Transcoding and gRPC-Web. By removing Quarkus and using Vert.x 5 directly, we could add gRPC Transcoding support to the Accrescent server, completely obviating the need for a gRPC-Gateway translation proxy. That’s one less server to maintain and deploy. What about the protobuf plugin?

Simplifying code generation

The reason we wrote a protobuf plugin in the first place is because we didn’t want to use gRPC-Web (because of the weakly maintained client library and the Envoy proxy requirement), we couldn’t use Connect (because there’s no Java server support), and gRPC Transcoding had no existing code generator which met our requirements. So to simplify this setup, we needed to do one of the following:

1. Find an existing gRPC Transcoding code generator.
2. Add Java support to Connect.
3. Find a way to use gRPC-Web without a proxy or the official client library.

(1) led to a dead end. (2) is far beyond our capabilities and would need significant buy-in from the Buf team to implement. But (3) is already half-solved; we just demonstrated that gRPC-Web doesn’t actually need Envoy in our setup because Vert.x 5 serves gRPC-Web on its own without a proxy.

Remember how I said we’ll come back to Connect later? The official connect-web library can be used to call Connect APIs, which isn’t much help to us. But it can also be used to call — you guessed it — gRPC-Web APIs! And like our protobuf plugin, connect-web uses protobuf-es to generate its TypeScript message types, meaning it still solves all of the problems with OpenAPI generation listed earlier without any of the custom code generation.

Our final code generation pipeline is as simple as this:

protobuf schema ➝ protobuf-es types

and our architecture has only the two pieces we began with:

A successful experiment

We evaluated a lot of mixed solutions to unify our API definitions: gRPC-Web, gRPC Transcoding, protobuf plugins, gRPC-Gateway, Vert.x gRPC, and Connect. But in the end, we accomplished what we were after: a strongly typed single source of truth for our API with generated clients and no additional plugins or proxies. As a result, we can now fearlessly add and update console features in a single location without worrying about drift between applications. You can find the new API schema in our GitHub repository under the permissive Apache 2.0 license.

And this post only scratches the surface of what we’ve accomplished recently. In our next blog post, we’ll talk about the deep changes we’ve made to the Accrescent server to make it simpler, faster, more secure, and all the more ready for opening up app submissions to everyone.

If you want to support our mission of building a private, secure, and user-friendly Android app store for everyone, consider donating to support the project. Consider joining our community or contributing on GitHub to ask questions and make a difference. And as always, thank you to our past and current sponsors and supporters for making Accrescent possible! We wouldn’t be here without you, and we’re excited to share the rest of what we’ve been working on and what’s next.

Footnotes

1. Protocol buffers does not have official support for TypeScript. However, other high-quality implementations do, such as protobuf-es. ↩︎

2. We’ll talk more about Quarkus and why we migrated to it in the first place in a later blog post in this series. ↩︎