Wire Support For Swift, Part 1

We’re excited to announce support for Swift in Wire. Wire already supports compiling your protocol buffer files into Java and Kotlin, and today Swift joins that family.

Protocol buffers are a powerful and efficient way to transfer data between devices. The format was created by Google and they provide excellent documentation with more information.

Along with the specification, Google maintains an official compiler for the protocol buffer format: protoc. The protoc compiler can generate code from .proto files for many languages, each of which is implemented as a plugin to the compiler. For Swift, the official plugin is primarily maintained by Apple.

So if there’s an official proto compiler, why build a new one? The original reason for writing the Java version of Wire was technical: Android binaries are limited in the number of methods they can have, and the official protoc output had a lot of methods, many of which were only in support of features that we considered unnecessary and confusing (and which, for the most part, Google ended up removing in Proto3). It was also an opportunity to create proto APIs that felt more idiomatic. As our organization and codebase scaled, however, Wire also became the tool we used to help protos scale with us. Since Wire isn’t the official compiler we’ve been able to simplify the feature set some, thus allowing us to streamline the output, all while staying compatible with the protocol buffer wire format.

Wire does omit a few protocol buffer features in order to create a simple, intuitive, idiomatic API. For Square, these omitted features all have reasonable alternatives, and the advantage of having those features was outweighed by the cost they impose on the clarity of the API. So, here’s what’s missing:

Groups

Protocol Buffers v2 (“Proto2”) has the concept of groups, which are effectively a message and a field combined. A group, however, can be implemented as an actual message and an actual field without losing any functionality, and so Wire has never supported them. With Proto3 Google similarly considered them an unnecessary complication and dropped them from the spec.

Dynamically Loaded Extensions

Proto2 supports adding fields to a message from another file using extensions. Those extensions can be compiled separately and loaded dynamically at runtime, thus allowing different libraries to contribute to the same proto message without having to generate all of the code in one compilation.

Protos with extensions

message Foo {
  optional int32 bar = 1;

  extensions 100 to 199;
}

extend Foo {
  optional int32 baz = 100;
}

Wire supports extensions, but they must all be compiled together at the same time. The resulting code is less error prone and easier to use as the extensions don’t need to be managed manually.

Protoc extensions in same compile

// Package_Name_Foo_Extensions is a constant that is generated by protoc
// and includes all of the extensions known at the time of compilation.
let foo = try Package_Name_Foo(
  serializedData: data,
  extensions: Package_Name_Foo_Extensions,
  partial: false,
  options: .init()
)

// The Package_Name_baz getter is generated automatically for the extension fields.
// But it will silently fail and return 0 if the appropriate extension wasn't passed to the init above.
let value = foo.Package_Name_baz

Wire extensions

let decoder = ProtoDecoder()
let foo = try! decoder.decode(Foo.self, from: data)

// In Wire the extension field is baked into the generated Foo object just like
// all other fields. The concept of extensions doesn't exist in the generated code.
let value = foo.baz

Additionally this means that Wire can provide definitive checking at compile time, generated objects are smaller, and the runtime code can be simpler and faster. With protoc’s dynamic extensions it’s possible for two extensions to use the same numeric tag identifier, resulting in a runtime error, whereas Wire’s compile-time limitation lets you know something is wrong much earlier. It also means that we can alert to name conflicts, which removes the need for the name mangling used to ensure that field names are unique in dynamic extensions.

Proto3 mostly does away with extensions presumably for many of the same simplification and safety reasons.

Default Values

Probably the biggest difference between Wire-generated code and protoc-generated code is support for default values.

History

Default values in protocol buffers have had a rocky history.

In Proto2 an unset, optional field could have an explicit default value:

optional int32 value_with_default = 1 [default = 42];

If there is no explicit default, then an implicit default value is used (empty string for string fields, zero for number fields, null for messages, etc).

optional int32 value = 1;

// The field returns zero, even if a value wasn't set by the sender.
let value = fooProto.value;

Additionally, it is possible to differentiate between a set field and an unset field that’s returning a default value:

let value: Int32
if fooProto.hasValue {
  // Use the explicit value from the sender
  value = fooProto.value
} else {
  // No value sent. Do some fallback behavior.
  value = 10;
}

In Proto3 default values were simplified and the concepts of optional fields and explicit default values were both removed. If a field doesn’t have a value set for it in the message bytes then it will return the implicit default value (empty string, zero, etc). Instead, Proto3 introduced box types, like Int32Value for each scalar. The semantics around defaults became simpler, but at the cost of having additional types to understand and implement in each compiler.

More recently, optional fields are being phased back into Proto3 with v3.12, making it possible to test whether a field was present in the message bytes or not.

The Wire API

Wire has no support for default values, and thus all of the above can be simplified into this statement:

Optional fields are nullable. If a value was included by the sender then the field will have a value. If no value was sent, then the field will be null.

This is a lot easier to understand, and in practice we never used the default values anyway.

Why No Default Values

Let’s explore the reasoning for omitting default value support. If you have an optional field, then the value and behavior you get from that field has several possibilities. For these examples we’ll call the consuming code “the client” and the producing code “the server”:

1. The field is set by the server. The client will use that value.

2. The field is not set by the server. The client can use the default value for the field.

   1. In Proto2 this value might be an explicit default value set in the proto definition.
   2. In Proto3, or Proto2 when no explicit default is set, the default value will be the standard zero, empty string, empty array, etc.
3. The field is not set by the server. The client can check that it was not set and apply some client-default behavior.
   1. In Proto2, or Proto3 (v3.12 and later) this can be done by specifying the optional cardinality keyword
   2. In Proto3 this can also be done by using a box type, like IntValue

This means that each field has three possible states (explicit value, default value, or unset), and thus the API needs to provide ways to differentiate between them. The output produced by protoc handles this mostly the same in all languages, by adding a separate “hasBar” and “clearBar” method.

Protoc generated field example

private var _bar: Int32?
public var bar: Int32 {
  get {return _bar ?? 0}
  set {_bar = newValue}
}
/// Returns true if `bar` has been explicitly set.
public var hasBar: Bool {return self._bar != nil}
/// Clears the value of `bar`. Subsequent reads from it will return its default value.
public mutating func clearBar() {self._bar = nil}

It’s quite a bit of code for a single field. And, importantly, in our usage at Square, we basically never wanted to use the default value (case 2 above). Explicit defaults are dicey because you never know what value was encoded in the mobile client you’re sending to at the time it was compiled (or if there was one), so it’s safer to explicitly send the value. Implicit defaults are almost never what we want as zero is rarely the correct numeric value for something and an empty string is rarely valid data and so requires checking anyway. In the few cases where zero is the right value, that can still be set explicitly in the client using case 3 above.

By removing support for default values in Wire we can then greatly simplify the API. The same field in Wire looks like:

Wire generated field example

If the field was set by the server then it has a value. If the field wasn’t set then the field is nil. This lets us write idiomatic Swift when consuming our generated protos:

// Use a default when the field isn't set
let value = foo.bar ?? 0

// Use different behavior when the field isn't set*\
if let value = foo.bar {
  // Use the set value
} else {
  // Hide fields or other custom behavior since the field wasn't set
}

Wire has a few other useful features that were designed to help protos scale with a large organization.

Roots and Prunes

One of the most powerful tools that Wire offers is the ability to intelligently trim the tree of messages defined in your proto files. By omitting types and fields that aren’t needed you can simplify the generated API, and reduce the size of your shipping binary. There are several scenarios where it’s convenient to include only some of the message tree, such as:

• You share a proto file between two clients and each client only needs some of the messages defined in that file.
• You share a proto message between two clients, but each only needs a subset of the fields.
• Your server code requires additional metadata on a message that your client doesn’t care about.
• You deprecated fields and your current client versions don’t need them.
• You deprecated enum cases and don’t want to have to handle them (and be warned about them) in your switch statements.
• You have message definitions for upcoming features but don’t need to use them yet.

Wire has offered the ability to prune out specific fields or types for quite some time, and it does so intelligently, removing fields that use pruned types so that there are no dangling references. With this new release for Swift we’ve also enhanced the pruning functionality and have completely removed the need to deal with .proto files directly. Instead, the Wire compiler works directly with the message definitions within .proto files and scopes the generated content based on semantic definitions within the tree.

With the new version of Wire, rather than listing .proto files you can instead provide a directory which contains all available .proto files, and a list of “root” types for which generated Swift (or your language of choice) objects should be emitted. Wire will build the dependency tree from the given .proto files, trim out anything that isn’t referenced, and generate objects for the types you specified (and their dependencies). You can also explicitly “prune” types and fields from the tree to cover the scenarios mentioned above. In all cases Wire will ensure that the dependency tree is complete, and will appropriately handle edge cases like type nesting and options.

MyProtos:
  roots:
    # This message and all of its dependencies will be included
    - yoda.teachings.LukeTraining
  prunes:
    # This message, which would normally be included as part of the
    # LukeTraining message, will not be included
    - yoda.teachings.Fear
    # Additionally, we can trim specific fields or enum cases (such as deprecated ones)
    - yoda.teachings.Options#try

Modules

Swift introduced a new challenge that didn’t exist with Kotlin and Java: modules. Kotlin and Java both use fully-qualified package names, but Swift modules are defined by their compilation unit, and thus namespaces aren’t declared at the type or file level. This meant that we needed to build a new packaging system for Swift that could deal with Swift module namespacing and imports.

We decided that the easiest way for a caller to define modules was to make those definitions handled directly by Wire. A single manifest file defines the modules, their names, their dependencies, and the content roots and prunes mentioned above.

In this example manifest the DarkSide and LightSide modules would depend on and import the CommonProtos module:

CommonProtos:
  roots:
    - jedi.Lightsaber
    - jedi.MindTrick
    - jedi.TheForce
    - jedi.Anakin

DarkSideProtos:
  dependencies:
    - CommonProtos
  roots:
    - darkside.*
    - jedi.Lightning
  prunes:
    - jedi.Mercy

LightSideProtos:
  dependencies:
    - CommonProtos
  roots:
    - lightside.*
    # Import the rest of the Jedi powers not already in CommonProtos
    - jedi.*
  prunes:
    # Remove unused lightsaber colors
    - jedi.LightsaberColor#red
    # Remove deprecated field. Use green_lightsaber instead.
    - lightside.Luke#blue_lightsaber
    # Remove dark-side-only types
    - jedi.Lightning

The Swift protoc plugin offers something similar for defining modules, but because Wire’s system works at the type level rather than the file level you have additional control to put generated types in the module where you want them.

Built-In Redaction

As a financial company we’re extremely careful about how we handle sensitive information. A common engineering mistake is to leak such information into your logging, which can then end up in a logging database somewhere or even in a third party logging service. If this happens it’s often very difficult to go back and scrub the sensitive data from those databases.

With Wire, we wanted to avoid this situation as best we can, and so Wire supports automatic redaction by tagging a field with the redacted option:

optional string email = 1 [(squareup.protos.redacted_option.redacted) = true];

In any language that Wire supports (now including Swift), this will redact the field when you print a description of the object:

"Foo(id: 1234, email: <redacted>)"

While Wire supports all of the features that many people will need, there are a few that are still on the roadmap. Wire does not yet support gRPC services in Swift (Kotlin support is available). Additionally, Proto3 support is a work in progress, and currently Wire can parse the syntax and emit appropriate objects with proper cardinality semantics, but some of the other Proto3 features, like box types, will be added in the future.

Swift support in Wire is ready for use as of 3.3.0-alpha1, and can be included in your project using CocoaPods. Manifest support is currently considered experimental while we finalize the syntax, and will require using an experimental flag.

Add Wire To Your Podfile

# Add the Wire compiler so that it is downloaded and available.
# CocoaPods will download the source and build the compiler directly,
# so you'll need Java installed.
pod 'WireCompiler', '3.3.0-alpha1'

# Add the Wire runtime to do the serializing/deserializing
pod 'Wire', '3.3.0-alpha1'

Then run pod install to get the dependencies and build the Wire compiler

Build Your Protos

Then you’ll use Wire to compile your protos into Swift files:

java -jar ./Pods/WireCompiler/compiler.jar \
  "--proto_path=<directory containing .proto files>" \
  "--swift_out=<directory where the generated .swift files go>" \
  "--experimental-module-manifest=<path to manifest yaml file>"

With Swift support, we’re really excited to now have all of our platforms using Wire for protobufs. It’s provided a cleaner and safer experience for our developers while also speeding up the app. We’ll dig into the performance and engineering considerations in part two of this post.

We hope you find Wire’s Swift support useful, and if you have any issues or feature requests, send us a ticket.