Download directly

We’re back with part 2 of our Buck episode. We continue our chat with Uber Engineer, Gautam. We dive into the details of how buck is used, how you can use it and how it can help you and your team. Learn about how to use it with Gradle and more with OkBuck.

Show Notes

Build systems

Misc

Resources

Contact

Transcript

This is Part 2 of a two-part interview. Part 1 can be found at http://fragmentedpodcast.com/episodes/68/

Donn Felker: As an Android developer, I’m very familiar with the Gradle build system, so I’m apprehensive about leaving for something brand new—there’s probably a large learning curve, and so on. But I hear that you guys have created a Gradle plugin for Buck called okBuck. Can you tell us a little bit about what okBuck is, and maybe why we would want to use it as compared to regular Buck?

Gautam Korlam: That’s a great question. We were in that same exact boat, because change is hard (especially when you have hundreds of developers working on a codebase). All our developers were familiar with Gradle, so we wondered, “Hey, is there any way that we can use our existing structure along with Buck?” It would have been a nightmare to force a large migration on all our developers, bringing all the cognitive overhead from the tooling changes.

So we started looking at an open-source project called okBuck. It wasn’t originally written by us, but by a guy in Beijing named Piasy Xu. He wrote it as a hobby project, and it was building very basic Android projects by the time we discovered it. At the time, a lot of things in it were not flexible enough to actually be considered production ready, so we went in and almost rewrote it from scratch over three to four months. We brought it to a high-quality state that can be considered production ready.

We use a lot of different tools at Uber. For instance, we pretty much use everything under the sun that Google puts out. All of that needs to work in alternate build systems. Sometimes there are no convenient analogs, so we had to bridge the gap, and that’s where okBuck comes in.

Kaushik Gopal: Let’s just step back a little. Right now, this is a little confusing to me. You’re saying that okBuck is a Gradle plugin—but fundamentally, Gradle and Buck are on opposite sides, so how does that work? Right now, my mind is split into two parts. There’s Gradle, and there’s Buck—but wait, both of these are working together? I don’t understand.

GK: Let’s say that you have a project which is currently being built with Gradle, and you want to try out this new system from curiosity. In other words, you don’t want to spend all the time and effort it takes to migrate over. What will you be looking for? “Is there anything that can make the transition automatic for me?” If you think of a project converter, okBuck is kind of like a build system converter.

The way it’s currently written, it’s very easy to convert a Gradle project into any other build system. If you remember Maven from back in the day, Android Studio used to come with something called a Maven to Gradle converter, which did something similar.

KG: Oh yeah, I remember those days.

GK: It’s similar to that, but there’s actually a huge gap between the concepts of the two build systems. There’s a level of complexity that needs to be bridged so that developers don’t have to spend time wondering “Hey, how does this work, exactly?”

To make that even easier, okBuck sets up something for you called a Buck wrapper. It’s very similar to the Gradle wrapper, so we kind of stole the name.

DF: So it’s essentially buckw.

GK: Exactly. In fact, we called it that. If you want to build an APK, you just say ./buckw build apk. If you want to test something, you say ./buckw test module Or if you lend something, it’s ./buckw lend [whatever]. It bridges the gap, taking the overhead of understanding the system away. That’s why okBuck is a very quick way for you to try Buck and see if it’s right for your project, and what value it can bring to your project or your codebase.

KG: Suppose I want to try Buck. What would my first steps be?

DF: How do we get started?

GK: okBuck has a sample project containing a lot of different examples. It also has a very extensive wiki and migration guide. But 90% of the time, you can just apply the plugin, and it should just work.

Admittedly, there are particular cases—especially in large, enterprise-level builds—where you may want to customize it, or you may have a bunch of legacy Gradle hackiness somewhere in your build that doesn’t work with Buck.

Another company that uses okBuck very extensively right now is AirBnB. Once we open-sourced it, they were pretty interested. They have basically the same problem (along with a lot of other companies that are hitting this issue): they started with Gradle, but they’re reaching this point of scale where it’s just not working out for them and they need alternatives. But they don’t have the bandwidth or time to go research, and that’s where this solution fits in. AirBnB basically did their conversion in three days to a week. Two engineers sat down, understanding what the system looks like, and just tried it out.

KG: So I guess our first step would be to go and look at the examples and instructions in the README, and that would help us understand where to move from there.

GK: I also gave a talk in September at DroidCon NYC which covers what the migration path looks like and the kinds of gotchas that users might encounter.

DF: You said that okBuck is something that you and Airbnb are using. Is it production ready, or are you guys just testing it out still? To me (and from what I can tell, to Kaushik), it’s very interesting, and I kind of want to go play with it. If I’m able to convince clients that, “Hey, this is going to cut down your build time significantly, which is going to increase your developer productivity, which affects your bottom line”—is this something that I can go use today?

GK: We’ve been using it to build the new Uber Android app ever since we started developing it six months ago. At this point, all of Uber uses Buck to build for Android (as well as iOS), and we use okBuck on a daily basis. So it goes through a lot of churn as we make sure to add any new features that people request. For example, we added support for things like SQLite, Square, Lambda, and the Gradle transform plugin. It actually supports a lot of those use cases out of the box.

It’s really a situation where you can try it out, and if something’s broken, it’s easy for us to add that support, because it’s super easy to extend. Also, we get a lot of excellent contributions to okBuck that make it easy for us to make these changes.

DF: There’s a good question. Is okBuck open-source?

GK: Yes, it’s completely open-source. You can check out all the code at https://github.com/uber/okbuck.

DF: It looks like you guys are MIT licensed too, for those who are wondering.

KG: Excellent, that’s a good license to have.

GK: Also, in terms of the quality of the product itself, Airbnb isn’t the only one using it. I think Lyft has been using it for a while, and the guys at Square are currently evaluating it.

Also, Facebook has set up a Buck-focused Slack, where a bunch of companies are usually chatting about different elements of Buck, how it’s being used, and the unique situations that they’re facing. That’s open for people to join, if they’re interested in learning more.

KG: We’re all software engineers, so we understand that not everything in the land of Android development is sunshine and roses. Right now, hearing about Buck’s advantages is pretty much just sunshine and roses, but I think it’s also fair to say that there have to be some disadvantages, right? For example, one thing that I’ve heard is that external dependencies aren’t supported right out of the box. Is that still true? I remember that that was thrown around a lot in the early days of Buck, when people were super frustrated with Gradle.

GK: Those still aren’t supported in the broad sense. With Gradle, you have a lot of finetuning abilities, so you can say, “Hey, I want this dependency, but not the transitive dependencies that it brings in.” Actually, there’s no concept of Maven in Buck yet, but you have to understand that Buck was built for the multi-people situation that Facebook is currently in—one where the whole codebase is filled with all sorts of JARs and external dependencies.

That may not work for everyone, which is why okBuck bridges that gap as well. It uses Gradle’s caching solution. You basically get the exact same dependencies that you would get in Gradle, but it downloads them to a local cache that Buck can actually reuse to build the codebase. If you just use Buck, it still doesn’t do external dependencies. There’s a basic command called buck fetch that can fetch external code, but there’s no dependency management in the Maven sense. So okBuck is definitely useful for people who are coming from the Maven world and want to bridge that gap.

KG: Essentially, I guess that the disadvantage (if you can call it that) is that all those dependencies would be pulled locally, versus having direct references.

GK: Exactly. Having them locally can bloat up your repository if you’re not using things like Intellifast, for example.

Those kinds of things are not documented really well on the Buck website. They just say, “Hey, you can have this checked in, and you can pretty much go with it.” The fact of the matter is that Facebook has tailored their BCS solution to scale really well. They use Material, and they have a lot of custom in-house things that make it very easy to work with large-scale codebases. But that’s not within everyone’s reach.

okBuck basically says, “It may not matter on small-scale projects, but eventually, as your project grows, your build will grow with you and not slow you down.” So a comes from here at Uber. Our codebase has grown from 40 modules at the get go 3 months ago to 170 modules in the released app. That doesn’t even count any of the external libraries that are used for networking and other stuff like that. That’s just the new app. But we haven’t actually changed, in terms of the build speed. It’s exactly the same.

We also have something called a network cache, which I forgot to mention. Buck comes with something called a remote build cache, which means that if you build something on CI (for example), it’s cached on the network cache. Let’s say that CI has the build master, and I come in tomorrow and say, “git rebase. Okay, I have the latest master.” Then I try buck build, and it says, “Build done in 3 seconds.” What just happened there? It downloaded the exact apk for that job from the network cache, and it says, “Okay, you don’t really need to build anything. It’s already been built on a different machine; you can just go ahead and use it.”

DF: That’s magical.

KG: That’s brilliant. So basically, it’s a shared build cache.

GK: We can do that because builds are reproducible. You couldn’t do it if they weren’t equivalent on different machines.

KG: That makes sense. So it’s like coming full circle in terms of reproducibility.

GK: Currently, our build times on a fresh clone from the repo are 12 seconds on Wi-Fi and 3 seconds on Ethernet.

KG: That’s pretty cool.

GK: Uber has also open-sourced our Buck HTTP cache implementation. That’s linked to in the README of okBuck, if you’re interested in trying it out. I think the Airbnb folks already use it for their builds.

DF: Kudos to you folks.

KG: I guess there are two points to be made. First, this is a network cache. Hopefully, you’re in a company and on Ethernet, but it depends on the internet. If your network connection is slow, don’t expect 3 seconds.

GK: It might be 30 seconds then, because you also have a local cache on your machine. There are multiple levels of cache.

KG: You mentioned the documentation, and that’s something that we should let our listeners know about. At this point, the documentation is not necessarily up to snuff.

GK: We found that lot when we were working in Buck, actually. There are lots of things inside the Buck codebase that are not immediately obvious from the public documentation. That’s not problematic, generally, because you can typically look at the source code and get a good understanding of what’s going on once you dig in, but it was definitely a surprise to us when we initially started to start develop with Buck.

DF: Before we hopped on this call for the podcast, I took a look (over the last couple of days) at the Buck website. I started hopping into the docs, thinking, “You know what? I’m going to see if I can try this out.” But what I immediately felt was an apprehension to learn it—not because I didn’t want to learn something new. I love learning. But it seemed like a big thing, like when you first learn Dagger or dependency injections. It’s a very large, steep learning curve that turned me off. Have you seen that with Buck as well? Is there a large or steep learning curve?

GK: For some parts of it, yes. If you’re doing a very basic build, most of the systems have very, very basic examples on their websites. But they won’t list, for example, how to hook up annotation processors, do build flavors, or other instant things like those. It may not be easy to access that information, so while there’s not a steep learning curve, there’s definitely a longer one. It will take you longer to find that information.

But okBuck just says, “Okay, that’s too much information. Let’s try to bring it all together into a nice API interface. Just build that thing, and we’ll get out of the way.” That’s why we really like okBuck.

KG: The other question that I had about Buck is that, while I’m very enthused to try it out, one of the biggest advantages of Gradle is the concepts of product flavors and build variants. Does Buck support the same thing, because that’s actually a pretty useful feature for many folks?

GK: Certainly. Buck has no concept of build variants (as such), but okBuck supports them. We do the translation logic in such a way that it doesn’t really make a big difference. Let’s say that with Gradle, you’d do something like, assembleFlavor1Debug. With Buck, you’d say, buck build Flavor1Debug instead. We did that translation because Buck doesn’t really have a concept of build flavors or variants, but you can map what Gradle does to say, “Hey, when you build Flavor1Debug and Flavor2Debug, you’re pretty much recompiling twice.” All the source code is compiled twice and treated as two separate targets, so we kind of do the mapping that way.

KG: So that’s where okBuck comes into play.

GK: If you did that manually, it can take quite a lot of time. I think some of the largest build files we have are our 5,000-10,000 lines of Buck files for really complex builds.

KG: Even beyond time, I would imagine that complexity is the tricky thing, right? These are things that you want to be careful about, especially if you’re building multiple variants.

GK: In terms of other features, I think that okBuck supports unit tests and espresso tests. Gradle transforms have been added recently.

We’re working on Kotlin support right now because the community is moving towards it very heavily. Kotlin build times are not exactly where they’re supposed to be, as far as the developer feedback I’ve been hearing goes. That’s very important to us. We do have some internal tooling in Kotlin, but we don’t have it in production yet. We basically want to have a way to build it at scale, so we’re definitely looking at that in the next few weeks or so.

DF: You said that it supports testing, like Android tests and unit tests. Do you know if the unit tests are faster? I would have a hard believing that the Android tests would run faster, because you’re waiting on espresso, online resources, and stuff like that. But what about the unit tests themselves?

GK: Unit tests are faster, purely because it can run multiple targets in a very interesting way. It runs multiple threads for different targets. Let’s say I change one single file somewhere in the codebase. Since it knows how that bubbles up, it’ll say, “Only these particular test cases have to be rerun,” and it’ll just run that subset of test targets, which makes it very fast.

KG: I have one question about Kotlin support, because I’m sure our listeners would want to ask this. You mentioned that Kotlin build times are significantly longer than what people want. Would Buck be able to solve that problem in some way, or is it something that’s inherently a problem with generating Java classes from Kotlin?

GK: Sure, I can talk a little bit about that. I forgot to mention one important detail: Buck does something called ABI (application binary interface) checking. Let’s say that you have two classes, A and B, in two different modules. B depends on A. Let’s say you change A. Gradle says, “Hey, A is changed, so B has to rebuild itself.” But if you changed a private method that doesn’t really affect anything in B, Buck won’t rebuild B. It will just rebuild A, and it’ll pack the text files together directly inside the APK.

When you go with Kotlin, the same thing would apply. For every JAR that comes out of a module, it creates this ABI JAR with all of the public APIs. Especially if you’re iterating on something really quick or you’re changing methods, it’s similar to instant turn on the command line, which can skip a significant portion of the build.

KG: How does that play into the Kotlin support? Does it apply the same logic to Kotlin?

GK: Yes, because all of these targets are actually generating JAR files at the end of the day. Buck doesn’t really care how the JAR file is materialized. It could be from Kotlin, Scala, Ruby, or Java. It doesn’t really matter.

KG: So because of the way incremental compilation is approached, you get its benefits across other languages. That’s pretty slick.

Okay, so all of this is amazing, but what’s the future for okBuck? What’s on the road map? You mentioned Kotlin support. That’s pretty cool. Are there any other things that you want to tell us about to get us even more excited?

GK: Well, at this stage, we have so many modules that the slowest part of our build is the time we actually take to run okBuck, because it’s still based in Gradle. So I just made some changes that basically make it run these things in parallel. We’re looking at incremental task execution.

At this point, we’re doing more polishing of our incremental build support. Some of our APIs are still not finalized, so we’re trying to make them stable. Also, we ought to have feature parity (at the end of the day) with whatever the Android Gradle plugin has. At this point, we’re at about 90%, but some features are still not supported. We’re slowly whittling down all the features that people are requesting and trying to make sure that it’s airtight.

At this point, Kotlin is the next big thing for us, along with making okBuck faster and scalable. People might be tempted to switch all the way to Buck, but then you’d lose access to the Gradle plugins that the community puts out. Square might put out a great Gradle plugin tomorrow, and you’d be thinking, “Oh man, I wish I was still using Gradle so I could use that.”

KG: Wow, that’s fantastic. It’s a lot to digest, but I’m super amped up now. For folks who are listening, what are some good resources to look at, and how do I get started?

GK: If you want to just admire how fast it is, there’s a test project that we put up to stress test Android Studio and Gradle. The funny thing is that it’s now used by the tools team and the Gradle guys to ignition test their performance on every release. It’s a very buggy project.

KG: So it’s a legit thing, not some convoluted project that just shows how Buck is doing better?

GK: It’s basically our project, but we removed all the code and obfuscated all the project names. It’s like an Uber project that has no actual information in it, just dependencies. That gives you an idea of how fast it is. There’s also the okBuck Github page, where you have the actual project; the DroidCon NYC talk; and a Buck Slack that Facebook maintains, where that people can hop in and get an understanding of what’s currently going on.

KG: Fantastic.

GK: I want to mention something else really quickly. There’s support for generating Bazel from okBuck now. The Bazel team in New York said, “Hey, this is a great project. Let’s try adding Bazel support”—so they actually did. The next version of okBuck will probably ship with Bazel, so you can compare all three systems side-by-side if you wanted to.

KG: Ooh, that’s very interesting. Gautam, thank you so much. I can’t wait to hop off this call and start exploring okBuck more.

DF: I have shiny object syndrome times 5.

GK: You should definitely try it out. It’s low investment. Just apply the plugin and see if it works out. There is no upfront cost to do the migration.

KG: Thank you so much for spending your time and sharing your wisdom on okBuck. I think this is going to be great. Our listeners will definitely appreciate it.

In fact, if they want to reach out to you, what’s an easy way to do that?

GK: I’m on Twitter (@kageiit). I’m not super active, but I do monitor it from time to time, so that might be a good way to reach out to me.

KG: All right, that makes sense. Donn, if folks want to reach out to you after you’ve tried Buck…

DF: The best way, again, is Twitter (@donnfelker), but the real question is: how do they get ahold of you, Kaushik?

KG: I’m @kaushikgopal on Twitter. If you give me some time, I’ll also try Buck, and that’s a good place for folks to reach out to me.