API-First Development and OpenAPI

I, like many others, got introduced the world of web application development through PHP. The world was a simpler time then: MVC was just a buzzword, Ruby on Rails was someone’s pet project, and JavaScript frameworks were little more than compatibility shims.

Web applications then were built similarly to desktop applications: A request would generate a thread, a thread would process the request, and along the way, a “view” (HTML) would be generated and sent to the client. Many applications were little more than procedural shell scripts that had interesting ways of concatenating strings. Some applications built dozens of layers of abstraction and called themselves Enterprise-Scale. It was beautiful. Until it wasn’t.

Most of these web applications, like so many Visual Basic 6 applications that came before them, became utterly long in the tooth. Systems would crumple under layer after endless layer of poor software architecture, system design, scale, technical debt, and technical rot. Furthermore, priceless amounts of valuable intellectual property — business rules, processes, workflows — were trapped under the weight of tightly coupled responsibilities and lack of any form of re-usability. Take a company like Facebook, who invested millions in R&D to build its own derivative of PHP and its entire runtime in order to make their existing codebase scale without a complete rewrite.

We’ve made a lot of advancements since this time. Obviously MVC and MVVM were a major help, but they were really only part of the solution. We could build better monolithic applications with re-usable libraries, but what if we wanted to build scalable architectures that could be split among many computing resources? And what if we wanted to support desktop, web, and mobile all with as much re-usability and centralized control as we can manage?

Eventually, correctly, we became an API driven world. SOAP and REST popularized the notion of distributed software API’s and how simple Request/Response protocols can power most every business process we could want. I’m finding it seems less and less commonplace for folks to choose to be “full-stack” developers for their entire careers. Most seem to end up finding their way into one of a few popular camps, e.g.: “services guy”, “UI guy”, “data guy”, etc. This is not necessarily because technology has become that much more complicated — I’ve had to write a web application in straight C and it was miserably complicated! Instead, it’s really that the expectations of what software should be able to do have outpaced how much simpler our layers of abstraction can make things.

Given a strict separation of concerns, it wasn’t hard to find teams that would inadvertently turn application development into a very waterfall process.

More and more we found ourselves needing to prevent this:

For many, API-First development was born.

First, know yourself.

I want to talk about OpenAPI, and I will get there, but first I want to talk about API’s. It’s easy to think of an API as a tool, a shuttle, a means to an end. Often times we rush in head-first with a shiny UI mock-up and fuzzy business requirements and start building API’s purposed to do that thing. That is probably OK sometimes. I generally think it’s worthwhile to take a step back and think about your API’s as the product. In today’s world of SaaS and point solutions, API’s are as much the product as the whizzbang UI and underlying data pool/lake/ocean. Just as we get that urge to produce ever shinier/cleaner buttons and a leaner/faster database or business process, we should look to build more and more beautiful API’s.

What is a beautiful API? Beauty is in the eye of the beholder, of course, but overall I like to judge beauty on a few key factors: consistency, simplicity, and technical soundness.

Consistency is the idea that one part of your API looks like every other part of your API. A common set of rules and patterns is applied so that my expectations can carry from one experience to another. As I learn and discover your team’s API’s, the knowledge that I gain easily carries over to other aspects of the API surface.

Simplicity is the idea that the API and its models “make sense”. Do your models and paths reflect the intention of your domain? Do they group information into ways that are easily understood and processed? Would a consumer be able to easily discover what they need and want? This is obviously extremely subjective, but again, so is beauty.

Technical soundness means that API not only solves the problems it sets out to solve (and well) but also does so in a way that conforms to norms established by the broader zeitgeist of the technical community. Examples of this include API’s correctly utilizing the common HTTP verbs, utilizing HTTP response codes, following normative standards such as REST or SOAP, best-practices, etc. Technical soundness can also represent how well the API matches up to what a consumer would actually need to do from a technical perspective. If the common action I need to perform minute-to-minute requires 1000’s of API requests, it’s probably not technically sound.

Once a team has soul-searched and identified how it wants its API’s to look and function, it is now ready for API first development.

API First Development

In a nutshell, API-first development simply means that the first, most waterfall aspect of application development is the design of the API’s. API’s are built to solve a specification. Instead of asking yourself how your entity diagrams and database are going to work, ask yourself how you, as a consumer, would want to work with data and perform behaviors to solve your business problems. Then, design your API to solve its problems to the best of its technical ability and be beautiful while doing it.

Once an API is “designed”, it becomes a codified specification. Your database guys can start working on their entity modeling, services guys can build their Netflix-scale messaging architectures, and the UI guys can start prototyping and simulating the existence of the services through Mocking.

Teams can begin to achieve staggered completion of their swim-lanes with less overall interaction and dependency. Feedback can be gathered iteratively, and the overall software development process can move forward faster. That’s the goal of API-first.

While this is all well and good, a new set of problems reared their head: collaboration and tooling.

A wild OpenAPI approaches!

OpenAPI got its start in 2011 as Swagger. In those days, API design was relegated to a variety of unstructured tools, most not well-suited to the task: Word documents, Visio, whiteboards, post-it notes, email threads, XML or even pseudo-code. Maintenance and extension were often difficult, some were darn-right expensive (Visio), and there was ultimately little in the way of tooling that could improve a team’s overall productivity. Some tools such as XSD came along over the years, but historians will account for the fact that they were complete garbage, objectively, full-stop, end of discussion.

While SOAP had its share of crappy tooling (XSD, SoapUI, etc.), REST was gaining in popularity for its leanness, simple access over HTTP, and general interoperability. REST, however, had no real tooling on its side, so a collaboration was a major pain-point. Furthermore, documenting API’s for others to easily consume and stay up-to-date was a task of human engineering. REST was never going to reach critical mass without a solution. Eventually, Swagger was born. Swagger is a JSON/YAML markup specification for documenting RESTful API surfaces. For many it’s the WSDL of REST, but with a lot less pain and horrible memories.

Directly from the Swagger website, Swagger is a specification format that can also be converted into, for example, auto-generated API documentation portals

Swagger has gone through its own storied history. It was eventually bought by SmartBear (who rightly did not want to be saddled with being The SOAP Company) and was fully open-sourced and its specification was opened up to a consortium. That specification was renamed to OpenAPI.

Life with OpenAPI

Over the years and across various versions, OpenAPI has given us more and more flexibility to express the needs and desires of our API’s. The core components of a REST API design are its a) paths and b) models. These components together form a schema or protocol that can express a nearly infinite set of possibilities.

OpenAPI offers up a myriad of features to ensure API’s are documented fully. Paths can specify versions and query tokens and parameters, security specifications (JWT/oAuth/basic/etc.), HTTP headers, multi-part request bodies, a dizzying array of response types, and some amazing modeling tools that reflect the needs of both static and dynamic type systems present in our backend services today.

Also included in the specifications are validations, metadata, enumerations, defaults, and even documented examples.

Models in OpenAPI largely represent types that you can define in JSON, with a couple of additions. Models are inherently interfaces and can be flexible compositions/extensions of other models. This allows us to model, to great extent, the varied flexibility of models we are likely to support in our API’s, whether we are using a strict static typing language like C# via ASP.NET or something much more dynamic like NestJS via TypeScript.

It’s all in the Tooling

Part of what makes OpenAPI amazing is its tooling. Its open-source nature has provided us with a boon of both completely free and paid offerings to make creating, maintaining, communicating and documenting REST API’s easier. Examples are endlessly abound.

There are tools that allow for creation, such as a personal favorite of mine, Stoplight Studio. Tooling can also reverse-engineer existing API’s into OpenAPI, scaffold client and server code, build testbeds and mock services, generate auto-documentation and sandbox portals, visualize model and API dependencies, and more. OpenAPI also empowers existing tools we are all familiar with such as Postman.

Stoplight Studio is like a graphical IDE for your REST API design. While this won’t win as many coolness points as editing YAML in vim, it can dramatically aid the learning curve and discovery of OpenAPI.

Getting back to API-First Development

We’ve talked about the benefits of separating application logic into a web services API layer. We’ve discussed the benefits of documenting our API’s in a structured format. Now I want to discuss how this actually fits into the team’s workflow.

Our team starts with a design phase. The design phase can begin with a fresh OpenAPI document, or — as often the case you are extending existing API’s — you can start with an existing OpenAPI document. Features or business requirements are discussed as a team and members will take responsibility to put those requirements into a draft for review.

Once in draft, the team reviews the work, matching it up to the product requirements and the agreed-upon perception of beauty. Our team’s workflow looks something like this:

By publishing these design artifacts, the concepts can be easily codified and shared. UI can build mocks, services can lay testbeds, QA can build test plans, etc.

Eventually, the API your team builds will need to be extended or modified. This leads to a conundrum: Where is the source of truth? Do we go back to those previous design artifacts and change them? One challenge teams face is that sometimes the implementation details change the API surface slightly. Or the fast pace of agile development introduces changes to the API that did not go through the typical design process. Either way, the problem is the same: your API got out of date with your design artifacts.

This is an important facet of where tooling comes into place. Earlier I mentioned that tools exist for OpenAPI that allow it to reverse-engineer existing API into OpenAPI documents. On the .NET Core side, we use Swashbuckle, as an example. To put it visually, the concept of the source of truth looks like this:

Stay Tuned

API-first development didn’t start with REST and it won’t end with REST either. gRPC defines its specifications in ProtoBuf. As more and more applications need a massive horizontal scale, messaging patterns are becoming more commonplace and right now documentation for messaging patterns is a very messy landscape. AsyncAPI hopes to change that by being the OpenAPI for messaging protocols. Just as UI/UX designers prototype their designs in Xd or Figma, I would encourage architects, product designers, and engineer-entrepreneurs to take pride in their API designs.

Hopefully, this article has given a little bit of insight into the importance of API design and how spending a little bit of time upfront can empower a more productive team.