TeaDSL: A Multi-language SDK Solution for All API Gateways

This article explores the crucial challenges related to OpenAPIs and introduces a tailored solution, TeaDSL, a multi-language SDK solution for all API gateways.

By Puling

In the field of cloud computing, OpenAPIs are unambiguously acknowledged as the most vital tools for developers. OpenAPI allows programmers to include instructions in their programming, like sending text messages, organizing resources, and managing permissions. It simply allows performing another OpenAPI call if one is not enough. As more companies are opting for open platforms for flexibility, OpenAPIs are increasingly favored as an indispensable part of the integration infrastructure. However, a question remains as to whether the OpenAPIs are really consumer-friendly. This article explores the crucial challenges related to OpenAPIs and introduces tailored solutions.

Background

Puling is an engineer at Alibaba Cloud. He is an ardent lover of open source technologies and is one of China's top experts active on GitHub. In his sixth year with the company, he considered quitting his job to join a startup. And while he was deliberating the decision and pondering his work, he felt that Alibaba Cloud had earned its reputation for providing excellent cloud computing services and products, which were popular choices among startups. But even so, since he had written many SDKs for Node.js, he knew that there was still room for improvement in Alibaba Cloud's developer experience. He decided, out of his passion for programming, that he would stay and try to improve the situation. He then joined Alibaba Cloud Open Platform and dedicated himself to the optimization of SDKs. TeaDSL is a patented solution developed by Puling and his team. The following sections share his thoughts on and understanding of using TeaDSL as a multi-language SDK solution.

Problems with OpenAPIs

An old practice for OpenAPI design was creating a service interface, sketching out the documentation, then leaving the OpenAPI to customers without further refinement. The provider would assume that the design part was complete, and not care about the user experience.

Figure 1. 1st-generation OpenAPIs are composed of interfaces and have ill-defined documentation.

Well, here are the problems with this approach. First of all, the vaguely defined parameters are barely tested and there is no guarantee that they will work. Secondly, in order to ensure the security of the OpenAPIs, we must implement authentication/authorization measures, for example, with digital signatures. The providers must ensure the uniqueness of each customer's signature, making it an error-prone process. Thirdly, providers must repackage the OpenAPIs for customers that write in other programming languages.

But that's not all. Sometimes after you've successfully tested your OpenAPIs, you may run into errors, like a connection timeout or wrong return value. As OpenAPIs are never going to be completely stable, you must prepare for the inevitable challenges during the versioning, like changing data structures or making incompatible changes.

Not only are the consumers displeased with the hard-to-use OpenAPIs, but sometimes the maintainers also get weary of following up. After you publish an OpenAPI, you will find that writing brief documentation will no longer suffice, since you must also provide detailed descriptions of each parameter and even of the signing algorithm to facilitate programming in other languages. But afterward, you may notice that the signing algorithm is only good for some consumers, and the majority still rely on you to create an SDK. So, you build and provide an SDK in the language that you are most familiar with.

Figure 2. 2nd-generation OpenAPIs can be implemented by SDKs, but this implementation is only supported by a few languages.

As your OpenAPIs are being consumed in more and more languages, you must cater to the need. In my case as a Java developer, not only do I need to write the SDK for Java, I must also provide SDKs in all the other languages. Otherwise, the customers will not be able to implement the intelligent code completion on IDEs.

In short, the application of OpenAPIs could be very complicated:

Good API documentation is the most basic requirement.
An SDK is required for better developer experience, and features like the detail encapsulation and IDE auto-completion must be implemented.
Documentation must come with code samples to demonstrate real-world use-cases.
CLI can also be provided to make writing Bash scripts easier.
Test cases are a must for practicing continuous integration and performance assurance.

To think that the providers must not only satisfy all these needs but also do so in multiple languages, it’s critical to understand why the design process encompasses so much detail and requires hours of redundant labor work. In addition, it’s imperative to maintain consistency throughout, since even a nuanced deviancy could lead to changes further downstream, and introduce problems with the customers.

Figure 3. As the number of consumers increases, OpenAPI providers need to develop a more comprehensive set of tools and offer support for more programming languages.

To create a simplified experience, an API gateway might be used to solve these problems and provide solutions for signature validation, throttling, and the generation of SDKs and documentation.

But as there were too many API gateways around, let alone the different types, signature algorithms, and serialization formats, the workload for completing these tasks increases exponentially.

Figure 4. Multiple kinds of OpenAPIs and API gateways are used as the platform of a company grows.

If problems like OpenAPI/SDK inconsistencies, documentation errors, or demo data errors occur, it is because the work is simply too trivial, and not too difficult. Just as we learned from our own experience, many repetitions of even the simplest task will become challenging in the end.

How TeaDSL Works

TeaDSL is a domain-specific language (DSL) designed by the Alibaba Cloud Open Platform SDK team. It is applied in the following scenarios:

It is used as an intermediate language that supports different kinds of API gateways. It supports the description of different types of OpenAPIs in an API gateway in a uniform fashion.
It is translated to generate code in other languages. In other words, it allows the generation of SDKs in different languages with unified intermediate representation.
We can consider a collection of OpenAPIs as a library on account of the intermediate representation, and then write code samples on this basis to achieve the unified generation of code samples for multiple languages.

Therefore, the core capability of TeaDSL is to describe OpenAPIs with an intermediate syntax, which is much similar to a programming language, and realize the integration of SDK creation, OpenAPI description, and code sample generation in a variety of programming languages.

Before TeaDSL, we had to develop independent workflows to create SDKs for OpenAPIs in different API gateways, which meant that we had to undergo repetitive process redesigns for different API gateways.

Figure 5. The workload for providing SDKs in N programming languages for M gateways is M * N.

With TeaDSL acting as an intermediate layer, workloads are converged and made efficient as SDK providers can now focus on the transformations from OpenAPIs in the gateways to TeaDSL and the final generation.

Figure 6. The workload becomes M + N with the aid of TeaDSL.

As shown in the preceding diagrams, TeaDSL reduced the workload from M * N to M + N, meaning that it could raise the efficiency to a greater degree for a higher volume workload.

TeaDSL can be used to create an OpenAPI architecture from scratch with support for the OpenAPI code sample and test case generation.

The following section describes how TeaDSL handles OpenAPIs and API gateways of different styles and different programming languages.

TeaDSL and API Gateways

Different API gateways or OpenAPIs of different products may have very dissimilar styles. One could even say that the OpenAPIs of each style have a particular metadata format. The preferred approach, in this case, is to adopt standardized formats for OpenAPI descriptions, for example, Swagger. Built around the OpenAPI Specification, Swagger is backed by a large ecosystem of tools and has become the industry standard for RESTful OpenAPI design.

However, not all existing OpenAPIs are RESTful, which means that if an OpenAPI is not RESTful, you cannot use Swagger tools to document or generate an SDK for the OpenAPI.

In order to rectify this situation, we have designed a two-step operation:

Establish a new set of standards applicable to all OpenAPI styles.
Develop a tooling ecosystem based on these standards to expand the functionality.

With these two steps, we would be able to utilize powerful tools for each and every OpenAPI, whether RESTful or not, with no need for migration.

Design of New Standards

We found that OpenAPIs are always based on the HTTP protocol stack, no matter what parameters are present in the endpoint and regardless of what protocol is used to transport data, be it JSON, XML, or even a custom protocol. The central theme remains the same: HTTP. Therefore, we have designed a rudimentary data model as follows:

{
  protocol: string, // http or https
  port: number,         // tcp port
  host: string,         // domain
  request: {
    method: string, // http method
    pathname: string, // path name
    query: map[string]string, // query string
    headers: map[string]string,  // request headers
    body: readable      // request body
  },
  response: {
    statusCode: number, // http method
    statusMessage: string, // path name
    headers: map[string]string,  // response headers
    body: readable      // response body
  },
}

One can compare OpenAPIs to buildings. Their styles may vary, yet all OpenAPIs have a nearly identical composition. The only things that differentiate one from another are the handlings and permutations of compositional elements. The essential differences between OpenAPI styles are distinct serialization processes. We separated the serialization process from the data model and extracted the data structure.

For example, the following data model is more easily visualized:

model User {
  username: string,
  age: number
}

The data to be transmitted to different gateways can be JSON-formatted or XML-formatted, but either way, they are both readable streams of byte data.

toJSON(user: User): string
toXML(user: User): string

The result is as follows:

__request.body = toJSON(user);
__request.body = toXML(user);

Then, we wrapped an OpenAPI request/response pair to a method:

api getUser(username: string): User {
  __request.method = 'GET';
  __request.pathname = `/users/${username}`;
  __request.headers = {
    host = 'hostname',
  };
} returns {
  var body = readAsJSON(__response.body);
  return body;
}

Although the preceding code cannot be executed, it demonstrates that different gateways and styles will be supported, which is implemented as follows:

The request/response protocol, or HTTP, is used as the core model.
Several methods, such as toJSON, toXML, and readAsJSON, are introduced to separate the data structure from the serialization process.
The process is wrapped into a method.

These methods have different implementations in different programming languages, but we ensure consistency with a uniform signature:

function toXML(data: $Model): string;
function toJSON(data: $Model): string;

The above explanation shows how TeaDSL supports different gateways and OpenAPI styles. In this abstract process, different gateways are handled similarly with methods, only the data structure is presented.

TeaDSL and Programming Languages

Using one description format to describe OpenAPIs is only half the work. TeaDSL still needs to take the description language as a single source and generate output in different languages. We used to take the template-based code generation as the means of multilingual support. However, this still has some drawbacks:

Easy as it seems, this method results in inflexible and hard-to-maintain code.
The generated code is prone to naming conflicts and syntax errors.

As mentioned earlier, we've revamped this approach into using a DSL that has lexical, syntax, and semantics rules and runs a series of validations before generating code in the target language. These functions of DSL are not available in templates.

In general, DSLs may not enjoy wide applicability, but some DSLs have been steadily gaining popularity among front-end engineers, such as CoffeeScript, Babel, JSX, and TypeScript. For this reason, we have studied the design of different programming languages and devised our own custom syntax. We also draw on the translation mechanism of compilers and implemented multi-language support with a single model.

The following diagram illustrates the phases of TeaDSL:

The following are the three major application scenarios of TeaDSL:

Create SDKs in multiple languages.
Create OpenAPI-related code samples in multiple languages.
Create OpenAPI-related test cases in multiple languages.

Use TeaDSL to generate output to different target scenarios through strong validation of the intermediate language to deliver full language support. This saves time and effort for maintainers as they no longer need to spend hours writing code samples for different languages. TeaDSL will be automatically updated with the latest language support features, with no need for additional programming.

Summary

The main features of TeaDSL are that it supports OpenAPIs of different styles and the generation of SDKs and code samples in multiple languages. The ultimate goal is to achieve consistency in all OpenAPI usage scenarios, such as definition, documentation, SDK generation, and CLI support. The uniform, professional, and consistent performance of TeaDSL significantly alleviates the workload of OpenAPI providers and reduces the frequency of human error.

Check out the Alibaba Cloud SDKs using TeaDSL. On Alibaba Cloud Open Platform, we are extending TeaDSL through integration with more accessible tools to build a more compatible ecosystem than Swagger.

Community

TeaDSL: A Multi-language SDK Solution for All API Gateways

Background

Problems with OpenAPIs

How TeaDSL Works

TeaDSL and API Gateways

Design of New Standards

TeaDSL and Programming Languages

Summary

JacksonTian

You may also like

Comments

JacksonTian

Related Products

OpenAPI Explorer

API Gateway

EMAS HTTPDNS

China Gateway Solution