Stellar - A library for API Programming

Stellar is a set of Idris 2 libraries for building interactive programs by programming declaratively with APIs using dependent types. API-programming a new programming paradigm where the unit of program manipulation isn’t an object, a function, or a type, but an API.

When working with complex software involving multiple APIs it is easy to lose track of how the different systems interact. Stellar alleviates this by building programs as a pipeline of API-transformers that translate between layers of APIs of different kinds.

Every API is a Request/Response Pair

Every API in Stellar is described by the same data structure:

record API where
    request : Type
    response : request -> Type

The request and the response are types that describe what counts as a valid value for a given API. This single definition a surprising range of systems:

• HTTP — requests are HTTP requests, responses are HTTP responses.
• A command line interface — requests are List String, responses are String.
• A database — requests are queries, responses are tables, confirmations or errors.

Because all of these share the same request/response building block, we can combine complex programs out of API-transformers.

Mapping across APIs with API-transformers

Say you want to write an HTTP server for a web service. You are dealing with at least three interfaces:

• HTTP itself
• The parsed endpoints of the server
• A database storing your data

Stellar lets you write the server as a pipeline:

-- HTTP ~> API ~> Database
app = router |> queries |> execDB

router is an API transformer translating bare HTTP requests into API endpoints, queries turns each endpoint into a database query, and execDB runs database queries and provides a result.

The database result is then translated back into endpoint responses and, in turn, encoded in HTTP responses.

This example is pseudocode, simplified for the purpose of explanation. The full story is theoretically involved but explains with absolute precision what is happening and why. For the theoretical explanation, see the container-compendium

What are API-transformers Concretely

An API-transformer is a type that is parameterised over two APIs. The most natural way of interpreting it is as a pair of methods, one to convert the “request” part of the APIs and one to convert the “response”.

At a high level, if we have two APIs A1 and A2 with their own request/response which we call A1.request, A1.response, A1.request, and A2.response, then an API-transformer must convert from A1.request to A2.request, and also convert back from responses of A2.response to responses A1.response.

Conceptually, it can help to think of A1 as the “high-level, main API” and A2 as the “lower-level, delegated API”. In this way, an API transformer delegates tasks to a lower-level api dealing with more details, from an API that is maybe more abstract, or closer to the user than the bytes.

There is a nice illustration we can draw from this structure, by laying our request/response pairs vertically, and drawing the maps that make up the API-transformer we see that mapping the request goes forward but mapping the response goes backward:

The corresponding type in Idris is a record with two fields:

record (=&>) (source, target : API) where
  mapRequest : source.request -> target.request
  mapResponse : (orig : source.request) ->
                target.response (mapRequest orig) -> source.response org

Mapping requests is straightforward, but mapping responses is a bit more involved. Let’s first look a the last line and ignore the arguments, we have target.response -> source.response . This is what converts responses from the delegated API to the main API. We notice that this type depends on orig, stating that any reinterpretation of a response must be done with regards to a fixed request. Finally, we notice that target.response takes mapRequest orig in argument ensuring that the response we obtain comes from the message we translated. This removes any possible doubt as to the origin of a response. A response is always refering to a previous message, and we keep track of any translation we have done previously to obtain that response.

This covers the main definition of API and API-transformer, the actual implementation is different for performance reasons, but they are equivalent.

API substrates

When you are designing your application, you assume both client and server use the same API directly. But the reality of deploying both programs assumes that they communicate via a common protocol through which you serialise and deserialise the data that makes up your API. We call this third protocol the substrate.

The substrate explains what happens in the real world: each party will lose structural information in order to communicate via the common protocol. But the data is present while designing your system. This guarantees that both client and server do not go out of sync and this synchronisation can be typechecked and versionned before you perform any deployment, detecting mismatches ahead of time. Whenever we are deploying we are relying on the common substrate which cannot detect API mismatches, only protocol errors

Stellar currently provides only two substrates, CLI and HTTP, but more are on the way.

Modules

Stellar is split into focused packages, each sharing the same API abstraction. Browse the source and per-module README on GitLab:

Getting the source

Everything lives in a single repository:

git clone https://gitlab.com/avidela/stellar.git
cd stellar

The project is built with pack. For example, to build the API package:

cd api && pack build stellar-api.ipkg

A note on usability

This repository is a prototype of an idea that can blossom into the future of programming. Current tools aren’t quite good enough to provide the optimal user experience we can expect from programming at the level of APIs, but we are working on it. This project is not the final form of the API-programing idea, but a prototype to validate the idea. We’re developping the technology to make API-programming a completely natural part way of programming at glaive.