romana

Cloud-native networking. A project of Manifest Networks

Swagger as you Go

Gregory Golberg romana , , , ,

As someone who offers a REST API, we at Romana project wanted to provide documentation on it via Swagger. But writing out JSON files by hand seemed not just tedious (and thus error-prone), but also likely to result in an outdated documentation down the road.

Why not automate this process, sort of like godoc? Here I walk you through an initial attempt at doing that — a Go program that takes as input Go-based REST service code and outputs Swagger YAML files.

It is not yet available as a separate project, so I will go over the code in the main Romana code base.

NOTE: This approach assumes the services are based on the Romana-specific REST layer on top of Negroni and Gorilla MUX. But a similar approach can be taken without Romana-specific code — or Romana approach can be adopted.

The entry point of the doc tool for generating the Swagger docs is in doc.go.

The first step is to run Analyzer on the entire repository. The Analyzer:

  1. Walks through all directories and tries to import each one (using Import()). If import is unsuccessful, skip this step. If successful, it is a package.
  2. For each *.go and *.cgo file in the package, parse the file into its AST
  3. Using the above, compute docs and collect godocs for all methods

The next step is to run the Swaggerer — the Swagger YAML generator.

At the moment we have 6 REST services to generate documentation for. For now, I’ll explicitly name them in main code. This is the only hardcoded/non-introspectable part here.

From here we, for each service, Initialize Swaggerer and call its Process() method, which will, in turn, call the main workhorse, the getPaths() method, which will, for each Route:

  1. Get its Method and Pattern — e.g., “POST /addresses”
  2. Get the Godoc string of the Handler (from the Godocs we collected in the previous step)
  3. Figure out documentation for the body, if any. For that, we use reflection on the MakeMessage field — the mechanism we used for sort-of-strong-dynamic typing when consuming data, as seen in a prior installment on this topic.

This is pretty much it.

As an example, here is Swagger YAML for the Policy service.

See also

Building on Gorilla Mux and Negroni

Gregory Golberg romana , , , ,

The combination of Negroni and Gorilla MUX is a useful combination for buildng REST applications. However, there are some features I felt were necessary to be built on top. This has not been made into a separate project, and in fact I doubt that it needs to — the world doesn’t need more “frameworks” to add to the paralysis of choice. I think it would be better, instead, to go over some of these that may be of use, so I’ll do that here and in further blog entries.

This was borne out of some real cases at Romana; here I’ll show examples of some of these features in a real project.

Overview

At the core of it all is a Service interface, representing a REST service. It has an Initialize() method that would initialize a Negroni, add some middleware (we will see below) and set up Gorilla Mux Router using its Routes. Overall this part is a very thin layer on top of Negroni and Gorilla and can be easily seen from the above-linked source files. But there are some nice little features worth explaining in detail below.

In the below, we assume that the notions of Gorilla’s Routes and Handlers are understood.

Sort-of-strong-dynamic typing when consuming data


While we have to define our route handlers as taking interface{} as input, nonetheless, it would be nice if the handler received a struct it expects so it can cast it to the proper one and proceed, instead of each handler parsing the provided JSON payload.

To that end, we introduce a MakeMessage field in Route. As its godoc says, “This should return a POINTER to an instance which this route expects as an input”, but let’s illustrate what it means if it is confusing.

Let us consider a route handling an IP allocation request. The input it needs is an IPAMAddressRequest, and so we set its MakeMessage field to a function returning that, as in


func() interface{} {
    return &api.IPAMAddressRequest{}
}

Looks convoluted, right? But here is what happens:

The handlers of routes are not called directly, they are wrapped by wrapHandler() method, which will:

  1. Call the above func() to create the pointer to IPAMAddressRequest
  2. Use the unmarshaller (based on Content-Type) to unmarshal the body of the request into the above struct pointer.

Voila!

Rapid prototyping with hooks

Sometimes we prototype features outside of the Go-bases server — as we may be calling out to various CLI utilities (iptables, kubectl, etc), it is easier to first ensure the calls work as CLI or shell scripts, and iterate there. But for some demonstration/QA purposes we still would like to have this functionality available via a REST call to the main Romana service. Enter the hooks functionality.

A Route can specify a Hook field, which is a structure that defines:

  • Which executable to run
  • Whether to run it before or after the Route‘s Handler (When field)
  • Optionally, a field where the hook’s output will be written (which can then be examined by the Handler if the When field is “before”). If not specified, the output will just be logged.

Hooks are specified in the Romana config, and are set up during service initialization (this is in keeping with the idea that no Go code needs to be modified for this prototyping exercise).

Then during a request, they are executed by the wrapHandler() method (it has been described above).

That’s it! And it allows for doing some work outside the server to et it right, and only then bother about adding the functionality into the server’s code.

If this doesn’t seem like that much, wait for further installments. There are several more useful features to come. This just sets the stage.

IDRing

Gregory Golberg romana , , , , ,

A major feature of the Romana Project is topology-aware IPAM, and an integral part of it is the ability to assign consecutive IP addresses in a block (and reuse freed up addresses, starting with the minimal).

Since IPv4 addresses are essentially 32-bit uint, the problem is basically that of maintaining a sequence of uints, while allowing reuse.

To that end, a data structure called IDRing was developed. I’ll describe it here. It is not yet factored out into a separate project, but as Romana is under Apache 2.0 license, it can still be reused.

  • The IDRing structure is constructed with a NewIDRing() method, that provides the lower and upper bound (inclusive) of the interval from which to give out IDs. For example, specifying 5 and 10 will allow one to generate IDs 5,6,7,8,9,10 and then return errors because the interval is exhausted. 

    Optionally, a locker can be provided to ensure that all operations on the structure are synchronized. If nil is specified, then synchronization is the responsibility of the user.

  • To get a new ID, call GetID() method. The new ID is guaranteed to be the smallest available.
  • When an ID is no longer needed (in our use case — when an IP is deallocated), call ReclaimID().
  • A useful method is Invert(): it returns an IDRing whose available IDs are the ones that are allocated in the current one, and whose allocated ones are the available ones in the current one. In other words, a reverse of an IDRing with min 1, max 10 and taken IDs from 4 to 8 inclusively is an IDRing with the following
    available ranges: [1,3], [9,10].
  • You can see examples of the usage in the test code and in actual IP allocation logic.
  • Persisting it is as easy as using the locker correctly and just encoding the structure to/decoding it from JSON.

Kabuta: GoClipse meets Delve

Gregory Golberg romana , , , , ,

I am addicted to having good development tools, specifically, I like my debuggers visual. Once I was so annoyed by inability to debug a common use case of calling a PL/SQL stored procedure from Java, I made it into a single-stack cross-language debugger project, for example.

And I’m also a fan of Eclipse (well, see above).

So when Romana went with Go (pun definitely intended) as our language of choice, I quickly found GoClipse, and it was nice.

But then I learned about Delve and I wanted to have the best of both worlds — the functionality of Delve and the graphics of GoClipse.

So here’s an idea… Eclipse (from which GoClipse is, obviously, derived) understands GDB/MI, and can represent it visually. So all that remains is to adapt Delve’s API to GDB/MI.

And so I started it as a Kabuta project. Let’s hope I don’t abandon it…