Today’s yak shaving

So I wanted to catch up on Go. I like my IDEs, so I got my Go 1.19.2, my GoClipse 0.16.1, and off to remember how to ride this bicycle again. Or rather ride this yak, because…

I like IDEs mostly for two things: interactive, graphical debugging (for that see the P.S. section) and content assist. And we have a problem:
The solution seems to be to install a fork of gocode. Ok, I got content assist now. On to trying the next sine qua non of IDE, interactive debugger. Wait… There’s no GDB on M1? Yikes.

For debugger, sure, I could use LLDB, as suggested, or, even better, Delve, but I want it in the IDE (how many times do I have to say that, in 2022?) A few GUI frontends exist, but I am partial to Eclipse platform (I can be as picky with my tools as any tradesman).

Is it time to dust off Kabuta, an abandoned attempt to adapt Delve API to GDB/MI interface that GoClipse speaks? Ok why not, let’s go get that sucker… Done.

But how do I get it into GoClipse? A StackOverflow post suggests it is done via “New Project wizard”, but not so fast, Mr. Yak Jockey:
Ok, let’s hack around it but first let’s file an issue on GoClipse… Wait, say it ain’t so… GoClipse is no longer maintained? My world collapses. To Bruno’s point that:

I’ve seen the issues Eclipse has, both external (functionality, UI design, bugs), and internal (code debt, shortage of manpower). I was holding on to the hope it would recover, but now I don’t see much of a future for Eclipse other than a legacy existence, and I realize exiting was long overdue.

I just a few months ago got fed up with how much of a torture it is to step through the code in IDEA (and no waving of dead chickens seems to help) and went back to Eclipse for Java.
Do I fork GoClipse?
P.S. The workaround for the import is to use default ~/go as GOPATH. In other words, using /Users/gregory/src/github.com/debedb/kabuta worked fine:

Swagger as you Go

As someone who offers a REST API, we at Romana project wanted to provide documentation on it via Swagger. But writing out JSON files by hand seemed not just tedious (and thus error-prone), but also likely to result in an outdated documentation down the road.

Why not automate this process, sort of like godoc? Here I walk you through an initial attempt at doing that — a Go program that takes as input Go-based REST service code and outputs Swagger YAML files.

It is not yet available as a separate project, so I will go over the code in the main Romana code base.

NOTE: This approach assumes the services are based on the Romana-specific REST layer on top of Negroni and Gorilla MUX. But a similar approach can be taken without Romana-specific code — or Romana approach can be adopted.

The entry point of the doc tool for generating the Swagger docs is in doc.go.

The first step is to run Analyzer on the entire repository. The Analyzer:

  1. Walks through all directories and tries to import each one (using Import()). If import is unsuccessful, skip this step. If successful, it is a package.
  2. For each *.go and *.cgo file in the package, parse the file into its AST
  3. Using the above, compute docs and collect godocs for all methods

The next step is to run the Swaggerer — the Swagger YAML generator.

At the moment we have 6 REST services to generate documentation for. For now, I’ll explicitly name them in main code. This is the only hardcoded/non-introspectable part here.

From here we, for each service, Initialize Swaggerer and call its Process() method, which will, in turn, call the main workhorse, the getPaths() method, which will, for each Route:

  1. Get its Method and Pattern — e.g., “POST /addresses”
  2. Get the Godoc string of the Handler (from the Godocs we collected in the previous step)
  3. Figure out documentation for the body, if any. For that, we use reflection on the MakeMessage field — the mechanism we used for sort-of-strong-dynamic typing when consuming data, as seen in a prior installment on this topic.

This is pretty much it.

As an example, here is Swagger YAML for the Policy service.

See also

Building on Gorilla Mux and Negroni

The combination of Negroni and Gorilla MUX is a useful combination for buildng REST applications. However, there are some features I felt were necessary to be built on top. This has not been made into a separate project, and in fact I doubt that it needs to — the world doesn’t need more “frameworks” to add to the paralysis of choice. I think it would be better, instead, to go over some of these that may be of use, so I’ll do that here and in further blog entries.

This was borne out of some real cases at Romana; here I’ll show examples of some of these features in a real project.

Overview

At the core of it all is a Service interface, representing a REST service. It has an Initialize() method that would initialize a Negroni, add some middleware (we will see below) and set up Gorilla Mux Router using its Routes. Overall this part is a very thin layer on top of Negroni and Gorilla and can be easily seen from the above-linked source files. But there are some nice little features worth explaining in detail below.

In the below, we assume that the notions of Gorilla’s Routes and Handlers are understood.

Sort-of-strong-dynamic typing when consuming data


While we have to define our route handlers as taking interface{} as input, nonetheless, it would be nice if the handler received a struct it expects so it can cast it to the proper one and proceed, instead of each handler parsing the provided JSON payload.

To that end, we introduce a MakeMessage field in Route. As its godoc says, “This should return a POINTER to an instance which this route expects as an input”, but let’s illustrate what it means if it is confusing.

Let us consider a route handling an IP allocation request. The input it needs is an IPAMAddressRequest, and so we set its MakeMessage field to a function returning that, as in


func() interface{} {
    return &api.IPAMAddressRequest{}
}

Looks convoluted, right? But here is what happens:

The handlers of routes are not called directly, they are wrapped by wrapHandler() method, which will:

  1. Call the above func() to create the pointer to IPAMAddressRequest
  2. Use the unmarshaller (based on Content-Type) to unmarshal the body of the request into the above struct pointer.

Voila!

Rapid prototyping with hooks

Sometimes we prototype features outside of the Go-bases server — as we may be calling out to various CLI utilities (iptables, kubectl, etc), it is easier to first ensure the calls work as CLI or shell scripts, and iterate there. But for some demonstration/QA purposes we still would like to have this functionality available via a REST call to the main Romana service. Enter the hooks functionality.

A Route can specify a Hook field, which is a structure that defines:

  • Which executable to run
  • Whether to run it before or after the Route‘s Handler (When field)
  • Optionally, a field where the hook’s output will be written (which can then be examined by the Handler if the When field is “before”). If not specified, the output will just be logged.

Hooks are specified in the Romana config, and are set up during service initialization (this is in keeping with the idea that no Go code needs to be modified for this prototyping exercise).

Then during a request, they are executed by the wrapHandler() method (it has been described above).

That’s it! And it allows for doing some work outside the server to et it right, and only then bother about adding the functionality into the server’s code.

If this doesn’t seem like that much, wait for further installments. There are several more useful features to come. This just sets the stage.

IDRing

A major feature of the Romana Project is topology-aware IPAM, and an integral part of it is the ability to assign consecutive IP addresses in a block (and reuse freed up addresses, starting with the minimal).

Since IPv4 addresses are essentially 32-bit uint, the problem is basically that of maintaining a sequence of uints, while allowing reuse.

To that end, a data structure called IDRing was developed. I’ll describe it here. It is not yet factored out into a separate project, but as Romana is under Apache 2.0 license, it can still be reused.

  • The IDRing structure is constructed with a NewIDRing() method, that provides the lower and upper bound (inclusive) of the interval from which to give out IDs. For example, specifying 5 and 10 will allow one to generate IDs 5,6,7,8,9,10 and then return errors because the interval is exhausted. 

    Optionally, a locker can be provided to ensure that all operations on the structure are synchronized. If nil is specified, then synchronization is the responsibility of the user.

  • To get a new ID, call GetID() method. The new ID is guaranteed to be the smallest available.
  • When an ID is no longer needed (in our use case — when an IP is deallocated), call ReclaimID().
  • A useful method is Invert(): it returns an IDRing whose available IDs are the ones that are allocated in the current one, and whose allocated ones are the available ones in the current one. In other words, a reverse of an IDRing with min 1, max 10 and taken IDs from 4 to 8 inclusively is an IDRing with the following
    available ranges: [1,3], [9,10].
  • You can see examples of the usage in the test code and in actual IP allocation logic.
  • Persisting it is as easy as using the locker correctly and just encoding the structure to/decoding it from JSON.

Kabuta: GoClipse meets Delve

I am addicted to having good development tools, specifically, I like my debuggers visual. Once I was so annoyed by inability to debug a common use case of calling a PL/SQL stored procedure from Java, I made it into a single-stack cross-language debugger project, for example.

And I’m also a fan of Eclipse (well, see above).

So when Romana went with Go (pun definitely intended) as our language of choice, I quickly found GoClipse, and it was nice.

But then I learned about Delve and I wanted to have the best of both worlds — the functionality of Delve and the graphics of GoClipse.

So here’s an idea… Eclipse (from which GoClipse is, obviously, derived) understands GDB/MI, and can represent it visually. So all that remains is to adapt Delve’s API to GDB/MI.

And so I started it as a Kabuta project. Let’s hope I don’t abandon it…