IDRing

A major feature of the Romana Project is topology-aware IPAM, and an integral part of it is the ability to assign consecutive IP addresses in a block (and reuse freed up addresses, starting with the minimal).

Since IPv4 addresses are essentially 32-bit uint, the problem is basically that of maintaining a sequence of uints, while allowing reuse.

To that end, a data structure called IDRing was developed. I’ll describe it here. It is not yet factored out into a separate project, but as Romana is under Apache 2.0 license, it can still be reused.

  • The IDRing structure is constructed with a NewIDRing() method, that provides the lower and upper bound (inclusive) of the interval from which to give out IDs. For example, specifying 5 and 10 will allow one to generate IDs 5,6,7,8,9,10 and then return errors because the interval is exhausted. 

    Optionally, a locker can be provided to ensure that all operations on the structure are synchronized. If nil is specified, then synchronization is the responsibility of the user.

  • To get a new ID, call GetID() method. The new ID is guaranteed to be the smallest available.
  • When an ID is no longer needed (in our use case — when an IP is deallocated), call ReclaimID().
  • A useful method is Invert(): it returns an IDRing whose available IDs are the ones that are allocated in the current one, and whose allocated ones are the available ones in the current one. In other words, a reverse of an IDRing with min 1, max 10 and taken IDs from 4 to 8 inclusively is an IDRing with the following
    available ranges: [1,3], [9,10].
  • You can see examples of the usage in the test code and in actual IP allocation logic.
  • Persisting it is as easy as using the locker correctly and just encoding the structure to/decoding it from JSON.

Content assist

It looks like you are researching razors. I think you are about to go off on a yak-shaving endeavor, and I cannot let you do that, Dave.

What I would really like my DWIM
agent to do. That, and to stop calling me Dave.

Being lazy and impatient, I like an idea of an IDE. The ease of things like autocompletion, refactoring, code search, and graphical debugging with evaluation are, for the lack of a better word, are good.

I like Eclipse in particular — force of habit/finger memory; after all, neurons that pray together stay together. Just like all happy families are alike, all emacs users remember the key sequence to GTFO vi (:q!) and all vi users remember the same thing for emacs (C-x C-c n) – so they can get into their favorite editor and not have to “remember”.

So, recently I thought that it would be good for a a particular DSL I am using to have an auto-completion feature (because why should I remember ). So I thought, great, I’ll maybe write an Eclipse plugin for that… Because, hey, I’ve made one before, how bad could it be?

Well, obviously I would only be solving the problem for Eclipse users of the DSL in question. And I have a suspicion I am pretty much the only one in that group. Moreover, even I would like to use some other text editor occasionally, and get the same benefit.

It seems obvious that it should be a separation of concerns, so to speak:

  • Provider-side: A language/platform may expose a service for context-based auto-completion, and
  • Consumer-side: An editor or shell may have a plugin system exposed to take advantage of this.

Then a little gluing is all that is required. (OK, I don’t like the “provider/consumer” terminology, but I cannot come up with anything better — I almost named them “supply-side” and “demand-side” but it evokes too much association with AdTech that it’s even worse).

And indeed, there are already examples of this.

There is a focus on an IDE paradigm of using external programs for building, code completion, and any others sorts of language semantic functionality. Most of MelnormeEclipse infrastructure is UI infrastructure, the core of a concrete IDE’s engine functionality is usually driven by language-specific external programs. (This is not a requirement though — using internal tools is easily supported as well).

  • Atom defines its own API

And so I thought – wouldn’t it be good to standardize on some sort of interaction between the two in a more generic way?

And just as I thought this, I learned that the effort already exists: Language-server protocol by Microsoft.

I actually like it when an idea is validated and someone else is doing the hard work of making an OSS project out of it…

Rethinking data gravity

At some point I remember having a short chat with Werner Voegels about taking spot instances to extreme in a genuine market in which compute power can be traded. His response was “what about data gravity?” to which my counter was — but by making data transfer into S3 free (and, later, making true the adage about not underestimating the bandwidth of a truck full of tape) you, while understanding the gravity idea, also provide incentives to not make it an issue. As in — why don’t I make things redundant? Why don’t I just push data to multiple S3 regions and have my compute follow the sun in terms of cost? Sure, it doesn’t work on huge scale, but it just may work perfectly fine on some medium scale, and this is what we’ve used for implementing our DMP at OpenDSP.

Later on, I sort of dabbled in something in the arbitrage of cost space. I still think compute cost arbitrage will be a thing; 6fusion did some interesting work there; ClusterK got acquired by Amazon for their ability to save cost even when running heavy data-gravity workload such as EMR, and ultimately isn’t compute arbitrage just an arbitrage of electricity? But I digress. Or do I? Oh yes.

In a way, this is not really anything new — it is just another way to surface the same idea that