It has been quite a while since I've posted anything here - almost an entire
year, which doesn't normally happen. Things have been pretty busy, though. I got
married, changed jobs, and found plenty of trouble along the way. There was a
3-month period where life was happening and I did almost nothing in the open
source world, which is clearly visible on my
github profile activity stream.
However, in the last year, I have a few interesting things in technology that I
think are worth mentioning.
Oaf is the ruby framework I developed and
mentioned in my previous post.
Although it hasn't seen any major popularity, the tool has definitely been
helpful to me on a number of occasions where I wanted to slap down a mock API
really quick that actually performed some function. As an example, I used it to
prototype an orchestration REST API which wrapped fabric. I think I probably
wrote about 50 lines of bash scripts to make what I wanted happen. I definitely
don't regret the time I spent making Oaf,
especially with the continuous test lessons that I learned from it.
I played around with parallelizing YUM downloads in RHEL6 with a small plugin
called yum-rocket. It substituted
urllib in place of
URLGrabber to allow parallel downloads at the cost of two
things: single-file progress indication, and HTTP keepalive's. The progress
indication I really did not care about so much as I could see how many files had
downloaded / how many total downloads. The keepalive connections, however, were
an interesting thing to observe. Even though
yum-rocket could be downloading using
as many threads as possible, in some cases the original
performed better. I think it really came down to the mirrors which were selected
for the particular session. yum-rocket
would also pull from multiple source repositories, which in some cases would get
around QOS/bandwidth limitations enforced on a per-connection level from remote
hosts. At times this proved to be extremely helpful, especially in conjunction
yum-fastestmirror plugin, which would essentially allow you to
download from the "fastest-N" mirrors concurrently. There are definitely cases
where this hurt performance though, for instance if you have a local mirror with
everything you need but you have configured yum-rocket to span up to 5 mirrors,
you would probably get worse performance than just sticking to your one local
mirror. All in all, good learning experience in plugin systems for interpreted
This one was really me going off on a tangent with a little idea I had to
create slide decks in my terminal. At
the time, I worked for Cisco Systems, and while my role there didn't ordinarily
have me creating slide decks, I loathed powerpoint so much that I made a shell
script to display some text, pause, and advance to the next slide when I was
ready. I used it a few times at Cisco, and it worked well. Having some initial
success with it, I added things like horizontal rules, color text, in-slide
pause, and more. I presented at PuppetConf 2013 with
slide.sh, which went well, despite
spilling over into GitHub's timeslot. Sorry, dudes.
pakrat is a YUM repo snapshotter tool for
use with RHEL6 (almost exclusively).
For a while, Keith Chambers and I were
really into snapshotting package repositories as a way of version controlling an
entire operating system. RightScale was doing it pretty much how we wanted to do
it, so we made a ghetto prototype using a curl script with a few loops in it. It
worked surprisingly well, and even when we didn't check on it for a few months,
whenever we would come back to it, it was chugging along, doing it's thing. I
decided to make a more formal implementation, although I made a few mistakes in
doing so. My first mistake was insisting on using the existing YUM libraries.
This worked great on RHEL distributions, but anywhere outside of RHEL, it was
pretty much impossible to use. And now that RHEL6 and its version of the YUM
libraries will soon be obsolete, the library would make little sense to use
anywhere else. The other mistakes were more minor, things like the command line
syntax and whatnot. Nonetheless, pakrat
was another good learning experience and actually had a fairly decent progress
Since early this year, I have been helping
@hashicorp implement some features in their
peer discovery/orchestration framework known as
serf. Hashicorp (both
@mitchellh) are just awesome to work with. I
helped with a few things like the key rotation
feature, among other
The time I've spent working on serf and
learning from the team has definitely not been mis-spent.
While working on serf,
@armon and I discussed the idea of using "one-time
pads" as a way of ensuring perfect forward security during Serf's key exchange.
The final design ended up a little different, but the conversation led me to
create go-otp, which is a very small
reference implementation of the OTP concept and how it can be applied using
Also while working on serf, I cleaned up
some of the command-line output by writing a simple little library called
columnize for golang. It will take a
list of strings, and produce perfectly aligned columns of output without relying
on tabs (
\t). I've used it in a number of side projects and it is still
currently in use in both serf and
go-semver is a library providing easy
access to semantic versioning logic in golang. It allows you to parse version
numbers and compare them.
I spent about 6 months trying to work with Ubuntu as a base operating system, as
the product I was working on at the time was running OpenStack and the
enterprise support was best for Ubuntu. I really hated using
and found the standard tooling to be nauseating, but thankfully a smart guy
@smira realized the same and did something about it
with aptly. While young in its life, aptly
offers some impressive features, including repository snapshots and snapshot
merging. I worked briefly with @smira on aptly, and
eventually created a quick ruby wrapper around its cli, called
ruby-aptly. While not the most elegant
thing I've written, it made writing higher-level logic around how to compose our
release of debian packages much easier than it could have been.
While developing a new project in golang, I found it odd that POSIX basic
(non-ERE) regualr expressions were not implemented anywhere that I could find.
This didn't bother me until I wanted to provide a command-line interface that
allows passing a string glob in using the '*' character as a wildcard. There
were a few IO-related packages that had functionality close to what I wanted,
but no exact matches. I ended up implementing just the globbing part as part of
a new golang package called go-glob.
During development of another project, I found the need to scan license text for
different software packages and guess what license type the text described. I
also saw a need for a standardized set if string identifiers for license types
which just doesn't exist right now. I ended up created
go-license in golang to satisfy both
After making a career change and joining a team of engineers who think purely in
cloud concepts, I saw a fairly common pattern in some things we were doing that
could be standardized, so I spent a week over nights and weekends creating
huck, which is a pretty small ruby framework for tossing
messages into a queue and executing logic based on it from the remote end. I
debated open sourcing the framework to myself, as I know there are probably
better ways for some of huck's use cases to be implemented,
but nonetheless, I decided to realease it because a framework is better than
re-implementing something 5 different times using different sets of scripts that
are all doing something similar.
So all in all, summer 2013 to summer 2014 was a pretty okay year in code for me,
but I still think I can do better and hopefully can prove that during this year.
I have been hacking away on a new project which I plan to open source sometime
in the coming months, and there will likely be more small golang libraries to
support it as I develop more on it.