Sep 29, 2014 - nicstat Patched for EC2

Get the Source

Get the patched nicstat utility here. It has several bugs fixed, of interest to anyone running Linux, whether in EC2 or on physical hardware.

The Whole Story

I recently learned about a useful performance monitoring utility that I wasn’t aware of - nicstat. Somewhat coincidentally, I also discovered that my coworker Brendan Gregg was coauthor of this utility for Solaris back when he worked at Sun.

nicstat is modeled very much after iostat, vmstat, and similar tools. It works great, but I found network utilization (%Util column) was not being calculated correctly in EC2. This was no surprise, as a xen guest has no idea about AWS imposed bandwidth throttles, but I thought it was strange that nicstat reported an interface speed of 1.410Gbps when ethtool reports a 10Gbps interface for c3.4xl - the instance type I was running this on.

Strange, but since nicstat has a -S option where the bandwidth can be overridden, no problem. So I tried that - using measured bandwidth from iperf, but that didn’t work either and nicstat kept reporting a 1.410Gbps rate.

After perusing bugs, I found the cause for the strange 1.410Gbps value - on 10Gbps interfaces there is an integer overflow that happens resulting in a bogus value. Great - so this mystery was solved, but why doesn’t the -S option work?

It turns out that nicstat will ignore the -S option if it gets a valid result from the SIOCETHTOOL call (getting it from the underlying hardware). Normally, you would expect command line options to override values, but that wasn’t the case. So, I coded up a workaround for that. Great - except, the utilization values were still wrong!

Back to the source again, where I found a bug in the implementation where utilization is always computed as a half-duplex link on Linux. Easily fixed - and now I have a working version of nicstat where I can provide an appropriate -S override for any particular EC2 instance type (and in our use at Netflix, we have a wrapper script that provides an appropriate value for each type).

As the current author of nicstat isn’t actively maintaining it, I’ve forked the source to nicstat 1.95 on github and fixed several bugs. (3 of which I reported) that have not yet made their way into the official version.

If you use nicstat, you may find this repository handy, rather than having to apply several patches manually.

$ nicstat -S eth0:2000 -l
Int      Loopback   Mbit/s Duplex State
eth0           No     2000   full    up
lo            Yes        -   unkn    up

$ nicstat -n 1
    Time      Int   rKB/s   wKB/s   rPk/s   wPk/s    rAvs    wAvs %Util    Sat
23:19:29     eth0 34157.0 34140.3 39139.6 39136.8   893.6   893.3  14.0   0.00
23:19:30     eth0 37426.4 37462.9 42513.8 42546.7   901.5   901.6  15.3   0.00
23:19:31     eth0 35787.9 35787.2 41224.3 41224.3   889.0   888.9  14.7   0.00
23:19:32     eth0 25482.9 25482.2 33962.9 33961.9   768.3   768.3  10.4   0.00
23:19:33     eth0 29245.6 29244.4 36125.4 36125.4   829.0   829.0  12.0   0.00
^C

Sep 21, 2014 - Smoked Ribs on WSM with Pitmon

While still under active development, I’ve been using my pitmon BBQ pit monitoring software along with my CyberQ WiFi for a few cooks already. Today, I smoked 2 racks of pork ribs which turned out great.

Finished ribs

I used the 3-2-1 Method of cooking ribs, using a base of yellow mustard (the secret to great smoking) and a mix of commercial, off-the-shelf pork rubs. I cooked over a mix of apple and cherry wood in my WSM (Weber Smokey Mountain) smoker. For phase 2, where the ribs are foiled, I added some apple juice and honey.

While the ribs didn’t have a particularly prominent smoke ring, they had great flavor. The small ribs were a bit overdone (falling off the bone - you can see a lonely bone above) where the larger ribs required just a bit of work to get the meat off the bone. Overall, some of the best ribs I’ve ever made.

Here’s the whole cook graph from pitmon. You can see each stage pretty readily, including where some chunks of wood caught fire at the start of stage two when I had the smoker open for 5 minutes or so while I wrapped each 1/2 rack in foil, driving smoker temps up for a bit. For the final stage, where I was checking the ribs frequently, you can see how this affects stability of temperature - compare to the first 3 hours where the smoker was never opened.

Ribs on pitmon

Ribs going on the WSM - two full racks cut in half to make them a bit more manageable:

Ribs on pitmon

After smoking, just about to be wrapped in foil for stage 2:

Ribs on pitmon

I was telling the family how disappointing it is ordering ribs from some place like Black Angus when you can make ribs in the backyard that are so much better.

Sep 19, 2014 - Reviving an old Roomba

In 2005 we bought a Roomba red and used it for a couple of years before moving onto our sailboat where we didn’t need it anymore. I passed it on to my sister, who used it for a couple more years before she also didn’t need it anymore and put it into storage until now.

I wasn’t surprised that the original NiMh battery would no longer take a charge, but $26 on Amazon was all it took to fix that - and the Roomba is once again alive, much to the annoyance of our dog.

Roomba red

This Roomba only cost $150, and works great - especially at picking up dog hair. I think iRobot has really lost it’s way, and as a result a huge portion of the market. The cheapest current Roomba is about $350, but most models are in the $450-$600 range. While our Red was near the bottom of the line when we bought it 9 years ago, I can’t see any reason why they would be so much more expensive today - even accounting for inflation, with reduction in manufacturing costs and technology getting cheaper, these should be less expensive today than they were 9 years ago.

As a result, iRobot now has much more competition from companies like Neato. Today, I’d never buy an overpriced Roomba, but at least our 9 year old model is still working great!

Sep 14, 2014 - Jekyll Clean Theme Published on jekyllthemes.org

My Jekyll Clean Theme has been published on jekyllthemes.org.

jekyll-clean

Give it a try, feel free to provide feedback and pull requests, and don’t forget to merge from upstream often to keep your fork up to date.

Sep 13, 2014 - wpagui - The Dark Horse Wifi Manager

Lots of folks use NetworkManager or wicd without realizing wpasupplicant already has a decent GUI (and CLI) for managing wifi networks. To use it effectively you have to configure wpasupplicant for roaming, but not only is that pretty easy to do, it’s well documented (on Debian systems, in /usr/share/doc/wpa_supplicant/README.Debian.gz).

Why not just use NetworkManager or wicd? A few reasons:

  • NetworkManager is huge, and requires large bits of either KDE or Gnome in order to use the GUI. Since I’m using i3wm, I don’t really want to fill my disk up with a bunch of Gnome stuff simply to connect to wireless.
  • wicd is crufty and janky.
  • Both NetworkManager and wicd have bugs where they can’t scan or connect in environments that have large numbers of access points.

That last one is the killer for me. At work there are many dozens of visible access points in most areas, and both NetworkManager and wicd can’t cope.

Configuration

Setting up wpasupplicant for roaming requires editing a couple of files to enable it, but actual access point configuration is done via wpa_gui or wpa_cli as you prefer.

Modify /etc/network/interfaces so your wireless interface looks like:

auto wlan0
allow-hotplug wlan0
iface wlan0 inet manual
    wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf
iface wlan0 inet dhcp
iface wlan0 inet6 auto

And create /etc/wpa_supplicant/wpa_supplicant.conf, making sure it’s owned by root and chmod 600:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

Don’t forget to add yourself to the netdev group as needed.

Now, use wpa_gui (or wpa_cli if you wish) to scan and configure your wireless networks.

The wpa_supplicant.conf will have plaintext passwords - some people consider this an issue, but since all my laptops use LUKS encrypted volumes, it’s no risk for me in a powered off system. And if someone did compromise a running system they could certainly see the key if they had root access - but I can also steal that directly from wpasupplicant on a running system with NetworkManager or wicd. So, in summary - there is no additional risk in using wpasupplicant directly on a properly secured system.

Update 2015-02-08: See here for an update for the following, which automates this so manually restarting the interface is no longer necessary.

wpasupplicant manages wifi, but not dhcp (where NetworkManager and wicd handle both). Sometimes, this means I don’t get an ipv4 DHCP address until I restart dhcp. Although I only need to stop and restart dhclient, I tend to prefer using ifup since it’s less to type:

$ sudo ifup wlan0

Eventually I’ll modify wpasupplicant’s postup action to do this automatically, but I don’t find this that big of a deal - certainly no worse than having to manually disconnect/connect to access points that you end up having to do with NetworkManager anyway (and in other operating systems - I see MacOS users having to do this constantly).