Aug 3, 2014 - Cornell Chicken

Yesterday I cooked this Cornell Chicken recipe in my Weber Smokey Mountain (WSM) and it turned out great!

The Cornell Chicken recipe comes from Robert Baker who was a professor at Cornell University and founded the Food Science Institute at that college.

There are variations on the original Cornell recipe, and the baste I used (recipe link above, from virtualweberbullet.com) uses a base of dijon, cider vinegar, and rosemary and it came out splendidly! I used a blend of cherry and apple woods.

I butterflied the chicken and brined it first, but this is a very versatile recipe that could be cooked with any chicken parts or preparation and in any grill or BBQ - gas, charcoal, a Traeger, etc.

The WSM smoking away:

WSM Smoking

Chicken cooking:

Chicken Cooking

Finished product:

Chicken Done

Give it a try, you won’t be disappointed.

Jul 23, 2014 - VOIP Solution Round FOUR

Oh no, if you’ve been reading this blog for a long time I already know what you are thinking - here we go, switching VOIP providers yet again. Don’t worry, Flowroute continues to be my SIP provider of choice (they are, quite simply, awesome).

No, this round was prompted by a hardware failure when our 5 year old Linksys PAP2TNA bit the dust. A couple of days ago my wife went to use the phone - so she hit the Talk button to take it off-hook, but then realized she needed some additional information so hung up. The ATA was making a dial-tone, because I could hear it from across the room. Two minutes later she hit the Talk button again, and got… silence. No dial-tone, just nothing.

Hmmm, that’s weird - so I go and take a look at the Linksys ATA and find it boot-looping. The power LED would flash green, the LAN LED would flash green, then the power LED would go red and it would reboot itself. I cycled the power, even left it unplugged for the day, but it appears that it decided to randomly fry itself.

Well, time for a new ATA. I had heard about Obihai’s affordable ATAs and found that the local Fry’s had an OBi100 in stock for around $40. The OBi100 is a very basic model, supporting just a single POTS line, but that’s all we need.

OBi devices have two modes of configuration - you can browse directly to the device on your local network to configure services, or auto-provision through the ObiTalk website. Although I’m familiar enough with SIP configuration that I don’t mind direct device config, I decided to provision it via ObiTalk.

Flowroute isn’t one of Obihai’s “certified” providers, but it’s just SIP and I found a config guide at Flowroute’s website. The config guide was for direct device config, but the fields are the same, so it was easy enough to provision.

And it worked! Well, actually it only sort of worked. I could make outbound calls via Flowroute, but inbound calls to my DID would just result in a busy signal. At this point, I wasn’t sure if it was something in the SIP negotiation preventing Flowroute from routing my DID, or something in the OBi device, or just something I screwed up in the config.

So, I gave up on ObiTalk - I deleted my device from ObiTalk, reset the OBi device to factory defaults, and then started manual SIP configuration. Guess what? It worked - fully! Both inbound and outbound calls were working fine.

At this point, I still wasn’t sure where the problem was - all the possibilities were still, well, possibilities. So this morning I opened a support case with Flowroute. Realize that I’m about as small potatoes of a customer that Flowroute can have - I give them just a couple of bucks a month for DID, E911, and per-minute usage. Yet, each time I’ve opened a support case they have been responsive, thorough, and just plain fantastic.

Today was no exception, with the support person providing the key information that they have seen issues on OBi devices where X_InboundCallRoute is misconfigured. The result of this is that everything is great on the Flowroute end, but the OBi device doesn’t know what to do with the call and gives a busy signal. She suggested making sure the value for this was “ph”. That seemed to ring a bell - I remember noticing that my manual configuration had this value, but the provisioned value from ObiTalk did not.

I thanked Flowroute support, but they didn’t stop there - they gave me instructions on doing a tcpdump so that if I wasn’t able to get it to work from ObiTalk, they could look at the SIP conversation to debug it. I was stunned! I’ve worked with dozens and dozens of vendors and providers over the years, and given several of them tcpdumps which they had no clue what to do with. In contrast, Flowroute was willing to dig this deep, into a problem for which I already had a workaround. That’s pretty cool. I can think of a few companies who could learn from this.

I am happy to report that after re-provisioning my Obi100 from ObiTalk, and then going into the advanced settings to reset X_InboundCallRoute, I have a fully working, auto-provisioned device! It’s pretty annoying that ObiTalk corrupts, for some unknown reason, the correct value and overwrites it with a wrong one. I don’t know why they do this, but it nearly resulted in me returning my OBi100 to the store for a refund.

Flowroute - thanks again for your help and support in getting this fully working!

And for anyone reading this - if you are paying your phone company, cable company, or VOIP provider more than $5 a month for phone service you are getting seriously ripped off. Cancel that, pick up an OBi100 for $40, and go set up an account at Flowroute. They will provision a DID instantly in the area code of your choosing, and you’ll be up and running pretty darn quick. By using auto-provisioning through ObiTalk and following the config guide mentioned previously, you don’t have to know anything about SIP or configuring an ATA to get up and running.

Manually configuring a SIP device is not a particular fun task if you aren’t familiar with telephony - there are literally hundreds of configuration options, and it’s not always obvious what you are supposed to do. The config guide mentioned above will walk you through manual configuration through the device wizard on the OBi, so it’s not that hard, but some people might find doing it all through ObiTalk attractive (and perhaps want the additional features it provides as well).

I don’t know if Flowroute can port existing numbers, but we switched to bouncing through a Google Voice number several years ago, so our friends+family know to use that number, not the ANI (CallerID) number they see when we call them.

One final note - Obihai used to work directly with Google Voice, but Google disabled that service. Obihai is doing an incredibly horrible job of getting the word out, as all of their web pages still say that this is a feature of the device - certainly they sell more devices with this false claim in their advertising, but I wonder how many get returned solely because of it. At any rate, it’s yet another case of Google killing off a useful service, but I can see how they weren’t making any money off it. Update 2015-02-16: Google Voice through Obihai might be available again, according to what I read at the ObiTalk site, but I haven’t tried it to validate, as it requires flashing a special firmware on the Obihai and what I have is already working…

Jul 1, 2014 - Microsoft Takes Down no-ip Domains

Today I discovered that I was unable to resolve DNS to my server at home, which I access via a noip.com domain. Why? Because Microsoft was granted a federal court order to seize 22 of noip.com’s domains!

This was done without any attempt by Microsoft to resolve any perceived issues with noip.com, and Microsoft claims this was done due to noip.com domains hosting malware. This is an amusing statement when you consider the vast amount of spam and malware that comes from hotmail.com and other Microsoft properties - or as a result of Windows being insecure to begin with.

Here’s the statement issued by noip.com: http://www.noip.com/blog/2014/06/30/ips-formal-statement-microsoft-takedown/ and the full text of this is below, as well.

Just another reminder to use open-source operating systems and stop supporting companies such as Microsoft, Apple, and even Google.

We want to update all our loyal customers about the service outages that many of
you are experiencing today. It is not a technical issue. This morning, Microsoft
served a federal court order and seized 22 of our most commonly used domains
because they claimed that some of the subdomains have been abused by creators of
malware. We were very surprised by this. We have a long history of proactively
working with other companies when cases of alleged malicious activity have been
reported to us. Unfortunately, Microsoft never contacted us or asked us to block
any subdomains, even though we have an open line of communication with Microsoft
corporate executives.

We have been in contact with Microsoft today. They claim that their intent is to
only filter out the known bad hostnames in each seized domain, while continuing
to allow the good hostnames to resolve. However, this is not happening.
Apparently, the Microsoft infrastructure is not able to handle the billions of
queries from our customers. Millions of innocent users are experiencing outages
to their services because of Microsoft’s attempt to remediate hostnames
associated with a few bad actors.

Had Microsoft contacted us, we could and would have taken immediate action.
Microsoft now claims that it just wants to get us to clean up our act, but its
draconian actions have affected millions of innocent Internet users.

Vitalwerks and No­-IP have a very strict abuse policy. Our abuse team is
constantly working to keep the No-­IP system domains free of spam and malicious
activity. We use sophisticated filters and we scan our network daily for signs
of malicious activity. Even with such precautions, our free dynamic DNS service
does occasionally fall prey to cyber scammers, spammers, and malware
distributors. But this heavy-handed action by Microsoft benefits no one. We will
do our best to resolve this problem quickly.

Update

Microsoft has now settled with no-ip.com and admitted wrongdoing. The details of the settlement were not made public, but I sure hope it was enough money.

Microsoft has reviewed the evidence provided by Vitalwerks and enters into the
settlement confident that Vitalwerks was not knowingly involved with the
subdomains used to support malware.

More at http://www.noip.com/blog/2014/07/10/microsoft-takedown-details-updates/

Here’s a good technical read on how Microsoft doesn’t understand DNS and the mistakes they made in blocking all 5 million domains instead of just the few hundred they claim were involved in the malware: http://www.unchartedbackwaters.co.uk/pyblosxom/microsoft_noip_dos

In case it’s not obvious - you should think twice if your networking infrastructure uses Microsoft DNS, DHCP, or other network services. Not only do they have a clear history of not following the RFCs, it’s pretty obvious they just honestly don’t understand how things work. It was suggested here to send Microsoft a copy of the DNS and Bind book - not a bad idea!

Jun 24, 2014 - A Weird Headhunter Email

Lake many engineers, I tend to get a lot of email from recruiters and head-hunters. Ever since I deleted my linkedin account the situation has gotten much better, yet I still get emails on occasion.

Last week I received the following email from a major technology company. I’ll leave a few details obscured, such as the company name - but I guarantee it’s a company you know if you are reading this. Maybe it’s because of that fact I was so surprised to receive a letter like this. I’ve obscured a couple of other minor details such as people and place names.

As an additional amusement, the subject of the email was “Our call on Friday”, as if we had scheduled something explicitly.

So weird, but read on!

Clearly you have been quite popular with my colleagues here in [company withheld]
over the past few years; I realize that you continue to hear from us asking if you
are interested in chatting about career opportunities at [company withheld] (yes,
I am one of them too). Since many of our outreach notes haven't caught your eye,
I thought I might try to convince you as to why I may be the one worth responding
to:

The address we have on your last resume says you lived in Diamond Mine, Oregon.
I grew up on Black Diamond Mine Road in Oregon Springs, Colorado. Not bad.

You maintain websites in your free time. I have a website I work on in my free
time too. Granted, I've only mildly played in the HTML and it's really a blog,
but I make up in journalism what I lack in coding.

You like gaming. I like gaming. Although I'm sure I wouldn't stand a chance
against you, because my definition of "gaming" is playing this entertaining
app (read: not really gaming).

Does that lightly peak your interest? If I've even remotely caught your attention,
do mind returning my email and setting aside some time to chat with me later this
week about your career activities and goals? I can give you a call at 12PM PST
this Friday if you are available. What would be your preferred number?

Lastly, I'm attaching the most recent version of your resume that we have on file.
If you are so inclined to send an updated version, I'd gladly exchange them in our
system. That way we can stop talking to you about old news and get excited about
what you're up to now. :)

Looking forward to chatting with you this week!

Kayley
[company withheld]

Grasping at straws like that to find commonality makes this borderline stalking. It’s been a long, long time since I’ve done any gaming, and I guess this blog is the website I maintain (not an impressive body of work). As mentioned, I obscured people, street, city, and state names for obvious reasons of privacy.

Another bit of oddness is that the attached resume was 7 years old, yet I know for a fact they had a more recent one since I had talked to them a couple of times since then, and I know the colleagues of which she spoke had access to newer ones. Again, very very odd and not the sign of a top-notch recruiter.

I love the errors too - such as specifying the time as PST (it’s PDT). Even a recruiter should know that bit of technical detail, right? At least she got AM vs PM right! At least, I assume she didn’t want to talk to me at midnight. :-)

Oh, and any recruiters or head-hunters reading this - no we don’t have a call scheduled, and no I don’t want to talk to you, and yes I know I’m not on linkedin anymore.

Mar 29, 2014 - systemd is in the trees

You read that right - systemd is “in the trees… It’s coming!” *

If you pay attention to things happening in the world of Linux, you probably know that systemd will be replacing sysvinit for Debian systems.

In the last couple of days I’ve switched 5 Debian testing/Jessie systems from sysvinit to systemd. Overall, it’s been a smooth transition, with just a couple of problems I ran into which were not ultimately systemd issues.

It’s easy to test out, because you can selectively boot with systemd as long as you don’t install systemd-sysv, which enables systemd by default. I think this is a good way to test, and if the system boots OK and everything works as expected, you can then go all-in and enable systemd by default.

I also discovered the debian systemd implementation doesn’t run /etc/rc.local by default, but that’s easy to solve as well. (IMPORTANT: This is no longer the case.)

The debian wiki has excellent documentation on systemd and includes links for other common distributions as well.

As far as the two (non systemd related) issues I ran into:

  • On one server system, any USB related operations would hang with a kernel call stack dumped to dmesg. It turned out that this was some issue related to upgrading to the 3.13 kernel and had nothing to do with systemd. I verified this because I ran into the same issue when booting with sysvinit under the kernel. I was able to workaround this problem by unplugging a certain USB device (an external hard drive) until after the system booted. This system is working fine with systemd, and the external drive, I just have to unplug it if I need to reboot. Had I simply rebooted before trying out systemd I would have found this issue and not been debugging the wrong thing.

  • On my Acer C720 chromebook (yes, it runs Debian), the kernel would panic when I attempted to boot with init=/bin/systemd. I’m a bit embarrassed to admit what the cause was… Well, I forgot one little important step in this process: I forgot to install systemd!

Installing systemd

Make sure your system is fully up to date and running the latest kernel. I.e.

$ sudo aptitude full-upgrade

At this point you should reboot if any major updates, such as a new kernel, were installed. You only want to be debugging one problem, not two as I described in my issues above!

Now is a really good time to make a full backup of your system, as well.

To start with, we’ll install systemd alongside sysvinit:

$ sudo aptitude install systemd

This will likely bring along a few dependencies, such as libpam-systemd, libsystemd-daemon0, libsystemd-journal0, and libsystemd-login0.

Testing systemd

Now you’ll need to reboot and manually edit the boot line in grub to enable systemd. To do this, hit ‘e’ at the grub boot menu for your kernel, and append init=/bin/systemd to the line that starts with linux. It might look something like the following, depending on the options for your specific kernel:

linux   /vmlinuz-3.13-1-amd64 root=/dev/mapper/root-root ro quiet init=/bin/systemd

Press F10 or ctrl+x to boot. If all goes well, and it should, your system will boot and run like normal. You can tell if systemd is in use if PID 1 is systemd* and not **init, for example:

$ ps -p1
PID TTY          TIME CMD
  1 ?        00:00:01 systemd

Watch out! some ps options will show it as /sbin/init, so be careful not to confuse yourself.

At this point, we can also use systemctl:

$ systemctl status
[long output deleted]

The nice thing about testing this way is that the systemd switch is not persistent. If you reboot again, you are back to sysvinit.

If you are happy at this point, and your system is stable, you can install systemd-sysv, which will remove sysvinit. You can also run in this testing configuration for a week or so if you prefer.

Enabling systemd by default

Once you do this, there’s no going back! If your system started ok with systemd, it should be safe to do this - just be aware that you absorb some risk…

$ sudo aptitude install systemd-sysv

This will remove sysvinit and make systemd the default. Next, reboot:

$ sudo shutdown -r now

Your system should reboot cleanly with systemd. Once again, you can use systemctl status to verify that everything looks good.

Longer term testing

If you aren’t ready to commit to systemd, you can always modify your grub configuration so that it boots with systemd by default, but if you run into trouble you can always disable it again. Note that if you have a non-booting system due to systemd you’ll have to boot a rescue livecd/liveusb and modify /boot/grub/grub.cfg manually.

To configure grub to boot with systemd by default, add init=/bin/systemd to the GRUB_CMDLINE_LINUX_DEFAULT line of /etc/default/grub such that it looks something like:

GRUB_CMDLINE_LINUX_DEFAULT="quiet init=/bin/systemd"

Note that yours may look different.

Configuring rc.local under systemd

IMPORTANT: The following section should no longer be necessary now that this functionality is there by default. If you are using an older version of systemd, or you find that there is no built-in service for rc.local, the following is still useful. Not sure if you already have an rc.local service? Run the systemctl status command shown below - if you have the service, it will be running, if you don’t then that command will result in error.

Finally, let’s get rc.local working under systemd.

Create the following file as /etc/systemd/system/rc-local.service:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU General Public License as published by
#  the Free Software Foundation; either version 2 of the License, or
#  (at your option) any later version.

[Unit]
Description=/etc/rc.local Compatibility
ConditionPathExists=/etc/rc.local

[Service]
Type=forking
ExecStart=/etc/rc.local start
TimeoutSec=0
StandardOutput=tty
RemainAfterExit=yes
SysVStartPriority=99

[Install]
WantedBy=multi-user.target

(Note: In a previous version of systemd service files needed to be executable. This is no longer the case, so this article has been updated to reflect this).

$ sudo systemctl enable rc-local

After rebooting, you can use systemctl to check that everything worked as expected:

$ systemctl status rc.local
rc-local.service - /etc/rc.local Compatibility
   Loaded: loaded (/etc/systemd/system/rc-local.service; enabled)
   Active: active (exited) since Sat 2014-03-29 15:49:01 PDT; 1h 15min ago
  Process: 1097 ExecStart=/etc/rc.local start (code=exited, status=0/SUCCESS)

And that’s all there is to it! So far, systemd is a rather nice upgrade over the aging sysvinit. It will take some time for all your various services and programs to get updated with systemd init scripts, but until then the default systemd compatibility mode will still start all those scripts for you.

* Yes, this is a reference to the Kate Bush lyric from Hounds of Love, originating in the movie Curse of the Demon aka Night of the Demon. Since systemd is about managing system daemons, somehow the line “It’s in the trees… It’s coming!” popped into my head as I was thinking about a good title for this article.