Aug 19, 2013 - Are your financial institutions' websites developed with agile practices?

If so, you are lucky - because mine sure aren't. Seems like every bank or other financial institution that I do business with is about a decade or so behind in web technology. They have very, very long and infrequent software development cycles, don't support recent (much less latest) client technologies, and have "major new feature releases" that are pretty darn uninspiring.

I can understand the importance of moving slow when it comes to people's finances, but it seems they don't really have a distinction between basic usability and those things that could result in serious financial exposure. In these days of distributed systems there's no excuse to muddle UIs with the backend.

I have three quick stories from the last year or so across two different financial institutions. They will remain nameless, although one of them (which comprises the first two stories) I'm in the process of closing accounts down, and in the third I have a great personal relationship with individual people - and even though their technology is pretty bad, the institution itself is excellent.

Story #1 - Please Downgrade Your Browser

So I've been using the institution's website several times a week with no problem, then suddenly it stops working from Chrome - I was unable to login, it would just redirect me to the login page. Maybe Chrome updated, I'm not really sure (my Linux distribution handles updates so well I seldom pay any attention into what get's updated). So, I try Firefox and it works fine, I am able to login to my account, so I fire off an email to customer support to let them know Chrome 17 (this is May 2012) doesn't work with their site. Following is the response:

I apologize that you have encountered this difficulty. Unfortunately, higher versions of chrome are unsupported with our website. When a browser version is unsupported the functionality will be intermittent at best. Sometimes it will work fine for months, and then one day stop working all together.

In order to continue using Chrome without issue, please use one of the following supported versions:

· Chrome: Versions 11 or 12

Once again, I do apologize for any inconvenience this may have caused. We are continually working on updating our supported browsers, but at this time those are the only truly supported versions.

Realize that the institution's website doesn't do anything complex - absolutely nothing that an update from Chrome 16 to 17 should cause inability to login. It's only because of using non-w3c compliant practices that this would happen in the first place, but it's absolute madness to suggest I would downgrade back 5 or 6 versions of Chrome! I'm not even sure how I would do that, does Google even have archives where you can download old versions?

At least they didn't tell me to use IE. That's the response I've gotten from a number of customer support interactions for various things over the years, even after telling them I'm not on Windows.

I don't remember how long it was before I could use the site again in Chrome, but it was at least several months, and I received no notification or follow-up, I just tried one day and it worked.

How would a more modern site deal with this? First of all, they would be testing not only the latest stable versions of browsers, but the bleeding edge, and be prepared. They would also be agile with the ability to test and push new releases daily, not monthly, to resolve fundamental usability issues. Good customer support would also dictate closing the loop with the customer, rather than leaving them hanging.

Story #2 - We Will Resolve This In 48 Hours

My paycheck was deposited into one institution, then partially transferred to another institution (the one in story #1) via an inter-bank ACH transfer initiated by the second institution. This had been working fine for a couple of years, then one day my transfer doesn't go through. Did they notify me of the failure? Nope. Did the account transfer page show anything interesting? Nope - the transfer just disappeared, unlike past transfers which had a history, it just vanished - like it never happened.

So, being the reasonable person that I am, I called customer service and opened a ticket with them. They said they would look into it, and it should be resolved in 48 hours. One month later they resolved the issue. You read that right, it took a month for their 48 hour fix. For the first 3 weeks I called them several times a week for a status, and their answer was always 48 hours - even after we were in this situation for 3 weeks. All they would tell me is that they were "investigating the issue" and that it would be resolved soon. Not knowing what was happening, and assuming that they would get the transfer through "real soon now" we ended up going low on funds in that institution and were basically unable to use our account. If I had known it would be weeks I would have deposited via other means, but when I'm told it's only a couple of days because they are nearly done with the fix I guess I was overly optimistic. To make matters worse, when I was concerned about automatic, scheduled payments, they told me there wouldn't be a problem because the problem was on their end - fool me once, shame on you... It was a problem because those payments still went through, even though they told me all was fine. As it turns out, they finally admitted it was a software bug in their ACH system, but for this institution it was too late as we had already taken our banking elsewhere.

How would a more modern company deal with this? This isn't strictly a site issue, but it was a back-end software problem, and repeatedly telling me it would be resolved in 2 days when in fact it would take 30 did nothing to help. I'm sure they didn't actually know, but better to give the customer a pessimistic answer than an unreasonably optimistic one. I arranged my finances based on what they told me and would have reacted completely differently otherwise. Additionally, I had to keep calling them for status, as they would never call me back after the promised 48 hours. They finally did call me back when it resolved, but by then it didn't matter anymore.

Story #3 -  We Are Experiencing Known Performance Issues

On a Saturday morning I logged into my account at this institution to check some activity. At least, that's what I was trying to do - the institution's landing page didn't mention a thing, and it was only after logging in that they displayed a page that the site was down for maintenance for the whole weekend.

Really? Down for the whole weekend for an upgrade? This is 2013!  And with zero notification in advance that the site would be down (they have my email address). And I had to login to even find out.

The last couple of companies I've worked at have prided themselves on the ability to do live rolling upgrades with no site outage. This is not a hard thing to do these days with a good architecture. Being down for 2 days would mean going out of business for a lot of places, but for a financial institution it's considered normal, I guess.

OK great, so now it's Monday morning and I try to login again. Right, so the page times out. I reload, and after several minutes, I finally get a page. Clicking on anything results in a similar pattern - either a timeout or page load time of several minutes. I tried to use the online feedback form, but that timed out too.

A call to customer service resulted in a long wait time with a message that they were aware of site performance issues due to huge demand for the upgraded site. I guess that's one way to put it, but clearly they did insufficient testing and weren't ready for Monday morning. There was no reason for me to waste support's time at that point, so I hung up and tried to login again later.

How great was the new site? It sucks just as bad as the old one did. It's really pretty bad.

How would a more agile company deal with this? They would test things in advance, do rolling updates rather than having a site outage for 2+ days, and route just a portion of traffic to the new site until being comfortable that all is well prior to a full cut-over.

What's the lesson from all of this? Financial institution IT is well behind the times and could learn a lot of things from agile companies who release continually. Again, I understand the financial risk, but a good architecture would insulate those details from the website and it's usability for customers.

May 31, 2013 - Using etckeeper in Linux Mint (or any Debian/Ubuntu distribution)

My systems tend to live for a long time with very frequent updates. By this, I mean that I don't reinstall the system, but continually upgrade the same Linux installation. While I apply upgrades to patches frequently (daily), this also applies to major distribution upgrades as well.

Performing in-place distribution upgrades is easier in some distributions than others - as with most things in life there are tradeoffs, so distributions likes raw Debian, or LMDE (Linux Mint Debian Edition) are always rolling upgrades, but tend to lag far behind Ubuntu or other mainline distributions (Linux Mint, Arch, Gentoo etc). On the flip-side, there are distributions who strongly discourage doing in-place upgrades, recommending users back-up and restore the system for each upgrade.

While Linux Mint is in that latter category, it's actually quite easy to do an in-place upgrade without losing anything. Although some people have reported issues doing so, I've never had any insurmountable issues with doing it. In fact, one of my computers last had a full install of Linux Mint 11, and is now running Linux Mint 15, having been upgraded each time.

In fact, in some cases doing upgrades have given me greater stability and options than a fresh install, such as when there was a kernel 3.8 issue in Linux Mint 15 that would have prevented boot, but instead I was able to simply boot under an older 3.5 kernel from Linux Mint 14, which I used until a newer kernel was available with the issues fixed.

Wait, isn't this a blog about etckeeper? Oh, right! :-) All of the discussion above is a great reason for why you might want to keep your /etc configuration files under version control. As packages are upgraded, or as you change configurations over time, you can easily diff configurations. And if you do a reinstall type upgrade, you can save off your entire /etc directory along with the version history, for comparison.

Whenever a package has an updated configuration file it's a bit less stressful to just answer "Yes, overwrite my customized configuration with the package maintainers version". Although it always saved back-up files previously, it wasn't always easy to find all the files related to a specific upgrade. With etckeeper, it's trivial - just diff the commits!

Setting up etckeeper is easy, and in it's simplest form you don't have to manage it at all - as you apt-get (or use aptitude, or other higher level package manager utilities), your /etc configuration will be automatically versioned. I recommend git for the version control system (VCS), which is what is in the examples below.
  • Install etckeeper (and git if you don't already have it):

 $ sudo apt-get install git etckeeper 
  • Update /etc/etckeeper/etckeeper.conf, commenting all VCS lines except git:
 $ sudo vi /etc/etckeeper/etckeeper.conf 
  • The first few lines of that file should look like:
 # The VCS to use. 
  • And finally, initialize the etckeeper repository:
 $ sudo etckeeper init         
 $ sudo etckeeper commit "initial version" 
  • This will create an initial commit of all your configuration files as they are now.
For the simplest use-case, that's all there is to using etckeeper! Whenever you do any system changes through the apt-get package manager (directly or indirectly), etckeeper will automatically get executed to version your configuration files in /etc before and after any package changes are applied to your system.

You can also explicitly commit changes when you manual change configuration files, and use the commit command to track them. See the manpages and the etckeeper site for more information. 

And finally, because this just uses git under the covers, you can use git commands to diff versions, etc - just cd to /etc and use git as root.

Feb 27, 2013 - T-Mobile Please Get Your Act Together!

I'll start by saying yes, this is a moderately whiny complaint blog. But if it helps one other person, then it's worth it.

T-Mobile has a great $30 pre-pay plan - unlimited text, unlimited data, 100 minutes talk. For our usage, this plan is perfect and pretty darn cheap. Sure, the "unlimited" data means that overage happens at a lower speed, but we've never come close to that limit.

Because this plan is such a good fit for people who don't use their phones as phones much, this is a great plan. However as far as I can figure out, T-Mobile doesn't want anyone to actually use this plan. It's easy, when you know the tricks, but forget trying to get T-Mobile to help you get this plan. Their customer support can't get you this plan (and will even go so far as to not tell the truth, insisting that you have to buy a new phone at Walmart). With the information in this post, you'll learn how to activate this plan and save a ton of money.

I've been a generally happy T-Mobile customer for a 8 years now, and I've had up to 4 lines on the account, though for the last 2 years we've had 2 pre-pay and 1 post-pay. The sole post-pay has been my Android phone, with the 2 pre-pay being feature phones used by other members of the family. You'd think that being a good customer in good standing for a long time would matter, but alas, in today's business world, it doesn't - they really just don't care.

In the last 9 months or so I've gone through a spree of updating my Android phone from a Samsung Galaxy 1, then a Samsung Galaxy 2, and now a Nexus 4. I've had a 2 year contract (which just expired), but swapped the phones around under the contract without even notifying T-Mobile. I bought the Galaxy 2 off a coworker, and the Nexus 4 straight from Google, so T-Mobile wasn't involved in the purchase of any device aside from the Galaxy 1 that I used when I started my 2 year contract.

I had to do some hacking to cut the regular SIM card down to microSIM for the Nexus 4, but that turned out to be really easy with some guidance from a coworker who had already done the same thing. Seriously, it's really easy to make a microSIM. Don't pay T-Mobile $50 for a replacement SIM, just grab some scissors and do it for free. You've got nothing to lose, and if you screw it up (but you won't, it's easy), then you can still pay T-Mobile for a replacement SIM. Don't buy a SIM cutter, don't buy a cutting pattern, just find someone with another microSIM so you get the outline right, cut it a bit big, and trim to size. If I can do it, you can do it.

When I bought the Galaxy 2, I sold my Galaxy 1 to a friend of mine. But when I bought the Nexus 4, I gave my wife the Galaxy 2 as an upgrade to her feature phone. The $30 pre-pay plan was appealing, so I bought a SIM card for $1 from the T-Mobile online store. It came a few days later, I activated it under a new number, with no problem, then called T-Mobile's horrible offshore pre-pay customer service to cancel her old number and transfer this number to the new number. This is really important: Do not, under any circumstances, attempt to activate the plan or SIM via T-Mobile customer service. They cannot and will not help you. You need to buy a SIM online and activate it online, then only call customer service to port your number afterwards.

Fast forward a few weeks until my 2-year post-pay contract is up. I wanted to switch my phone to the same $30 pre-pay plan. I know the plan works, because my wife has been using it for a few weeks, but unlike her plan where we were converting pre-pay to pre-pay, I was converting post-pay to pre-pay. Cancelling post-pay requires a conversation with customer service, but I thought it would be easy - and it is, now that I know the secret, but boy did I go about it the wrong way (by trying to go about it the right way)!

My initial plan was to call up post-pay and cancel. Then I would buy a pre-pay SIM for $1 from the online store, activate it, and just like with the Galaxy 2, be set and done with this. Sounds simple, right?

So I called up post-pay customer service and told them I wanted to cancel. I get an amazingly awesome CSR (customer service rep), who first tells me he can get my current $79 plan for $55. This is a decent discount, but $55 is obviously more than $30, so I told him what I planned to do - that I was going to cancel, and just reactivate with a pre-pay SIM. He agreed with me that requiring customers to cancel just to activate pre-pay was silly, and he said he could override pre-pay and give me the $30 plan I wanted. Pretty awesome, right?! So, the guy sets me up and says I'll get an SMS within 24 hours, and that I'll need to then go activate with that code to switch to pre-pay.

I can't say enough good things about this post-pay CSR who helped me. Unlike every pre-pay CSR, and most other post-pay CSRs, he really wanted to help me - and understood that ultimately the end state would be the plan I wanted, so T-Mobile might as well make it easy on everyone involved and just do it.

Unfortunately, it didn't quite work out that way - pre-pay denied the account conversion. Of course they couldn't be bothered to actually call me, or notify me - instead they just blocked the conversion silently. Two days later I called, as I hadn't received the SMS to activate the new plan. This is where things got ugly. Post-pay wouldn't help me, and transferred me to pre-pay. Pre-pay told me that I couldn't get this plan and that it was only available if you bought a new phone from Walmart (This is obviously untrue, and a total lie on T-Mobile's part, as I had activated this exact same plan previously with the Galaxy 2!). I got really angry with the pre-pay CSR, who just started to give me a bunch of run-around. Asking me for account and SIM card numbers (which I never needed before), and finally just refusing to help me, even when provided with this information as well as the confirmation reference from my call two days previous. I asked, several times, to just transfer me to someone who could help me, but she basically refused. I was really surprised, customer service isn't the way it used to be, that's for sure. Sure, I was getting pretty angry at this point, but only because she wouldn't listen to me when I knew that I was right.

It was at this point that I remembered I had ordered a second SIM card when I bought the $1 SIM for activating the Galaxy 2 for my wife! I totally forgot about this - since they are only $1, I bought two just in case I needed another pre-pay SIM for some reason in the future.

Thirty minutes later, it was all done - I did the conversion from post-pay to a pre-pay plan that T-Mobile says there's no way you can get. And here's how you do it:

  1. Wait until you contract is nearly up. Don't blame me if you screw this part up and have contract cancellation fees!
  2. Go to the T-Mobile online store and order a new pre-pay SIM card for your phone for $1 plus shipping.
  3. Wait a few days for the SIM card to arrive.
  4. Activate the SIM card online, and choose the $30 plan with unlimited text, unlimited data, 100 minutes talk. And do not, under any circumstance, call T-Mobile customer support as part of the activation! Don't forget to port your old number over at the appropriate place in the process (and if you can't do it at the time of activation, you can always do that bit later, as we did for my wife's phone).
  5. Cancel your old plan.

That's all there is to it - that's how you do the impossible, to get the plan that T-Mobile says you cannot have.

Jan 19, 2013 - Atari Inc. Business Is Fun

If you are interested in the history of Atari, I would encourage you to grab a copy of "Atari Inc. Business Is Fun" by Curt Vendel and Marty Goldberg. This is a big book - 800 pages of narrative, pictures, and internal documents that tell the unabridged history of this pioneer video game company. What Atari was doing in the 70s and early 80s shaped not only the video game industry, but the entire computer industry.

As an added bonus, there's even a picture of me in this book! My father was an engineer for Atari from the beginning until the fall of Atari in 1984.

Go grab yourself a copy by clicking below - you won't be disappointed!


Nov 30, 2012 - The fun of working in a large bureaucratic company

The following is a real email from a decade ago, with only the names removed, from a mid-sized software development company that was very bureaucratic. As you can probably imagine after reading this, the company had a very old-school, throw it over the wall, development lifecycle - long release cycles, little interaction between developers and QA, and no coding until the design is "perfect". They even had a "no prototyping" policy - one of the engineering upper-ups said that a professional engineer should be experienced enough to know what they need to build the first time and it's not OK to throw away code. I'm not kidding! You can imagine how productive this environment was, and the resulting quality of the code.

If there's one reason to move to an agile lifecycle with a flexible work environment, just avoiding condescending, morale destroying emails like this is enough!



To: Development
Subject: Miscellaneous Items

There is standards that we need to follow for safety, cost and equity purposes. Unfortunately, I'm finding that I need to step in here and remind everyone of these standards and policys. These are not new standards. They have been in place for quite some time. So let me be the one to remind you of the following:

1) Chairs - Each associate is issued one task chair. These are the ones you are sitting in. A manager is issued one task chair and one side chair. A Director is issued one task chair and two side chairs. The conference rooms have been furnished with the appropriate amount of chairs depending on the size of the room. Teams areas have been issued appropriate chairs. That's it. Therefore, there will be no side chairs issued to an 8X8 cubicle. I know some of you have modified cubicles that would accommodate an additional chair, but this is not going to happen. Those that currently have an additional chair, will return them to me. Those that are requesting an additional chair can stop asking. Now then, if someone wants to write me a check for $26,950 from their own personal account, I would be more than happy to provide an additional chair for all cubicles. And can not bring a chair from home either.

2) Coffepots, toasters, microwaves, refrigerators, etc are not to be used in the cubicles. Period, end of story.

3) Cubicle modifications - all modifications need to be approved by facilities. This is not your personal property and therefore is not yours to modify using nails, molley bolts, screws, saws, paint, etc.

4) Any of that old furniture that you became attached to (lamps, tables, bookcase, etc) is not approved for placement here. However, I'm told that you may be able to take it home (contact [name withheld] if interested).

If you have further questions or objections, please feel free to see me personally and I will help to explain.

[name withheld]
Vice President Software Development