Tech Debt

This is a concept I first learned about at Yammer while working there in the engineering department.  It took a little while to fully realize the implications and how to incorporate it into the way I explain things to upper management. Here is the most succinct way I have come up with. It will include elements of my velocity of business equations as those are closely related, but don't worry, I'll get back to the point.

Decisions have costs. Those costs can be in hardware, in immediate man hours, in long term man hours, or in moral. Let me rephrase it. You can choose money on equipment, money on employees or money on productivity. Seem overly simplified? It is, most people know that part although rarely use it in their decision making. What they don't account for is the non immediate ramifications. The debt incurred along with the immediate costs. This is true for IT, which I will talk about int his article, and development, which Kris Gale does a much better job at here.

I'll re-phrase... there is very rarely true cost cutting, there is only cost re-allocation. Only if something was truly wasteful to begin with is money saved. Usually true savings require investment, which is counter intuitive. When you understand Tech Debt however this will make sense.

This is the single most harmful error accountants and MBA types make when dealing with technology costs. They see only red, expense, they have never been trained to evaluate the total cost, or fully appreciate opportunity costs. It's not their fault, pity them and educate them on the realities of business in the modern age. They are working off antiquated philosophies. This is spending applied to velocity of business.

Simplifying the equations that define these thought processes isn’t hard:

Is productivity gain < cost where productivity gain = value of worked produced factored by time and cost of time + opportunity cost of moving slower

Let me touch on two key areas of spending in detail so what I mean by each of this things is clearer.

Personell

Just like equipment, people cost money. Usually even more than equipment. This is the absolute easiest place to either save yourself a ton of money, or get yourself way into debt. Why do I keep using the word debt? Because that's what's being created.

Direct productivity via expertise and experience

If you have someone not experienced enough to design a thing do it, often they do it wrong. In doing it wrong you then have to spend time and money to fix it. By wrong I mean not well thought through. In IT especially this can be VERY expensive. The problem is the cost is not immediate, the solution may seem to work.

Example: A proxy server is put in to conserve bandwidth by two different sysadmins.

Method 1:

Because the company has Active Directory servers already your sysadmin just add the IIS built in proxy and push out settings via GPO that change all http traffic to the proxy, disallow any other outgoing HTTP and HTTPS traffic.

Method 2:

Your architect comes up with a solution using PAC files and WPAD to distribute settings and point it at a squid server running on linux. The PAC file has a fallback to general internet and no firewall changes were implemented as the goal was bandwidth reduction, not security.

Both took the same time to implement, both solved the problem. For the first month these look like the same solution. Month two however the proxy server has a hardware issue. Method 1 users are now stuck without internet until the system is re-built or the GPO settings can re-propagate. Method 2 the config file automatically re-directs traffic and likely no one notices except for a few https refreshes required and maybe slower internet.

Which solution cost your company more in month two? Method 1 has a debt being paid in lost productivity for the company, incurred from a decision made that did not fully think through the consequences of the solution.

People design solutions and thus the people you hire will determine the quality and effectiveness of your solutions. In the example above perhaps spending a little on salary for your sysadmin saves you 2x that increased cost in lost productivity and velocity of business.

Moral

Moral as a component of productivity and therefore velocity of business cannot be underestimated. There have been more studies than I can count that come to one common sense conclusion: Happy workers work harder, and are more loyal. Therefore decisions that negatively effect moral impact your salary structure via productivity and thus creates a debt that must be paid and calculated into the cost of the decision. The inverse is obviously just as true, decisions and expenditures that have a positive impact on moral have a positive impact on productivity.

Equipment

Here is the absolute easiest place to see the equation of debt, and the most common place mistakes are made in spending due to bad policy.

Workstations

Lets say you have a group of 20 engineers who make $100k a year. This equates to ~$50 an hour. A decision is being made whether to purchase them the $1000 standard workstation or a $1500 workstation they want. There are many ways to make this decision but the only right one is the following:

The difference is $500 or 10 hours. Will you, over the course of one year, gain 10 hours of extra work from each engineer for this investment? If yes, buy. If no, don’t. See how simple that is?

All you had to do was convert the cost to hours. The old adage “time is money” has never been as true as in today’s high paced business environment.

How about if you got 20 hours extra in a year for that same $500 because compile/wait times went down? Now by spending $500 you have gained $1000 in productivity. Pretty good investment eh? It’s not hard to imagine scenarios where this would be true.

Now imagine you chose not to do this, and the productivity was set at 20 hours. This decision just cost you the full cost of the computer again, making it a $2000 purchase, $1000 for the machine, $1000 for 20 hours of productivity debt. Not such a good deal now is it?

What if it’s one of the same engineers is asking for a $100 fancy keyboard? The usual reaction is an immediate and hard no, you don’t need a wizbang keyboard. This is wrong. Again we are talking about 2 hours. Calculate the likelihood the employee with get a moral boost from the generous gift that they work an extra two hours that year. Likely this has a positive outcome. Now calculate the likelihood the engineer tells their friends how cool your company is because they got a wizbang keyboard. Recruitment budget anyone?

So does this always mean spend money? No. It means think differently about your spending. Think in terms of time as well as money. Think in terms of opportunity costs and velocity of business, not just the bottom line. There will be times when the equations above come out negative, probably quite a few. In that case however you have a well thought out explanation people can easily understand rather than just “we don’t have the money” when turning down or modifying a purchase request.

Software

So now we apply this equation to software. Lets start with communication software like Yammer.

When deciding to implement such a product first look at price, 9 minutes per month per engineer. Now look at potential benefit. Will you get 9 minutes of productivity out of the solution per month? Notice also I have not listed the dollar amount as it’s been converted to time in my head.

Now lets look at a harder decision like Zendesk vs Groove. Zendesk is 2 hours per support rep per month, assuming a $50k/year salary. Groove is 0.6 hours, nearly 1/4 the time. Thus, it is necessary to carefully examine the features of the more expensive option and ensure they are worth the extra expense. Maybe Zendesk’s superior process automation will save your agents 4 hours a month. In that case the money to productivity equation is positive. If your needs are simple and those features are not going to save you time, you go with Groove. Choosing wrong could put you 1-2 hours a month * number of support reps in debt.

Maintenance

The biggest cost left out or underestimated in a Total Cost of Ownership calculation, and therefore creates the most debt, is maintenance.

Compare a solution that seems cheaper and one that seems more expensive. If you discover one is harder to use, and therefore will add time to tasks for the lifecycle of the software over the other option, you need to add this time as a cost for each and every employee which will be using the software. Only then will you get the true cost. The cheaper solution is not always cheaper, some times it puts you in productivity debt from technology (tech debt).

Even length and impact of maintenance should be considered. In today’s environment of 24/7 always on employees, a two hour down time for maintenance once a month rather than an in-line upgrade could mean a lot of potential lost productivity. These lost hours must be accounted for in the decision making.

Getting out of debt

Getting out of Tech debt is not easy once you are in it, that is why it’s best to avoid it from the beginning. I talk more about that in my time management series, but it can be done, just like you can pay off credit cards with careful financial management. It just means making some hard choices.

The biggest suggestion is focus on productivity multiplier projects. These are projects that buy you time rather than create time syncs. It’s the equivalent of getting a second job to pay off your credit card debt, and is going to suck just as much. Realize you aren’t going to get anything more right away, you are paying for previous bad decision making.

Second is don’t be afraid to cut ties. Whether this is a bad piece of software, an unproductive employee, a bad policy or process, an idea that just isn't panning out etc. Humans resist change, they want to be nice, but much like financial management, tech debt management involves some hard choices.

Focus on these two areas, and make good decisions going forward, and it will get better, I promise.