For technologists, the very idea of having a practically unlimited pool of resources to pull from is a dream come true. It’s like all the benefits of working for a big, huge, well-financed behemoth of an organization without actually having to – well – work for a big, huge, well-financed behemoth. On top of that, it’s like having all new servers – ALL THE TIME!!! As with most things that excite technologists, they should terrify those who hold the purse strings. The cloud is no exception. Just because the changing cost structure (fixed to variable) is obvious, doesn’t mean the solutions are.
Take a brand new project as an example. Prior to the cloud, the team sat down to guesstimate (woefully) the compute needs for the project and spec’d out the associated hardware. Finance then reviewed (and halved) that number, servers were ordered, installed, configured some months later and we were all on our merry way. For all its downsides, this approach did have an unheralded benefit – inherent cost control. No matter how much you ran those servers, they weren’t going to cost your organization any more (save for electricity and HVAC, but work with me here). If you really needed more compute, you pulled together a compelling case for it and proceeded to beg, borrow, steal, grovel, and gather at the proverbial caracas of a failed project to grab whatever you could repurpose. The sheer effort required to actually justify and buy new hardware was in itself a deterrent to doing so.
Enter the cloud. Any organizational leader from the PM to the finance office finds him/herself in the unenviable position of playing Ebeneezer Scrooge to the technologists Bob Cratchit on an on-going basis. Everyone inherently knows that treating the cloud the same way as hardware makes little sense and obviates much of the benefit of going to the cloud. On the other end of the spectrum, empowering technologists to take full advantage of the nearly unlimited capacity of the cloud and asking them nicely to “keep an eye on costs” and “turn things off they aren’t using” isn’t likely to succeed. It amounts to putting 2 year olds in a room full of toys and asking them to put old toys away before getting out new ones to play with. Having considerable hands-on experience with both (techies and 2 year olds) believe me, this doesn’t work. At least with the 2 year old, I don’t have to pay for the toys left on the floor! I’ve seen projects with literally hundreds of unused volumes sitting around because “oh, we didn’t know that those stayed there when we shut the servers down” – never mind the very obvious list of 750 volumes right there on the screen, but I digress.
Avoiding The Cloud-Blown Budget Fate
Alas, there are a number of relatively easy to implement practices you can put in place that still permit an organization to take advantage of the cloud, but control some of the cost risks associated with diving right in. Bear in mind that if I sound negative on developers that’s not my intent. They, like everyone else, are looking to get their jobs done as quickly and efficiently as humanly possible. It’s our job as leaders to balance that desire with the realities of our respective budgets and create a culture that buys into that.
1. Lightweight Architecture Diagrams – It’s easy to see why reasonable and responsible advanced planning gets overlooked when it comes to the cloud. After all, the initial dollar figures tend to be relatively low (ever heard: “we need to play with it to see how much it will cost” – sounds a lot like “we have to pass the bill for you to see what’s in it”) and “we can always shut-down whatever we don’t need”. Don’t fall into this trap! These are justifiable reasons for keeping a process simple and lightweight, but not for abandoning it altogether. Insist on at least a notional architecture diagram, with justifications of instance types, sizes and exactly who (and how) they will get “shut down”. Not only does this get everyone on the same page, it serves as a bit of a cloud knowledge test to help you avoid the “we didn’t know” later.2. Least Privilege – It ain’t just for security any more. Limiting the number of individuals with permissions to incur costs is critical to managing costs. AWS in particular provides incredibly powerful tools for authoring very granular permissions. It takes some time to learn and use them, but it’s well worth the investment. Permissions for activities that can incur considerable costs (specifically creating new machines and volumes) should be limited to “the select few”. Furthermore, automatic start-up and shut-down activities is remarkably more predictable than having humans do it. Machine roles are your friend! 3. Clear Accountability – Clarify roles, responsibilities, and expectations for cost control to everyone on the team. If someone has the ability (responsibility) to incur costs, they also have the accountability for managing those costs. Use whatever objectives, measurements or incentives your organization places at your disposal to drive home this point. 4. Premature Optimization – Is AWESOME! This is perhaps the biggest departure from traditional architectures where automation and optimization are always things you’ll get to “eventually” but never do because the ROI is very hard to justify for these endeavors. In the cloud, however, the ROI for these activities is both tangible and immediate. Listen to your technical team and heed their advice on spending some time (and budget) to implement key optimizations and automations – and don’t be shy about asking them to justify the ROI. Spending $1,000 to save $20 doesn’t make any sense in any hosting environment. But spending $20 to save thousands is a no-brainer. Which brings me to… 5. 3rd Party Tools – AWS in particular has a host of tools for managing and alerting on costs, but they are all targeted at the single-account level. If you manage multiple AWS accounts, consider 3rd party tools that provide robust cost data across multiple accounts. If cost monitoring is time consuming and difficult or requires multiple account logins, nobody is going to do it. It’s well worth the investment in a tool that will improve compliance and visibility without a bunch of extra work.
It’s a brave new world that’s going to require not only a new set of tools to manage, but also new ways of thinking. Constantly juggling cost and agility can be a tricky balancing act, but a few reasonable practices put in place early on can go a long way. It’s true what they say about an ounce of prevention.