AWS Cloud Cost and Usage Reporting

Last week was quite busy for us, as we were preparing to launch new feature – interactive AWS cost and usage reports. Well, it’s not totally new, as our users were getting Amazon’s cloud costs and usage reports weekly on their mailboxes, but these new in-app reports take it to whole new level! Lets take a look.

AWS Cost and Usage Weekly Reports

Every user that has connected their Amazon Web Services with CloudVertical account is receiving weekly Cost and Usage reports on their mailbox at the beginning of every week. Reports cover basic figures, trends, numbers and graphs that give you high level overview of your cloud performance. This is our way of keeping our users always on top of their cloud spend and usage, without bothering them to login to our app and explore all the data on their own.

We’ve received a lot of great feedback on those reports. Still, some users wanted to dig a bit deeper in their report data and be able to compare cost and usage from different weeks. Responding to this demand, we expanded “Reports” section of our app to not only let users turn reports on and off, but to give them access to reports archive and ability to interact with data used to generate report.

Interactive Reports

Under “Reports” tab every user that has connected their Amazon cloud account with CloudVertical can see their reports list. Right now CloudVertical offers two types of reports: AWS Cost Report and AWS Usage report. On that list you can turn reports on and off (if for whatever the reason, you don’t want to get weekly report on your mailbox). But now you can also expand the list to view reports archive. Of course – if you have just signed up and we’ve just started processing your data, reports won’t be available to you just yet, as we won’t have enough data to create them.

From here you’ll be able to view your reports in details. Just click “View” to move into interactive report view. First thing you’ll notice is beautiful interactive graph representing how amount of running instances, average cost per instance and CPU utilization were changing in time. You can zoom and scroll graph to see more detailed data for specific period.

Below you’ll see data that you got in your AWS usage report sections: key figures on EC2 Instances, EBS volumes and Network usage and cost. But there is one more thing: by clicking “Show Details” button you can expand those sections to see much more detailed information that was used to calculate summary values (and more!).

As mentioned above, that’s just a first iteration of CloudVertical’s Interactive Reports feature. There’s still much to do in that area and our dev team is busy working on improvements. If you have any questions or feedback, we’d love to hear from you – here in comments, on Twitter or via email.

10 Simple Steps to Reduce your AWS Cloud Costs

Cloud Computing is a fantastic way to get access to world-class, extremely resilient and scalable infrastructure – for a pay-as-you-go price – with no minimum entry cost. However once you get past the initial win of getting access to Infrastructure as a Service rather than ‘investing’ in hardware it becomes clear that the Cloud has a few cost surprises.

So with that in mind, here is 10 Simple Steps to Reduce your Amazon Web Services – AWS  - costs.

1. Scale DOWN as well as UP

Every application is different, and every business has different tolerances for uptime and performance. Find yours. Agree it. Lets say when an instance hits 80% CPU usage average for more than 5 minutes – you add another instance to your pool; and when it hits 20% usage average for more than 5 minutes, you terminate it. Set auto-scaling rules based on acceptable thresholds – and remember – auto-scaling is not only for scale-UP, you can scale down in line with business activity. Maybe you don’t need your entire instance pool online at 10pm on Friday.

2. Pre-pay if you have a known minimum usage

For most people, the Cloud isn’t about temporary one-off resources, it’s a about not having to buy equipment, and getting access to a massively scalable platform, on-demand. That means there’s likely to be a minimal load that will always be running. Figure out what this is, and buy Reserved Instances to match it. You can pay On-Demand prices in line with your Scaling, but for you minimal load, pre-pay for instances. You’ll easily save 35%+ over 12 months.

3. Stop Your Instances!

Are you or your developers working 24×7? No? Then why is your dev environment on all the time? What about that payroll server that gets used once a month? Or your back office applications that get hammered from 9-5, and then nothing out of hours and at the weekends. If you shut servers off out of business hours, just as an example, you’re going to save 70% of your costs (168 hours in a week : you should only pay for 40 of them in this case) - you don’t pay for servers that are not running.

Bonus Tip: If you can’t turn them off – resize them. Use an EBS volume and re-map it to a smaller server for those applications that need to be available 24×7, but whose load drops dramatically outside of standard hours.

4. Watch out for Waste – Cloud Sprawl

The Cloud is a variable resource, which means it’s easy to add resources and forget about them. VM sprawl happens internally all the time – but the cost can be managed with periodic audits, whereas in the Cloud, you pay for everything – whether or not you’re not using it. Prime examples here on AWS are Elastic IPs that you have detached from an instance, but have not released back to the pool – are charged per hour; also any EBS volumes you have mounted to instances, and then terminated the instance but forgot to get rid of the EBS volume, can start to add up.

5. Set Smart Alerts

Alerts don’t only need to be used to in times of crisis. Set alerts based on a daily budget, or specific usage alerts per instance, or across your cluster. Use these to manage your costs when usage is below a certain amount – you should be able to reduce the resources you have deployed.

6. Cost is a proxy for Usage

Not to harp on about the variable nature of the Cloud, but it is the primary fundamental difference between Cloud Computing and On-Premise Infrastructure. Because it’s a variable resource and your cost goes up or down in line with your usage – if your cost suddenly jumps up, or massively drops, it’s likely that something unplanned has happened – either a mis-configured auto-scaling group, or script, or a few instances have gone off-line (and you’re no longer billed for them …). Use Cost as a base level to monitor your Usage – you can do it right down to an hourly level – and get a feel for what the ‘normal’ is. Any deviations should be questioned.

7. Spot the Difference

AWS have a fantastic marketplace that sells off unused capacity for a (normally) heavily discounted price. Spot instances seem to run at an average of 40% of the on-demand price, so effective use of spot instances can save you big time. Not every workload suits this – but even aside from one-off data crunching jobs – adding spot instances to your pool makes a lot of sense if you want to ‘average’ a good performance benchmark, but you can tolerate minor service degradation in favour of the huge cost savings that can be made.

8. Understand Performance

Performance doesn’t mean your CPU and RAM usage – those metrics are really only useful to see what your server is doing. What if your application is massively inefficient in how it’s using those resources? Or a recent deployment has caused a memory leak, or processes to stall? Get a performance management tool  to analyse your log files and events – and try not just to get a high level of usage from your infrastructure, but to make to understand what your application performance benchmark is, and how to map your costs against that line. This is really the holy-grail of effective Cloud management.

9. The Clouds ‘Dirty Little Secret’

The web is abuzz about how Cloud enables startups to get going for near zero, and how businesses can now start AND SCALE in ways never before possible, the capital requirement to get up and running has completely left a startups financial model. Enterprises are disintermediating their IT departments and firing up instances so they can test new ideas and run their own applications. Great! But the ‘Dirty Little Secret’ is that Cloud is only cheap when you start; and only if you take advantage of its variable nature. For a static load – say 5 instances that run 24×7 at maximum efficiency – the Cloud can be more expensive that running managed or co-located infrastructure.

The major caveat here of course is that outsourcing is very attractive in most cases – but this is a common argument – the best Cloud use cases involve a hybrid model – we’ll cover this in another post. Understand your total ‘loaded server cost’ which is how much your internal or colocated server costs to depreciate, maintain and power – and compare that to the Cloud – baring in mind it’s not a like for like comparison unless you have a large redundant infrastructure in-house.

10. Know the Numbers

It’s amazing how many people don’t actually know the detail of their Cloud costs. What’s driving change? What will the estimated cost this quarter be? If we add 1,000 users or records, how will that impact cost? How efficient is our usage? Its common for someone on the management team to press for budgeting and analysis too – especially as the cost grows month on month. (Hint: that’s why we launched CloudVertical!)

In a later post I’ll go into specific use cases and how to manage costs in a more granular way, but for now I hope the few simple tips above are the start to getting control over your Cloud costs.