Adding Memory Metrics for EC2

To ensure that your EC2 instances are properly sized in accordance with your real needs, clear and comprehensive insight into your EC2 performance metrics is a must. CloudWatch is a great Amazon service providing basic performance metrics on running instances. The metrics are collected on the hypervisor level, so the existing instance metrics include:

 

  • CPU utilization
  • I/O (bytes and operations)
  • Network (bytes and operations)

 

However the memory and swap utilization metrics are missing – these metrics are operating system specific and the hypervisor (at least at Amazon) does not have any visibility into the OS. Of course, without metrics on memory, it’s not so easy to make sizing decisions.

 

Custom CloudWatch Metrics

 

Amazon partially overcomes this deficiency by allowing customers to push any custom metrics into CloudWatch. They even wrote a set of very convenient sample scripts to push memory, swap and some other metrics. Amazon's original scripts may be found here for Linux and here for Windows. These scripts perform the main job of the data collection on the host, and then pushing these metrics into CloudWatch. However, Amazon scripts are just a sample: you must manually install a (sometimes long) list of prerequisites. You must manually create a sampling schedule and you must manually do that for any instance you wish to activate the monitoring on.

 

Making Things Better with Cloudyn Scripts

 

Cloudyn went one step further. We used Amazon scripts as a base, and wrapped it with a simple oneline installer. Cloudyn scripts check and install the prerequisites on most of the popular platforms, copy Amazon package and add an execution schedule. In order to allow the automated installation, we made the installers command-line based. The installation needs run as root on Linux, or as an administrative user on Windows. Cloudyn one-line installation command may be added into your Chef recipe or any other automated configuration script, so that any new machine is launched with the memory monitoring. To be able to push the CloudWatch metrics you need to make sure to have prepared the credentials for the IAM user with the appropriate permissions. Keep in mind the AWS user needs CloudWatch access. See Appendix A below for the security policy required for this IAM user.

 

Cloudyn Scripts on Linux

 

Let’s take a look at how to get things going on Linux.

 

Installing the package

 

To install the package on a Linux server, run the one-liner below (substitute the AAA and SSS with the actual IAM user access and secret key) as root:

curl http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.sh | AWSAccessKeyId=AAA AWSSecretKey=SSS Action=Install InstallLWP=Yes sh

This command will install all the prerequisites, download Amazon package to push the custom metrics, configure it and define a schedule to push the metrics every 15 minutes. Amazon script itself won’t be executing as root, but as a user with very limited permissions.


 

Uninstalling the package

 

To uninstall the package and remove the schedule 

 

curl http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.sh | Action=Uninstall sh

 

Checking the metrics for the last 2 hours

 

To see what metrics were pushed to CloudWatch in the last 2 hours, run

 

curl http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.sh | Action=GetCWData sh

 

 


Getting help/viewing all options

 

Finally, to see all the usage options, run:

 

curl http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.sh | Action=Usage sh

 

Cloudyn Scripts on Windows

 

Windows installation requires PowerShell 2.0 or higher to run and supports Windows Server 2008 and 2012.

 

Installing the package

 

Depending on your command processor, run the appropriate one-liner below (substitute the AAA and SSS with the actual IAM user access and secret key) as an administrative user.

 

For cmd.exe

a Install on cmd.exe (Command Prompt)

Powershell -c $o=new-object system.net.webclient; $o.downloadfile(\”http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs\”, \”%temp%\1.vbs\”); if($?){cscript %temp%\1.vbs /I /A AAA /S SSS}

 

 

For powershell:

b – Install on PowerShell

$o=new-object system.net.webclient; $o.downloadfile(”http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs”,$env:temp\\1.vbs”); if($?){cscript $env:temp\\1.vbs /I /A AAA /S SSS}

 

 

This command will install all the prerequisites, download Amazon package to push the custom metrics, configure it and define a schedule to push the metrics every 15 minutes.

 

Uninstalling the package

To uninstall the package and remove the schedule.

 

For cmd.exe:

c - Uninstall on cmd.exe

Powershell -c $o=new-object system.net.webclient; $o.downloadfile(\”http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs\”, \”%temp%\1.vbs\”); if($?){cscript %temp%\1.vbs /U}

 

 

For powershell:

d - Uninstall on PowerShell

$o=new-object system.net.webclient; $o.downloadfile(”http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs”,$env:temp\\1.vbs”); if($?){cscript $env:temp\\1.vbs /U}

 

 

 

Checking the metrics for the last 2 hours

To see what metrics were pushed to CloudWatch in the last 2 hours, run:

 

For cmd.exe: 

e - Check metrics on cmd.exe

powershell -c $o=new-object system.net.webclient; $o.downloadfile(\”http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs\”, \”%temp%\1.vbs\”); if($?){cscript %temp%\1.vbs /G}

 

 

For powershell:

f - Check metrics on PowerShell

$o=new-object system.net.webclient; $o.downloadfile(”http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs”,$env:temp\\1.vbs”); if($?){cscript $env:temp\\1.vbs /G}

 

 

Getting help/viewing all options

 

Finally, to see all the usage options, run:

 

For cmd.exe:

g - Get help on cmd.exe

powershell -c $o=new-object system.net.webclient; $o.downloadfile(\”http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs\”, \”%temp%\1.vbs\”); if($?){cscript %temp%\1.vbs /H}

 

 

For powershell:

h - Get help on PowerShell

$o=new-object system.net.webclient; $o.downloadfile(”http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs”,$env:temp\\1.vbs”); if($?){cscript $env:temp\\1.vbs /H}

 

 

Viewing Memory in Cloudyn Reports

 

If you elect to utilize CloudWatch for memory metrics, Cloudyn will show these metrics in its reports and will use them in the sizing recommendations. For example, you may see below our memory report:

 

 

In this report it’s clearly visible that the specified instance is using most of its RAM, and occasionally starts using some swap.

 

How Much Will it Cost Me?

 

Currently, Amazon pricing is $0.3 per metric per instance per month.

We collect a total of 4 metrics per instance:

  • Used Memory (MB)
  • Used Memory (%)
  • Used Swap (MB)
  • Used Swap (%)

 

This equals just $1.2/month/instance for instance that runs 24x7.

 

By identifying which instance types are your largest cost drivers, you can push custom scripts for just those instances for 1 month (recommended). At that point you should be ready to confidently make sizing decisions based on our recommendations, and uninstall the custom metrics with no further charges by Amazon.

For more information about Cloud Watch pricing see: https://aws.amazon.com/cloudwatch/pricing/

  

Conclusion

 

Memory is a necessary metric for any serious instance sizing. Cloudyn brings it all together in a very convenient way, presenting all the metrics together and helping with the sizing decisions. We recommend periodically defining the memory metrics for short periods of time, analyzing the results and remove the memory metrics to save the cost. Now, with Cloudyn auto-installation scripts, this task is easy.

 

Appendix A

 

IAM user credentials required to read/write the CloudWatch metrics:

 

i - IAM Policy for CloudWatch Metrics

 

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Sid": "Stmt1498725148000",

            "Effect": "Allow",

            "Action": [

                "cloudwatch:GetMetricStatistics",

                "cloudwatch:ListMetrics",

                "cloudwatch:PutMetricData"

            ],

            "Resource": [

                "*"

            ]

        }

    ]

}