Forecast: Cloudy with a Chance of Low Visibility

Now that I have had a chance to re-coup from Shmoocon and the associated Shmoosnow Apocalypse, I wanted to get this post up. Great CON BTW! If you have the chance to go in the future, don't hesitate!

In December, I began noticing an uptick in scans looking for TCP 1080 (socks proxy) on my corporate firewalls. Not that unusual. But by New Years Day the scans began accounting for a large percentage of all deny's logged to my syslog servers. After some investigating, the fact that all source IP's were registered to Amazon's Ec2 Elastic cloud services became apparent. Egress filtering did not indicate any outbound connections to the IP addresses in question.

So began my adventures in reporting the issue to the Amazon abuse black hole. I initially reported the top source offender via ec2-abuse@amazon.com on Thursday January 7, 2010 and "promptly" received the following email on Monday January 11th.

Please file a report at
https://www.amazon.com/gp/html-forms-controller/AWSAbuse/

It is possible that the activity you see comes from an Amazon EC2 instance. This activity that you report was not, however, initiated by Amazon.

One of the biggest advantages of Amazon EC2 is that developers are given complete control of their instances. While the IPs may indicate that the network is Amazon's, our developer customers are the ones controlling the instances. You may learn more about EC2 at http://aws.amazon.com/ec2

That said, we do take reports of unauthorized network activity from our environment very seriously. It is specifically forbidden in our terms of use.

In order for us to identify the actual customer, please provide
* src IP
* dest IP (your IP)
* dest port
******************** Accurate date/timestamp and timezone of activity**************************
* Intensity/frequency (short log extracts)
* Your contact details (phone and email) Without these we will be unable to identify the correct owner of the IP address at that point in time.

Thank you

Best regards,

-EC2 Abuse Team

I obliged but cursed as I fought to fill out the report on the badly designed web form which kept throwing vague invalid input errors. The forms purpose is to facilitate the reporting of abuse between the reporter and Amazon EC2 customer while keeping both anonymous. So I attempted to keep a positive outlook with the hope that my time may assist an Amazon EC2 customer with a possible compromise.

After a week of no response, I followed up with their follow-up form located at here. During this time the scanning for open proxies on my firewalls had escalated and was accounting for more than 30% of all daily denied connections. So I began submitted the top source IP's (all Amazon EC2 addresses) with the associated logs. To date I have not received one response from any submissions.

So I wanted to share the breakdown of 30 days of logs acquired from my production firewalls. Destination port TCP 1080 made up 35.7% of all denied connections during the month of January 2010 (see breakdown of ports below).

Of those connection attempts, 43.7% of all source addresses resided from the same 10 addresses which were all registered to Amazon's EC2 Cloud services. All source addresses checked from the remaining sources were also registered to Amazon Ec2 Cloud services but for obvious reasons I did not check every source address. The amount of connection attempts has dropped since the end of January but are they still occurring at a good clip.

Others have noted abuse of Amazon EC2 cloud services in the past. Brian Krebs formerly of The Washington Post and now at Krebs On Security wrote about his experience with spammers leveraging Amazon EC2 services in July 2008. More recently, Amazon was found hosting command and control servers for the Zeus botnet. And while editing this post yesterday, I came across this article at ZDNet UK on subject. The article contains some good quotes from Rik Ferguson, Senior Security Adviser at Trend Micro.

"One of the things that persuades me personally that the cloud is absolutely a viable model and has longevity is that it has already been adopted by criminals," Ferguson said. "They are the people who are leading-edge adopters of technology that is going to work and going to stick around for a long time."

 "But now that criminals are moving into cloud services, what are you going to do? Block EC2 [Amazon Elastic Compute Cloud]? It becomes very much more difficult and I think that is an area that security companies and security professionals need to focus on."

Every ISP and ASP out there has to deal with issues similar to this. Let's face it the problem is not going away anytime soon. However, with the loss of visibility associated with the dynamic nature of cloud services, attractiveness of easy provisioning and setup for the non-technical, and the service providers desire to provide privacy for its customers, cloud services are certainly ripe for abuse. Let's hope providers, such as Amazon, can find a happy medium of providing needed services and privacy for its customers while offering others an effective mechanism for reporting and deterring any misuse and exploitation.

The Risk of Complexity

It is human nature to desire a shiny new technology based on marketing claims and feature promises. But many times during my career in information technology and security I have really questioned the “value add” of a particular solution or system. Will it really lower costs, improve employee performance, and facilitate collaboration? Will it provide the seamless interoperability between complex systems as advertised? Will it do all this and still provide stability and security? Or are we just attempting to throw complex technology at managerial, organizational, and performance issues as a fix?

Often, adding more complexity to technology will only make the issues associated with that technology more complex. These issues include security. Generally speaking, with more complexity comes less security. This is not necessarily because the ability to secure the technology does not exist but because it becomes out of reach due to resource limitations. These resource limitations include limitations in finances, time, and expertise. Complexity can increase the attack surface area of a network hence decreasing its security posture unless the proper training, planning, and defensive resources have been budgeted and obtained. Unfortunately, this is often not the case. Moreover, much of the technology used to secure and defend such solutions can increase the complexity of one’s information systems even further, potentially causing an endless loop of new features and defensive solutions.

Virtualization is a great example of this. The ability to virtualize operating systems, resources, and applications has many advantages in IT infrastructure and business. But the ease of virtualizing systems, combined with a lack of planning and available expertise in these products has the potential of creating an out of control scenario of misconfiguration and mismanagement. Proper change control, build procedures, code review, monitoring, disaster recovery planning, and documentation still need to be addressed. The security risk associated with virtualization needs to be assessed, managed, mitigated, and re-assessed on a regular basis. This can be a daunting task without the proper resources. Such resources may not have been factored in during the budgeting and planning process or may no longer exist during economic downturns.

I am not downplaying the incredible benefits of virtualization. I use virtualization too. However, much like any technology, it has its place and I don’t believe the “lets virtualize everything” mantra. The idiom of “don't put all your eggs in one basket” comes to mind. Doing so can be a serious mistake with dire consequences in assuring the confidentiality, integrity, and availability of data. I only use virtualization as an example, due to its prevalence in our industry and the complex baggage that often comes with it. There are dozens of other examples that could be used, but like most, I cite examples that I am familiar and comfortable with.

The recent compromise of Vaserv.com, a UK ISP, has been reported to affect over 100,000 hosted web sites which may never recover. Some have reported the attack was a result of vulnerability in the virtualization technology the web hosts were running on while others claim bad administrative practices are to blame. Some have questioned Vaserv’s disaster recovery and incident response procedures, or lack thereof. Most likely, it is a combination of these factors that contributed to this colossal failure. Was the complexity of the technology to blame? Was Vaserv.com naïve to think they could increase their profit margin by decreasing engineering and administrative costs through the use of virtualization? Or was the company putting all its “eggs in one basket” and ignoring the fundamentals of security?

These are only speculations on my part as I am, like most, not privy with the details of the compromise. The irony of this example is Vaserv.com was marketed as a low cost hosting solution. One may speculate that many companies and individuals chose their hosting services to save money only to incur a substantial financial loss associated with the incident. Some may feel I am simplifying the issue at hand but sometimes that is all that is needed.