– Lori MacVittie, senior technical marketing manager at F5 Networks (www.f5.com), says:

In the past two years there have been interesting stories floating around about what happens when IP addresses are “shared” in public cloud computing environments. You’ve no doubt heard how someone spun up an instance and was immediately blacklisted by some other website because the last application assigned that IP address was naughty on the Internet.

Organizations have struggled with such issues and admittedly they are somewhat unique to the shared, up and down nature of cloud computing. In the history of Internet this type of scenario has certainly happened with dynamically assigned IP addresses from ISPs and even statically assigned addresses that were formerly assigned to someone who got the IP address listed as a SPAMmer or otherwise tagged as a carrier of illegal or malicious intent. But these are not the only issues with shared IP addresses. It’s not really that the organization dealing with such skeletons in their IP address closet are experiencing reputation damage, it’s that technical – and often very static – security systems are tightly coupled to IP addresses. They haven’t adequately, yet, evolved to deal with the reality of highly volatile IP address assignment. We still use IP address as a “unique” identifier even though today, particularly with public cloud computing, it’s anything but.

And if you thought being tagged as guilty by (IP) association was bad enough, you haven’t considered the potential for application “identity” theft that could leave applications open to back-door attacks.

MISTAKEN IDENTITY
So let’s assume that two applications are deployed in a cloud computing environment. Let us further assume that both use multiple instances and a load balancing service to maintain availability and scalability. What happens when Application A drops an instance – and therefore its associated IP address – and that IP address is subsequently assigned to Application B?
Usually nothing bad, that’s for sure. That’s just another minutes’ work in a public cloud computing environment. But what if the load balancing service for Application A did not drop the IP address.

Yeah. Isn’t that interesting? Application A is going to break (assuming a rudimentary but commonly used in cloud computing round robin algorithm) 25% of the time. Every fourth request will result almost assuredly in either an error or the wrong content. But that’s would actual be a good thing, considering the alternative possibilities.

1. The customer who now “owns” the IP address is a legitimate customer whose only intention is to deploy an application in the cloud. The customer is angrified because this case of mistaken (IP) identity is costing them money in terms of bandwidth and processing power to constantly field requests that should not be directed to them in the first place. Those requests are not valid for their application, but they pay to process and reject them nonetheless.

2. The customer who now “owns” the IP address is an evil genius with a plan to now use the mistaken (IP) identity to serve up malicious traffic. Perhaps a browser vulnerability-based injection. Perhaps some content that looks like it belongs on the site that tricks the user into confirming their login details, providing an easy means of credential theft. Or perhaps it’s the perfect means of delivering what looks like valid content to the page that’s nothing more than a phishing attack. Oh, the options if you’re an evil genius just go on and on. After all, the IP address is trusted by the application it is serving content to, so a lot of the obstacles that attackers must navigate to deliver malicious content on other sites are eliminated in this scenario.

Think such a scenario is far-fetched? It’s not. It’s not widespread (yet), either, and it’s difficult to target an application in such a manner because of the (seemingly) random nature of IP assignment in cloud computing environments, but mass SQL injection attacks aren’t exactly targeted, either. They’re designed to take advantage of a vulnerability when one is found, and part of its usage is the discovery of such an existing vulnerability. A maliciously minded person could certainly set up such an attack method and then sit back – and wait. After all, most attackers aren’t using their money to perpetrate such attacks anyway, so they’re unlikely to care that they’ll wrack up a lot of money in instance costs while waiting for a successful scenario to come about.

There are a couple of possibilities for how such a scenario could (and did) happen. Based on the nature of load balancing and dynamic environments, however, it is easy enough to extrapolate how such a situation could easily arise.

First and foremost this scenario is enabled by a tight-coupling between “application” and “IP” address. This tight-coupling is a dependency that is no longer necessary in well-architected high-availability environments and, as we are learning, can be dangerous to the health and security of applications in highly dynamic environments such as cloud computing where IP addresses are shared and cannot be relied upon as a definitive identifier of anything. This is particularly impactful in load balanced environments, where IP addresses are still used to identify nodes and where operations has failed to separate the application identity from an IP address.

Load balancing works by applying an algorithm to choose a resource from a pool of resources dynamically. There are numerous algorithms that can factor in a number of variables (resource capacity, performance, connection limits, etc…) but all rely upon information gathered from the application to determine suitability to fulfill any given request. In healthy load balanced environments there is a very nearly symbiotic relationship between the load balancing service and the applications, as it is from the application and its supporting infrastructure that the load balancing service receives the status information crucial to making decisions.

In this particular scenario, it is likely a failure to properly monitor the application compounded a failure of process to remove a resource from the pool in the first place. In a cloud computing style environment, this may have been the result of an automation failure.

1. AUTOMATION FAILURE
The load balancing service was either local to the cloud provider or at the customer’s site and upon release of the instance/IP address, whatever automation or process that was in place that should have removed that IP address from the pool of available resources did not. This could have been due to a disconnect between the way in which the instance was de-provisioned; perhaps the customer used a method that was unanticipated by the provider and thus no corresponding entry point into an automated process was provided for, making it impossible for an automated response from the load balancing service. Or perhaps the load balancing service was external to the provider, and the resources integrated into an existing load balanced system internal to the customer’s data center, in a hybrid-style architecture and no such automation exists and instead relied upon manual intervention. Human error is a common instigator of operational failures so this is a distinct possibility as well.

2. MONITORING FAILURE
All load balancing solutions use some kind of monitoring to ensure that a resource is actually available. This monitoring occurs on a fixed interval basis as determined by the service/configuration, and can be implemented using a variety of techniques – from ICMP replies to advanced content verification. ICMP, the most rudimentary but least costly (in terms of bandwidth and processing) method, is wholly inadequate to determine the availability of an application because it is designed only to garner a response at the network (IP) layer. TCP-based monitoring, as well, is designed to garner a response at the application or web server layer, and is better than ICMP but still fails to determine whether an application is actually available in the sense that it is (1) running and (2) returning valid responses. Advanced health monitoring that verifies the availability at the network, server and application layer is the only certain method to guarantee true application availability. Such methods involve making a request and receiving a response from the application and then verifying the content is what was expected. If it is not, the node (member, resource) can be marked as “unavailable” and taken out of the rotation for the load balancing algorithm, ensuring that no further requests to that resource are made.

AUTOMATION needs CONTEXT
Regardless of what disconnect caused the load balancing service to fail to remove the instance when it was decommissioned, proper health monitoring techniques would have caught the problem and resolved it such that neither the original assignee of the IP address was serving up invalid content (or failures) and the inheritor of the IP address would not be inundated with requests that resulted in extraneous and unnecessary costs.

More importantly, however, proper health monitoring would prevent a scenario in which the (application) identity of what may be a trusted service can be misappropriated and used to further some evil genius’ plans. Automation of any kind – and make no mistake, health monitoring that can automatically pull a resource out of a pull is a form of automation – requires context to ensure it makes the right decisions at the right time. It’s simply not enough to PING or open a TCP connection and assume that means anything other than basic networking and application platform services are available. In an environment where networks and resources are shared, the ability to ascertain availability of an application is critical not only to the successful delivery of an application but as we are learning, its security as well.