Dividing a server into multiple virtual machines has brought down firms’ purchasing costs and allowed for more efficient use of existing hardware. However, virtualization also poses security risks and challenges, including managing a more complex network, additional layers of technology, potential data leaks as multiple virtual machines share common communication lines, and the threat of rogue machines.

The most common form of virtualization entails creating a layer–a hypervisor–between the cold, hard metal of the server and the virtual machines that sit on top of it. Each virtual machine has its own operating system and runs its own applications.

But someone who gains access to the hypervisor level could damage all the virtual machines, potentially bringing down multiple applications. “Server virtualization technologies are prone to security issues if the requisite security architecture and best practices are not in place,” says Eric Greenfeder, director of product management at San Francisco-based technology consultancy Primitive Logic.

Problems can also spread from one machine to another. “Security vulnerabilities in a single virtualized guest operating system can undermine the security of other virtual machines as well as the virtualization layer,” notes Greenfeder.

One difficulty is that the hypervisor layer exists outside the operating system–something without which most security applications such as firewalls and antivirus software cannot run. Security software vendors, virtualization technology providers and even hardware shops have all stepped forward to offer solutions.

Vulnerable OSs

Parag Patel, VP of alliances at Palo Alto, Calif.-based VMware, says his company’s new VMsafe allows security vendors to connect directly to the VMware infrastructure. “We’re enabling security products to have a lot more power,” he asserts.

A hypervisor layer, which is much thinner than the heavy operating systems on top of it, presents a tiny target, points out Patel. “The hypervisor has a much smaller footprint–so that gives you more protection, less holes, less vulnerability,” he says. “In fact, virtualization provides a more isolated and protected environment. A lot of vulnerability comes from operating systems.”

Intel Corp. is working on ways to build protection into the hardware of the server. “Intel developed Intel Virtualization Technology–hardware assists for virtualization–to increase the robustness and reliability in virtualization software,” says Radhakrishna Hiremane, product marketing engineer at Santa Clara, Calif.-based Intel.

Network-based security tools such as firewalls, intrusion detection systems and monitoring applications can protect a server from the outside, says Greenfeder of Primitive Logic.

OnPath Technologies, for one, provides network virtualization services, essentially creating separate networks within a single connection to keep data and messages isolated. The Marlton, N.J.-based company also provides monitoring tools to keep an eye on the network–and to shut down pieces of it quickly when necessary.

“If you were to have an application server or a file server on that storage device that contains sensitive financial data–trading records, customer data–and you were accidentally to plug in a network connection on the public side of the firewall, that sensitive data could be exposed to hackers or anyone else” with access, says OnPath president and CEO Peter Dougherty. “It’s very simple to expose corporate data in that manner, due to human error. Our products guard against that.”

OnPath currently has over 300 installations, including more than two dozen of the world’s largest securities firms, Dougherty says.

Leaky Machinery

When two or more virtual machines share a physical server, they also share the network cables, access to storage, and any other attached communication devices. As a result, data intended for one machine might wind up being read by another. Sensitive financial data could spill over into a less secure, or even public, environment.

Greenfeder recommends that administrators consider the security requirements of individual applications when deciding how to arrange them on virtual hosts. Virtual disk encryption can help safeguard data stored on a disk accessible by more than one virtual machine, and the same applies for networked storage, he says.

Some protection against data leakage can be built into the hypervisor level. Patel says that VMware’s hypervisor product provides isolation for virtual machines. “With the hypervisor, we have been able to create a lot of advanced features and functions because we can directly manage the hardware,” says Patel. “It allows virtual machines to be cordoned off, and create completely separate networks. It’s up to the user whether they want them to share information.”

While it may be difficult to keep up with the security patches, antivirus definitions and software upgrades for one machine, management overhead increases substantially when that machine has ten different virtual servers running on it. Manual oversight becomes a major challenge, and management software comes into play.

On the positive side, virtualization management tools create opportunities for system administrators to enforce security policies at system start-up and shutdown, when pausing virtual machines, cloning them or moving them, says Joe Fitzgerald, CEO of Mahwah, N.J.-based virtualization management software company ManageIQ. That allows security and compliance administrators to apply fine-tuned policies to their environment.

But to the extent that management software makes it easy to set up and create a virtual machine for legitimate uses, it also makes it possible for someone to do the same for nefarious reasons. “Virtual systems are easy to copy, and the availability of portable media makes it easy for a malicious operator to walk out with an entire production system that they can attack at their leisure,” explains Fitzgerald.

The speed at which virtual machines can be created also presents opportunities for worms and viruses to propagate within a corporate network, according to a study conducted by Tal Garfinkel, a graduate student in Stanford University’s computer science department.

“When worms hit conventional networks they will typically infect vulnerable machines fairly quickly,” Garfinkel says in the report. “Administrators can usually identify which machines are infected quite easily, clean up the infected machines, patch them to prevent re-infection, and rapidly bring the network back into its steady state.”

For networks populated with virtual machines, however, this approach doesn’t always work. “Infected virtual machines appear briefly, infect other machines, and disappear before they can be detected and their owner identified,” says Garfinkel.

Rogue Machines

Forgotten virtual machines don’t get patched on time and can provide viruses and worms a way into a corporate network. But the situation is worse when administrators don’t even know they were there to begin with.

Virtual machines can be created by individual users without any oversight from IT administrators–VMware offers free virtualization software that can be downloaded off the Internet. “Virtual machines are really easy to create, to the point where you can create one on your desktop,” says Richard Whitehead, director of product marketing at Waltham, Mass.-based Novell. “What you end up with is virtual sprawl. You have so many of these things … that you don’t know what virtual machines are out there, if they’re patched.”

A trusted employee can also create a virtual machine in order to have a safe place in which to release a virus, or do other things beyond the reach of prying eyes. “You could run an application then delete the entire virtual machine to erase your tracks,” says Whitehead.

Novell offers software to help track virtual machines, and also to lock down computers so that users cannot create the machines on their own. “The simplified side of virtual machines is that you can track them,” says Whitehead. “And your environment actually becomes more secure, in my opinion.”

Lemuel V. Cacho contributed to this report.

Article first appeared in Securities Industry News. (Paid subscription required.)