And You Thought All Spam Filters Were the SameAnd You Thought All Spam Filters Were the Same

Lab tests show differentiation among some leading products, though all provided strong results.

Spam threatens the effectiveness of email systems. Left unattended, spam clogs inboxes, reduces user efficiency, and overwhelms email systems. Spam can also let hackers gather confidential and proprietary information. It is essential for organizations to deploy top-tier email security solutions to mitigate the risks and problems associated with spam.

Email security appliances are evolving to be state-of-the-art defensive weapons against email-delivered network threats. Because spam, viruses, malware and other electronic attacks are delivered to email servers daily, it is important to choose the right product for your network. Miercom tested several email security appliances that are currently available:

* The Barracuda Spam & Virus Firewall 300
* The McAfee Email and Web Security 3100
* The McAfee Email Gateway 5000
* The WatchGuard Extensible Content Security (XCS) 570.

The objective was to evaluate the appliances for their content analysis accuracy (including false negatives, false positives and spam catch rate), virus detection capabilities, ease of use, installation and features. All our tests were designed to evaluate the devices in an out-of-the-box configuration and without any tuning.

With the constant emergence of new threats and attack vectors, email security solutions need to maintain a database with frequent updates, and stay current with the latest tools and techniques being used on email networks. To ensure that the spam filters could defend a network from the latest email threats, we tested them with a mix of Win32 malware samples and in-the-wild viruses. We used evasion techniques of archiving and compressing the virus samples. Filtering by attached file type or extension was disabled to get an accurate count for malware detection proficiency of the device. The appliances were tested with over 100,000 samples of malware and viruses.

Our analysis also included highly specific, stateful test cases that were built based on the state, structure and semantics of protocols, as well as their interdependencies on other protocols. Secure and robust targets should handle protocol-mutated packets by either dropping them or sending an error message. But an insecure target with protocol implementation flaws would respond abnormally or not at all. Miercom has observed, in other testing, reactions ranging from minor interruptions in legitimate traffic to complete appliance lock-ups due to mutation attacks causing untimely resets.

Our SMTP mutation attack consisted of 43,700 different stateful and stateless variants and attack vectors. Each variant/attack vector carried a single protocol mutation directed at each appliance tested. The different variants were implemented for SMTP Banners, DATA, EHLO, HELO, MAIL FROM, and RCPT- TO messages.

All the spam filtering vendors tested provided strong email security. The Barracuda Spam and Virus Firewall 300 attained high scores for blocking nearly all spam, although it was weakest in protecting against archived viruses and malware. The McAfee Email and Web Security 3100 achieved the highest scores for blocking malware and archived attachments, and had the lowest false positives rate. The McAfee Email Gateway 5000 scored highest for blocked spam percentage. The WatchGuard XCS 570 received high marks for having few false positives while scoring lower for overall blocked spam count.

Barracuda Spam and Virus Firewall 300 (Barracuda Spam Firewall 300)-version 3.5.12.0025
The Barracuda Spam & Virus Firewall 300 protects a network from spam, phishing, viruses and spyware attacks. It is an integrated hardware and software solution that is designed to work with all email servers, defending environments from small business to corporate.

This model consists of a 1U mini rack-mount chassis with one 10/100 port. It can support from 300 to 1,000 active email users and 250 domains with an 8 GB message log, and 10 GB quarantine file.

The firewall can be clustered for greater capacity and accessibility, and outbound filtering is done for attachment inspection, rate controls and encryption.

Our tests show that the Barracuda appliance blocked 99.14% of spam email and registered only 3 false positives when deployed in a production environment. Testing for email virus and malware protection, the Barracuda 300 scored a block rate of 70.15% against archived viruses and malware sent as email attachments.

We found the Barracuda Spam & Virus Firewall 300 to be easy to set up with no software to install or network modifications required and needing minimum ongoing administration. Updates are delivered automatically by Barracuda through an advanced technology center. While testing, security updates were distributed hourly for continuous protection against the latest threats. Miercom trained the Barracuda 300 for our own filtering preferences during testing.

We validated that administration is done through a web-based interface using secure remote access. Multiple domain support, journaling, delegated domain administration and help desk combine to make the appliance very easy to use and administer.

The Barracuda Spam & Virus Firewall 300 was able to filter all email messages and then display the number of emails received, with statistics on how they were processed. The Barracuda appliance offered the most detailed message logging and reporting, providing the administrator with reports, graphs and statistics through a web-based interface.

Support options include technical support, hourly spam and virus definition updates, Barracuda Reputation databases, fingerprint and intent analysis definitions, and spam and spam image rules. There is no per user license fee but there is the Barracuda Networks Updates subscription that offers the latest virus updates, spyware and content definitions. Barracuda offers multi-year subscriptions for a cost savings.

McAfee Email and Web Security 3100 appliance–version 5.5
The McAfee Email and Web Security 3100 appliance delivers anti-virus and malware protection capabilities, built-in compliance tools, usage policies, zero day threat protection, and comprehensive reporting and monitoring. The McAfee scanning engine identifies, sterilizes and guards emails from worms, viruses, rootkits, Trojans and other threats. Automated updates are sent to the appliance with the latest virus signature files to stop electronic attacks.

The 3100 is a 1U rack-mounted unit with one 10/100/1000 Ethernet port. It can support approximately 300 active email users.

Real-time malware protection reduces contact with the latest network threats. When a suspicious file is detected, it is sent to the McAfee Labs where it is examined. If recognized, a response for quarantine is issued immediately.

The 3100 comes with TrustedSource, a program that recognizes spam and other harmful email. TrustedSource drops network connections from known malware sources.

Advanced content filtering scans both in- and out-bound emails and attachments. Content-based policies can be created with rule wizards and built-in dictionaries.

Our tests show that the McAfee EWS 3100 achieved the second highest spam blocking percentage, at 99.65% and had the lowest number of false positives, registering only one false positive. The appliance attained 99.37% on blocking archived emails and malware. It blocked all SMTP mutation attacks.

We noted the following features of the appliance although these were not part of our review: the McAfee EWS 3100 has integrated URL filtering to block or permit HTTP requests based on predefined URL filtering profiles for Internet access. McAfee SiteAdvisor is a web reputation technology that judges website reputations from detailed content inspection and alerts you of unsafe websites.

Capacity can be increased by built-in clustering and load-balancing capabilities. Appliances can be added to a centrally managed cluster that provides consolidated reporting and updating.

The 3100 has turnkey installation and policy wizards that had us up and running in no time. We could manage the appliance through a browser with the quick-start wizard. The dashboard showed us a consolidated view of our status at a glance.

The McAfee ePolicy Orchestrator (ePO) software provides centralized policy management and graphical reporting. The ePO software manages systems, networks, data and compliance solutions. You can manage your McAfee Email and Web Security appliance along with the rest of your McAfee security, including other McAfee products securing your network, data, and endpoints. Comprehensive graphical reports keep you informed of the network's security status. The appliance manages anti-virus, anti-spyware, anti-spam, anti-phishing, and web filtering solutions for gateway devices, email servers, and desktop systems.

McAfee Email Gateway 5000 appliance
The McAfee Email Gateway 5000 appliance protects against email-borne threats and data loss via email. It analyzes the behavior of Internet articles such as IP addresses, Internet domains, specific URLs, images, and email messages. TrustedSource calculates the risk of interacting with these articles and can avoid them by relying on signature-based defenses.

This model is a 1U rack-mounted appliance with four 10/100/1000 Ethernet ports. It can support medium to large enterprises.

With inbound traffic, the Email Gateway 5000 protects against viruses, malware, phishing, directory harvest (DHA), Denial of Service (DoS), and bounceback attacks. Our analysis did not include outbound tests, but McAfee claims that the Email Gateway protects against outbound data loss by using six different encryption techniques along with alerts, re-routing, quarantine, blocking and creating customizable actions. Sensitive information is protected by integrated data loss prevention (DLP). It can analyze structured data (account numbers, credit card numbers) and unstructured data (company sensitive information, engineering schematics). Prebuilt dictionaries are used for regulations such as HIPAA, GLBA, and SOX.

Our tests show that the Gateway 5000 appliance achieved the highest spam blocking percentage of all four products tested at 99.79%; and registered 22 false positives while protecting against viruses, malware, phishing, directory harvest, DoS, bounce-back attacks, zero-hour threats, and spam surges when deployed in a production environment.

The Email Gateway uses McAfee Trusted Source technology to provide security against threats to the network. Through a global network of sensors and third-party intelligence sources, it analyzes the behavior of Internet objects or entities such as host IP addresses, Internet domains, specific URLs, images, and email messages and calculates the current reputation of working with these entities.

The McAfee EG 5000 features wizard-based installation, detailed reporting, real-time dashboard and alerts, scalability and stability. McAfee ePolicy Orchestrator (ePO) management platform integration allows for minimum administrative overhead. The appliance has exportable report logs, detailed reports and was flexible in policy creation.

The basic differentiation between the two McAfee products we tested is that the EG 5000 was added to the product line as a result of the company's IronMail acquisition. It is more scalable than the EWS 3100. However, the 3100 provides URL filtering in addition to email scanning.

WatchGuard Extensible Content Security (XCS) 570 appliance
The WatchGuard XCS 570 appliance protects against harmful email and attachments with multiple security layers, and stops threats at the network perimeter so that optimal bandwidth utilization is provided. Spam is quarantined on a local server and users can manage their quarantined messages. Message level redundancy works to prevent lost communications. Content filtering scans inbound email and protects the network from damaging messages and attachments that can be delivered by blended threats. The appliance uses real-time, cloud-based reputation services to examine sender information and content, including images, attachments, and embedded URLs.

Data loss is prevented through automatic blocking, rerouting and encrypting of suspicious messages and attachments by the appliance as it analyzes message contents. The system contains pre-defined and customizable compliance dictionaries for GLB, HIPAA, PCI and other regulations. The XCS 570 categorizes data files and is trained on what to recognize and what actions to take when such harmful data is found in outbound messages.

Our tests show that the WatchGuard XCS 570, which contains the Kaspersky anti-malware engine, achieved a high score of 99.01% with 4 false positives for all spam blocked by the appliance. For archived viruses and malware sent as attachments, the XCS 570 blocked 99.29%, and it blocked all SMTP mutation vector attacks. The XCS 570 supports Data Loss Prevention (DLP) with its ability to automatically block, quarantine, reroute, Bcc or allow messages based on user-defined policies. The 570 includes predefined regular expression for matching patterns of text. Email encryption is supported as an optional add-on, securing confidential messages to any recipient.

The system was observed to quarantine spam and suspect emails, directing spam to a local quarantine server and allowing end users to manage their quarantined messages, safe lists, and block lists from an easy-to-use, web-based interface.

We found the interface to be intuitive for configuring network security with easy set-up wizards. We used unified management views to view system actions in real-time.

LiveSecurity Standard Service is activated online when the product is registered, and it can be extended from 12 hours/5 days to 24 hours/7 days a week. A WatchGuard add-on subscription for mail encryption secures confidential messages for delivery to any recipient. It avoids the need for a dedicated server; thereby lowering costs when using encryption technology.

Conclusion
Test results demonstrated that the McAfee Email Gateway 5000 had the highest blocked spam percentage at 99.79% with the most false positives (valid emails filtered out) at 22. Administrators need to be aware that there are trade-offs to tighter security which may include an increase in the number of false positives. Barracuda Spam Firewall 300 produced a block rate of 99.14% with only three false positives, providing a strong balance between block rate and not blocking valid emails.

As email attacks become more sophisticated and mutations become commonplace, the ability to detect and remove mutated packets and infected attachments is very important. The WatchGuard XCS 570 exhibited strong capabilities in these areas, attaining Miercom Performance Verified Certification.

The results of our tests are summarized on the next page (Page 4) of this article, and a diagram and explanation of our test methodology can be found on Page 5.

Rob Smithers is CEO of Miercom, a leading testing lab and network consultancy. Rob can be reached at [email protected], and reviews can be arranged by contacting [email protected].

Test Bed Diagram and How We Did It
All testing was conducted at Miercom Labs in New Jersey. Each email security appliance was deployed in an out-of-the-box configuration without tuning or training the spam filter, and without any quarantine conditioning. All appliances were deployed in a production environment receiving 3,000 to 4,000 emails per day. This type of deployment was essential to accurately evaluate the behavior and performance of the product when deployed in an enterprise network. Once the messages were received, each was manually read and classified as spam or legitimate email. The spam blocking percentage and the total number of false positives were calculated.

To test for email malware detection capabilities of each product, we used automated scripts to send tens of thousands of emails with archived virus attachments. The attachments were not more than 5 MB in size with the spam filter platform configured to receive attached files no larger than 10 MB. Filtering by attached file type or extension was disabled to record the accurate malware detection proficiency of the device. The spam filter platform was tested with over 100,000 samples of malware and viruses, including zero day and in-the-wild viruses.

We used the Mu Test Suite by Mu Dynamics to perform security effectiveness assessments. This program was employed to conduct SMTP scans and mutation analyses with thousands of variations on valid service-level traffic.

The Ixia IxLoad was used to generate SMTP traffic during vulnerability testing of all appliances. IxLoad is a scalable solution for testing converged multiplay services and application delivery platforms. IxLoad emulates data, voice, video and protocols for performance testing.

The security assessment was conducted with IxDefend. It was used to generate exploits and attacks. Ixia's IxDefend is an advanced security assessment tool that can quickly find quality, resiliency, and security exposures across the broadest array of applications. It provides identification of known and zero-day threats in even the most hardened and complex protocol implementations. IxDefend tests over 40 protocols from link layer communications all the way up to application protocols. Each protocol in each bundle includes thousands of tests, each with its own detailed online documentation. IxDefend's tests provide the deepest possible protocol coverage.

The tests in this report are intended to be reproducible for customers who wish to recreate them with the appropriate test and measuring equipment. Contact [email protected] for additional details on the configurations applied to the system under test and test tools used in this evaluation. Miercom recommends customers conduct their own needs analysis study, and test specifically for the expected environment for product deployment before making a selection.