Emerald Spam Shield

PDF Spam gone wild

Emerald Technology, Inc. — Fri, 03 Aug 2007 08:36:20 GMT

Is anyone else sick of PDF spam yet?

This has to be one of the dummest forms of spam yet. Outlook does not auto preview PDF files. And since we all know that spammers target Outlook what is the point? You would have to double click the PDF and launch Reader to see the stock image embedded in the PDF. Some of the new ones now include only text, and some are now zipping the PDF to get around the PDF block some companies have put into place.

Sad. I guess not enough idiots bought the pump and dump stock from just PDF spam, now they have to send millions more. I received over 5,000 in ONE email box yesterday. Wow. Like anyone would bother to open all of those and buy some stock that way?

What you should do

First, if you do not need PDF's in your company attachment list, just remove them. Set PDF as a blocked attachment. This is not a great solution, but it works.

Make sure your userlist is uploaded and wildcard receive is turned OFF.

Turn on Relay Delay. Yes, this slows down your first contact from remote users, but it really does work well for this type of spammer. If you can't afford to have email delays during the day, turn it on Friday night and leave it until Monday. That will help you with the huge deluge of spam you see first thing Monday morning.

Up the trust level of the RBLs, and set them to REJECT. Most of these new spammers are smart enough to stay off the RBLs, but it does help some.

Use a nonstandard port on your server. Spammers know that companies like Emerald exist and will try to get around us and connect direct to your server. Especially if your server is named mail.domain they will hit it more and more these days.

Spammers and bot nets getting smarter

They are now getting smarter in their blasting techniques. They will only send 10-20 at a time from a machine, and then let the machine stay idle for an hour or more. It means they have to run more bot nets, but it keeps their bots alive longer. The person with the machine probably does not notice a slight slowdown once an hour, and they stay useful to the spammer longer. I personally applaud the efforts of some of the ISP's now to block outbound port 25 from their residential customers. 99% of this traffic is probably bots sending spam.

Celeste with Small Business Trends Radio

Emerald Technology, Inc. — Fri, 09 Mar 2007 06:34:46 GMT

Celeste was interviewed for Small Business Trends radio!

I think you can tell she was pretty nervous (her first public speaking engagement).

She is talking about Email Archival and spam filtration. Click the button to listen!

Moving our Virus Scanner to ClamAV

Emerald Technology, Inc. — Wed, 28 Feb 2007 17:59:19 GMT

Moving our Virus Scanner to ClamAV

I wrote up a while back about my testing of ClamAV and why I thought it was a good product. It is, and I like it a lot. After that write up I spent more time testing, and was even more impressed. Some of their Phishing detection was better than any other AV product I tested. And the service native to Win32 is rock solid.

I ran over 500,000 spam messages through the test system. Not one fault. And I was pleasantly surprised that is detected a number of EBay scams, and other Phishing messages. That is an added bonus for us since we already scan for that type of stuff, but having a second opinion is a good thing.

The other big bonus for us is improved scan speed. Since the virus definitions are kept in RAM with the ClamD service, the clamdscan application does not have to load the database every email. It passes the data to the service and gets a reply over a local socket. This improves our scan speed, but it uses a LOT more CPU than our older solution. There is no free lunch, so we are willing to take the hit on CPU usage. CPU’s keep getting faster, and we will upgrade the processors on the servers if we need to in order to meet the load.

When is this going live?

This should be live on most of the servers by March 1, 2007. We expect all servers to be upgraded by March 4^th. You should not need to make any changes to your settings.

Country Code reporting useful again?

Emerald Technology, Inc. — Tue, 13 Feb 2007 15:02:54 GMT

These are interesting times to be in the spam industry for sure. Spammers are changing tactics on average about once every three weeks right now. And the rise of the stock spam is growing faster than anyone had ever thought possible. We are always evaluating our technique for effectiveness and changing to meet the times. Looks like it is time to implement the country code system again…

Country Codes – useful again?

Do you only receive email from certain countries? Would you like to be able to stop email based on the country it comes from?

We used to have a country blocking system, but over time it became very ineffective. Most of the spammers started using zombie machines from here in the USA. They were not sending from their originating servers. And in many cases they were using open relays in other countries as well.

We took a sample of 250,000 stock spams delivered over the past seven days and looked them up in the country code database. Any guess as to what percentage came from one country? 41% came from Russia, and 18% came from China directly. Wow. We did not expect that at all.

GeoIP database

We are going to start using the GeoIP Country database from MaxMind. They claim a 99% accuracy for countries, including AOL and other dialup providers that were almost always wrong in our old database. And they update their system every week. The pricing is very reasonable as well.

Update on our virus scanner plans

Emerald Technology, Inc. — Tue, 06 Feb 2007 10:50:23 GMT

Update on Virus Scanner changes

I have been looking at ClamAV for the past few days. I am very impressed. Not only have they managed to port the Unix anti-virus scanner to Windows, but they have done a good job of it. For those of you who do not know what ClamAV is I will provide a brief summary.

Clam AntiVirus is a GPL anti-virus toolkit for UNIX. The main purpose of this software is the integration with mail servers (attachment scanning). The package provides a flexible and scalable multi-threaded daemon, a command line scanner, and a tool for automatic updating via Internet. The programs are based on a shared library distributed with the Clam AntiVirus package, which you can use with your own software. Most importantly, the virus database is kept up to date .

They have also ported the base packages to Windows here. This is not a mere port. There is no Unix interop layer, and the ClamD (The service to use the windows terminology) is very small using less than 30 MB of RAM, and runs multi threaded very well.

If you are interested in their stand alone scanner take a look at the Clam Win site. This is the client experience application. It is a pretty good GUI, a scanning scheduler, auto downloading of updates, an Outlook plugin to scan attachments, and more. This is a LOT to offer for free. Now you have no reason not to install an anti-virus on all your PC’s. The only option missing from the system is a real time scan system. I personally have never really been a big fan of them since they can really impact your system performance. I almost always have mine turned off on my other systems.

Why use a client server anti-virus scanner?

Our current solution from F-Prot is not a client server system. So this is the basic flow that happens for each email:

· Email and file attachments are stored to disk

· Command line scanner is run

· Command line scanner has to load the entire virus database into ram

· File is scanned

· Result is returned and all memory is freed up

This is how we have done it since 2003 when we added the anti-virus scanning to the system. It is not very fast, but it does work. And since F-Prot was charging us a very good price I was willing to live with the performance hit. It has caused us a few backlogs over the years waiting for the scanners to finish files, but not enough to really worry about it. The “big guys” have nice SDK’s that you call in this manner:

· A service is started at Windows startup that loads the virus database

· Email and file attachments are stored to disk – OR – streamed across a socket to the service

· Command line scanner only contacts the service to tell it what to scan (very small memory footprint)

· File is scanned by the service and the result is returned

Obviously this is a much more efficient way of handling the files. But we could not afford to use most of these SDK based products due to costs. One vendor wanted over $4,000 for the SDK, and then to charge us over $10,000 per month for an unlimited mailbox license. No way is that cost effective.

Well, guess what? Clam AV has both of these models implemented. You can scan one file at a time, or use the service. Your choice. You can even install the scanner on a Unix machine and scan from the Windows to the Unix machine. Now that is what I call flexible.

The tests I performed

I have quite a huge sampling of spam and virus attachments on my drives since this is what we do for a living. So I took a small sample of 27,498 emails (246.3 MB) that arrived in our spam traps and ran them through the system. My first test was our existing F-Prot Version 3 engine. Then I tried the F-Prot Version 6 (new version). Then I also tried the Clam AV command line scanner, and the ClamD (think client / server) version as well. In all cases these files were in a temp directory on my drive locally. All of these tests were performed on an Intel Core 2 Duo CPU with 4 GB of RAM.

F-Prot Version 3
This is the system we have in place right now. It can scan at a decent speed, but is not the best solution for the reasons mentioned above. The files that were skipped are an oddity to me. None of them were really bad, they had bad encoding due to spammers doing crazy things, but the anti-virus scanner should not skip them in my opinion.
Files Tagged: 34
Files Skipped due to Error: 29
Total Time: 21:02

F-Prot Version 6
Not sure why they jumped the version numbers this way, but they did. This is the new model and will cost us a LOT more money if we were to deploy it. They have changed their licensing and we would no longer be able to use the versions that we have in place right now. F-Prot does have an SDK now, but it is not available to mere mortals like us without a lot of NDA paperwork and contracts - no thanks.
Files Tagged: 34
Files Skipped due to error: 24
Total Time: 1:18:34

ClamAV Command Line
This is the true apples to apples comparison because this version is a command line scanner just like F-Prot. It has different return codes, but it works basically the same way. Notices that zero files were skipped. Maybe they couldn’t decode them either, but they did scan what they found anyway. And they actually managed to tag two of the “bad” files from F-Prot as I-FRAME exploits. Kudos to ClamAV!
Files tagged: 36
Files skipped: 0
Total Time: 31:18

ClamAV Client Server
Now this is not the best comparison since F-Prot is command line and this is client server, but I am going to do it anyway. Same options as the ClamAV command line scan above, but with a much faster completion time. By far the fastest, and if I were to actually spin up two or more of these it can take advantage of the dual core chips much better since we are not duplicating the 25 MB virus database in ram for each instance. Running several threads did not provide a linear speed up mostly because the drive starts to become the limiting factor.

Single Threaded Test
Files tagged: 36
Files skipped: 0
Total time: 9:21

Five Thread Test
Files tagged: 36
Files skipped: 0
Total time: 5:47

Now I know some people will say that I could spin up multiple copies of the command line version as well. And you are correct, but I would be paying a much larger price in terms of memory usage. For each app that starts we would use approximately 28 MB of RAM. So starting five copies means you use 140 MB of RAM, where the ClamAV service still only uses that base 28 MB.

Should you get rid of your local anti virus?

Absolutely not. This system is not a replacement for other ways bad files get onto your system. And like any software it is not fool proof. Having more than one vendor scanning for bad attachments is always a good practice. You can just look at our service being your first line of defense, but you still need a local scanner.

What does all this mean?

OK, sorry for the rather long winded explanation as to why I think we are going to change our anti-virus system to ClamAV. It will mean less CPU usage, which means that we can continue to offer the best prices for our customers.

We are going to deploy the ClamAV system onto our spam traps and beta servers and evaluate it further. Make sure there are no crashes, memory leaks, etc. This is a normal deployment scenario for us whenever a component changes. We implement it internally, and then stage it to the beta servers. Our spam traps generate a TON of attacks and some really bad virus attachments, so it is a logical place for us to test this as well. If all goes well I expect to announce that we are moving everyone sometime in early March 2007.

As always, any concerns or feedback is greatly appreciated.

Anti Virus prices going up

Emerald Technology, Inc. — Tue, 30 Jan 2007 19:26:40 GMT

As we have discussed before we use F-Protect as our antivirus solution on our servers. Well, a new version of F-Protect has been released and they are changing the pricing model for servers. We have a corporate license agreement with them that expires in March, so look for prices to go up when we renew. I don't agree with their new pricing model, but we don't have much say in the matter either. (Let's just say the cost to us will be FIVE TIMES higher than what we pay now)

Pricing is now based on the number of mailboxes scanned. I don't understand why though. They don't pay for the CPU time, I do. There is no difference to them if I scan 10 files or 10 million files. I am the one paying the CPU time. The updates are the same. There is not even a real API to call their system, we use a command line app that returns status codes to us. That has been OK with us in the past because they were so inexpesive compared to the big guys. But the big guys have fancy API's that you call in process and avoid a lot of CPU overhead. I always figured it was a price / performance trade off. Now we are going to have to evaluate if F-Protect is the right antivirus vendor for us.

We had avoided most of the other vedors because they were extremely expensive for a small service like ours. Some of them were $4,000 or more for the server, and then you still had to pay on a per mailbox basis. That is just too expensive. We may have to look into ClamAV or some of the other smaller vendors out there. I had liked F-Prot because they always scored a 10/10 on all the antivirus roundup tests.

Anyone else have an anti virus solution that is cost effective for small business?

New File Formats for Office 2007

Emerald Technology, Inc. — Fri, 26 Jan 2007 21:25:15 GMT

New file extensions in Microsoft Office 2007

Microsoft has changed all their internal file formats for the new release of Office 2007. They are almost all XML based now, and they are open and documented. This is good news going forward, but could be a problem for a while until everyone gets upgraded. Fortunately they have provided the ability to “save as” an older format so you can send them to existing users, and they have added a few new nice changes as well.

PDF from Office
You can now save as a PDF direct from Office 2007 almost everywhere. It is not “built in”, but a free download from Microsoft’s site. The presenter at the show said that was due to legal reasons. Don’t expect to digitally sign or anything fancy with the PDF, but you can make basic PDF files with ease. And let’s face it, that is what 99% of the people do with the Acrobat system right now. Basic PDF output. That is why Adobe made the “Adobe Elements” product line. It could only make basic PDF’s, but they still charged you a license fee for it. You cannot import PDF files back into Office 2007 either. Only very simple output. So it is not a replacement for Acrobat, but it will meet the needs of most of your staff.

XPS File Format
XPS is a new file format for Windows Vista, but you can get viewers for it for legacy systems as well.

Here is the official text:

Document Experience

In Windows Vista, you can easily create a document in a paginated, fixed-layout format. An XML Paper Specification (XPS) Document is a page-by-page view of the document's content as it would be printed. In other words, it turns on-screen content into true electronic paper.
The XPS Document Viewer comes with Windows Vista and is also available for Windows XP and Windows Server 2003. The XPS Document Viewer allows you to open and read XPS Documents without using the original authoring application. You can also use the XPS Document Viewer to view and apply digital signatures to XPS Documents. The XPS Document Viewer is a Windows Rights Management Services (RMS)-enabled application so that any XPS Document protected with Windows RMS can be authenticated by the viewer.
2007 Microsoft Office system applications can create XPS Documents from within the application, making it very simple to create XPS Documents from applications you are already used to using. The Microsoft XPS Document Writer can also be used by any Windows-compatible application to easily create an XPS Document. The Microsoft XPS Document Writer is a print-to-file converter that creates XPS Document files.

Sounds a lot like a competitor to PDF, right? Here is the link to download the XPS viewer and more information at Microsoft.

Microsoft XPS Viewer

File Extension Changes

Most of the new file extensions are the same base, but with an X at the end for XML. So a Word .doc file becomes a .docx file.

These are the file extensions we will be adding to the default block lists of each customer. Feel free to edit them as always. Many of these can be used to install macro and plugin systems to Office 2007, so we are recommending you block them by default. It is only a matter of time until some hacker comes up with a clever way to use them to attack users.

XLSX, XLSM, XLSB, PRN, SLK, XLA, XLAM, XPS, DOCX, DOCM, DOTX, DOTM, MHT, MHTML, PSW, PWD

We will be adding these to accounts over the weekend and will post a note in the account alert when it is complete. If you do not want us to add them please let us know. If we see you have made major changes to your file list we will ask before making changes.

Anti Virus Update

Emerald Technology, Inc. — Wed, 24 Jan 2007 20:09:05 GMT

We are going to make a few changes to the anti virus detection system this week.

We currently use F-Prot for our anti virus scanning. They have started flagging some types of attachments as suspicious that we think should be tagged as a virus.

For example, one of our customers received an email with an EXE in it that contained a W32/Downloader.HYDY. This file is not actually a virus. But it does go out to the Internet and download the virus to install it on your machine. So, they return the code that it is suspicious.

We handle suspicious attachments by looking harder at the email. We add some points to the email for that response code, but if we don't find anything else we pass the email. The only files we had ever seen returned as suspicious in the past were VBScript files in zips, or encrypted zips that contained EXEs; things of that nature.

Our current thinking is to add a new handler and let you choose what to do with suspicious attachments. Tag them differently (something like maybe subject: [CAUTION] rather than simply tagging it as spam.

New Image Filter

Emerald Technology, Inc. — Wed, 24 Jan 2007 20:01:33 GMT

We released a new version of the image detection system yesterday.

Our current system was detecting images that were part of the background (like the snow for winter / flowers for spring) type of thing. These should now be safely ignored. We have already identified one older version of Outlook Express that does not attach them correctly (but modern Outlooks still show them to you). Please let us know if any others are tagged.

We have also changed the way we look at some types of image. Images that are part of a signature block should be safely ignored now as well.

Please let us know if you have any problems.

Autologout on the secure site

Emerald Technology, Inc. — Wed, 24 Jan 2007 18:51:53 GMT

You may notice that if you sit on a single page longer than fifteen minutes you will automatically be logged out. This is to release the server handles more quickly. We have seen some customers who login to the site and leave the browser window open for 12 hours or more. This can lead to errors on the site since session information expires much sooner than that.