The paper presents a novel framework called AutoRE that can automatically generate regular expression signatures to detect spam emails. AutoRE analyzes URLs contained in emails to group similar domains and merge signatures, allowing it to detect future botnets not seen in training data. While AutoRE showed promising results on a Hotmail dataset, it has weaknesses like not addressing proxy URLs and inability to detect image spam. The paper is technically sound but could improve organization by separating discussion of AutoRE and botnet characteristics more clearly.

1. Group Details:-
Dhara Shah z3299353
Imad Hashmi z3193866
Zuo Cui z3261136
Our Paper:- Y. Xie , F. Yu, K. Achan , R. Panigraphy , G. Hulten and I. Osipkov , Spamming Botnets: Signatures and Characteristics, in Proceedings of ACM SIGCOMM 2008, pp. 171-182, Seattle, USA August 2008.
Is this paper technically sound?
Paper is based on the experiments conducted on 3 months data collected from the Hotmail‟s Server. To simulate similar results we needed the algorithm or rules used in the AutoRE software to generate regular expression and data on which experiments could be conducted.
To get the details of the software we tried contacting the Authors but unfortunately could not receive any reply from them (proof attached in appendix). We suspect that as it‟s a Microsoft group research and commercial product details are confidential. Hence we tried looking at the open source spam detection software to understand working of AutoRE. We could not compare the techniques used by the open source Spam Detection Software and AutoRE as we didn‟t had all details of AutoRE.
There are a number of spam detection tools available both commercial and open source but none of them is based on signatures. The idea in this paper is genuine and novel because other content based filters do not generate signatures and rely on a complete scan of the email. Following are some of the rules used to identify a spam URL[3]. We discuss URLs only because AutoRE works with URLs only:
 Uses a numeric IP address in URL
 Uses %-escapes inside a URL's hostname
 Completely unnecessary %-escapes inside a URL
 Dotted-decimal IP address followed by CGI
 Uses non-standard port number for HTTP
 Has Yahoo Redirect URI
 Contains an URL-encoded hostname (HTTP77)
 URI contains ".com" in middle
 URI contains ".com" in middle and end
 URI contains ".net" or ".org", then ".com"
 URI hostname has long hexadecimal sequence
 URI hostname has long non-vowel sequence
 CGI in .info TLD other than third-level "www"
 CGI in .biz TLD other than third-level "www"

2. There is a long list of email header criteria which can be applied to identify spam but that is beyond the scope here.
Next was we tried collecting data from the University‟s Mail server to verify the characteristics about the spam emails mentioned in the paper (proof attached in the appendix). But due to security issues concerned with the university we couldn‟t get the data. Hence we redirected our yahoo, Gmail and hotmail accounts to Cse account. And then accessing the Cse account via “pine” utility. Pine is a text based email reader which enables us to see detailed email headers. We tried distinguishing the email header of the Spam Email and a legitimate Email. But as Cse doesn‟t have an anti spam technology applied to it, it relies on the University‟s server for this. We verified this by observing that all the emails coming to Cse are being forwarded by the University‟s server. Also we understood that even if the user marks a email as spam, the system does not categorize it as spam until it satisfy the basic property of burstiness. We classified few legitimate email-ids as spam but the email server never classified it as spam as they were never sending in bulk.
Result from Pine is as follows:-
INFPACM003.services.comms.unsw.edu.au ([149.171.193.26]) (IP doesn't match sender domain)
(for <dsha472@cse.unsw.edu.au>) By note With Smtp ;
Fri, 18 Jun 2010 20:23:12 +1000
Received: from mta156.mail.in.yahoo.com ([203.84.221.168]) by INFPACM003.services.comms.unsw.edu.au with SMTP; 18 Jun 2010 20:02:46
+1000
Received: from 68.142.207.198 (HELO web32405.mail.mud.yahoo.com)
(68.142.207.198) by mta156.mail.in.yahoo.com with SMTP; Fri, 18 Jun 2010 15:53:07 +0530
Received: (qmail 20395 invoked by uid 60001); 18 Jun 2010 10:23:04 -0000
Received: from [117.193.43.248] by web32405.mail.mud.yahoo.com via HTTP; Fri
, 18 Jun 2010 03:23:03 PDT
Received: From INFPACM001.services.comms.unsw.edu.au ([149.171.193.18])
(for <dsha472@cse.unsw.edu.au>) By note With Smtp ;
Fri, 18 Jun 2010 20:04:32 +1000
Received: from mta177.mail.in.yahoo.com ([202.86.5.206]) by INFPACM001.services.comms.unsw.edu.au with SMTP; 18 Jun 2010 19:52:33
+1000
Received: from 65.54.190.16 (EHLO bay0-omc1-s5.bay0.hotmail.com)
(65.54.190.16) by mta177.mail.in.yahoo.com with SMTP; Fri, 18 Jun 2010 15:34:22 +0530
Received: from BL2PRD0102HT003.prod.exchangelabs.com ([65.54.190.61]) by bay0-omc1-s5.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
Fri, 18 Jun 2010 03:04:00 -0700
Received: from BL2PRD0102MB009.prod.exchangelabs.com ([169.254.34.168]) by BL2PRD0102HT003.prod.exchangelabs.com ([169.254.220.82]) with mapi; Fri, 18
Jun 2010 10:03:59 +0000

3. Are the ideas and results presented in this paper novel?
In our opinion, the idea of framework AutoRE is significantly novel. Although in some previous works, regular expressions were used for spam detection which is based on URLs in the email content; AutoRE is quite different from them. As can be seen from reasons below:
First, AutoRE has ability to automatically generate regular expressions based on the discovered URLs. Currently, man-made regular expressions are required in most detection framework. With the rapid growth of the number of spam, it becomes increasing tough even impossible to generate regular expressions manually. By learning from some methods of worm detection system (Singh's research [2]), AutoRE generates spam signature automatically. Therefore, this technique reduces the workload of human being and improves the veracity of regular expressions.
Second, AutoRE has capacity to predict the future domain-agnostic botnets. Most of previous researches and current detection frameworks are aiming at specific individual botnet. However, for those botnets which have similar behaviours, AutoRE cannot detect them automatically and they can only take action to the domain of those botnets which have been captured. For those possible future domain, these previous research is helpless. However, AutoRE is able to analyse and group the domains which have similar behaviour, and then merge domain-specific regular expressions into domain-agnostic regular expressions, therefore, AutoRE obtain the ability of detecting the domain both currently and in the future which possess same behaviour.
From these points of view, AutoRE can be considered as an innovative framework in the field of spam detection.
Are there any weaknesses of this paper that you have not mentioned in your answers to the above questions?
One of the weaknesses is that AutoRE doesn‟t deal with proxy URL. These proxy URLs usually have no relevance to their redirect destination, so it is hard to group them by using AutoRE. Although they can be traced from redirecting destination and using this destination address to detect whether it is a spam or not by AutoRE, but the tracing process is exactly as spammer‟s wishes. Currently, this situation cannot be improved in this paper. Another weakness is that AutoRE cannot detect the increasing image spam. So authors could borrow ideas from other image spam detection framework (like Uemura research [1]), using image‟s information, such as URL, file name or size, to improve this framework.

4. Do you think the results of this paper are of practical significance?
Even though AutoRE was only tested randomly on Hotmail, the result was pretty compelling. As the author mentioned, the regular expression signatures can detect 10 times more spam than previous complete URL based signatures and it can reduce the false positive rate of detection of botnet spam and host significantly. AutoRE is able to capture an additional 16-18% of the spam that bypassed well known spam filters (e.g. spamhaus). Meanwhile, at the present time, both the transient nature of the attack and the fact that only a few spam sent by each botnet make it more difficult for previous spam filtering frameworks detecting and blacklisting the individual bots. Hence, AutoRE becomes more practical for helping existing spam filtering frameworks to detect spam. And most importantly, AutoRE is also capable of “predicting” future botnets regardless of domain name, and besides, it is also quite useful for the characteristic of current botnets.
However, there is no single framework that can be permanent suitable for all kinds of spam. If AutoRE is widely used in real time, spam senders will try to find weaknesses of this framework, and further, find a way to counter the weaknesses and hide spam from being detected by AutoRE. Thus, AutoRE needs to update frequently to make it more efficiently.
What is your assessment of the readability, organization and overall presentation of the paper?
The idea of the paper has been well described overall. The reader gets a fair idea about what the author wants them to understand as they proceed with the topics. There is however a few improvements deemed important. The abstract section of the paper gives an impression as the software AutoRE processes the complete email contents including body for signature generation which is not the case. As the algorithm works only on the URLs inside the email contents it should be mentioned in the abstract section that this is not a content based filtering system. Another point that we noted is the focus of the paper which seems divided between two different topics; AutoRE and Botnet characteristics. Although the paper addresses both of these topics but they seem unrelated sometimes as AutoRE generates signatures only on already received collection of emails. The way these spam emails are sent and how different botnet characteristics effect that may be better described in a separate paper with more details and then can be referred here as required by AutoRE. There is a lot of detail associated with topics like dynamic and static IP addresses, email sending behaviours of botnets and traffic correlations. A lot of data and statistics can be collected on these lines for analysis. The paper itself suggests that this is an interesting future direction because due importance cannot be given to all areas in a single paper.

5. If you were a reviewer whose recommendation is being sought by the editor of the journal or the conference proceedings on whether or not to publish this paper, what would be your recommendation?
This is a very important topic and a well known subject. The authors does not need to explain too much about the importance as there is a lot of investment already being done in the field of spam detection. The authors also have a complete working implementation of the algorithm which has been tested on real world data. With the success results claimed by the authors the idea seems to carry a lot of weight although the software has not been in practice for unknown reasons.
The paper is definitely worth publishing in a related conference. The low false positive rates of applying AutoRE signatures is significantly less than the existing mechanisms although it does not cover the complete email contents.
How can the work presented in this paper be improved?
The paper tries to solve a very important problem of spam emails using a mix of content based and non-content based filtering. With significantly low false positive rate and detection of high number of spam campaigns, the results are quite impressive. However we suggest that the work can be improved in a number of ways.
 Improvement of Signature
Since AutoRE generates a signature of the spam campaign which it applies to emails arriving later to find out similarities. This signature creation can be improved in a number of ways. Currently it involves only the URLs inside the email message. This signature generation mechanism is incomplete since a lot of spam emails do not contain URLs.
 Handling of Proxy URLs
The system at the moment does not work with proxy URLs. This means that a lot of different URLs redirecting to a single resource will not be picked up by the signature. This can be solved by building a blacklist database of all domains providing redirection services to spammers. A domain found in multiple subsequent emails is a good candidate for the blacklist database. It will not be possible for spammers to quickly register new domains for redirection services.

6.  Keeping signature up-to-date
AutoRE works on historic data. Since it generates spam signatures and identified spam emails based on historic data it is a big challenge itself to keep those signatures correct and up-to-date. If the signature expires the low false positive rate may change significantly and the system may lose its strength. The paper does not explain anything about it. Having a mechanism to update the signature will heavily boost the software performance.
 Detecting Image spam
A lot of spam emails today are sent in the form of images. The purpose of using images is to hide email contents from content based filters. This important feature should be dealt with by content based filtering systems like AutoRE. One way of doing this is to generate signature of the image as well. Some basic characteristics like image size, type and dimensions can be recorded inside the email signature to identify similar images in other emails. Advanced image signature algorithms like colour histograms might not be possible to apply at such mass scale but calculating an image hash might turn out to be useful.
 Dependence on Botnet burstiness
AutoRE heavily relies on the burstiness property of spamming botnets with the assumption that the botnets will be rented for a small time only. This can ultimately result in generation of a totally incorrect spam signature if botnet start throttling the sending speed. However this topic remains wide open because waiting for the right spam email to be used as signature data is not the option.
Reference:
[1]Uemura, M& Tabata, T 2008 „Design and Evaluation of a Bayesian- filter-based Image Spam Filtering Method‟, 2008 International Conference on Information Security and Assurance, 2008 IEEE
[2]S. Singh, C. Estan, G. Varghese, and S. Savage. Automated worm fingerprinting. In OSDI, 2004.
[3]Apache SpamAssassin

7. Appendix:-
Following are the proof of our efforts:-
1. Letter from the IT Department of UNSW

8. Email to Microsoft Team:-
Respected Sir/Madam, I am a student at The University of New South Wales,Sydney,Australia.
Your paper on "Spamming Botnets: Signatures and Characteristics" is my anchor
paper for a research study in my course "Advance Computer Networks".
First of all, I would like to appreciate the manner in which the paper is written,
It was very interesting and inspiring to go through the paper.
Secondly I needed a favour from you to help me in research study on your paper.
I would be highly obliged if you could help in my research study.
I understand your limitations and would highly appreciate any help you could
provide me. I am hoping for some kind of pointers to move ahead on my research
work all, I am expected is to do is try and conduct some experiments on anchor
paper to understand the topic well and if possible come up with any difficulties
not mentioned in paper. I would be waiting for your reply eagerly. Thanks and Regards, Dhara Shah Master of Engineering Science specialization Information Technology The University of New South Wales Student.
Inquiry Regarding your paper on "Spamming Botnets : Signatures and Characteristics"
Dharaben Shah You forwarded this message on 4/13/2010 12:36 AM.
Sent:
Tuesday, April 13, 2010 12:34 AM
To:
rina@microsoft.com

9. Our Diary
Release Date: - 11th March, 2010.
Read abstract of 8 topics each and nominated 2 topics per group member by 17th March, 2010.
Got the final selected topic by 19th March, 2010.
Till 28th March went through the anchor paper thoroughly and wrote one page write-up as a summary of the understanding of the paper.
On 28th March decided the approach ahead. Our approach was we listed the references mentioned in the anchor paper and each on us was assigned 8 of them. Our objective was to find where the references were used in the anchor paper and to write a small summary explaining its use in the anchor paper. The Deadline for this work was 4th April. Every Monday we discussed our progress as it was our lab time.
Next was we mailed to the researchers of our anchor paper and tried to get coding of the software mentioned in our anchor paper. Our efforts were futile as the software was not available commercially and being a Microsoft research details were not revealed to us. Hence we decided to move ahead and gather more literature to find a way to experiment the anchor paper.
Till 11th April we had been working around anchor paper only as it took us time understanding and finding a way to experimenting. From 11th April for 2 weeks (till 25th April) following task was assigned to the group members: - Dhara - working on anchor paper and finding way to conduct experiment on it. Imad – Future work and related work. Zuo – Past and related work.
Outcome: - Possible area of exploitation are creating Botnet and sending emails to test various mail service provider and see how they detect spam email .Proving difference between Regular Expression and Token Conjunction Signature. 2 page write-up on key findings of the paper, future and background work.
From 25th April to 12th May we are working on presentation as our presentation was on 13th May. After 13th May from 13th May to 20th May we tried getting data from University mail server and tried setting up mail server to get data to testify findings. Due to failure in setting up the mail server from 21st May to 27th May we tried getting University data and setting up Botnet. From 27th may to 25th June we tried collecting data through pine and applying for University Data.