MIME-aware Sieve filters

MailSite's sieve filter implementation has been enhanced to include MIME-aware parsing functionality and Body test capabilities.

These extensions define mechanisms to perform tests on MIME body parts by looping through the MIME structure.  Message parsing efficiency is greatly improved over previous implementations as only specified MIME body content-types are scanned (e.g. content-type text).  This eliminates the need to scan entire messages including binary parts when testing for string content.

These new extensions are currently only available via the advanced editor.  GUI support will be considered at a later stage.

1) MIME Header tests

Three new MIME tests have been added :filename, :type and :subtype.  The MIME test is coded as "x_mime" to denote its newly implemented status.

The syntax is:

x_mime [COMPARATOR][MATCH-TYPE]

Where COMPARATOR can be:
:filename
:type
:subtype
and MATCH-TYPE can be:
:is <value list>
:regex <regular expression>
:contains <value list>

A new control command: "x_foreverypart" has been introduced.  It acts as an iterator to walk though every Mime part of a message, including nested parts, and apply the specified block to each of them.

---

1.1) :filename examines the "Content-Disposition:" header field for its "filename" parameter. If there is no "Content-Disposition:" header field, then it will look at the "Content-Type:" header field for the "name" parameter.

Typical example filter:

require ["regex","reject","x_mime","x_foreverypart"];

/*
RuleName: Mime Filename Test
RuleDescription: Reject attachments with the .bat, .com and .exe extensions.
*/
if true {
   x_foreverypart {
if  anyof (x_mime :filename :regex "^.+\\.(bat|com|exe)$")
   {reject "Executable attachments rejected by filter";
   stop;
}}}

Sample message content detected:

Content-Disposition: attachment;
filename="GenKeyCli.exe"

Upgrade path suggestion:
Previous way to look for attachment extensions:
x_body:regex "filename=\".+\\.BAT\"",
x_body:regex "filename=\".+\\.EXE\"",

New improved MIME aware filter:
x_mime:filename :regex ["^.+\\.(bat|exe)$"]

Advantages:

• Scans only Mime attachment headers (avoids a full binary scan)
• Covers both "filename" and "name" attachment types

---

1.2) :type examines the "Content-Type:" header field type parameter.

Typical example filter:

require ["regex","reject","x_mime","x_foreverypart"];
/*
RuleName: Mime Type Test
RuleDescription: Reject messages with Image content
*/
if true {
   x_foreverypart {
if  anyof (x_mime :type :is "image")
      {reject "Image content rejected by filter";
       stop;
}}}

Sample message content detected:

Content-Type: image/jpeg; name="Picture.jpg"

---

1.3) :subtype examines the "Content-Type:" header field subtype parameter.

Typical example filter:

require ["regex","reject","x_mime","x_foreverypart"];
/*
RuleName: Mime SubType Test
RuleDescription: Reject messages with Image/Jpeg or Image/Gif content
*/
if true {
   x_foreverypart {
if  allof (x_mime :type :is "image", x_mime :subtype :regex ["jpeg","gif"])
      {reject "Image/Jpeg and Image/Gif content rejected by filter";
       stop;
}}}

Sample message content detected:

Content-Type:image/jpeg; name="Picture.jpg"

Upgrade path suggestion:

Previous way to look for this sort of content:

x_body :regex "Content-Type: image/jpeg"
x_body :regex "Content-Type: image/gif"

New improved MIME aware filter:
x_mime :type :is "image", x_mime :subtype :regex "jpeg|gif"

Advantages:

• Scans only Mime content-type image (avoids a full binary scan)
• Covers both subtypes in single more efficient regular expression
• Avoids possible miss-match with message body that contains the string "Content-Type: image/jpeg"

---

1.4) :subtype can also be used to determine, for example, if a message contains only a plain body part. Here we omit the x_foreverypart directive...

Typical example filter:

require ["regex","fileinto","x_mime"];

/*
RuleName: Mime SubType Test
RuleDescription: Redirect plain text messages
*/
if true {
if  x_mime :subtype :is "plain"
      {fileinto"plain text";
       stop;
}}

Sample message content detected:

Content-Type: text/plain; charset="iso-8859-1"

---

2) MIME-Aware Body Tests

The MIME Body test has been expanded to include three new variants :raw, :content, and :text.

Note: the "x_body" directive already existed, and all features previously supported will continue to function under the "x_body" directive.  The new features :raw, :content, :text are now available under the "body" directive which also includes the existing features.

The syntax is:

"body" [COMPARATOR] [MATCH-TYPE] [BODY-TRANSFORM]

Where COMPARATOR can be:
..existing functionality

and MATCH-TYPE can be:
..existing functionality

and BODY-TRANSFORM can be:

:content
:text
:raw

---

2.1) If the body transform is " :content", only MIME parts that have the specified content-types are selected for matching.  The following example shows the use of content-type "text" to match specified string.

Typical example filter:

require ["body","regex","reject"];

/*
RuleName: Body Content
RuleDescription: File message based on text content
*/
if true {
if  body :content "text" :contains "project plan"
      {fileinto"project/plan";
}}

---

2.2) ":text" body transform is a short hand method of using :content "text" to match strings or regular expressions against UTF-8 encoded text extracted from a message.

Typical example filter:

require ["body","regex","reject"];
/*
RuleName: Body Content
RuleDescription: Reject message based on work matching
*/
if true {
if  body :text :regex "\\b(expletive1|expletive2|expletive3)\\b"
       {reject "Message rejected by text content filter";
       stop;
}}

Upgrade path suggestion:

Previous way to look for particular words:

x_body:regex "(([ ]|^|<[^>]>)[,.;:'?!\"]*([a-zA-Z]+[-])?(expletive1|expletive2|expletive3)([-][a-zA-Z]+)?[,.;:'?!\"]*([]|$|</[^>]>))",
x_body:regex "(\\b> expletive1>\\b|\\bexpletive2\\b|\\bexpletive3\\b)"

New improved MIME aware filter:
body :text :regex "\\b(expletive1|expletive2|expletive3)\\b"

Advantages:

• Scans only MIME body content-type text elements of the message (avoiding a full binary scan)
• Removes the need to cover complex HTML coding in the rule
• More efficient regular expressions

---

2.3)": raw" looks at every byte after the headers right up to the end of the message, including all attachments and mime boundaries, without removing any transfer or character set encoding.  By using :raw the Mime body structure is effectively ignored: all multipart boundaries and Mime headers are therefore included as part of the text being matched. Using "body :raw" is therefore likely to be an expensive CPU operation as the full binary of the message (excluding headers) will be included as target content for scanning.

Typical example filter:

require["body","regex","reject"];
/*
RuleName: Body 1
RuleDescription: Match text against full content without removing transfer and character encoded body content
*/
if true {
if  body :raw :contains "this matching text"
      {reject "Message rejected by raw content filter";
       stop;
}}