Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Rspamd symbols

1,039 views

Published on

Symbols and composite rules in Rspamd

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Rspamd symbols

  1. 1. SYMBOLS AND RULES RSPAMD
  2. 2. WHAT ARE SYMBOLS AND RULES DEFINITIONS RULE SYMBOL SCORE GROUP DESCRIPTION WEIGHT * true/false OPTIONS Dynamic part Static part ∑ Results FLAGS
  3. 3. WHAT ARE SYMBOLS AND RULES WHY DO WE NEED SYMBOLS RULE SYMBOL_ALLOW SYMBOL_DENY SYMBOL_UNKNOWN Either of symbols
  4. 4. WHAT ARE SYMBOLS AND RULES WHY DO WE NEED SYMBOLS RULE MAP1 MAP2 MAP3 Multiple symbols
  5. 5. WHAT ARE SYMBOLS AND RULES WHY DO WE NEED SYMBOLS RULE1 SYMBOL1 RULE2 Dependency
  6. 6. WHAT ARE SYMBOLS AND RULES RULES ▸ Rules define what is executed: ▸ Regexps expression ▸ Lua code ▸ Plugin logic ▸ Each rule can be associated with one or many symbols ▸ Rule can depend on other rules identified by associated symbols ▸ Each rule can define the current dynamic weight (usually from 0 to 1)
  7. 7. WHAT ARE SYMBOLS AND RULES SYMBOLS ▸ Symbols define meta-information of a rule: ▸ Name ▸ Static score ▸ Other data (description, group, flags, etc) ▸ Symbols can be: ▸ Normal: associated with exactly one rule ▸ Virtual: are not associated with rules but grouped with normal symbol) ▸ Callback: do not have name or score, just define common rule ▸ Special: have special purpose (e.g. composite symbols)
  8. 8. SYMBOLS SYMBOLS GROUPS SYMBOL1 SYMBOL2 SYMBOL3 SYMBOL4 Group1 SYMBOL3 SYMBOL4 SYMBOL5 SYMBOL6 Group2 GROUP LIMIT GROUP DESCRIPTION
  9. 9. SYMBOLS SYMBOLS GROUPS ▸ Groups join common symbols logically ▸ Groups can set joint limit for symbols scores enclosed ▸ Groups can be used in composite rules: ▸ SYMBOL5 && G:GROUP1 ▸ SYMBOL5 && (G:GROUP1 || !G:GROUP2)
  10. 10. RULES EXPRESSIONS IN RULES ▸ Expressions are used in: ▸ Regexp rules ▸ Composite symbols ▸ Expressions have common syntax: ▸ Logic operations: AND (&&), OR (||), NOT (!) ▸ Braces ▸ Limit operation: A + B + C > 2 ▸ Elements are called atoms
  11. 11. RULES REGEXP EXPRESSIONS ▸ Atoms are regular expressions (/re/flags): ▸ Header: Header=/re/H ▸ Mime (/P): scan text parts ▸ Body (/B): scan full undecoded body ▸ URL (/U): scan URLs found ▸ There is no order of regexps execution within an expression ▸ Same expressions are cached and executed once
  12. 12. COMPOSITE EXPRESSIONS COMPOSITES STRUCTURE SYMBOL3 SYMBOL4 SYMBOL5 SYMBOL1 Group2 NOT SYMBOL6 AND SYMBOL1 AND GR:GROUP2 AND !SYMBOL6 AND
  13. 13. COMPOSITE EXPRESSIONS COMPOSITES OPERATIONS SYMBOL3 SYMBOL4 SYMBOL5 SYMBOL1 Group2 NOT SYMBOL6 AND AND SYMBOL1 SYMBOL3 Symbols to remove Stage 1: Check Stage 2: Remove symbols
  14. 14. COMPOSITE EXPRESSIONS COMPOSITES STRUCTURE ▸ Composite atoms can include: ▸ Other symbols ▸ Groups (gr:) ▸ Other composites (with recursive references check) ▸ Composite operations can be the following: ▸ Remove symbol and weight (SYMBOL) ▸ Remove weight only (~SYMBOL) ▸ Remove symbol but preserve weight (-SYMBOL) ▸ Always remove symbol and weight (^SYMBOL)
  15. 15. COMPOSITE EXPRESSIONS COMPOSITES OPERATION ▸ If any composite proposes that a symbol should NOT be removed, then it is NOT removed: ▸ A & ~B and C & B: B will NOT be removed because of the first rule, but its weight will be removed ▸ A & -B and C & ~B: neither weight, nor symbol B will be removed ▸ Removal could be forced by “^” symbol: ▸ A & ^B and C & -B: weight and symbol B are both removed
  16. 16. PRACTICAL EXAMPLES A SIMPLE REGEXP EXPRESSION local reconf = config['regexp'] -- Define alias for regexp module -- Define a single regexp rule reconf['PRECEDENCE_BULK'] = { -- Header regexp that detects bulk email re = 'Precedence=/bulk/Hi', -- Default score score = 0.1, description = "Message marked as bulk", group = 'upstream_spam_filters' } rspamd.local.lua:
  17. 17. PRACTICAL EXAMPLES A MORE COMPLEX EXAMPLE rspamd.local.lua: local reconf = config['regexp'] -- Define alias for regexp module -- Define encodings types -- /X is undecoded header local subject_encoded_b64 = 'Subject=/=?S+?B?/iX' local subject_encoded_qp = 'Subject=/=?S+?Q?/iX' -- Define whether subject must be encoded (contains non-7bit characters) local subject_needs_mime = 'Subject=/[x00-x08x0bx0cx0e-x1fx7f-xff]/X' -- Final rule reconf['SUBJECT_NEEDS_ENCODING'] = { -- Combine regexps re = string.format('!(%s) & !(%s) & (%s)', subject_encoded_b64, subject_encoded_qp, subject_needs_mime), score = 3.5, description = "Subject contains non-ASCII chars but it is not encoded", group = 'headers' }
  18. 18. PRACTICAL EXAMPLES A MORE COMPLEX EXAMPLE rspamd.local.lua: local reconf = config['regexp'] -- Define alias for regexp module -- Define encodings types -- /X is undecoded header local subject_encoded_b64 = 'Subject=/=?S+?B?/iX' local subject_encoded_qp = 'Subject=/=?S+?Q?/iX' -- Define whether subject must be encoded (contains non-7bit characters) local subject_needs_mime = 'Subject=/[x00-x08x0bx0cx0e-x1fx7f-xff]/X' -- Final rule reconf['SUBJECT_NEEDS_ENCODING'] = { -- Combine regexps re = string.format('!(%s) & !(%s) & (%s)', subject_encoded_b64, subject_encoded_qp, subject_needs_mime), score = 3.5, description = "Subject contains non-ASCII chars but it is not encoded", group = 'headers' }
  19. 19. PRACTICAL EXAMPLES A MORE COMPLEX EXAMPLE rspamd.local.lua: local reconf = config['regexp'] -- Define alias for regexp module -- Define encodings types -- /X is undecoded header local subject_encoded_b64 = 'Subject=/=?S+?B?/iX' local subject_encoded_qp = 'Subject=/=?S+?Q?/iX' -- Define whether subject must be encoded (contains non-7bit characters) local subject_needs_mime = 'Subject=/[x00-x08x0bx0cx0e-x1fx7f-xff]/X' -- Final rule reconf['SUBJECT_NEEDS_ENCODING'] = { -- Combine regexps re = string.format('!(%s) & !(%s) & (%s)', subject_encoded_b64, subject_encoded_qp, subject_needs_mime), score = 3.5, description = "Subject contains non-ASCII chars but it is not encoded", group = 'headers' }
  20. 20. PRACTICAL EXAMPLES COMPOSITES EXAMPLE local.d/composites.conf: # Ignore forged recipients in case of mailing list composite "FORGED_RECIPIENTS_MAILLIST" { # MALLIST symbol is preserved expression = "FORGED_RECIPIENTS & -MAILLIST"; } # Ignore forged sender if a message has been forwarded composite "FORGED_SENDER_FORWARDING" { # Symbols from `forwarding` group are removed expression = "FORGED_SENDER & g:forwarding"; } # Ignore forged sender if a message has been from the mailing list composite "FORGED_SENDER_MAILLIST" { # Symbol 'FORGED_SENDER' is forced to be removed expression = "^FORGED_SENDER & -MAILLIST"; }
  21. 21. PRACTICAL EXAMPLES COMPOSITES EXAMPLE local.d/composites.conf: # Ignore forged recipients in case of mailing list composite "FORGED_RECIPIENTS_MAILLIST" { # MALLIST symbol is preserved expression = "FORGED_RECIPIENTS & -MAILLIST"; } # Ignore forged sender if a message has been forwarded composite "FORGED_SENDER_FORWARDING" { # Symbols from `forwarding` group are removed expression = "FORGED_SENDER & g:forwarding"; } # Ignore forged sender if a message has been from the mailing list composite "FORGED_SENDER_MAILLIST" { # Symbol 'FORGED_SENDER' is forced to be removed expression = "^FORGED_SENDER & -MAILLIST"; }
  22. 22. PRACTICAL EXAMPLES COMPOSITES EXAMPLE local.d/composites.conf: # Ignore forged recipients in case of mailing list composite "FORGED_RECIPIENTS_MAILLIST" { # MALLIST symbol is preserved expression = "FORGED_RECIPIENTS & -MAILLIST"; } # Ignore forged sender if a message has been forwarded composite "FORGED_SENDER_FORWARDING" { # Symbols from `forwarding` group are removed expression = "FORGED_SENDER & g:forwarding"; } # Ignore forged sender if a message has been from the mailing list composite "FORGED_SENDER_MAILLIST" { # Symbol 'FORGED_SENDER' is forced to be removed expression = "^FORGED_SENDER & -MAILLIST"; }
  23. 23. PRACTICAL EXAMPLES COMPOSITES EXAMPLE local.d/composites.conf: # Ignore forged recipients in case of mailing list composite "FORGED_RECIPIENTS_MAILLIST" { # MALLIST symbol is preserved expression = "FORGED_RECIPIENTS & -MAILLIST"; } # Ignore forged sender if a message has been forwarded composite "FORGED_SENDER_FORWARDING" { # Symbols from `forwarding` group are removed expression = "FORGED_SENDER & g:forwarding"; } # Ignore forged sender if a message has been from the mailing list composite "FORGED_SENDER_MAILLIST" { # Symbol 'FORGED_SENDER' is forced to be removed expression = "^FORGED_SENDER & -MAILLIST"; }
  24. 24. PRACTICAL EXAMPLES COMPOSITES EXAMPLE local.d/composites.conf: # Ignore forged recipients in case of mailing list composite "FORGED_RECIPIENTS_MAILLIST" { # MALLIST symbol is preserved expression = "FORGED_RECIPIENTS & -MAILLIST"; } # Ignore forged sender if a message has been forwarded composite "FORGED_SENDER_FORWARDING" { # Symbols from `forwarding` group are removed expression = "FORGED_SENDER & g:forwarding"; } # Ignore forged sender if a message has been from the mailing list composite "FORGED_SENDER_MAILLIST" { # Symbol 'FORGED_SENDER' is forced to be removed expression = "^FORGED_SENDER & -MAILLIST"; }

×