Downloaded 19 times
Uploaded byVsevolod Stakhov
rspamd-hyperscan
This document discusses using Hyperscan to match regular expressions in Rspamd. It describes how Hyperscan can match many regular expressions against the same data faster than joining expressions with pipes in PCRE. However, Hyperscan has slower compile times and does not support some PCRE features. The document proposes a "lazy hyperscan compile" approach where Rspamd initially uses PCRE alone and later switches to the pre-compiled Hyperscan database to get the speed benefits without the initial compile overhead. It shows how this approach improves Rspamd performance for both small and large messages.
rspamd-hyperscan
- 1. Hyperscan in Rspamd WhenPCRE is not enough
- 2. Problem definition • Needto match many regular expressions for the same data: len: 610591, time: 2492.457ms real, 882.251ms virtual regexp statistics: 4095 pcre regexps scanned, 18 regexps matched, 694M bytes scanned using pcre • No need to capture patterns • Need to match both UTF8 and raw data • Need to support Perl compatible syntax
- 3. Naive solution • Joinall expressions with ‘|’: /abc/ + /cde/ => /(?:abc)|(?:cde)/ • Won’t work: /abc/ + /a/ => /(?:abc)|(?:a)/ -> the second rule won’t match for ‘abc’
- 4.
- 5. Rules and Hyperscan From Subject TextPart HTML Part Attachment Header RE class Header RE class Text Parts class Raw Body class Whole message m1 m2 m3 m4 m5 m6 MIME message Hyperscan regexps
- 6. Problems with Hyperscan •Slow compile time (like 100 times slower than PCRE+JIT) • Many PCRE features are unsupported (lookahead, look-behind) • Is not packaged for the major OSes (requires relatively fresh boost)
- 7. Lazy hyperscan compile HScompiler Main Scanners Compiled class Compiled class Compiled class Use PCRE only Compile to disk
- 8. Lazy hyperscan compile HScompiler Main Scanners Compiled class Compiled class Compiled class Use PCRE only All done Broadcast message
- 9. Lazy hyperscan compile HScompiler Main Scanners Compiled class Compiled class Compiled class Switch to hyperscan Loaded from cache Wait for ‘recompile’ command
- 10. Lazy hyperscan compile •Does not affect the starting time • If nothing changes then scanners loads pre- compiled expressions on startup (using crypto hashes) • Recompile command is used to force recompilation process: rspamadm recompile
- 11. Unsupported features Pre-filter mode Text /re1/pre /re2/ /re3/pre /re4/ /re5/ /re6/ /re7/ Scan Maybematch? Match Match Check PCRE Match! Replaces unsupported constructs with "weaker" supported ones, guaranteeing a match superset:
- 12. Unsupported features Approximation Try hyperscan Usehyperscan Try prefilter Use PCRE Use prefilter SuccessFail SuccessFail Timeout
- 13. Results • Use Hyperscanfor the vast majority of RE rules: len: 610591, time: 2492.457ms real, 882.251ms virtual regexp statistics: 4095 pcre regexps scanned, 18 regexps matched, 694M bytes scanned using pcre len: 610591, time: 654.596ms real, 309.785ms virtual regexp statistics: 34 pcre regexps scanned, 41 regexps matched, 8.41M bytes scanned using pcre, 9.56M bytes scanned total
- 14. Results • The defaultrules all compile with Hyperscan without prefiltering mode: compiled 235 regular expressions to the hyperscan tree loading hyperscan expressions after receiving compilation notice hyperscan database of 235 regexps has been loaded hyperscan database of 4118 regexps has been loaded • Works for most of the SA rules (some are prefiltering): • Speeds up both small and large messages
- 15.