1. libinjection
New Techinques in Detecting SQLi Atttacks
iSEC Partners Open Forum
Gilt Group, New York, September 6 2102
Nick Galbreath @ngalbreath nickg@client9.com

2. http://client9.com/20120906/
Follow along at
your own pace

3. First presented at
Black Hat USA 2012
http://client9.com/20120725
iSEC Partners party at Bellagio

4. The Next 15 Minutes
• You know what SQLi is and why it's
important
• Why detecting SQLi is a hard problem
• Why current solutions aren't so good
• The libinjection algorithm and library

5. It's Easy to Get Started
with Regular Expressions
s/UNIONs+(ALL)?/i
‣ At least two open source WAF
use regular expressions.
‣ Failure cases in closed-source
WAFs also indicate regexp.

6. 1992 SQL Spec
625 pages plain text
bit.ly/10fmhZ

7. 2003 SQL Spec
128 pages pure BNF
bit.ly/OB5vfW

8. Integer Forms
• [0-9]+
• 0x[0-9a-fA-F]+ 0xDEADbeef
MySQL, MSSQL
0x is case sensitive
• 0x MSSQL only
• x'DEADbeef' PgSQL
• b'10101010' MySQL, PgSQL
• 0b010101 MySQL

9. Floating Point
• digits
• digits[.] http://bit.ly/Qp6KTu
• digits[.]digits • digits[.]digits[eE]digits
• digits[eE]digits • [.]digits
• digits[eE][+-]digits • [.]digits[eE]digits
• digits[.][eE]digits • [.]digits[eE][+-]digits
• digits[.]digits[eE][+-]digits • "binary_float_infinity" (O)
Optional starts with [+-]
Optional ending with [dDfF] (Oracle)

10. Money Literals
• MSSQL has a money type.
• -$45.12
• $123.0
• +$1,000,000.00 Commas ignored
• Haven't experimented with this yet.
• Does it auto-cast to a float or int type?

11. Ridiculous Operators
• != not equals, standard • ||/ cube root (pgsql)
• <=> mysql • ** exponents (oracle)
• <> mssql • # bitwise xor (pgsql
conflicts with mysql
• ^= oracle comment)
• !>, !< not less than mssql
• / oracle
• !! factorial (pgsql)
• |/ sqaure root (pgsql)

12. Strings,
Charsets,
and Comments
Such a cluster that I defer you to my DEFCON 20 talk.
http://client9.com/20120727/

13. (?:)s*whens*d+s*then)|(?:"s*(?:#|--|{))|(?:/*!s?d+)|(?:ch(?:a)?rs*(s*d)|(?:(?:(n?and|x?or|
not)s+||||&&)s*w+()
(?:[s()]cases*()|(?:)s*likes*()|(?:havings*[^s]+s*[^ws])|(?:ifs?([dw]s*[=<>~])
(?:"s*ors*"?d)|(?:x(?:23|27|3d))|(?:^.?"$)|(?:(?:^["]*(?:[d"]+|[^"]+"))+s*(?:n?and|x?or|not||
||&&)s*[w"[+&!@(),.-])|(?:[^ws]w+s*[|-]s*"s*w)|(?:@w+s+(and|or)s*["d]+)|(?:@[w-]+s(and|
or)s*[^ws])|(?:[^ws:]s*dW+[^ws]s*".)|(?:Winformation_schema|table_nameW)
(?:"s**.+(?:or|id)W*"d)|(?:^")|(?:^[ws"-]+(?<=ands)(?<=ors)(?<=xors)(?<=nands)(?<=nots)(?<=|
|)(?<=&&)w+()|(?:"[sd]*[^ws]+W*dW*.*["d])|(?:"s*[^ws?]+s*[^ws]+s*")|(?:"s*[^ws]+
s*[Wd].*(?:#|--))|(?:".**s*d)|(?:"s*ors[^d]+[w-]+.*d)|(?:[()*<>%+-][w-]+[^ws]+"[^,])
(?:d"s+"s+d)|(?:^admins*"|(/*)+"+s?(?:--|#|/*|{)?)|(?:"s*or[ws-]+s*[+<>=(),-]s*[d"])|
(?:"s*[^ws]?=s*")|(?:"W*[+=]+W*")|(?:"s*[!=|][ds!=+-]+.*["(].*$)|(?:"s*[!=|][ds!=]+.*d+$)|
(?:"s*likeW+[w"(])|(?:siss*0W)|(?:wheres[sw.,-]+s=)|(?:"[<>~]+")
(?:unions*(?:all|distinct|[(!@]*)?s*[([]*s*select)|(?:w+s+likes+")|(?:likes*"%)|(?:"s*like
W*["d])|(?:"s*(?:n?and|x?or|not ||||&&)s+[sw]+=s*w+s*having)|(?:"s**s*w+W+")|(?:"s*[^?
ws=.,;)(]+s*[(@"]*s*w+W+w)|(?:selects*[[]()sw.,"-]+from)|(?:find_in_sets*()
(?:ins*(+s*select)|(?:(?:n?and|x?or|not ||||&&)s+[sw+]+(?:regexps*(|soundss+likes*"|[=d]
+x))|("s*ds*(?:--|#))|(?:"[%&<>^=]+ds*(=|or))|(?:"W+[w+-]+s*=s*dW+")|(?:"s*iss*d.+"?w)|
(?:"|?[w-]{3,}[^ws.,]+")|(?:"s*iss*[d.]+s*W.*")
(?:[dW]s+ass*["w]+s*from)|(?:^[Wd]+s*(?:union|select|create|rename|truncate|load|alter|delete|
update|insert|desc))|(?:(?:select|create|rename|truncate|load|alter|delete|update|insert|desc)s+(?:
(?:group_)concat|char|load_file)s?(?)|(?:ends*);)|("s+regexpW)|(?:[s(]load_files*()
(?:@.+=s*(s*select)|(?:d+s*ors*d+s*[-+])|(?:/w+;?s+(?:having|and|or|select)W)|(?:ds+group
s+by.+()|(?:(?:;|#|--)s*(?:drop|alter))|(?:(?:;|#|--)s*(?:update|insert)s*w{2,})|(?:[^w]SETs*@w
+)|(?:(?:n?and|x?or|not ||||&&)[s(]+w+[s)]*[!=+]+[sd]*["=()])
(?:"s+ands*=W)|(?:(s*selects*w+s*()|(?:*/from)|(?:+s*d+s*+s*@)|(?:w"s*(?:[-+=|@]+s*)+
[d(])|(?:coalesces*(|@@w+s*[^ws])|(?:W!+"w)|(?:";s*(?:if|while|begin))|(?:"[sd]+=s*d)|
(?:orders+bys+ifw*s*()|(?:[s(]+cased*W.+[tw]hen[s(])
(?:(select|;)s+(?:benchmark|if|sleep)s*?(s*(?s*w+)
(?:creates+functions+w+s+returns)|(?:;s*(?:select|create|rename|truncate|load|alter|delete|update|
insert|desc)s*[[(]?w{2,})
(?:alters*w+.*characters+sets+w+)|(";s*waitfors+times+")|(?:";.*:s*goto)
(?:procedures+analyses*()|(?:;s*(declare|open)s+[w-]+)|(?:creates+(procedure|function)s*w+s*
(s*)s*-)|(?:declare[^w]+[@#]s*w+)|(execs*(s*@)
(?:selects*pg_sleep)|(?:waitfors*delays?"+s?d)|(?:;s*shutdowns*(?:;|--|#|/*|{))
(?:sexecs+xp_cmdshell)|(?:"s*!s*["w])|(?:fromW+information_schemaW)|(?:(?:(?:current_)?user|
database|schema|connection_id)s*([^)]*)|(?:";?s*(?:select|union|having)s*[^s])|(?:wiifs*()|
(?:execs+master.)|(?:union select @)|(?:union[w(s]*select)|(?:select.*w?user()|(?:into[s+]+
(?:dump|out)files*")
(?:merge.*usings*()|(executes*immediates*")|(?:W+d*s*havings*[^s-])|(?:matchs*[w(),+-]+

14. Proven Fail
‣ At Black Hat USA 2005, Hanson and
Patterson presented:
Guns and Butter: Towards Formal Axioms of
Validation (http://bit.ly/OBe7mJ)
‣ …formally proved that for any regex validator, we
could construct either a safe query which would be
flagged as dangerous, or a dangerous query which
would be flagged as correct.
‣ (summary from libdejector documentation)

15. Can we do better?

16. libinjection

17. Key Insight
‣ A SQLi attack must be parsed as SQL within
the original query.
‣ SQL has a rigid syntax
‣ it works, or it's a syntax error.
‣ Compare this to HTML/XSS rules
‣ "Is it a SQLi attack?" becomes
"Could it be a SQL snippet?"

18. Only 3 Contexts
User input is only "injected" into SQL in three
ways:
‣ As-Is
‣ Inside a single quoted string
‣ Inside a double quoted string
Means we have to parse input three times.
Compare to XSS

19. Identification of
SQL snippets
without context is hard
‣ 1-917-660-3400 my phone number or an
arithmetic expression in SQL?
‣ @ngalbreath my twitter account or a SQL
variable?
‣ English-like syntax and common keywords:
union, group, natural, left, right, join, top,
table, create, in, is, not, before, begin, between

20. Existing SQL Parsers
‣ Only parse their flavor of SQL
‣ Not well designed to handle snippets
‣ Hard to extend
‣ Worried about correctness
... so I wrote my own!

21. Tokenization
‣ Converts input into a stream of tokens
‣ Uses "master list" of keywords and functions
across all databases.
‣ Handles comments, string, literals, weirdos.

22. 5000224' UNION USER_ID>0--
[ ('...500224', string),
('UNION', union operator),
('USER_ID', name),
('>', operator),
('0', number),
('--.....', comment) ]

23. Meet the Tokens
‣ none/name ‣ group-like operation
‣ variable ‣ union-like operator
‣ string ‣ logical operator
‣ regular operator ‣ function
‣ unknown ‣ comma
‣ number ‣ semi-colon
‣ comment ‣ left parens
‣ keyword ‣ right parens

24. Merging,
Specialization,
Disambiguation
‣ "IS", "NOT" ==> "IS NOT" (single op)
‣ "NATURAL", "JOIN" => "NATURAL JOIN"
‣ ("+", operator) -> ("+", unary operator)
‣ (COS, function), (1, number) ==>
(COS, name), (1, number)
functions must be followed with a
parenthesis!

25. Folding
‣ This step actually isn't needed to detect, but
is needed to reduce false positives.
‣ Converts simple arithmetic expressions into a
single value (don't try to evaluate them).
‣ 1-917-660-3400 -> "1"

26. Knows nothing about SQLi
‣ So far this is purely a parsing problem.
‣ Knows nothing about SQLi (which is evolving)
‣ Can be 100% tested against any SQL input
(not SQLi) for correctness.
‣ Language independent test cases
$ cat test-tokens-numbers-floats-003.txt
--TEST--
floating-point parsing test
--INPUT--
SELECT .0;
--EXPECTED--
k SELECT
1 .0
; ;

27. Fingerprints
‣ The token types of a user input form a hash or
a fingerprint.
‣ -6270" UNION ALL SELECT 5594, 5594, 5594, 5594, 5594, 5594,
5594, 5594, 5594, 5594, 5594, 5594, 5594, 5594, 5594, 5594,
5594, 5594, 5594, 5594, 5594, 5594, 5594, 5594, 5594, 5594,
5594, 5594, 5594, 5594, 5594, 5594, 5594, 5594, 5594, 5594,
5594, 5594, 5594, 5594, 5594# AND "JWWQ"="JWWQ
‣ becomes "sUk1,1,1,1,1,1,1,1,&"
‣ Now let's generate fingerprints from
Real World Data.
‣ Can we distinguish between SQLi and benign
input?

28. Training on SQLi
‣ Parse known SQLi attacks from
‣ SQLi vulnerability scanners
‣ Published reports
‣ SQLI How-Tos
‣ > 32,000 total
‣ Since Black Hat, donations from
‣ modsecurity
‣ qualys
‣ > 50,000 total

29. Training on Real Input
‣ 100s of Millions of user inputs from Etsy's
access logs were also parsed.
‣ Large enough to get a good sample (Top 50
USA site)
‣ Old enough to have lots of odd ways of query
string formatting.
‣ Full text search with an diverse subject
domain

30. How many tokens are
needed to determine if
user input is SQLi or not?

31. 5
no matter how long the input

32. 622 out of 1,048,576 are SQLi
&1o1U &f()o &f(1) 1&((f 1&((k 1&(1, 1&(1o 1&(f( 1&(k( 1&(k1 1&(kf 1&(kk 1&(kn 1&(ko 1&(v) 1&1Bf 1&1Uk 1&1f( 1&1o( 1&1o1
1&1of 1&1ok 1&1on 1&1oo 1&1ov 1&f(( 1&f() 1&f(1 1&f(f 1&f(k 1&f(n 1&f(v 1&k(1 1&k(f 1&k1k 1&kUk 1&kk1 1&n() 1&no1 1&o(1
1&o1o 1&so1 1)&(1 1)&(f 1)&(k 1)&(n 1)&1B 1)&1U 1)&1f 1)&1o 1)&f( 1)&o( 1)()s 1))&( 1))&1 1))&f 1))&o 1)))& 1)))) 1)));
1)))B 1)))U 1)))k 1)))o 1));k 1))B1 1))Uk 1))Un 1))k1 1))kk 1))o( 1))o1 1))of 1))ok 1))on 1),(1 1);k& 1);k( 1);kf 1);kk
1);kn 1);ko 1)B1 1)B1& 1)B1c 1)B1o 1)U(k 1)Uk1 1)Ukf 1)Ukk 1)Ukn 1)Unk 1)k1 1)k1c 1)k1o 1)kks 1)o(1 1)o(k 1)o(n 1)o1B
1)o1f 1)o1k 1)o1o 1)of( 1)ok( 1)ok1 1)on& 1)ono 1,(f( 1,(k( 1,(k1 1,(kf 1,1), 1,1)o 1,1B1 1,1Uk 1,f(1 1,s), 1;k&k 1;k((
1;k(1 1;k(o 1;k1, 1;kf( 1;kks 1;kn( 1;kn, 1;knc 1;ko( 1;kok 1B1 1B1,1 1B1,n 1B1Uk 1B1c 1B1k1 1B1ks 1Bf(1 1Bf(f 1Bk(1
1Bn,n 1Bnk1 1U(k1 1U(kf 1U1,1 1Uc 1Uk 1Uk(1 1Uk(k 1Uk(n 1Uk1 1Uk1, 1Uk1c 1Uk1f 1Uk1k 1Uk1n 1Uk1o 1Ukf 1Ukf( 1Ukf,
1Ukk( 1Ukk, 1Ukk1 1Ukkk 1Ukkn 1Ukn& 1Ukn( 1Ukn, 1Ukn1 1Uknc 1Uknk 1Ukno 1Ukns 1Uko1 1Ukok 1Uks, 1Uksc 1Ukv 1Ukv, 1Ukvc
1Un,1 1Un1, 1Unk( 1Unk1 1Unkf 1Uon1 1f()k 1k1U( 1k1Uk 1k1c 1kU1, 1kf(1 1kk(1 1kksc 1knkn 1n&f( 1nUk1 1nUkn 1nk1c 1nkf(
1o((( 1o((1 1o((f 1o(1) 1o(1o 1o(f( 1o(k( 1o(k1 1o(kf 1o(kn 1o(kv 1o(n) 1o(s) 1o1)& 1o1)o 1o1Bf 1o1Uk 1o1f( 1o1kf 1o1o(
1o1o1 1o1of 1o1oo 1o1ov 1of() 1of(1 1of(f 1of(n 1of(s 1ok(1 1ok(k 1ok1 1ok1, 1ok1c 1ok1k 1okf( 1oks, 1oksc 1okv, 1onos
1oso1 ;kknc Uk1,1 Uk1,f Uk1,n Ukkkn f((k( f((kf f()&f f()of f(1)& f(1)o f(1,f f(1o1 f(f() f(f(1 f(k() f(k,( f(k,n f(n()
f(v,1 k()ok k(1)U k(ok( k(vv) k1,1, k1,1c k1,1k k1,f( k1k(k k1o(s k;non kf(1) kf(1, kf(f( kf(n, kf(v: kk(f( kk1kk kk1nk
kk1vk kk1vn kn1kk kn1vk kn1vn ko(k( ko(kf kok(k kv) kvk(1 n&(1) n&(k1 n&(o1 n&1f( n&1o( n&1o1 n&1of n&1oo n&f(1 n&f(f
n&k(1 n&o1o n)&(k n)&1f n)&1o n)&f( n))&( n))&1 n))&f n)))& n))); n)))k n)))o n));k n))kk n))o( n))o1 n))of n))ok n);k&
n);k( n);kf n);kk n);kn n);ko n)k1o n)kks n)o(k n)o1& n)o1f n)o1o n)of( n)ok( n,(f( n,(k( n,(k1 n,(kf n,f(1 n:o1U n;k&k
n;k(( n;k(1 n;kf( n;kks n;kn( n;ko( n;kok nUk(k nUk1, nUkn, nUnk( nk1Uk nkf(1 nkksc nnn)U nno1U no(k1 no(o1 no1&1 no1Uk
no1f( no1o( no1o1 no1of no1oo nof(1 nok(1 nok(k o1kf( oUk1, of()o of(1) ok1o1 okkkn ook1, s&(1) s&(1, s&(1o s&(f( s&(k)
s&(k1 s&(kf s&1Bf s&1Uk s&1c s&1f( s&1o( s&1o1 s&1of s&1on s&1oo s&1os s&1ov s&f(( s&f() s&f(1 s&f(f s&f(v s&k&s s&k(1
s&k(o s&k1o s&kc s&knk s&ko( s&ko1 s&kok s&kos s&n&s s&no1 s&nos s&o(1 s&o(k s&o1o s&okc s&oko s&os s&sos s&v:o s&vos
s&vso s)&(1 s)&(k s)&1B s)&1f s)&1o s)&f( s)&o( s))&( s))&1 s))&f s))&n s))&o s)))& s))); s)))B s)))U s)))k s)))o s));k
s))B1 s))Uk s))Un s))k1 s))kk s))o( s))o1 s))of s))ok s),(1 s);k& s);k( s);kf s);kk s);kn s);ko s)B1 s)B1& s)B1c s)B1o
s)U(k s)Uk1 s)Unk s)k1 s)k1c s)k1o s)kks s)o(1 s)o(k s)o1B s)o1f s)o1k s)o1o s)of( s)ok( s)ok1 s,1), s;k&k s;k(( s;k(1
s;k(o s;k1, s;k1o s;k; s;k[k s;k[n s;kf( s;kkn s;kks s;kn( s;knk s;knn s;ko( s;kok s;kvc s;kvk s;n:k sB1 sB1&s sB1Uk
sB1c sB1os sU((k sU(k( sU(kk sU(kn sU(ks sUk(k sUk1 sUk1& sUk1, sUk1c sUk1k sUk1o sUkf( sUkk1 sUkkk sUkn( sUkn, sUkn1
sUknk sUkok sUkv, sUkvc sUn(k sUnk1 sUnkf sUno1 sf(1) sf(n, sf(s) sk)&( sk)&1 sk)&f sk);k sk)B1 sk)Uk sk)Un sk)k1 sk)kk
sk)o( sk)o1 sk)of sk)ok sk1&1 sk1Uk sk1c sk1o1 sk1os skU1, skks skksc sn,f( sno1U so((( so((k so((s so(1) so(1o so(f(
so(k) so(k1 so(kk so(kn so(ko so(ks so(os so(s) so1&1 so1&o so1&s so1Bf so1Uk so1c so1f( so1kf so1o( so1o1 so1of so1ok
so1oo so1os so1ov sof() sof(1 sof(f sof(k sok&s sok(1 sok(k sok(o sok(s sok1 sok1, sok1c sok1o sokc sokf( sokn, soknk
soko( soko1 sokok sokos sonk1 sono1 sonos sos sos&( soso( sosos sov&1 sov&s sov:o sovo1 sovok sovos sovov sovso v:o1)
vUk1, vok1,

33. The Library
On GitHub Now
~500 Lines of Code
One file + data
No memory allocation
No threads
No external dependencies
Fixed stack size
>100k checks a second

34. tada
#include "sqlparse.h"
#include <string.h>
int main()
{
const char* ucg = "1 OR 1=1";
// input should be normalized, upper-cased
// You can use sqli_normalize
// if you don't have your own function
sfilter sf;
return is_sqli(&sf, ucg, strlen(ucg));
}
$ gcc -Wall -Wextra sample.c sqlparse.c
$ ./a.out
$ echo $?
1

35. What's Next?
• Change API to allow passing in
fingerprint data or a function. Allows
upgrades without code changes.
• Can we reduce the number of tokens?
String, variables, numbers are all just
values.
• Folding of comma-separated values?
1,2,3,4 => 1
• Can we just eliminate all parenthesis?

36. Help!
• More SQLi from the field please!
• False positives welcome
• More test cases with exotic SQL to test
parser.
• Ports to other languages (the language-
neutral test framework should make this
easier).
• Compiling on Windows (mostly tested on
Mac OS X and Linux)

37. Slides and Source Code:
http://www.client9.com/libinjection/
Nick Galbreath
@ngalbreath
nickg@client9.com
Sept 20 OWASP NY
SQL obfuscation and libinjection
Oct 25, OWASP USA, Austin, Texas
Continuous Deployment and Security
Thanks