SciLifeLab Coffee & Code, Sept 25th 2020. An introduction to regular expressions at the SciLifeLab / NGI Sweden "Coffee 'n code" talk. Aimed at people who sort-of-know what regexes are, but find them a bit terrifying..  Watch the talk on YouTube: https://youtu.be/2Yp6kvdUMxM

1. REGEXES
PHIL EWELS, COFFEE ’N CODE
2019-09-24

2. REGEXESREGEXES

3. REG EXULAR PRESSIONS

4. REG EXULAR PRESSIONS
TL;DR; use this website:
https://regex101.com/

5. https://twitter.com/garabatokid

6. BASICS

7. AREGEXISAPATTERN
USEDTOSEARCHTEXT

8. AREGEXISAPATTERN
USEDTOSEARCHTEXT
/regex/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.

9. AREGEXISAPATTERN
USEDTOSEARCHTEXT
/regexp/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.

10. QUANTIFIERS
/regexp?/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.

11. QUANTIFIERS
/regu?l?a?r? ?exp?r?e?s?s?i?o?n?/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.

12. QUANTIFIERS
?
*
+
0 or 1
1 or more
0 or more
{2} Exactly 2
{0,4} Between 0 and 4

13. CHARACTERCLASSES
/reg[a-z ]*ex[a-z]*/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.

14. CHARACTERCLASSES
[abc]
[^abc]
[a-z]
Any of these characters
Anything in this character range
Anything except these characters

15. METACHARACTERS
(SHORTCUTS)
/regw*s?exw*/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.

16. METACHARACTERS
(SHORTCUTS)
w
s
Any word character
d
A whitespace character
A number
[A-Za-z0-9_]
[ trnf]
[0-9]
D Anything that’s not a number[^0-9]

17. THEDOT
/reg.*/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.
n
n
n
n

18. THEDOT
. Matches any character except a newline (n)
*(Except sometimes it also matches newlines)

19. ANCHORS
/^w/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.

20. ANCHORS
^
$
Start of the string
b
End of the string
A word boundary

21. THEPIPE
/reg|exp/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.

22. GROUPS
/re(gu|ss)/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.

23. ESCAPING
/(shortened/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.

24. LEVELTWO

25. GREEDYMATCHING
/regw*s?exw*/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.
(no newlines this time)

26. GREEDYMATCHING
/reg.*exw*/
(no newlines this time)
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.
A greedy quantifier

27. GREEDYMATCHING
/reg.*?exw*/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.
A lazy quantifier

28. REGEXMODIFIERS
/reg.*exw*/U
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.
Anything after the final / is a modifier

29. REGEXMODIFIERS
g
m
Global - don’t return after first match
i
Multiline - ^ and $ match start and end of each line(not the whole string)
Insensitive - case insensitive
U Ungreedy - Make all quantifiers ungreedy
s Single line - Make the dot match newlines too

30. GROUPMODIFIERS*
/re(gu|ss)/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.
*(not always available in all languages)
matches: [gu, ss, ss]

31. NON-CAPTURINGGROUPS
/re(?:gu|ss)/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.
matches: []

32. NON-CAPTURINGGROUPS
/([0-9]+)(?:st|nd|rd|th)/ This text contains the 1st, 2nd and 3rd
match but doesn’t match 4, 5 and 6.
matches: [1, 2, 3]

33. LOOKAHEADS
/regex/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.

34. LOOKAHEADS
/regex(?!p)/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.
A negative lookahead

35. LOOKAHEADS
/regex(?=p)/
A regular expression (shortened
as regex or regexp; also referred to
as rational expression) is a sequence
of characters that define a search pattern.
A positive lookahead

36. COMMONCONFUSION

37. FORGOTTOESCAPE
/s./ Don't forget to escape special characters.
Otherwise stuff will break.

38. FORGOTTOESCAPE
/s./ Don't forget to escape special characters.
Otherwise stuff will break.

39. UNEXPECTEDGREEDINESS
/a.*a/ Greedy can be dangerous at times

40. UNEXPECTEDGREEDINESS
/a.*?a/ Greedy can be dangerous at times

41. UNEXPECTEDEMPTINESS
/(aaa|bbb)*/ Greedy can be dangerous at times
Found a match!

42. UNEXPECTEDEMPTINESS
/(aaa|bbb)+/ Greedy can be dangerous at times
No match

43. SPECIALCHARACTERCONTEXTS
/a?/ Zero or one a characters
/a*?/ Zero or more a characters, ungreedy
/(?!a)/ Negative lookahead group for a character
/^a/ An a at the start of the string
/[^a]/ Any character except an a

44. REGEXENGINEDIFFERENCES
Basic regular expressions (BRE)
Extended regular expressions (ERE)
Perl-compatible regular expressions (PCRE)
grep sed vi
awk egrep =~
Many code languages
https://unix.stackexchange.com/questions/119905/why-does-my-regular-expression-work-in-x-but-not-in-y (thanks to Andreas Kähäri)

45. TOPTIPS

46. JUSTGOOGLEIT
There’s a Stack Overflow post
for every Regex I’ve ever
needed
ADAYMAYCOMEWHENILEARN
HOWREGEXESACTUALLYWORK
BUTITISNOTTHISDAY

47. USEREGEX101.COM

48. DON’TMATCHMORETHANYOU
NEEDTO
/(?:It's good to)?s*keep it simple: (d+),?s*(d+),?s*(d+)./
It's good to keep it simple: 1, 2, 3.

49. DON’TMATCHMORETHANYOU
NEEDTO
/(d+)/g
It's good to keep it simple: 1, 2, 3.

50. MULTIPLESIMPLEREGEXESARE
BETTERTHANONEBEHEMOTH

51. BREAKITOVERSEVERALLINES,
ADDCOMMENTS
my ($id, $description) = $line
=~ m/ ^ # Start of line
> # Fasta header
(S+) # Identifier
s+
(S+) # Description?
/x;
Credit: Johan Viklund, discussion on #code Slack channel (2020-05-15)

52. KNOWWHENNOTTOUSEAREGEX
https://stackoverflow.com/a/1732454

53. KNOWWHENNOTTOUSEAREGEX
"a" in "abc"
10000000 loops, best of 5: 20.2 nsec per loop
import re
pattern = re.compile(“a")
pattern.search("abc")
2000000 loops, best of 5: 136 nsec per loop
~ 7x faster

54. IFSPEEDDOESMATTER…
..ignore most of the previous tips
• Match more than you need to
• Smush everything into one regex
• Fail as fast as possible
• Avoid large character classes
• Be as specific as possible with quantifiers
• Use possessive quantifiers
• For long strings with lots of no-matches, unroll your loops

55. IFSPEEDDOESMATTER…
@SRR4292758.1 HWI-ST570:204:C83FNACXX:5:1101:1194:2176 length=50
AGAATNACAGCGGAAAATCGCTATCCGATTTACCCCGTAAACTGAATGAT
+SRR4292758.1 HWI-ST570:204:C83FNACXX:5:1101:1194:2176 length=50
CCCFF#2=AFAFHHIIIIGGEGDGFBDBFHIHHGDDFHIIGIHIIEHGIH
@SRR4292758.2 HWI-ST570:204:C83FNACXX:5:1101:1214:2181 length=50
CAAGANATGAAATCAGAAGTAAGAAAAAACATTTTGTTAAACAACTAAAA
+SRR4292758.2 HWI-ST570:204:C83FNACXX:5:1101:1214:2181 length=50
@@@DA#2ADBFFFFIII@F@FEFAHFFIIECGFIIIFFEFFFEIIIIBFI
@SRR4292758.3 HWI-ST570:204:C83FNACXX:5:1101:1155:2205 length=50
CTAAGTATTTTAGATAGACTAGACTAGGATNTATGTACAGATCCTCCTAG
+SRR4292758.3 HWI-ST570:204:C83FNACXX:5:1101:1155:2205 length=50
CCBFFDE:DFHHHHIIJJJIIJIJJGHJGG#2AFGHIIIIHIIJIJJJIJ
/=(d+)/
(116Mb, ~1M matches)
2.3066 seconds
/@SRR.{58}(d{2})/
0.8115 seconds
/^@SRR.{58}(d{2})$/
7.4183 seconds
Simple FastQ file

56. https://xkcd.com/208/

57. https://regexcrossword.com/

58. PhilEwelshttps://github.com/ewels/
https://twitter.com/tallphil/
https://phil.ewels.co.uk/
https://scilifelab.se/
https://ngisweden.scilifelab.se/