Python and automated statistics generation in Wikipedia using Jupyter, pywikibot, seaborn and GeoJSON

1. Wikipedia: the encyclopedia
anyone can edit…
even with Python
Miguel-Angel Monjas
mmonjas@gmail.com
miguelangelmonjas@wikimedia.es
Meetup Python Madrid, ICAI-ICADE, 2018-04-12

2. About me
Miguel-Angel Monjas:
● Telecom engineer, data scientist,
innovation coach… in Ericsson Spain
● Proud wikipedian since December 2004
● Member of Wikimedia Spain (wikimedia.es)
Author’s portrait © 2016-2018 Miguel-Angel Monjas
@mmonjaspro
http://www.linkedin.com/in/mmonjas
https://github.com/miguel-angel-monjas/
2

3. Wikipedia?
Old stuff but…
3

4. Pages
5

5. Pages:
history
6

6. Pages:
Wikitext
7

7. Pages:
Wikitext
8

8. Categories
9

9. Users
10

10. Users
11

11. Images
12

12. Images
13

13. Images
14

14. Images
15

15. Sites
16

16. Wikidata
Items
17

17. A Python bot framework:
pywikibot
Concepts and how-to
18

18. Bots:
A definition
A bot is a software application
that performs automated tasks.
Typically, bots perform tasks
that are both simple and
structurally repetitive, at a
much higher rate than would
be possible for a human alone.
In the Wikimedia projects, a
bot is any software application
that modifies any element of
the project, by uploading
pictures, updating a page…
either massively or not
19

19. pywikibot
● Python library and
collection of tools that
automate work on
MediaWiki sites.
● https://www.mediawiki.org/
wiki/Manual:Pywikibot
● https://doc.wikimedia.org/p
ywikibot/api_ref/pywikibot.
html
20

20. pywikibot: Site
21

21. pywikibot: Page
22

22. pywikibot: Page
23

23. pywikibot: Page
24

24. pywikibot: Page
25

25. pywikibot: Page
26

26. pywikibot: FilePage
27

27. pywikibot: FilePage
28

28. pywikibot: FilePage
29

29. 30
pywikibot: Category

30. 31
pywikibot: Category

31. pywikibot: User
32

32. Editing in Wikipedia
33

33. Editing in Wikipedia
34

34. 35

35. Editing in Wikipedia
36

36. 37

37. MediaWiki API
Another way to interact with the projects
38

38. The
MediaWiki API
● The MediaWiki API is a web
service that provides access
to wiki features, data, and
meta-data over HTTP, via a
URL usually at api.php
● https://www.mediawiki.org
/wiki/API:Main_page
● https://www.mediawiki.org
/wiki/API:FAQ
39

39. URL elements
https://commons.wikimedia.org/w/api.php ?
action=query &
format=json &
titles=File:Madrid - Puerta Alcala 01.jpg &
prop=globalusage &
guprop=url|namespace &
gulimit=500
END POINT
ACTION FORMAT
QUERY-SPECIFIC
PARAMETERS
METHOD: POST
40
QUERY TYPE

40. 41

41. 42
IMAGE TITLE
PAGE
NAME SPACE
PROJECT

42. 43

43. Extension:Kartographer
Maps capabilities in Wikimedia projects
44

44. Extension:
Kartographer
● A MediaWiki extension that
adds maps capabilities to
Wikimedia projects
● https://www.mediawiki.org/
wiki/Help:Extension:Kartogr
apher
● Based on OpenStreetMap
● Enabled by the <mapframe>
tag within Wikimedia pages
● Content must be valid
GeoJSON
(http://geojson.org/)
45

45. 46

46. IMAGE
POINT
LINK
47

47. TAG
MAP CENTER
Wikitext
MAP SIZE
48

48. FEATURE TYPE: POINT
COORDINATES
IMAGE
LINKMARKER FEATURES
GeoJSON
49
MARKER

49. PAWS (Pywikibot: A Web Server)
Local Pywikibot deployment not needed
any more
50

50. ● PAWS is a Jupyter Notebook
Server provided by the
Wikimedia Cloud Services.
● Pre-integrated with Pywikibot
and many other Python
packages
● pip, Git available…
● Authenticated (OAuth) with
your Wikimedia account.
● https://paws.wmflabs.org
● http://paws-
public.wmflabs.org/paws-
public/User:YOURUSERNAME/
51

51. 52

52. 53

53. 54

54. 55

55. 56

56. 57

57. 58

58. The home directory (minus
secret credential files) is public
by default
59

59. Attribution-ShareAlike 4.0
(CC BY-SA 4.0)
Except where otherwise noted, this work by Miguel-Angel Monjas is licensed under
https://creativecommons.org/licenses/by-sa/4.0/
60

60. Thank you!
mmonjas@gmail.com
miguelangelmonjas@wikimedia.es
61