What we did, learnt, how we build the data science system with Python and Django during the Antarctic Circumnavigation Expedition in 2016. Presented in Pycon-UK 2017.

1. Python and Django during the Antarctic
Circumnavigation Expedition (ACE)
PyCon UK, 2017. Carles Pina i Estany (carles@pina.cat)

2. What do we do for work?
Carles
C++/Qt software engineer at Mendeley
Some Python (scripts at work, evenings and weekends, Python code Dojos, etc.)
Using GNU/Linux for a long time
Long time ago: system administrator / network administrator
2 / 45

3. What do we do for work?
Carles
C++/Qt software engineer at Mendeley
Some Python (scripts at work, evenings and weekends, Python code Dojos, etc.)
Using GNU/Linux for a long time
Long time ago: system administrator / network administrator
Jen
Worked for 3.5 years at the British Antarctic Survey (BAS) as a data manager
Now works for the Swiss Polar Institute, following up on the data management of ACE
More into wildlife and science (especially birds) than programming
2 / 45

4. What were our roles on the ship?
As a team we:
managed the data of all 22 science projects
set up the data management recording system (Django);
set up the data storage and back-ups;
liaised with scientists to back up data, record metadata in a database and create
metadata records;
telecommunications setup:
onboard network;
e-mail system;
file uploader / Dropbox equivalent;
connecting science equipment.
3 / 45

5. What is ACE?
Antarctic Circumnavigation Expedition
4 / 45

6. Copyright: Swiss Polar Institute 5 / 45

7. R/V Akademik Tryoshnikov
6 / 45

8. 7 / 45

9. 8 / 45

10. Why is ACE different?
Usually science expeditions focus on a small area of the Antarctic
9 / 45

11. Why is ACE different?
Usually science expeditions focus on a small area of the Antarctic
Only other similar expedition was Discovery II in the 1950's
9 / 45

12. Why is ACE different?
Usually science expeditions focus on a small area of the Antarctic
Only other similar expedition was Discovery II in the 1950's
Sampled and studied marine and terrestrial habitats
9 / 45

13. Why is ACE different?
Usually science expeditions focus on a small area of the Antarctic
Only other similar expedition was Discovery II in the 1950's
Sampled and studied marine and terrestrial habitats
First expedition of the Swiss Polar Institute
9 / 45

14. Why is ACE different?
Usually science expeditions focus on a small area of the Antarctic
Only other similar expedition was Discovery II in the 1950's
Sampled and studied marine and terrestrial habitats
First expedition of the Swiss Polar Institute
It was organised very quickly
9 / 45

15. Science
10 / 45

16. Copyright Swiss Polar Institute 11 / 45

17. 12 / 45

18. 13 / 45

19. 14 / 45

20. Data management
15 / 45

21. What is it?
Backup all the collected data
Keep a log of what's happening:
ultimate goal - to be able to describe where, when, by whom and for what reason the
data were collected
Keep a database of all samples collected
Create metadata records of all data sets so they can be discovered
16 / 45

22. What is it?
Backup all the collected data
Keep a log of what's happening:
ultimate goal - to be able to describe where, when, by whom and for what reason the
data were collected
Keep a database of all samples collected
Create metadata records of all data sets so they can be discovered
Why?
Backup to avoid losing data and to archive it properly for the future
The data can be made publicly available after the 2 year embargo
Make data easier to discover and usable by others in the future
16 / 45

23. We had many IT tasks
Setup the data storage (2 NAS, 64 TB, 2 UPS, etc.)
17 / 45

24. We had many IT tasks
Setup the data storage (2 NAS, 64 TB, 2 UPS, etc.)
Create utilities to backup from USB hard disks and network-attached computers
17 / 45

25. We had many IT tasks
Setup the data storage (2 NAS, 64 TB, 2 UPS, etc.)
Create utilities to backup from USB hard disks and network-attached computers
Create an intranet webpage for the data entry, utilities and to disseminate information on
board
17 / 45

26. We had many IT tasks
Setup the data storage (2 NAS, 64 TB, 2 UPS, etc.)
Create utilities to backup from USB hard disks and network-attached computers
Create an intranet webpage for the data entry, utilities and to disseminate information on
board
Design and implement an e-mail system that works with an unstable 128 kbit connection
for 80 people
17 / 45

27. We had many IT tasks
Setup the data storage (2 NAS, 64 TB, 2 UPS, etc.)
Create utilities to backup from USB hard disks and network-attached computers
Create an intranet webpage for the data entry, utilities and to disseminate information on
board
Design and implement an e-mail system that works with an unstable 128 kbit connection
for 80 people
Create a way to upload videos for the news (we had journalists on board)
17 / 45

28. We had many IT tasks
Setup the data storage (2 NAS, 64 TB, 2 UPS, etc.)
Create utilities to backup from USB hard disks and network-attached computers
Create an intranet webpage for the data entry, utilities and to disseminate information on
board
Design and implement an e-mail system that works with an unstable 128 kbit connection
for 80 people
Create a way to upload videos for the news (we had journalists on board)
Download data (ice images, weather models) for the scientists
17 / 45

29. We had many IT tasks
Setup the data storage (2 NAS, 64 TB, 2 UPS, etc.)
Create utilities to backup from USB hard disks and network-attached computers
Create an intranet webpage for the data entry, utilities and to disseminate information on
board
Design and implement an e-mail system that works with an unstable 128 kbit connection
for 80 people
Create a way to upload videos for the news (we had journalists on board)
Download data (ice images, weather models) for the scientists
Help scientists to connect different equipment (like snow flake counter)
17 / 45

30. We had many IT tasks
Setup the data storage (2 NAS, 64 TB, 2 UPS, etc.)
Create utilities to backup from USB hard disks and network-attached computers
Create an intranet webpage for the data entry, utilities and to disseminate information on
board
Design and implement an e-mail system that works with an unstable 128 kbit connection
for 80 people
Create a way to upload videos for the news (we had journalists on board)
Download data (ice images, weather models) for the scientists
Help scientists to connect different equipment (like snow flake counter)
Fix a winch (software part)
17 / 45

31. We had many IT tasks
Setup the data storage (2 NAS, 64 TB, 2 UPS, etc.)
Create utilities to backup from USB hard disks and network-attached computers
Create an intranet webpage for the data entry, utilities and to disseminate information on
board
Design and implement an e-mail system that works with an unstable 128 kbit connection
for 80 people
Create a way to upload videos for the news (we had journalists on board)
Download data (ice images, weather models) for the scientists
Help scientists to connect different equipment (like snow flake counter)
Fix a winch (software part)
...
17 / 45

32. science-data-management
Jen said: Can you help me build a database?
18 / 45

33. science-data-management
Jen said: Can you help me build a database?
I answered: yes! (and thought of MySQL and create table...)
18 / 45

34. science-data-management
Jen said: Can you help me build a database?
I answered: yes! (and thought of MySQL and create table...)
Jen also meant a web system to enter data in the database!
18 / 45

35. 19 / 45

36. Django
20 / 45

37. I knew:
Python
21 / 45

38. I knew:
Python
Flask
21 / 45

39. I knew:
Python
Flask
MySQL
21 / 45

40. I knew:
Python
Flask
MySQL
SQLAlchemy
21 / 45

41. I knew:
Python
Flask
MySQL
SQLAlchemy
Mustache
21 / 45

42. I knew:
Python
Flask
MySQL
SQLAlchemy
Mustache
But... How could I create the forms to enter data into the database?
21 / 45

43. I knew:
Python
Flask
MySQL
SQLAlchemy
Mustache
But... How could I create the forms to enter data into the database?
I asked Fran... and he said...
21 / 45

44. I knew:
Python
Flask
MySQL
SQLAlchemy
Mustache
But... How could I create the forms to enter data into the database?
I asked Fran... and he said...
Django!
21 / 45

45. I knew:
Python
Flask
MySQL
SQLAlchemy
Mustache
But... How could I create the forms to enter data into the database?
I asked Fran... and he said...
Django!
(I thought... nooooo...)
21 / 45

46. I knew:
Python
Flask
MySQL
SQLAlchemy
Mustache
But... How could I create the forms to enter data into the database?
I asked Fran... and he said...
Django!
(I thought... nooooo...)
Fran created an example... and I loved it!
21 / 45

47. One week of Django
I went back home and really enjoyed learning Django:
Writing Django models is easier than MySQL create table (with foreign keys, etc.)
Easier to change the models and migrate
We needed authentication of users
Permissions for tables were useful
Templates are very well integrated with models
All the admin part that we used has a good default UI
Easy to extend
22 / 45

48. One week of Django
I went back home and really enjoyed learning Django:
Writing Django models is easier than MySQL create table (with foreign keys, etc.)
Easier to change the models and migrate
We needed authentication of users
Permissions for tables were useful
Templates are very well integrated with models
All the admin part that we used has a good default UI
Easy to extend
And it has very good off-line documentation!
22 / 45

49. Django models
class StorageCrate(models.Model):
name = models.CharField(max_length=255, unique=True)
location = models.CharField(max_length=255)
description = models.CharField(max_length=255, null=True, blank=True)
comment = models.TextField(null=True, blank=True)
23 / 45

50. Django models
class StorageCrate(models.Model):
name = models.CharField(max_length=255, unique=True)
location = models.CharField(max_length=255)
description = models.CharField(max_length=255, null=True, blank=True)
comment = models.TextField(null=True, blank=True)
It creates the forms automatically
If the model didn't exist: creates the tables in the database
If the model already existed: creates the migration code
Deals with foreign keys, relations, etc.
It creates relational database models like we would have done
23 / 45

51. science-cruise-data-management
http://github.com/cpina/science-cruise-data-management
24 / 45

52. 25 / 45

53. 26 / 45

54. 27 / 45

55. 28 / 45

56. 29 / 45

57. In retrospect
Django was a very good tool to do what we had to do:
Without Django we would have taken much longer!
Django didn't get in the way as much as I had thought
Django commands were very easy to implement and useful for accessing the models from
command line parameters, widely used for many of our scripts
Django developers are very nice and helpful! Thanks for the concise error messages:
main.Project.title: (fields.E120) CharFields must define a 'max_length' attribute.
30 / 45

58. In retrospect
Django was a very good tool to do what we had to do:
Without Django we would have taken much longer!
Django didn't get in the way as much as I had thought
Django commands were very easy to implement and useful for accessing the models from
command line parameters, widely used for many of our scripts
Django developers are very nice and helpful! Thanks for the concise error messages:
main.Project.title: (fields.E120) CharFields must define a 'max_length' attribute.
Django documentation is very well written!
Admin forms were easy to adapt and change
30 / 45

59. 31 / 45

60. What is Django?
From the Django website:
Django makes it easier to build
better Web apps more quickly and
with less code.
32 / 45

61. What is Django?
From the Django website:
Django makes it easier to build
better Web apps more quickly and
with less code.
Data manager thinks:
We need to build a database
32 / 45

62. What is Django?
From the Django website:
Django makes it easier to build
better Web apps more quickly and
with less code.
Data manager thinks:
We need to build a database
Django for data managers?
32 / 45

63. Very useful Django features for us
Admin forms were easy to adapt and change (we could work around something if it
wasn't possible)
commands are fantastic!
33 / 45

64. Very useful Django features for us
Admin forms were easy to adapt and change (we could work around something if it
wasn't possible)
commands are fantastic!
Django things that were not so helpful...
Some modules use resources (CSS, js, etc.) from the Internet (e.g. debug module). This
didn't work for us
Hard to change (only via changing the template?) the main admin page
33 / 45

65. pip and rsync
pip wasn't "bad connection friendly":
No progress bar
We didn't see how to easily see the URIs being download (e.g. apt-get --show-uris to
download them overnight)
When bad networking: error messages weren't clear
34 / 45

66. pip and rsync
pip wasn't "bad connection friendly":
No progress bar
We didn't see how to easily see the URIs being download (e.g. apt-get --show-uris to
download them overnight)
When bad networking: error messages weren't clear
man rsync, --progress:
This option tells rsync to print information showing the
progress of the transfer. This gives a bored user something to
watch.
I'd say "it helps my impatience".
34 / 45

67. pip and rsync
pip wasn't "bad connection friendly":
No progress bar
We didn't see how to easily see the URIs being download (e.g. apt-get --show-uris to
download them overnight)
When bad networking: error messages weren't clear
man rsync, --progress:
This option tells rsync to print information showing the
progress of the transfer. This gives a bored user something to
watch.
I'd say "it helps my impatience".
Python documentation
More comprehensive off-line documentation could have been better (we relied on examples
and source code)
34 / 45

68. Django closing the circle
We wanted to thank Django somehow for all the fantastic help.
I thought of fixing at least one bug that we experienced during the expedition
35 / 45

69. Django closing the circle
We wanted to thank Django somehow for all the fantastic help.
I thought of fixing at least one bug that we experienced during the expedition
Bug #28120 reported and fixed! (thanks for the quick code review, merge...)
35 / 45

70. Python/Django ideas if you go on
an expedition
Take lots of code with you (e.g. Calibre, other projects, etc.). This was more useful than
books and documentation
36 / 45

71. Python/Django ideas if you go on
an expedition
Take lots of code with you (e.g. Calibre, other projects, etc.). This was more useful than
books and documentation
Use an IDE like Pycharm that allows you to navigate code of your libraries easier
36 / 45

72. Python/Django ideas if you go on
an expedition
Take lots of code with you (e.g. Calibre, other projects, etc.). This was more useful than
books and documentation
Use an IDE like Pycharm that allows you to navigate code of your libraries easier
Download the offline Django documentation: you will feel almost online!
36 / 45

73. Python/Django ideas if you go on
an expedition
Take lots of code with you (e.g. Calibre, other projects, etc.). This was more useful than
books and documentation
Use an IDE like Pycharm that allows you to navigate code of your libraries easier
Download the offline Django documentation: you will feel almost online!
Actually also download Wikipedia!
36 / 45

74. Appendix: Internet on the ship
37 / 45

75. Introduction
We had 2 Iridium systems
Initially one for calls, another one for Internet
This is, 128 kbits for (officialy) 80 people
The Iridium connections are very unstable
38 / 45

76. L. Wood, SaVi: satellite constellation visualization, First Annual CCSR Research Symposium
(CRS 2011), Centre for Communication Systems Research, 30 June 2011.
http://arxiv.org/abs/1204.3265
39 / 45

77. Iridium
66 Iridium satellites
40 / 45

78. Iridium
66 Iridium satellites
Iridium name: originally supposed to be 77 satellites: atomic number of Iridium
40 / 45

79. Iridium
66 Iridium satellites
Iridium name: originally supposed to be 77 satellites: atomic number of Iridium
There are spare satellites for when there are problems
40 / 45

80. Iridium
66 Iridium satellites
Iridium name: originally supposed to be 77 satellites: atomic number of Iridium
There are spare satellites for when there are problems
Signal on the north and south pole! But VERY slow and unstable: it connects and
disconnects because satellites are not geostationary
40 / 45

81. Iridium
66 Iridium satellites
Iridium name: originally supposed to be 77 satellites: atomic number of Iridium
There are spare satellites for when there are problems
Signal on the north and south pole! But VERY slow and unstable: it connects and
disconnects because satellites are not geostationary
Satellites are around 780 Km above the Earth (GPS and Inmarsat around 20,000 Km)
40 / 45

82. Iridium
66 Iridium satellites
Iridium name: originally supposed to be 77 satellites: atomic number of Iridium
There are spare satellites for when there are problems
Signal on the north and south pole! But VERY slow and unstable: it connects and
disconnects because satellites are not geostationary
Satellites are around 780 Km above the Earth (GPS and Inmarsat around 20,000 Km)
Sometimes the satellites are visible: Iridium flares!
40 / 45

83. Iridium
66 Iridium satellites
Iridium name: originally supposed to be 77 satellites: atomic number of Iridium
There are spare satellites for when there are problems
Signal on the north and south pole! But VERY slow and unstable: it connects and
disconnects because satellites are not geostationary
Satellites are around 780 Km above the Earth (GPS and Inmarsat around 20,000 Km)
Sometimes the satellites are visible: Iridium flares!
They are launching new satellites
40 / 45

84. 41 / 45

85. 42 / 45

86. 43 / 45

87. Thank you!
Questions?
44 / 45

88. Contact
Carles Pina i Estany (carles@pina.cat)
Jen Thomas (jenny.thomas@epfl.ch)
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Slides
Slides have been written using Markdown and rendered using remarkjs (https://remarkjs.com)
Slides: https://github.com/cpina/ace-it-presentation / https://cpina.github.io/ace-it-
presentation/
ACE: http://spi-ace-expedition.ch/
45 / 45