AGENDA • Big data dan data science • Arsitektur big data Drone Emprit • Contoh project LOFAR (Low Frequency Array) • Big data dan data science untuk astronomi dan kalender Islam

1. BIG DATA &
DATA SCIENCE
UNTUK KEMAJUAN ASTRONOMI
DAN PENATAPAN KALENDER ISLAM
Ismail Fahmi, Ph.D.
Direktur Media Kernels Indonesia (Drone Emprit)
Dosen Universitas Islam Indonesia
Wakil Ketua Komisi Infokom MUI Pusat
Ismail.fahmi@gmail.com
UHAMKA IT FORUM
28 APRIL 2021

2. 2
1981 – 1986 Madrasah Ibtidaiyah Muhammadiyah, Kenep, Balen, Bojoengoro, Jatim
1986 – 1989 SMP Negeri Balen, Bojonegoro, Jawa Timur
1989 – 1992 SMA Negeri 1 Bojonegoro, Jawa Timur
1992 – 1997 Undergraduate, Electrical Engineering, ITB, Indonesia
2003 – 2004 Master, Information Science, University of Groningen, NL
2004 – 2009 Doctor, Information Science, University of Groningen, NL
2009 – Now Engineer at Weborama (Paris/Amsterdam)
2014 – Now Founder PT. Media Kernels Indonesia, a Drone Emprit Company
2015 – Now Consultant at Perpustakaan Nasional, Inisiator Indonesia OneSearch
2017 – Now Lecturer at the IT Magister Program of the Universitas Islam Indonesia
2021 – Now Wakil Ketua Komisi Infokom, Majelis Ulama Indonesia Pusat
Ismail Fahmi, S.T., M.A., Ph.D.
Ismail.fahmi@gmail.com

3. AGENDA
• Big data dan data science
• Arsitektur big data Drone Emprit
• Contoh project LOFAR (Low Frequency Array)
• Big data dan data science untuk astronomi dan kalender Islam
3

4. TENTANG DRONE EMPRIT
4

5. DATA SCIENCE AND BIG DATA

6. THE BIG 3
6

7. DATA SCIENCE = IS + CS + KD + STATS
7

8. WHAT IS BIG DATA?
8

9. DATA GROWTH
9

10. DARI MANA DAPAT DATA BANYAK?
10

11. FROM DATA TO ACTIONS
11

12. WHAT IS BIG DATA ANALYTICS?
12
Big data analytics is the
often complex process of
examining large and
varied data sets, or big
data, to uncover information
-- such as hidden patterns,
unknown correlations,
market trends and customer
preferences -- that can help
organizations make informed
business decisions.

13. STAGES IN BIG DATA ANALYTICS
13

14. BIG DATA ANALYTICS – TOOLS
14

15. HOW ACTUALLY IT WORKS?
15

16. DRONE EMRPIT
BIG DATA ARCHITECTURE

17. MEDIA KERNELS ARCHITECTURE DIAGRAM
Twitter
Facebook
YouTube
Instagram
Online News
Physical Hardware
SOLR Index 1 SOLR Index 2 SOLR Index N
Data Lake Engine
Crawler &
Data
Ingest
Engine
Realtime
Job
Analytics Engine
SQL DB Engine
Scheduled
Job
Advanced
Analytics
Basic
Analytics
API Engine
Access
Control
Data
Advanced
Analytics
Basic
Analytics
UI Engine
Data Sources
Theme
Config
Logo
Features
Hostname
TikTok

18. MEDIA KERNELS FEATURES
18

19. 19
Social Media
Twitter
Facebook
Search
+
JSON
Detik (ID)
Reuters (EN)
Etc..
RSS
+
HTML
Gatra (ID)
Bloomberg (EN)
Etc..
HTML
Kaskus
Detik Forum
Etc..
HTML
Online News
Forums
Twitter Stream
JSON
Kompas
TEXT
Warta Ekonomi
Etc..
Print
PUSH
JSON
Subscriber
Projects
Storage
Search + Account
Crawler
RSS + HTML
Crawler
HTML Crawler
HTML Crawler
SOLR Nodes
Shard 1
SOLR Nodes
Shard N
Index Servers
Redis Queue
Cache Manager
Mentions
Storage
Keywords +
Accounts Filters
deletes
Sentiment
Analysis
Sentiment
Models
Backtrack
Filters
Sentiment
Analysis
Analyses
Control Room
Screens
Smart phones,
tablets
Desktops
Client(s)
Converter
System Architecture

20. SERVER ARCHITECTURE
Media Kernels installation with 12 servers:
• Maximum 2000 keywords
• Online news crawler & data ingestion (Server 1, 2, 3)
• Twitter, Facebook, Instagram, and YouTube crawlers (Server 4, 5, 6, 7, 8)
• Data lake and analytics (Server 6, 7, 8, 9, 10)
• Database and API (Server 11, 12)
• Web (Server 11, 12)
• NAS (Network Attached Storage)
20
Server 1
Server 2
Server 3
Server 4
Server 5
Server 6
INTERNET
Router
NAS
Server 7
Server 8
Server 9
Server 10
Server 11
Server 12

21. LOFAR PROJECT

22. 22
S2-S3
2003 - 2009
UNIVERSITY OF GRONINGEN
BELANDA

23. LOFAR PROJECT (2006-2021)
23
The Low-Frequency Array, or LOFAR, is a large radio telescope network located mainly
in the Netherlands, completed in 2012 by ASTRON, the Netherlands Institute for
Radio Astronomy and its international partners, and operated by ASTRON's radio
observatory, of the Netherlands Organisation for Scientific Research.

24. ABOUT LOFAR
• The project is based on an interferometric array of radio telescopes
using about 20,000 small antennas concentrated in 52 stations at
present.
• 38 of these stations are distributed across the Netherlands and were
funded by ASTRON.
• The six stations in Germany, three in Poland, and one each in France,
Great Britain, Ireland, Latvia, and Sweden, were funded by these
countries. A station in Italy will be built soon. Further stations may also
be built in other European countries.
• The total effective collecting area is approximately 300,000 square
meters, depending on frequency and antenna configuration.
• The data processing is performed by a Blue Gene/P supercomputer
situated in the Netherlands at the University of Groningen.
24

25. JARINGAN STASIUN ANTENA LOFAR
25

26. LOFAR NETWORK
26

27. FASILITAS KOMPUTASI LOFAR DI GRONINGEN
27

28. IBM BLUE GENE AT ZERNIKE BLD UNIV GRONINGEN
28
ZERNIKE BUILDING
UNIV GRONINGEN

29. ASTRON LOFAR/CIT BLUE GENE
• The Blue Gene P system consists of 3072 compute nodes, each
with 2 GB of memory.
• One compute node consists of 4 cores, providing a total of 12288
cores.
• A number of “I/O nodes” are used for the communication with
the compute nodes and perform the I/O operations between
compute nodes and data servers.
• Although the system can be used as one massive compute block
it has been divided in logical “partitions”. These partitions
communicate with the frontend of the system via their I/O nodes
and can be used separately.
• Jobs submitted to disjoint partitions can be run at the same time.
29

30. LOFAR DATA PROCESSING: LINUX CLUSTER
30
GbE
switch
GbE
switch
GbE
switch
GbE
switch
GbE
switch
GbE
switch
Auxiliary processing
section
Storage
system
Off-line processing/
user software
Off-line
Storage
On-Line
Long term Archiving
& Data Export
GbE
switch
On-Line pipeline processing
Blue Gene/L
WAN

31. LOFAR LONG TERM ARCHIVE
31

32. LOFAR COMPUTER NETWORK
32

33. HASIL LOFAR: IMAGE OF THE UNIVERSE
33
An international team of
astronomers created the
most sensitive images of the
Universe ever taken at low
radio frequencies. The use
of #LOFAR reveals images
of Milky Way like galaxies in
the most distant parts of the
Universe

34. MAP OF THE SKY: 25.000 BLACK HOLES
34
An international team of astronomers has produced a map of the sky at ultra-low radio
frequencies using LOFAR, revealing more than 25,000 active supermassive black holes in
distant galaxies.

35. MORE INFO: TWITTER LOFAR & ASTRON
35

36. BIG DATA & DATA SCIENCE DENGAN
CLOUD COMPUTING

37. ISRN UHAMKA JAKARTA
37

38. BIG DATA CHALLENGES
38

39. DATA SCIENCE SKILL SET
39

40. BIG DATA SKILL SHORTAGES
40

41. SOLUTION: RENT
41

42. SOLUSI GOOGLE BIG DATA – RENT
42

43. 43

44. MICROSOFT AZURE
44

45. MS AZURE CLOUD TECHNOLOGY
45

46. SERVER SENDIRI

47. DRONE EMPRIT ACADEMIC
HTTPS://DEA.UII.AC.ID
47

48. HOW IT WORKS
48
STEPS:
• Registration
• Propose keywords
• Analysis and publication
Dashboard
Access
REQUIREMENTS:
• Publish their analysis for public
using any medium at least 1
publication every 2 months.
USERS
• Students
• Researchers
• Lecturers
• Journalists
• Blogger
• Hoax buster
Admin

49. UII DATA CENTER
49
UII Servers

50. TOPICS BASED ON SDGS
(SUSTAINABLE DEVELOPMENT GOALS)
50

51. DRONE EMPRIT ACADEMIC DASHBOARD
51

52. PENUTUP

53. KESIMPULAN
• Project big data terkait astronomi biasanya membutuhkan fasilitas komputasi
yang besar, baik dari sisi penyimpanan (storage) maupun pemrosesan
(computing).
• Contoh project Lofar (Low Frequency Array) di Belanda, membutuhkan High
Performance Computing dengan server IBM Blue Gene yang disimpan di
Universitas Groningen, Belanda.
• Dengan adanya fasilitas HPC sendiri, memungkinkan peneliti dari seluruh
dunia untuk memanfaatkannya bersama-sama secara lebih efektif.
• Jika ukuran data dan komputasi relatif kecil, serta akses dan pengolahan
datanya menggunakan mobile apps, bisa menggunakan fasilitas cloud
computing yang disediakan secara komersial seperti Google Cloud dan
Microsoft Azure.
• Contoh Drone Emprit Academic yang menyimpan dan memproses data
besar, lebih efektif dan murah jika menggunakan server sendiri (dedicated,
bare metal) dibandingkan menggunakan cloud computing.
53

54. TERIMAKASIH
Ismail Fahmi, PhD.