Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
www.icm.edu.pl
InfiniCortex
A Supercomputer
that encompasses the world
Marek	Michalewicz
Interdisciplinary	Centre	for	Math...
www.icm.edu.pl
CSIRO,
Australia
A*STAR,
Singapore
Nano-
technology
BREAK
My life - in (computing scale) numbers
1,000x >10...
www.icm.edu.pl
0
250
500
750
1000
1250
1500
2009 2010 2011 2012 2013 2014 2015 2016
SGI KNL
NSCC Fujitsu
IBM BlueGene Q Cu...
4
Level 17 at Fusionopolis
A*CRC Datacenter 1
5
A*CRC Datacenter 2
Matrix Building at Biopolis
Mellanox	Metro-X	testing	since	early	2013
goal:	to	connect	HPC	resources	at	Fusionopolis	
with	storage	and	genomics	pipeli...
www.icm.edu.pl
NOT GRID!
NOT CLOUD!
NOT “Internet”!
InfiniCortex is …
www.icm.edu.pl
The	InfiniCortex uses	a	metaphor	of	a	human	brain’s	outer	layer,	the	Cortex,	consisting	
of	highly	connecte...
www.icm.edu.pl
InfiniCortex Components
1. Galaxy of Supercomputers
• Supercomputer	interconnect	topology	work	
by	Y.	Deng,...
www.icm.edu.pl
TITECH
(Tokyo)
A*STAR
(Singapore)
NCI
(Canberra)
Seattle
SC14
(New Orleans)
GA Tech
(Atlanta)
10Gbps Infini...
www.icm.edu.pl
www.icm.edu.plIn	collaboration	with	NCI	ANU	Andrew	Howard	and	Jakub	Chrzeszczyk
www.icm.edu.pl
10Gbps InfiniBand
100Gbps InfiniBand
100Gbps	InfiniBand	East-ward	link:	Singapore-trans-Pacific-USA-trans-A...
www.icm.edu.pl
largest spanning
InfiniBand network:
ring-around-the-world
with up-to100Gbps
sections Singapore-
USA
Infini...
www.icm.edu.pl
InfiniCloud 2015
True HPC Cloud around the Globe
www.icm.edu.pl
Running the pipeline on the virtual InfiniCloud cluster
Virtual
globally
distributed
cluster
CPU
Network
Ca...
www.icm.edu.pl
7 InfiniBand sub-nets
7 countries: Singapore,	USA,	Australia,	Japan,	
Poland,	France,	Canada
100Gbps Singap...
InfiniCortex demo, SC16, Salt Lake City,
UT, USA
www.icm.edu.pl
www.icm.edu.pl
No. Description
Quantit
y
Unit Price
Unit Price
S$
Total Price
S$
A*CRC
Spending
s
NSCC
Spending
s
Remarks
...
www.icm.edu.pl
NGSP Sequencers at B2 (Illumina + PacBio)
NSCC Gateway
GIS
NSCC
STEP 2: Automated pipeline analysis once se...
Science, Technology and Research Network (STAR-N) connects all
National Supercomputing Centre stakeholders: A*STAR, NUS, N...
www.icm.edu.pl
www.icm.edu.pl
Interdisciplinary Centre for Mathematical and
Computational Modelling, University of Warsaw
People at ICM:
...
www.icm.edu.pl
A selection of ICM’s services:
Computational grants allocation
• 350 active computational grants
• over 280...
www.icm.edu.pl
• VisNow
• Generic open-source platform for data analysis and visualization, developed from
scratch at ICM
...
Strength of ICM: projects
Timothy
Timothy is a novel large scale parallel
computational model allowing 3-D simulations
of ...
CERMINE
• Problem: Automatic extraction of information from PDFs with
scientific papers
• Learns automatically from data u...
www.icm.edu.pl
Cosmology
• Evolution of
the Universe
Epidemiology
• Agent Based Modelling of the Influenza
Disease spread ...
www.icm.edu.pl
PCJ is a library for Java language that helps to
perform parallel and distributed calculations.
• Actively ...
PSNC: 1
Cyfronet: 2.4
WCNS: 1
ICM: 1.3
TASK: 1
PIONIER Academic Network
Consortium coordinated by
PSNC, Poznan
7,500 km ow...
www.icm.edu.pl
Interdisciplinary Centre
for Mathematical and Computational Modelling
University of Warsaw, Poland
Poznan Supercomputing and
Networking Centre
SC16: booth
501
CYFRONET, Krakow
InfiniCortex Team
With	help	from:
SingAREN
A/Prof Francis Lee
Prof Lawrence
Wong
NTU
Stanley Goh
A*CRC
Tan Geok Lian (Netw...
(most) Project Partners 2014-2016
Huawei, HPE, Fujitsu, Aspera, Bright Cluster, Altair,
ByteScale, AristaFermiLab, George ...
www.icm.edu.pl
Bell's law of computer classes
Roughly	every	decade	a	new,	lower	priced	computer	class	forms	based	on	a	new...
PLNOG 18 - Dr Marek Michalewicz - InfiniCortex: Superkomputer wielki jak świat
Upcoming SlideShare
Loading in …5
×

PLNOG 18 - Dr Marek Michalewicz - InfiniCortex: Superkomputer wielki jak świat

55 views

Published on

YouTube: https://www.youtube.com/watch?v=MsXhC-Kb7SI&index=47&list=PLnKL6-WWWE_VNp6tUznu7Ca8hBF8yjKj2

Published in: Technology
  • Be the first to comment

  • Be the first to like this

PLNOG 18 - Dr Marek Michalewicz - InfiniCortex: Superkomputer wielki jak świat

  1. 1. www.icm.edu.pl InfiniCortex A Supercomputer that encompasses the world Marek Michalewicz Interdisciplinary Centre for Mathematical and Computational Modelling (ICM), University of Warsaw, Poland Institute for Advanced Computational Science, Stony Brook University, USA EmTech Asia, Singapore, 14 February 2017
  2. 2. www.icm.edu.pl CSIRO, Australia A*STAR, Singapore Nano- technology BREAK My life - in (computing scale) numbers 1,000x >100x >100x Giga Tera Peta 1,000,000,000x memory and 100,000,000x computing speed increase!
  3. 3. www.icm.edu.pl 0 250 500 750 1000 1250 1500 2009 2010 2011 2012 2013 2014 2015 2016 SGI KNL NSCC Fujitsu IBM BlueGene Q Cumulus IBM Power7 Cirrus SGI Aurora Fujitsu Fuji HP Axle ACRC old systems -10 computers! TFLOPS Growth of computational power in Singapore
  4. 4. 4 Level 17 at Fusionopolis A*CRC Datacenter 1
  5. 5. 5 A*CRC Datacenter 2 Matrix Building at Biopolis
  6. 6. Mellanox Metro-X testing since early 2013 goal: to connect HPC resources at Fusionopolis with storage and genomics pipeline at Biopolis A*CRC Metro-X testing team: Stephen Wong Tay Teck Wee Steven Chew
  7. 7. www.icm.edu.pl NOT GRID! NOT CLOUD! NOT “Internet”! InfiniCortex is …
  8. 8. www.icm.edu.pl The InfiniCortex uses a metaphor of a human brain’s outer layer, the Cortex, consisting of highly connected and dense network of neurons enabling thinking …. to deliver concurrent supercomputing across the globe utilising trans-continental InfiniBand and Galaxy of Supercomputers InfiniCortex is like a living global brain
  9. 9. www.icm.edu.pl InfiniCortex Components 1. Galaxy of Supercomputers • Supercomputer interconnect topology work by Y. Deng, M. Michalewicz and L. Orlowski • Obsidian Strategics Crossbow InfiniBand router 2. ACA 100 & ACE 10 • Asia Connects America 100 Gbps, by November 2014 • Asia Connects Europe 10Gbps, established February 2015 • SingAREN-TEIN-CC and SingAREN-Internet2 connections 3. InfiniBand over trans-continental distances • Using Obsidian Strategics Longbow range extenders • round the globe InfiniBand connectivity established by 2015 4. Application layer • from simplest file transfer: dsync+ • to complex workflows: ADIOS, multi-scale models
  10. 10. www.icm.edu.pl TITECH (Tokyo) A*STAR (Singapore) NCI (Canberra) Seattle SC14 (New Orleans) GA Tech (Atlanta) 10Gbps InfiniBand 100Gbps InfiniBand Enabling geographically dispersed HPC facilities to collaborate and function as ONE concurrent supercomputer, bringing the capability to address and solve grand challenges to the next level of efficiency and scale. InfiniCortex 2014 (phase 1)
  11. 11. www.icm.edu.pl
  12. 12. www.icm.edu.plIn collaboration with NCI ANU Andrew Howard and Jakub Chrzeszczyk
  13. 13. www.icm.edu.pl 10Gbps InfiniBand 100Gbps InfiniBand 100Gbps InfiniBand East-ward link: Singapore-trans-Pacific-USA-trans-Atlantic-Europe 10Gbps InfiniBand West-ward link: Singapore-Europe (via TEIN*CC) InfiniCortex 2015 InfiniBand ring-around the World TITECH (Tokyo) NCI (Canberra) Seattle GA Tech (Atlanta)SC15 (Austin,TX) PSNC (Poznan) A*STAR (Singapore) URCA (Reims)
  14. 14. www.icm.edu.pl largest spanning InfiniBand network: ring-around-the-world with up-to100Gbps sections Singapore- USA InfiniBand routing across 7 subnets over 4 continents
  15. 15. www.icm.edu.pl InfiniCloud 2015 True HPC Cloud around the Globe
  16. 16. www.icm.edu.pl Running the pipeline on the virtual InfiniCloud cluster Virtual globally distributed cluster CPU Network Cancer mutation calling pipeline Output of mutations geopipeline-compute001 | australia | geopipeline-compute002 | australia | geopipeline-compute003 | singapore | geopipeline-compute004 | europe | geopipeline-frontend001 | nova |
  17. 17. www.icm.edu.pl 7 InfiniBand sub-nets 7 countries: Singapore, USA, Australia, Japan, Poland, France, Canada 100Gbps Singapore-Austin 10-30Gbps rest of network ~15 Universities and Research entities ~40 partners and growing HPC InfiniCloud over 4 continents
  18. 18. InfiniCortex demo, SC16, Salt Lake City, UT, USA
  19. 19. www.icm.edu.pl
  20. 20. www.icm.edu.pl No. Description Quantit y Unit Price Unit Price S$ Total Price S$ A*CRC Spending s NSCC Spending s Remarks 1 100G link from SG-LA 1 USD 105,000 149 100 149 100 0 74 550 50/50 Co-sharing between NSCC and Internet2- Sponsored 2 100G link from LA to Austin 1 USD 100,000 142 000 142 000 0 0 Sponsored by Centurylink/SCINET 3 100G Cards for SLIX 4 66 250 SGD 66 250 265 000 0 0 Sponsored by Ciena/3D 4 Cross-Connect at Los Angeles 1 USD, 10 000 14 200 14 200 0 0 Sponsored by SCINET 5 10G SG-Europe Link 1 EURO 16,000 24 800 24 800 0 12 400 50-50 Co-sharing between NSCC and TEIN*CC- Sponsored 6 30G Poznan to Austin 1 USD 100,000 142 000 142 000 0 0 Sponsored by GEANT/Internet2/Pioneer/Poznan/Scinet 7 Connection from SCINET to ACRC Booth USD 7000 9 940 9 940 9 940 0 8 100G Network Switches 2 300 000 SGD 300 000 600 000 0 0 Sponsored by Arista 9 Long Haul InfiniBand Switches and Routers 38 USD 135,000 191 700 7 284 600 0 0 34 InfiniBand switches and 1x InfiniBand Router was sponsored by Obsidian 4 x InfiniBand switches and 1x InfiniBand Router are A*CRC existing equipments using FY14 funds. 10 Long Haul InfiniBand Routers 2 USD 12,500 17 750 35 500 0 0 1 x InfiniBand Router was A*CRC existing equipments using FY14 funds 1 x InfiniBand Router was sponsored by Obsidian 11 40G Infiniband Switches 2 USD 22,500 31 950 63 900 0 0 Sponsored by Obsidian Strategics for inter-booth connection 12 Servers 10 USD 15,000 21 300 213 000 0 0 Sponsored by HPE/Mellanox/Intel/NVIDIA 13 Storages 1 USD 52,260 74 210,00 74 210 0 0 Sponsored by DDN
  21. 21. www.icm.edu.pl NGSP Sequencers at B2 (Illumina + PacBio) NSCC Gateway GIS NSCC STEP 2: Automated pipeline analysis once sequencing completes. Processed data resides in NSCC 500Gbps Primary link Data Manager STEP 3: Data manager indexes and annotates processed data. Replicate metadata to GIS. Allowing data to be search ed and retrieved Data Manager Compute Tiered Storage POLARIS, Genotyping and other Platforms in L4~L8 Tiered Storage STEP 1: Sequencers stream directly to NSCC Storage (NO footprint in GIS) Compute 1 Gbps per sequencer 10 Gbps 1 Gbps per machine 100 Gbps 10 Gbps Genomic Institute of Singapore - National Supercomputing Centr (GIS-NSCC) Integration
  22. 22. Science, Technology and Research Network (STAR-N) connects all National Supercomputing Centre stakeholders: A*STAR, NUS, NTU and Industrial users with 100Gbps + InfiniBand links. NUS NTU A*STAR Fusionopolis SingAREN Global Switch A*STAR Biopolis WOODLANDS 100Gbps InfiniBand 500Gbps Infinera Cloud Express 10/40/100Gbps InfiniBand/IP SELETAR CHANGI NOVENA OUTRAM ONE-NORTH JURONG Singapore InfiniBand connectivity • A high bandwidth network to connect the distributed login nodes • Provide high speed access to users (both public and private) anywhere • Support transfer of large data-sets (both locally and internationally) • Builds local and international network connectivity (Internet 2, TEIN*CC) • ASEAN, USA Europe, Australia, Japan, Middle East
  23. 23. www.icm.edu.pl
  24. 24. www.icm.edu.pl Interdisciplinary Centre for Mathematical and Computational Modelling, University of Warsaw People at ICM: • researchers, • software developers, • IT administration, • HPC experts Hardware • 5 HPC installations, 2.1 PetaFlops peak performance (CPU’s and GPU’s). • dedicated installation for DataScience Premises • Supercomputer and Data center at Białołęka • Backup installation at Kampus Ochota A base:
  25. 25. www.icm.edu.pl A selection of ICM’s services: Computational grants allocation • 350 active computational grants • over 280 scientific publications in 2016 • 180 millions cpu hours in 2016 Meteo - numerical weather forecast • 4 independent models, first service in production in Poland • over 150 millions of visits a year Virtural Library • Access to the scientific publications for all academics in Poland • In collection: Springer, Elsevier, Wiley, ACS, and others • An access to single article within countrywide licence ~ 1€, instead of ~ 30€
  26. 26. www.icm.edu.pl • VisNow • Generic open-source platform for data analysis and visualization, developed from scratch at ICM • Multiplatform, extensible, Java software • Supporting large-scale datasets • Image-based analysis of tissue regeneration process • Classic approach to image-based features in MRI imaging • Deep Learning techniques for machine learning classification • Novel T2 mapping based imaging • Quantitative automated assessment of tissue pathology for Achilles tendon Strength of ICM: projects
  27. 27. Strength of ICM: projects Timothy Timothy is a novel large scale parallel computational model allowing 3-D simulations of cell colonies growing and interacting with variable environment in previously unavailable tissue scale. Vessel with concentration gradient of oxygen in the surrounding tissue 3-D solic tumor development in oxygen environment (necrotic core)
  28. 28. CERMINE • Problem: Automatic extraction of information from PDFs with scientific papers • Learns automatically from data using modern machine learning methods (CRF, SVM, k-means, …) • Award winning ESWC 2015 SemPub Best Performing Approach • Web service http://cermine.icm.edu.pl • Used by: Strength of ICM: projects
  29. 29. www.icm.edu.pl Cosmology • Evolution of the Universe Epidemiology • Agent Based Modelling of the Influenza Disease spread across Poland • Census data based realistic and georeferential society model • Model includes many details: - daily commuting and long-therm mobility • - immunological distance Computational and algorithmic support for scientific projects:
  30. 30. www.icm.edu.pl PCJ is a library for Java language that helps to perform parallel and distributed calculations. • Actively developed • Local and remote memory access • Resilience • Program deployment • Able to work on the multicore systems connected with the typical interconnect such as ethernet or infiniband providing users with the uniform view across nodes. • HPC Challenge Class 2 Best Productivity Award • http://pcj.icm.edu.pl
  31. 31. PSNC: 1 Cyfronet: 2.4 WCNS: 1 ICM: 1.3 TASK: 1 PIONIER Academic Network Consortium coordinated by PSNC, Poznan 7,500 km own fiber Five Academic Supercomputing Centers, combined ~6.7 PFLOPS Polish Academic Supercomputing and Networking Landscape, 2016
  32. 32. www.icm.edu.pl Interdisciplinary Centre for Mathematical and Computational Modelling University of Warsaw, Poland
  33. 33. Poznan Supercomputing and Networking Centre SC16: booth 501 CYFRONET, Krakow
  34. 34. InfiniCortex Team With help from: SingAREN A/Prof Francis Lee Prof Lawrence Wong NTU Stanley Goh A*CRC Tan Geok Lian (Networking) Lim Seng (Networking) Dr Jonathan Low (H/W, S/W, Applications) Dr Dominic Chien (S/W, Applications) Dr Liou Sing-Wu (S/W, Applications) Dr Gabriel Noaje (S/W, Applications) Lukasz Orlowski (S/W, Applications) A/Prof Tan Tin Wee (PI) Yves Poppe, International connectivity Prof Yuefan DengDr Marek Michalewicz (PI) Dr David Southwell
  35. 35. (most) Project Partners 2014-2016 Huawei, HPE, Fujitsu, Aspera, Bright Cluster, Altair, ByteScale, AristaFermiLab, George Washington University Team Europe GEANT, TEIN, France: University of Reims, Poland: PSNC, ICM
  36. 36. www.icm.edu.pl Bell's law of computer classes Roughly every decade a new, lower priced computer class forms based on a new programming platform, network, and interface resulting in new usage and the establishment of a new industry. Computer classes that conform to the law • mainframes (1960s) • minicomputers (1970s)[3] These are essentially replaced by clusters of PCs for a twenty-year (1965-1985) lifespan. [4] • personal computers and workstations evolving into a network enabled by Local Area Networking or Ethernet (1980s) • web browser client-server structures enabled by the Internet (1990s) • cloud computing, e.g. Amazon Web Services,{2006} or Microsoft Azure (2012) • hand held devices from media players and cell phones to tablets, e.g.Creative, iPods, BlackBerrys, iPhones, Smartphones, Kindles, iPads (c. 2000-2010) • Wireless Sensor Networks that enable sensor [5] and actuator interconnection, enabling the evolving Internet of Things. (c. >2005) Can we consider InfiniCortex to be a new class of computers in Bell’s Law sense?

×