This presentation accompanied a practical demonstration of Amazon's Elastic Computing services to CNET students at the University of Plymouth on 16/03/2010.
The practical demonstration involved an obviously parallel problem split on 5 Medium size AMIs. The problem was the calculation of the Clustering Coefficient and the Mean Path Length (Based on the original work done by Watts and Strogatz) for large networks. The code was written in Python taking advantage of the scipy, pyparallel and networkx toolkits
The AWS Cloud provides a number of building blocks for your Deep Learning applications, such as GPU instances and the Deep Learning AMI. In this session, we will present them and show you how can deploy them. Of course, we will run some demos, involving popular tools such as nvidia-docker, Jupyter or MXnet.
This presentation is intended to provide an overview of Cloud computing along with the future scopes yet to be delivered and steps involved in building clouds using open source: EUCALYPTUS.
Fascinating Tales of a Strange TomorrowJulien SIMON
In recent months, Artificial Intelligence has become the hottest topic in the IT industry. Of course, this has happened before, often with disappointing results: in this talk, we’ll explain why it is different this time. In particular, we’ll explain how Deep Learning — a subset of AI — differs from traditional Machine Learning and how it can help you solve complex problems such as computer vision or natural language processing. We’ll also look at how you can add Deep Learning capabilities to your own applications and we'll explain why leveraging Cloud technology can help you get there faster.
Intel and Amazon - Powering your innovation together. Eran Shlomo
In these slides we go over the current joined offering from Intel and amazon, the coming great technologies and how the two companies are creating synergy that boost your innovation and productivity.
This was presented in TLV AWS loft Mar 2017.
The AWS Cloud provides a number of building blocks for your Deep Learning applications, such as GPU instances and the Deep Learning AMI. In this session, we will present them and show you how can deploy them. Of course, we will run some demos, involving popular tools such as nvidia-docker, Jupyter or MXnet.
This presentation is intended to provide an overview of Cloud computing along with the future scopes yet to be delivered and steps involved in building clouds using open source: EUCALYPTUS.
Fascinating Tales of a Strange TomorrowJulien SIMON
In recent months, Artificial Intelligence has become the hottest topic in the IT industry. Of course, this has happened before, often with disappointing results: in this talk, we’ll explain why it is different this time. In particular, we’ll explain how Deep Learning — a subset of AI — differs from traditional Machine Learning and how it can help you solve complex problems such as computer vision or natural language processing. We’ll also look at how you can add Deep Learning capabilities to your own applications and we'll explain why leveraging Cloud technology can help you get there faster.
Intel and Amazon - Powering your innovation together. Eran Shlomo
In these slides we go over the current joined offering from Intel and amazon, the coming great technologies and how the two companies are creating synergy that boost your innovation and productivity.
This was presented in TLV AWS loft Mar 2017.
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...EUDAT
Giuseppe will present the differences between high-performance and high-throughput applications. High-throughput computing (HTC) refers to computations where individual tasks do not need to interact while running. It differs from High-performance (HPC) where frequent and rapid exchanges of intermediate results is required to perform the computations. HPC codes are based on tightly coupled MPI, OpenMP, GPGPU, and hybrid programs and require low latency interconnected nodes. HTC makes use of unreliable components distributing the work out to every node and collecting results at the end of all parallel tasks.
Visit: https://www.eudat.eu/eudat-summer-school
This is a 2 hours overview on the deep learning status as for Q1 2017.
Starting with some basic concepts, continue to basic networks topologies , tools, HW/Accelerators and finally Intel's take on the the different fronts.
High Performance Computing in the Cloud is viable in numerous use cases. Common to all successful use cases for cloud-based HPC is the ability embrace latency. Not surprisingly then, early successes were achieved with embarrassingly parallel HPC applications involving minimal amounts of data - in other words, there was little or no latency to be hidden. Over the fulness of time, however, the HPC-cloud community has become increasingly adept in its ability to ‘hide’ latency and, in the process, support increasingly more sophisticated HPC use cases in public and private clouds. Real-world use cases, deemed relevant to remote sensing, will illustrate aspects of these sophistications for hiding latency in accounting for large volumes of data, the need to pass messages between simultaneously executing components of distributed-memory parallel applications, as well as (processing) workflows/pipelines. Finally, the impact of containerizing HPC for the cloud will be considered through the relatively recent creation of the Cloud Native Computing Foundation.
Hadoop Cluster Configuration and Data Loading - Module 2Rohit Agrawal
Learning Objectives - In this module, you will learn the Hadoop Cluster Architecture and Setup, Important Configuration files in a Hadoop Cluster, Data Loading Techniques.
I have studied on Big Data analysis and found Hadoop is the best technology and most popular as well for it's distributed data processing approaches. I have gathered all possible information about various Hadoop distributions available in the market and tried to describe most important tools and their functionality in the Hadoop echosystems in this slide show. I have also tried to discuss about connectivity with language R interm of data analysis and visualization perspective. Hope you will be enjoying the whole!
Explores the notion of "Hadoop as a Data Refinery" within an organisation, be it one with an existing Business Intelligence system or none - looks at 'agile data' as a a benefit of using Hadoop as the store for historical, unstructured and very-large-scale datasets.
The final slides look at the challenge of an organisation becoming "data driven"
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...EUDAT
Giuseppe will present the differences between high-performance and high-throughput applications. High-throughput computing (HTC) refers to computations where individual tasks do not need to interact while running. It differs from High-performance (HPC) where frequent and rapid exchanges of intermediate results is required to perform the computations. HPC codes are based on tightly coupled MPI, OpenMP, GPGPU, and hybrid programs and require low latency interconnected nodes. HTC makes use of unreliable components distributing the work out to every node and collecting results at the end of all parallel tasks.
Visit: https://www.eudat.eu/eudat-summer-school
This is a 2 hours overview on the deep learning status as for Q1 2017.
Starting with some basic concepts, continue to basic networks topologies , tools, HW/Accelerators and finally Intel's take on the the different fronts.
High Performance Computing in the Cloud is viable in numerous use cases. Common to all successful use cases for cloud-based HPC is the ability embrace latency. Not surprisingly then, early successes were achieved with embarrassingly parallel HPC applications involving minimal amounts of data - in other words, there was little or no latency to be hidden. Over the fulness of time, however, the HPC-cloud community has become increasingly adept in its ability to ‘hide’ latency and, in the process, support increasingly more sophisticated HPC use cases in public and private clouds. Real-world use cases, deemed relevant to remote sensing, will illustrate aspects of these sophistications for hiding latency in accounting for large volumes of data, the need to pass messages between simultaneously executing components of distributed-memory parallel applications, as well as (processing) workflows/pipelines. Finally, the impact of containerizing HPC for the cloud will be considered through the relatively recent creation of the Cloud Native Computing Foundation.
Hadoop Cluster Configuration and Data Loading - Module 2Rohit Agrawal
Learning Objectives - In this module, you will learn the Hadoop Cluster Architecture and Setup, Important Configuration files in a Hadoop Cluster, Data Loading Techniques.
I have studied on Big Data analysis and found Hadoop is the best technology and most popular as well for it's distributed data processing approaches. I have gathered all possible information about various Hadoop distributions available in the market and tried to describe most important tools and their functionality in the Hadoop echosystems in this slide show. I have also tried to discuss about connectivity with language R interm of data analysis and visualization perspective. Hope you will be enjoying the whole!
Explores the notion of "Hadoop as a Data Refinery" within an organisation, be it one with an existing Business Intelligence system or none - looks at 'agile data' as a a benefit of using Hadoop as the store for historical, unstructured and very-large-scale datasets.
The final slides look at the challenge of an organisation becoming "data driven"
Integrate Hue with your Hadoop cluster - Yahoo! Hadoop Meetupgethue
This talk will describe how Hue can be integrated with existing Hadoop deployments with minimal changes/disturbances. Romain will cover details on how Hue can leverage the existing authentication system and security model of your company. He will also cover the Hive/Shark/Pig/Oozie best practice setup for Hue.
http://www.meetup.com/hadoop/events/125191612/
Hadoop World 2011: The Hadoop Stack - Then, Now and in the Future - Eli Colli...Cloudera, Inc.
Many people refer to Apache Hadoop as their system of choice for big data management but few actually use just Apache Hadoop. Hadoop has become a proxy for a much larger system which has HDFS storage at its core. The Apache Hadoop based "big data stack" has changed dramatically over the past 24 months and will chance even more over the next 24 months. This talk talks about trends in the evolution of the Hadoop stack, change in architecture and changes in the kinds of use cases that are supported. It will also talk about the role of interoperability and cohesion in the Apache Hadoop stack and the role of Apache Bigtop in this regard.
Simplified Data Management And Process Scheduling in HadoopGetInData
If you want to stay up to date, subscribe to our newsletter here: https://bit.ly/3tiw1I8
Proper data management and process scheduling are challenges that many data-driven companies under-prioritize. Although it might not cause troubles in short run, it becomes a nightmare when your cluster grows. However, even when you realize this problem, you might not see that possible solutions are so close... In this talk, we share how we simplified our data management and process scheduling in Hadoop with useful (but less adopted) open-source tools. We describe how Falcon, HCatalog, Avro, HDFS FsImage, CLI tools and tricks helped us to address typical problems related to orchestration of data pipelines and discovery, retention, lineage of datasets.
The Hadoop Cluster Administration course at Edureka starts with the fundamental concepts of Apache Hadoop and Hadoop Cluster. It covers topics to deploy, manage, monitor, and secure a Hadoop Cluster. You will learn to configure backup options, diagnose and recover node failures in a Hadoop Cluster. The course will also cover HBase Administration. There will be many challenging, practical and focused hands-on exercises for the learners. Software professionals new to Hadoop can quickly learn the cluster administration through technical sessions and hands-on labs. By the end of this six week Hadoop Cluster Administration training, you will be prepared to understand and solve real world problems that you may come across while working on Hadoop Cluster.
How Big Data and Hadoop Integrated into BMC ControlM at CARFAXBMC Software
Learn how CARFAX utilized the power of Control-M to help drive big data processing via Cloudera. See why it was a no-brainer to choose Control-M to help manage workflows through Hadoop, some of the challenges faced, and the benefits the business received by using an existing, enterprise-wide workload management system instead of choosing “yet another tool.”
Case study of amazon EC2 by Akash BadoneAkash Badone
Introduction to Amazon EC2, Historical Trends, Elastic Map Reduce (EMR), Dynamo DB, RDS, S3, EBS, Iaas, Getting started with EC2 from scratch. Creating key pairs, Launching an instance and types of the instance.AWS services, virtualization and XEN hypervisor with cost (according to on-demand services).
This presentation contains an overview about the hot topics internet of things.
Presentation contins an intro to the modern embedded systems industry with simple technical concepts
Keynote given at BOSC, 2010.
Does the hype surrounding cloud match the reality?
Can we use them to solve the problems in provisioning IT services to support next-generation sequencing?
Slide presentasi ini dibawakan oleh Andri Yadi, CEO DyCode dalam acara CodePolitan Meetup X TIA DevTalk. Membahas tentang seluk beluk IoT, perkembangan dan sisi bisnisnya.
Chaos Engineering - The Art of Breaking Things in ProductionKeet Sugathadasa
This is an introduction to Chaos Engineering - the Art of Breaking things in Production. This is conducted by two Site Reliability Engineers which explains the concepts, history, principles along with a demonstration of Chaos Engineering
The technical talk is given in this video: https://youtu.be/GMwtQYFlojU
Rethinking the cloud_-_limitations_and_oppotunities_-_2011_nexcomhybrid cloud
This was presented for the subject, Rethinking the Cloud, at NEXCOM 2011.
This presentation includes cloud business and strategy, and covers overall cloud technologies such as virtualization, storage, network, and cloud platform.
Enterprise-Ready Private and Hybrid Cloud Computing TodayRightScale
RightScale User Conference NYC 2011:
Enterprise-Ready Private and Hybrid Cloud Computing Today
Rich Wolski - Founder and CTO, Eucalyptus
In this session, we'll discuss the use of Eucalyptus and RightScale to build enterprise-grade cloud computing environments. By combining on-premise clouds with Amazon Web Services (AWS) through a common cloud management interface, Eucalyptus and AWS form a coherent platform for reliable and cost-effective enterprise cloud computing. The RightScale Cloud Management Platform delivers the high-level framework for cost-effectively automating and managing this ensemble of technologies.
A brief presentation of our work in establishing the MSc in Health Data Science and MSc in Health Informatics programes at Swansea University.
This was presented at the "Best Practices" set of talks at the UK eHealth Week conference (http://ukehealthweek.com/)
A practical workshop focusing on the HANDI-HOPD platform to host and process electronic health records that is to be held at the upcoming Open Innovation Conference (http://www.digitalhealthassembly.com/) in February 2015
A poster for the "Lure of the new" conference that was held in Plymouth University on 20-22 March 2013. The conference marked the launch of Plymouth University's COGNITION institute too. (http://cognition.plymouth.ac.uk/)
An introduction to Digital Image Processing as a continuation of a classic Digital Signal Processing course delivered at the University of Plymouth (2011)
The presentation that accompanied the paper submission to ITAB2010. The paper will become available from IEEE Xplore (http://ieeexplore.ieee.org/Xplore/dynhome.jsp)
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Amazon Elastic Computing 2
1. Amazon Elastic Computing 2 AthanasiosAnastasiou Signal Processing And Multimedia Communications Research Group University of Plymouth - UK
2. Topics How Did We Get Here? Enabling Technologies Amazon Elastic Computing Why? What? How? A Quick Demonstration Exploring Complex Networks Further Reading & Resources
3. How Did We Get Here?(Enabling Technologies) 1939 The (Modern) Computer Is Born Almost instantly people start thinking about connecting many units (CPUs) together… 1960 The (Modern) Network Is Born 1964 The ‘Virtual Machine’ Is Born 1967 Paper on Amdahl’s Law 1970 The Internet Is Born (ARPANET) (Modern) Distributed Computing Is Born 1975 The Personal Computer Is Born Mass production of CPUs!!! 1988 SoftPC Is Released
4. How Did We Get Here?(Enabling Technologies) 1990 The World Wide Web Is Born A worldwide network of computers…Hmmm Computer Clusters (Local or over the internet) 1991 Linux Is Born 1998 VMWare patents its virtualisation techniques 2002 GRID Computing Bridging together a variety of technologies into ONE system. 2005 Today Cloud Computing Resources (Virtual Computers And Storage Devices) are remotely accessible on demand by some other system over a network (the internet)
5. Amazon Elastic ComputingWhy? On Demand Remote Access To Resources Computational Rent access to computer(s) Storage Rent storage space Easy, Cheap, Available Loose Restrictions Server instances, Databases, Bandwidth etc By Itself An Enabling Technology To: Commercial Projects Scientific Projects
6. AmazonElasticComputingWhat? (1/3) Amazon Online Enterprise Elastic Claiming Resources According To Your Needs Computing CPUs Computational Time What About Storage? Amazon Cloud Storage (S3) Create Disks Mount them on your filesystem Treat them like any other disk space Amazon Elastic Computing Offers Just The Infrastructure
8. Amazon Elastic ComputingWhat? (3/3) Amazon Elastic Computing Offers Just The Infrastructure User Registration Billing User Manage AMIs Manage I.Ps Manage Storage Store AMIs Services CloudWatch Auto Scaling Load Balancing
16. OK, Let’s Do Something With It!!! Time Consuming Tasks 3D Rendering Computational Fluid Dynamics Simulation Search Through A Large / Huge Domain
17. About The Demonstration Search Through A Large Domain Networks Duncan Watts, Steven Strogatz, 1998, Collective Dynamics of ‘Small World’ Networks Networks Abstract construction with many practical applications Nodes Edges Structure Lattice Random Function Structure affects the emergent functionality What if a network is just a little bit random?
18. Exploring Complex Networks Lattice Random Small World Rewiring Probability (p) Different p values lead to networks with varying structures. How can we characterise these networks?
27. Exploring Complex Networks Networks with the ‘Small World’ property are everywhere… Friendships The Internet The Brain
28. Exploring Complex Networks Let’s try and replicate Watts & Strogatz’s results! Based on Python Scipy PyParallel Networkx Amazon Elastic Computing A custom AMI based on Fedora All necessary software already installed 1 Small Instance (Acting as a “Coordinator”) 5 Medium Instances (Acting as “Workers”)
29. Further Reading & Resources The timeline was created with material from the following sources Computer History, Network History, Virtualisation Technology History, Linux Development Timeline, Super Computers Timeline Some Noteworthy Parallel Processing Projects Where YOU can take part! SETI@home Folding@Home Some Noteworthy Virtualisation Software The XenHupervisor (and cloud computing infrastructure) Oracle’s Virtualbox
30. Further Reading & Resources Amazon Web Services A huge resource about amazon’s cloud computing infrastructure Google App Engine Specifically targeted to web applications. Or, build your own cloud! With Ubuntulinux Python The official web page Scipy An example of a service that “integrates” with Amazon Cloud Computing Perhaps the natural evolution of Cloud Computing PiCloud
31. Further Reading & Resources Mapping The Internet For some HUGE graph datasets! The Internet Mapping Project The Opte Project Books SelimAkl, Parallel Computation: Models And Methods Behrooz, Pahrami, Introduction To Parallel Processing J. Rittinghouse & J. Ransome, Cloud Computing Implementation, Management and Security
32.
Editor's Notes
This is a brief introduction to Amazon’s Cloud computing service. But before we get into this it would be useful to see how did we get here. What were those technologies that enabled Amazon to create its elastic computing service?We will then look into the three key questions about Amazon Elastic Computing.Why? Or in other words what was the need or driving force behind its inceptionWhat Is it?How can we benefit from it?We will then move on to a quick demoAnd finally, for those of you that are more interested I have put together a brief list of further references you can look into.
The reason why this list extends so far in the past is to illustrate the point that this idea of decentralised and distributed processing is in fact almost as old as computers themselves.WWII (unfortunately) and events associated with it accelerated the invention of the modern electronic computer. Alan Turin and the rest of the code breakers in Bletchley park have access to Colossus! The “first programmable digital electronic computing device”…..They split their tasks amongst two of these computers to speed up their code breaking work (!)The evolutions that lead to the birth of the internet start in early 60s. In the mean time, IBM manufactures a series of mainframe computers that run an operating system that abstracts the hardware of a complete computer and uses the term “Virtual Machine”. By 1970, ARPANET is started, again (unfortunately) as a military project. By 1975, a key step is taken. The personal computer is born (!) Which inevitably leads to the mass production of CPUs which in turn means that computational power becomes accessible and affordable by everyone.In 1988 we see the development of SoftPC, a software emulator (You are probably familiar with game machine emulators? This one was a software emulator for the x86 platform)
In 1990, with the inception of HTML and other technologies, the internet acquires a “face” (HTML pages) and starts taking the shape that we know it today. Around the same time, various software toolkits (PVM, MPI) are developed that enable parallel processing on “common” cheap personal computers. Computer clusters and projects that distribute tasks over a very large pool of computers start taking shape. Perhaps the most popular of these projects was (and still is) SETI.1991 Another key step is taken. LinusTorvalds starts working on Linux initially as a uni project. He uses the internet to reach out to other talented people who start putting together Linux. The biggest advantage that Linux provided was that its code was open and available and therefore modifiable. If a “bottleneck” was discovered it was easy for a knowledgeable person to rectify it. As the operating system matures, people start writing software for it. There are no barriers to development, no additional costs to purchase costly development tools and licences and no pressure to generate revenue. Consequently, software is offered for free…Linux gradually conquers the server market.1998 VMWare is granted a patent for its virtualisation techniques…Eventually, it will lead to VMWare as we know it today. Although back then it did not create big waves, you can see that Vmware was in the making for a long time. Also around the same time, some free open source tools start to develop (For example bochs)2002 Various communication, computational and storage technologies have now matured enough to enable GRID computing. GRID computing attempts to abstract various underlying technologies to make a network of computers to appear operating as one. However, this computer might be composed of heterogeneous hardware connected over a heterogeneous network, storing and exchanging data over a number of different technologies without the end user having to mind the details of each system separately.2005 Cloud computing takes its first steps. Cloud computing is where the computers, operating systems, parallel computing and virtualisation software come together to offer remotely accessible resources on demand. Although major players such as Google and Amazon seem to be the major driving forces of this technology, the cloud computing concepts and capabilities continue to develop and grow quickly.And this brings us to today!
Through the rest of this talk, we are going to be looking at one Cloud Computing platform called Amazon Elastic Computing 2.Why should you (or anyone) care about it? Because it offers cheap remote access to resources in an easy way. This basically means that you can rent some computational time or storage capacity and pay by what you use (we will cover pricing later on).So, to make it more relevant to you, imagine that you are working on some project that requires a network of computers or that you would like to have a go at setting up a server with specific capabilities (web server, database server, LDAP server, anything you can think of)… Renting your own server (collocated or stand-alone) would mean something like tens of pounds for a few months (or per month for a ‘stand-alone’) or hundreds of pounds for a year. You would still be restricted in terms of software, bandwidth, number of databases, number of email accounts, etc. With this technology you could rent 10-20 ‘virtual computers’ at a fraction of the equivalent ‘real server’ cost.And of course, let us not forget, that Cloud Computing is itself an enabling technology to a number of commercial and scientific projects. People do find value in this technology to employ it in their businesses or projects (and we will see a few that do later on)
What is Amazon Elastic Computing? The name does a good job at explaining thisAfter all this, it would be good to just keep in mind that Amazon Elastic Computing offers just the infrastructure. In other words, unless your need is 20 networked computers available from the internet…you still have a bit of work to do. This means that you would still need to write the software that runs over this system. We will see what this means in a minute. First of all we need to take a look at a rough sketch of Amazon Elastic Computing and introduce some terminology.
Here is a rough sketch of the key entities in Amazon’s Cloud Computing.If you think about it, given the availability of enabling technologies, the structure of the whole system seems to be following “common sense”. If you pose yourself the question “How would I do this?” and start outlining your answers you would pretty much end up with something like this……Come to think of it, you could end up with something better! So give it a try anyway!!!Users access the service over the internet. Obviously, the service resides in a set of “real machines” or servers that are already networked. Through these servers you can launch ‘virtual servers’. We need a name for these. They are called AMIs from Amazon Machine Image. These are networked with each other on a “virtual network” but, through the use of software switches, are also networked with the real servers (the outside world), the “real network” within Amazon and eventually the internet.We must also point out two more servers that live in this network. The Amazon S3 storage server and a DNS server. You can think of the S3 server as virtualised disk space that belongs to a user and is accessible from the virtual machines. The DNS server makes it possible for the virtual servers to be accessible from ‘the outside world’ or anyone over the internet. Each virtual machine gets an internal name and an external name. If you are trying to access the machine from another computer within the network you can use the internal name while if you are trying to access the machine from the internet you use the external name. As you would expect, machine names and IP addresses are not the same each time an AMI is launched. If you want to uniquely identify a ‘virtual computer’ within this network you can (purchase and) use a static IP that can be binded uniquely to a machine.
Already from this brief description you can see that there are a few tasks that need to be carried out at infrastructure level.We need a framework to register users, bill them for what they use, provide them with tools that makes this infrastructure available to them and also provide services that add value such as Cloudwatch to monitor the ‘health’ of each server, Auto scaling with which you can launch more instances as the server load is increasing and finally provide Load Balancing for large installations.This is what Amazon Elastic Computing is about…You might be wondering, at what cost does all this come to? Let’s take a look at this issue
So now, let’s take a look at HOW does it workPLEASE NOTE: Prices depicted in this slide are as of 15/03/2010
The first step in using Amazon’s services is to register for it. I am not going to go into full detail about this step because it is already covered extensively by Amazon’s documentation at the provided link.Once registration is complete, a user gains access to the amazon web services management console which can be used to manage all available products. If you wanted to have access to a machine you can do it through SSH or SCP for a secure console or secure transfer of files respectively.Let’s see how this looks like.
General Overview
AMIs that are already shared by others.One thing to notice here is the variety of distributions
An overview of the available AMIs and underlying architecture to launch in
Spot instances. Variable pricing according to demand (!)….A computational stock market (!) :-D
OK, let’s take a small pause here. We are about to see what can we do with all this infrastructure but before we go there, are there any questions about the infrastructure so far?
OK, so let’s do something with all these nice little toys!We most commonly turn to parallel computation when we are faced with something that can not be done through:Clever mathematics!Optimising Code!! (or clever programming)Here are a few problems that remain hard even after investing a lot of clever mathematics and programming!!!!3D Rendering: How does light propagate through space and objects? Movies like UP, Toy Story, Wall-e, Shrek, Final Fantasy, etcComputational Fluid Dynamics: How does a fluid flow around an object? To do vehicle Design (Car, Train, Aircraft, Ship, Spaceship etc)Simulation:How would something behave in a given condition? (Before we build it, or while it is still alive)‘Small’ How would an airplane fly? ‘Extra Large’ What if the polar ice caps melt?High resolution weather simulation (and prediction) on Earth (or any other planet) high resolution full brain simulationSearch Through a Large / Huge Domain. This doesn’t mean necessarily literal search. Say for instance, find a name in a list of names. It could also mean, find one image inside Flickr’s huge dataset or What is the average distance between the codewords of a given code? OrWhat is the output of a model for different parameters? And other applications.We will actually look at one of these exploration applications.
We are now going to talk about networks and in particular Complex Networks, focusing on the brilliant work of Watts & Strogatz. This was published on Nature in 1998.
Up until Watts & Strogatz’s paper, graph theory related work employed models of networks. These models provided constructions that either had some well defined structure or were completely random!....But no one had ever looked at the characteristics of networks that leave in between these two extremes. Watts & Strogatz came into this while working on sociology. The nodes in their networks are individuals and the edges represent friendships.They created a model that could return networks that were somewhere in between of lattices and random networks and also studied many real life networks.They found that these networks that were in between order and disorder had some very interesting properties and they also found that this structure is very common in nature. They called these networks, the Small World networks (!)
What they did in order to characterise them was to use two metrics. The clustering coefficient and the mean path length…Here is how they are calculated.And you probably can do this mentaly for these networks over here but what about…
…this network? With just 128 nodes…
….Or this network which is actually a rendering of the connected parts of the internet (Millions of nodes, gazilions of edges)
You might be thinking….Why do we have to study these networks…..Here is why.