This is the keynote talk i delivered at GeekCamp.SG 2014
The main purpose of the talk is to create an awareness, if not existent, in the community when it comes to choosing and wanting to building a distributed system.
This presentation is not meant to be a survey of distributed computing through the ages but hopefully it serves as a good starting point in which the journeyman can start from.
I want to thank Jonas, CTO of Typesafe, as his work in Akka strongly influenced my own and i hope it would help you in the way his work helped me.
Introduction to distributed systems
Architecture for Distributed System, Goals of Distributed system, Hardware and Software
concepts, Distributed Computing Model, Advantages & Disadvantage distributed system, Issues
in designing Distributed System,
This is the keynote talk i delivered at GeekCamp.SG 2014
The main purpose of the talk is to create an awareness, if not existent, in the community when it comes to choosing and wanting to building a distributed system.
This presentation is not meant to be a survey of distributed computing through the ages but hopefully it serves as a good starting point in which the journeyman can start from.
I want to thank Jonas, CTO of Typesafe, as his work in Akka strongly influenced my own and i hope it would help you in the way his work helped me.
Introduction to distributed systems
Architecture for Distributed System, Goals of Distributed system, Hardware and Software
concepts, Distributed Computing Model, Advantages & Disadvantage distributed system, Issues
in designing Distributed System,
Senthilkanth,MCA..
The following ppt's full topic covers Operating System for BSc CS, BCA, MSc CS, MCA students..
1.Introduction
2.OS Structures
3.Process
4.Threads
5.CPU Scheduling
6.Process Synchronization
7.Dead Locks
8.Memory Management
9.Virtual Memory
10.File system Interface
11.File system implementation
12.Mass Storage System
13.IO Systems
14.Protection
15.Security
16.Distributed System Structure
17.Distributed File System
18.Distributed Co Ordination
19.Real Time System
20.Multimedia Systems
21.Linux
22.Windows
A distributed system is a collection of computational and storage devices connected through a communications network. In this type of system, data, software, and users are distributed.
The primary reasons for using parallel computing:
Save time - wall clock time
Solve larger problems
Provide concurrency (do multiple things at the same time)
Distributed Operating System,Network OS and Middle-ware.??Abdul Aslam
Define Distributed Operating System, Network Operating System and Middle-ware? Differentiate between DOS, NOS and Middle-ware? Define the goals of each? ???
Senthilkanth,MCA..
The following ppt's full topic covers Operating System for BSc CS, BCA, MSc CS, MCA students..
1.Introduction
2.OS Structures
3.Process
4.Threads
5.CPU Scheduling
6.Process Synchronization
7.Dead Locks
8.Memory Management
9.Virtual Memory
10.File system Interface
11.File system implementation
12.Mass Storage System
13.IO Systems
14.Protection
15.Security
16.Distributed System Structure
17.Distributed File System
18.Distributed Co Ordination
19.Real Time System
20.Multimedia Systems
21.Linux
22.Windows
A distributed system is a collection of computational and storage devices connected through a communications network. In this type of system, data, software, and users are distributed.
The primary reasons for using parallel computing:
Save time - wall clock time
Solve larger problems
Provide concurrency (do multiple things at the same time)
Distributed Operating System,Network OS and Middle-ware.??Abdul Aslam
Define Distributed Operating System, Network Operating System and Middle-ware? Differentiate between DOS, NOS and Middle-ware? Define the goals of each? ???
A brief report on Client Server Model and Distributed Computing. Problems and Applications are also discussed and Client Server Model in Distributed Systems is also discussed.
Apache Hive is a data warehousing system for large volumes of data stored in Hadoop. However, the data is useless unless you can use it to add value to your company. Hive provides a SQL-based query language that dramatically simplifies the process of querying your large data sets. That is especially important while your data scientists are developing and refining their queries to improve their understanding of the data. In many companies, such as Facebook, Hive accounts for a large percentage of the total MapReduce queries that are run on the system. Although Hive makes writing large data queries easier for the user, there are many performance traps for the unwary. Many of them are artifacts of the way Hive has evolved over the years and the requirement that the default behavior must be safe for all users. This talk will present examples of how Hive users have made mistakes that made their queries run much much longer than necessary. It will also present guidelines for how to get better performance for your queries and how to look at the query plan to understand what Hive is doing.
Presentation during the Big Data Developers in Paris organized by IBM. Subject : Social Data (application of machine learning to a data challenge proposed by DrivenData (Data Science to save the world) ).
Abdellah Lamrani Alaoui & Samed Atouati
This Slide talks about multithreading in python with sample code snippets.
Connect me at
https://nextsrini.blogspot.com/
https://www.youtube.com/channel/UCqIHkbuf1uGiN8QXwWX5dkQ
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
3. Outline
Introduction to Distributed Computing
Parallel vs. Distributed Computing
History of Distributed Computing
Parallelization and Synchronization
Networking Basics
4. Computer Speedup
Moore’s Law: “The density of transistors on a chip doubles every 18
months, for the same cost” (1965)
Image: Tom’s Hardware and not subject to the Creative
Commons license applicable to the rest of this work.
5. Scope of problems
What can you do with 1 computer?
What can you do with 100 computers?
What can you do with an entire data
center?
6. Distributed problems
Rendering multiple frames of high-quality
animation
Image: DreamWorks Animation and not subject to the Creative Commons license applicable to the rest of this work.
8. Distributed problems
Indexing the web (Google)
Simulating an Internet-sized network for
networking experiments (PlanetLab)
Speeding up content delivery (Akamai)
What is the key attribute that all these examples have in common?
9. Parallel vs. Distributed
Parallel computing can mean:
Vector processing of data
Multiple CPUs in a single computer
Distributed computing is multiple CPUs
across many computers over the network
10. A Brief History… 1975-85
Parallel computing was
favored in the early
years
Primarily vector-based
at first
Gradually more thread-
based parallelism was
introduced
Image: Computer Pictures Database and Cray Research Corp and is not subject to the Creative Commons license
applicable to the rest of this work.
11. A Brief History… 1985-95
“Massively parallel architectures” start
rising in prominence
Message Passing Interface (MPI) and
other libraries developed
Bandwidth was a big problem
12. A Brief History… 1995-Today
Cluster/grid architecture increasingly
dominant
Special node machines eschewed in favor
of COTS technologies
Web-wide cluster software
Companies like Google take this to the
extreme
14. Parallelization Idea
Parallelization is “easy” if processing can be
cleanly split into n units:
work
Partition
problem
w1 w2 w3
15. Parallelization Idea (2)
w1 w2 w3
Spawn worker threads:
thread thread thread
In a parallel computation, we would like to have as
many threads as we have processors. e.g., a four-
processor computer would be able to run four threads
at the same time.
18. Parallelization Pitfalls
But this model is too simple!
How do we assign work units to worker threads?
What if we have more work units than threads?
How do we aggregate the results at the end?
How do we know all the workers have finished?
What if the work cannot be divided into
completely separate tasks?
What is the common theme of all of these problems?
19. Parallelization Pitfalls (2)
Each of these problems represents a point
at which multiple threads must
communicate with one another, or access
a shared resource.
Golden rule: Any memory that can be used
by multiple threads must have an
associated synchronization system!
20. What is Wrong With This?
Thread 1: Thread 2:
void foo() { void bar() {
x++; y++;
y = x; x+=3;
} }
If the initial state is y = 0, x = 6, what happens
after these threads finish running?
21. Multithreaded = Unpredictability
Many things that look like “one step” operations
actually take several steps under the hood:
Thread 1: Thread 2:
void foo() { void bar() {
eax = mem[x]; eax = mem[y];
inc eax; inc eax;
mem[x] = eax; mem[y] = eax;
ebx = mem[x]; eax = mem[x];
mem[y] = ebx; add eax, 3;
} mem[x] = eax;
}
When we run a multithreaded program, we don’t
know what order threads run in, nor do we know
when they will interrupt one another.
22. Multithreaded = Unpredictability
This applies to more than just integers:
Pulling work units from a queue
Reporting work back to master unit
Telling another thread that it can begin the
“next phase” of processing
… All require synchronization!
23. Synchronization Primitives
A synchronization primitive is a special
shared variable that guarantees that it can
only be accessed atomically.
Hardware support guarantees that
operations on synchronization primitives
only ever take one step
24. Semaphores
A semaphore is a flag Set: Reset:
that can be raised or
lowered in one step
Semaphores were
flags that railroad
engineers would use
when entering a
shared track
Only one side of the semaphore can ever be red! (Can both be
green?)
25. Semaphores
set() and reset() can be thought of as
lock() and unlock()
Calls to lock() when the semaphore is
already locked cause the thread to block.
Pitfalls: Must “bind” semaphores to
particular objects; must remember to
unlock correctly
26. The “corrected” example
Thread 1: Thread 2:
void foo() { void bar() {
sem.lock(); sem.lock();
x++; y++;
y = x; x+=3;
sem.unlock(); sem.unlock();
} }
Global var “Semaphore sem = new Semaphore();” guards access to
x&y
27. Condition Variables
A condition variable notifies threads that a
particular condition has been met
Inform another thread that a queue now
contains elements to pull from (or that it’s
empty – request more elements!)
Pitfall: What if nobody’s listening?
28. The final example
Thread 1: Thread 2:
void foo() { void bar() {
sem.lock(); sem.lock();
x++; if(!fooDone)
y = x; fooFinishedCV.wait(sem);
fooDone = true; y++;
sem.unlock(); x+=3;
fooFinishedCV.notify(); sem.unlock();
} }
Global vars: Semaphore sem = new Semaphore(); ConditionVar
fooFinishedCV = new ConditionVar(); boolean fooDone = false;
29. Too Much Synchronization?
Deadlock
Synchronization becomes even
more complicated when multiple
locks can be used
Can cause entire system to “get
stuck”
Thread A: Thread B:
semaphore1.lock(); semaphore2.lock();
semaphore2.lock(); semaphore1.lock();
/* use data guarded by /* use data guarded by
semaphores */ semaphores */
semaphore1.unlock(); semaphore1.unlock();
semaphore2.unlock(); semaphore2.unlock();
(Image: RPI CSCI.4210 Operating Systems notes)
30. The Moral: Be Careful!
Synchronization is hard
Need to consider all possible shared state
Must keep locks organized and use them
consistently and correctly
Knowing there are bugs may be tricky;
fixing them can be even worse!
Keeping shared state to a minimum
reduces total system complexity
32. Sockets: The Internet = tubes?
A socket is the basic network interface
Provides a two-way “pipe” abstraction
between two applications
Client creates a socket, and connects to
the server, who receives a socket
representing the other side
33. Ports
Within an IP address, a port is a sub-address
identifying a listening program
Allows multiple clients to connect to a server at
once
34. What makes this work?
Underneath the socket layer are several more
protocols
Most important are TCP and IP (which are used
hand-in-hand so often, they’re often spoken of as
one protocol: TCP/IP)
TCP
IP header Your data
header
Even more low-level protocols handle how data is sent over
Ethernet wires, or how bits are sent through the air using 802.11
wireless…
35. Why is This Necessary?
Not actually tube-like “underneath the hood”
Unlike phone system (circuit switched), the
packet switched Internet uses many routes at
once
you www.google.com
36. Networking Issues
If a party to a socket disconnects, how
much data did they receive?
… Did they crash? Or did a machine in the
middle?
Can someone in the middle
intercept/modify our data?
Traffic congestion makes switch/router
topology important for efficient throughput
37. Conclusions
Processing more data means using more
machines at the same time
Cooperation between processes requires
synchronization
Designing real distributed systems requires
consideration of networking topology
Next time: How MapReduce works