Cloud computing provides scalable and elastic resources over the internet. There are three main service models: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). IaaS allows users to deploy and run operating systems and applications, while PaaS allows users to deploy consumer-created apps, and SaaS provides applications to users. There are challenges around availability, performance unpredictability, and security and privacy concerns as user data is stored externally by cloud providers.
Webinar presented live on April 19, 2017
The Cloud Standards Customer Council has published a reference architecture for securing workloads on cloud services. The aim of this new guide is to provide a practical reference to help IT architects and IT security professionals architect, install, and operate the information security components of solutions built using cloud services.
Building business solutions using cloud services requires a clear understanding of the available security services, components and options, allied to a clear architecture which provides for the complete lifecycle of the solutions, covering development, deployment and operations. This webinar will discuss specific security services and corresponding best practices for deploying a comprehensive cloud security architecture.
Read the whitepaper: http://www.cloud-council.org/deliverables/cloud-customer-architecture-for-securing-workloads-on-cloud-services.htm
Webinar presented live on April 19, 2017
The Cloud Standards Customer Council has published a reference architecture for securing workloads on cloud services. The aim of this new guide is to provide a practical reference to help IT architects and IT security professionals architect, install, and operate the information security components of solutions built using cloud services.
Building business solutions using cloud services requires a clear understanding of the available security services, components and options, allied to a clear architecture which provides for the complete lifecycle of the solutions, covering development, deployment and operations. This webinar will discuss specific security services and corresponding best practices for deploying a comprehensive cloud security architecture.
Read the whitepaper: http://www.cloud-council.org/deliverables/cloud-customer-architecture-for-securing-workloads-on-cloud-services.htm
Introduction of Cloud Computing & Historical Background
Cloud Service Models & Cloud Deployment Models
Benefits of Cloud Computing
Risks and Challenges
Future Trends in Cloud Computing
Edge Computing, Serverless Computing, AI & Machine Learning in Cloud, Security and
Compliance
Needs and Obstacles for Cloud Deployment
Conclusion
Speaker Presention by Irena Bojanova of the University of Maryland University...Tim Harvey
Irena Bojanova, Professor & Program Director in Information and Technology Systems at the University of Maryland University College, spoke at the Federal Cloud Computing Summit on Dec. 17, 2013 at the Ronald Reagan Building in Washington, D.C.
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
Introduction of Cloud Computing & Historical Background
Cloud Service Models & Cloud Deployment Models
Benefits of Cloud Computing
Risks and Challenges
Future Trends in Cloud Computing
Edge Computing, Serverless Computing, AI & Machine Learning in Cloud, Security and
Compliance
Needs and Obstacles for Cloud Deployment
Conclusion
Speaker Presention by Irena Bojanova of the University of Maryland University...Tim Harvey
Irena Bojanova, Professor & Program Director in Information and Technology Systems at the University of Maryland University College, spoke at the Federal Cloud Computing Summit on Dec. 17, 2013 at the Ronald Reagan Building in Washington, D.C.
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
Enhanced Enterprise Intelligence with your personal AI Data Copilot.pdfGetInData
Recently we have observed the rise of open-source Large Language Models (LLMs) that are community-driven or developed by the AI market leaders, such as Meta (Llama3), Databricks (DBRX) and Snowflake (Arctic). On the other hand, there is a growth in interest in specialized, carefully fine-tuned yet relatively small models that can efficiently assist programmers in day-to-day tasks. Finally, Retrieval-Augmented Generation (RAG) architectures have gained a lot of traction as the preferred approach for LLMs context and prompt augmentation for building conversational SQL data copilots, code copilots and chatbots.
In this presentation, we will show how we built upon these three concepts a robust Data Copilot that can help to democratize access to company data assets and boost performance of everyone working with data platforms.
Why do we need yet another (open-source ) Copilot?
How can we build one?
Architecture and evaluation
The Building Blocks of QuestDB, a Time Series Databasejavier ramirez
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeWalaa Eldin Moustafa
Dynamic policy enforcement is becoming an increasingly important topic in today’s world where data privacy and compliance is a top priority for companies, individuals, and regulators alike. In these slides, we discuss how LinkedIn implements a powerful dynamic policy enforcement engine, called ViewShift, and integrates it within its data lake. We show the query engine architecture and how catalog implementations can automatically route table resolutions to compliance-enforcing SQL views. Such views have a set of very interesting properties: (1) They are auto-generated from declarative data annotations. (2) They respect user-level consent and preferences (3) They are context-aware, encoding a different set of transformations for different use cases (4) They are portable; while the SQL logic is only implemented in one SQL dialect, it is accessible in all engines.
#SQL #Views #Privacy #Compliance #DataLake
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Discussion on Vector Databases, Unstructured Data and AI
https://www.meetup.com/unstructured-data-meetup-new-york/
This meetup is for people working in unstructured data. Speakers will come present about related topics such as vector databases, LLMs, and managing data at scale. The intended audience of this group includes roles like machine learning engineers, data scientists, data engineers, software engineers, and PMs.This meetup was formerly Milvus Meetup, and is sponsored by Zilliz maintainers of Milvus.
Global Situational Awareness of A.I. and where its headedvikram sood
You can see the future first in San Francisco.
Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. Every six months another zero is added to the boardroom plans. Behind the scenes, there’s a fierce scramble to secure every power contract still available for the rest of the decade, every voltage transformer that can possibly be procured. American big business is gearing up to pour trillions of dollars into a long-unseen mobilization of American industrial might. By the end of the decade, American electricity production will have grown tens of percent; from the shale fields of Pennsylvania to the solar farms of Nevada, hundreds of millions of GPUs will hum.
The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word. Along the way, national security forces not seen in half a century will be un-leashed, and before long, The Project will be on. If we’re lucky, we’ll be in an all-out race with the CCP; if we’re unlucky, an all-out war.
Everyone is now talking about AI, but few have the faintest glimmer of what is about to hit them. Nvidia analysts still think 2024 might be close to the peak. Mainstream pundits are stuck on the wilful blindness of “it’s just predicting the next word”. They see only hype and business-as-usual; at most they entertain another internet-scale technological change.
Before long, the world will wake up. But right now, there are perhaps a few hundred people, most of them in San Francisco and the AI labs, that have situational awareness. Through whatever peculiar forces of fate, I have found myself amongst them. A few years ago, these people were derided as crazy—but they trusted the trendlines, which allowed them to correctly predict the AI advances of the past few years. Whether these people are also right about the next few years remains to be seen. But these are very smart people—the smartest people I have ever met—and they are the ones building this technology. Perhaps they will be an odd footnote in history, or perhaps they will go down in history like Szilard and Oppenheimer and Teller. If they are seeing the future even close to correctly, we are in for a wild ride.
Let me tell you what we see.
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...sameer shah
"Join us for STATATHON, a dynamic 2-day event dedicated to exploring statistical knowledge and its real-world applications. From theory to practice, participants engage in intensive learning sessions, workshops, and challenges, fostering a deeper understanding of statistical methodologies and their significance in various fields."
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
2. Cloud computing
• Uses Internet technologies to offer scalable and
elastic services
“Elastic computing” refers to the ability of dynamically
acquiring computing resources and supporting a variable
workload
• Resources used for such services can be metered
Users can be charged only for the resources they used
• Service provider ensures the maintenance & security
• Service providers can operate
more efficiently due to
Specialization
Centralization
2
3. Cloud computing
• Lower costs for the cloud service provider are
passed to the cloud users
• Data is stored
Closer to the site where it is used
In a device
In a location-independent manner
• The data storage strategy can
Increase reliability
Increase security
Decrease communication costs
3
6. Software-as-a-Service (SaaS)
• Applications are supplied by the service provider
User does not manage or control the underlying cloud
infrastructure or individual application capabilities
• Potential Services:
Enterprise services: workflow management, group-ware
& collaborative, supply chain, communications, digital
signature, customer relationship management (CRM),
desktop software, financial management, geo-spatial, and
search
Web 2.0 applications: metadata management, social
networking, blogs, wiki services, and portal services
• Not suitable for real-time applications or for those
where data is not allowed to be hosted externally
• Examples:
Gmail, Google search, Google Docs
6
7. Platform-as-a-Service (PaaS)
• Allows a cloud user to deploy consumer-created or
acquired applications using programming languages
and tools supported by the service provider
• The user:
Has control over the deployed applications and, possibly,
application hosting environment configurations.
Does not manage or control the underlying cloud
infrastructure including network, servers, operating
systems, or storage.
• Not particularly useful when:
The application must be portable.
Proprietary programming languages are used.
The hardware and software must be customized to improve
the performance of the application. 7
8. Infrastructure-as-a-Service (IaaS)
• User is able to deploy and run arbitrary software,
including operating system (OS) and applications
• The user does not manage or control the
underlying cloud infrastructure but has control over
OSs, storage, deployed applications, and possibly
limited control of some networking components,
e.g., host firewalls.
• Services offered by this delivery model include:
server hosting, Web servers, storage, computing hardware,
operating systems, virtual instances, load balancing, Internet
access, and bandwidth provisioning
8
9. Cloud Services Models
Facilities
Hardware
Core
connectivity
Abstraction
API
Software as a Service
Facilities
Hardware
Core
connectivity
Abstraction
API
Integration and
middleware
Data Metadata
Applications
API
Presentation
Infrastructure as a Service
Facilities
Hardware
Core
connectivity
Abstraction
API
Integration and
middleware
Platform as a Service
9
11. Public Cloud
• Owned by the organization selling cloud
services
• Made available to the general public, or a
large industry group
• IaaS, PaaS, or SaaS for anyone to use
• Built for general-purpose deployment
• Public APIs
• Major players
AWS, GCP, Azure
11
12. Private Cloud
• Built and managed by an organization for itself
• Purpose-built for its own services
• Custom APIs
• Handles critical/custom services
• Might holds critical data
Legal reasons
Security concerns
• Major players
Google
Facebook
Microsoft
Baidu
12
13. Community Cloud
• Shared by several organizations
• Supports a community that has
shared concerns
13
14. Hybrid Cloud
• Composition of two or more
clouds (public, private, or
community)
• Bound by standardized
technology that enables
data and application
portability
14
15. Benefits of cloud computing
•Resources are shared
CPU cycles, storage, network bandwidth
•Multiplexing leads to a higher resource
utilization
When multiple applications share a system, their peak
demands for resources are not synchronized
•Resources can be aggregated to support
data-intensive applications
•Data sharing facilitates collaborative
activities
Many applications require multiple types of analysis of
shared data sets and multiple decisions carried out by
groups scattered around the globe 15
16. Benefits of cloud computing
•Minimal investment cost
Eliminates the initial investment costs for a private
computing infrastructure and the maintenance and
operation costs
•Cost reduction
Concentration of resources creates the opportunity to pay
as you go for computing
•Elasticity
Ability to accommodate workloads with very large peak-to-
average ratios
•User convenience
Virtualization allows users to operate in familiar
environments rather than in idiosyncratic ones 16
17. Challenges for cloud computing
• Availability of service
Users expect services to be available
Associated Cost
■ Lost revenue
■ Lost engagement
17
Source: Skybox terms of service: http://skyboxinnovations.com/terms-of-service/
■ Service level agreements (SLAs):
■ P50 or P95 latencies
■ Cost for not meeting levels
18. Challenges for cloud computing
• Value of Service
Users also expect services to stay relevant and provide
value
■ Continuously changing requirements from users
■ Important for services to stay relevant
18
19. Challenges for cloud computing
• User Engagement
Services providers need to keep their users engaged
19
Relevant services User engagement
Revenue
20. Challenges for Cloud Computing
• Standardization
Diversity of services, data organization, user interfaces
available at different service providers limit user mobility;
once a customer is hooked to one provider it is hard to
move to another
• Performance unpredictability
One of the consequences of resource sharing.
How to use resource virtualization and performance
isolation for QoS guarantees?
How to support elasticity, the ability to scale up and down
quickly?
20
21. Challenges for Cloud Computing
• Data transfer bottleneck
Many applications are data-intensive.
• Resource management
Are self-organization and self-management the solution?
• Security and confidentiality
Major concern
Addressing these challenges provides good research
opportunities!!
21
22. Cloud Computing – Holistic View
Delivery models
Infrastructure as a Service (IaaS)
Software as a Service (SaaS)
Platform as a Service (PaaS)
Deployment models
Private cloud
Hybrid cloud
Public cloud
Community cloud
Defining attributes
Massive infrastructure
Accessible via the Internet
Utility computing. Pay-per-usage
Elasticity
Cloud computing
Resources
Networks
Compute & storage servers
Services
Applications
Infrastructure
Distributed infrastructure
Resource virtualization
Autonomous systems
22
23. Cloud activities
•Service management and provisioning
including
Virtualization
Service provisioning
Call center
Operations management
Systems management
QoS management
Billing and accounting, asset management
SLA management
Technical support and backups
23
24. Cloud activities
•Security management including:
ID and authentication
Certification and accreditation
Intrusion prevention
Intrusion detection
Virus protection
Cryptography
Physical security, incident response
Access control, audit and trails, and firewalls
24
25. Cloud activities
•Customer services such as:
Customer assistance and on-line help
Subscriptions
Business intelligence
Reporting
Customer preferences
Personalization
•Integration services including:
Data management
Development
25
26. NIST cloud reference model
Carrier
S
e
c
u
r
i
t
y
P
r
i
v
a
c
y
Service
Consumer Broker
Service Provider
Auditor
Security
audit
Privacy
impact audit
Performance
audit
Service
Management
Business
support
Provisioning
Portability/
Interoperability
IAAS
IaaS
SaaS
Service Layer
PaaS
Carrier
Hardware
Facility
Physical resource
layer
Resource
abstraction and
control layer
Intermediation
Aggregation
Arbitrage
26
https://bigdatawg.nist.gov/_uploadfiles/M0008_v1_7256814129.pdf
27. Ethical issues
•Paradigm shift with implications on
computing ethics:
oThe control is relinquished to third party
services
oThe data is stored on multiple sites
administered by several organizations
oMultiple services interoperate across the
network
•Implications
oUnauthorized access
oData corruption
oInfrastructure failure, and
oservice unavailability
27
28. De-perimeterisation
• Systems can span the boundaries of multiple
organizations and cross the security borders
• Complex structure of cloud services can make it
difficult to determine who is responsible in case
something undesirable happens
• Identity fraud and theft are made possible by the
unauthorized access to personal data, also pose a
danger to cloud computing
28
29. Privacy issues
•Cloud service providers have already
collected petabytes of sensitive personal
information stored in data centers around
the world
•Acceptance of cloud computing will be
determined by privacy issues addressed by
these companies and the countries where
the data centers are located
•Privacy is affected by cultural differences;
some cultures favor privacy, others
emphasize community. This leads to an
ambivalent attitude towards privacy in the
Internet which is a global system
29
30. Cloud vulnerabilities
•Clouds are affected by malicious attacks and
failures of the infrastructure, e.g., power
failures
•Such events can affect the Internet domain
name servers and prevent access to a cloud
or can directly affect the clouds:
oIn 2004 an attack at Akamai technologies, caused a
domain name outage and a major blackout that affected
Google, Yahoo, and other sites
oIn 2009, Google was the target of a denial of service
attack which took down Google News and Gmail for
several days
oIn 2012, lightning caused a prolonged down time at
Amazon
30