Bloomberg is a large financial technology company that builds its own software, platforms, and hardware. It has over 3,000 developers located around the world and over 320,000 subscribers that are primarily financial professionals. Bloomberg uses Accumulo for its large-scale data storage and analytics needs, including storing over 3 petabytes of archived data. Accumulo provides security and access controls at scale through its visibility framework and high performance through features like its file system API.
Accumulo Summit 2016: Timely - Scalable Secure Time Series DatabaseAccumulo Summit
Some of the most popular metric visualization tools work really well for smaller deployments, but have issues dealing with large amounts of data. Timely started after an integration of OpenTSDB with Accumulo failed to meet our needs. Timely can be used to gain visibility into the performance of your networks and hardware, your Hadoop cluster and Accumulo database, and your application. In this talk we will cover the current implementation and APIs, security model, deployment models, and our roadmap.
– Speakers –
Dave Marion
Principal Software Engineer, Vistronix
Dave Marion is a Principal Software Engineer for Vistronix. He has been working on big data projects since 2010 and prior to that worked as a database engineer on large relational database projects. He is a veteran of the U.S. Navy and has a BS in Computer Information Systems from University of Maryland University College. He is a PMC member of Apache Accumulo and contributes to several other Apache projects.
Jim Klucar
Principal Software Engineer, Praxis Engineering
Jim Klucar is a Principal Software Engineer for Praxis Engineering. He has a BS in Electrical Engineering from Pennsylvania State University and a MS in Applied and Computational Mathematics from Johns Hopkins University. After a dozen years of developing high performance radar processing techniques, in 2010 he switched to developing Hadoop-based data warehouse and analysis systems. He has contributed to many open source projects including Apache Accumulo, Mesos and Myriad.
Drew Farris
Senior Associate, Booz Allen Hamilton
Drew Farris is a technology consultant at Booz Allen Hamilton who specializes in distributed computing, information retrieval and machine learning. He's a voting member of the Apache Software Foundation, on the Accumulo PMC and works with the Apache Incubator as a mentor for several projects.
— More Information —
For more information see http://www.accumulosummit.com/
The rise of data-driven sales / Veera Virintie, Vainu.io / Next 2016Luxus Worldwide
Organized on the 19th of August on the island of Lonna, Helsinki, Next 2016 was a day-long digital marketing summit that brought together marketers, innovators and business decision-makers to discuss the challenges and opportunities of digitalization.
In her presentation, Veera shared the story of Vainu, one of Europe's fastest growing startups. Key points included the rise of data driven sales and automating lead generation so that you'll never miss a sales opportunity again.
IT due diligence, software audit and software quality standards are very important for startups that want to sell to or partner with large companies and corporates. In this invited talk the importance of quality is discussed from a startup perspective.
Accumulo Summit 2016: Timely - Scalable Secure Time Series DatabaseAccumulo Summit
Some of the most popular metric visualization tools work really well for smaller deployments, but have issues dealing with large amounts of data. Timely started after an integration of OpenTSDB with Accumulo failed to meet our needs. Timely can be used to gain visibility into the performance of your networks and hardware, your Hadoop cluster and Accumulo database, and your application. In this talk we will cover the current implementation and APIs, security model, deployment models, and our roadmap.
– Speakers –
Dave Marion
Principal Software Engineer, Vistronix
Dave Marion is a Principal Software Engineer for Vistronix. He has been working on big data projects since 2010 and prior to that worked as a database engineer on large relational database projects. He is a veteran of the U.S. Navy and has a BS in Computer Information Systems from University of Maryland University College. He is a PMC member of Apache Accumulo and contributes to several other Apache projects.
Jim Klucar
Principal Software Engineer, Praxis Engineering
Jim Klucar is a Principal Software Engineer for Praxis Engineering. He has a BS in Electrical Engineering from Pennsylvania State University and a MS in Applied and Computational Mathematics from Johns Hopkins University. After a dozen years of developing high performance radar processing techniques, in 2010 he switched to developing Hadoop-based data warehouse and analysis systems. He has contributed to many open source projects including Apache Accumulo, Mesos and Myriad.
Drew Farris
Senior Associate, Booz Allen Hamilton
Drew Farris is a technology consultant at Booz Allen Hamilton who specializes in distributed computing, information retrieval and machine learning. He's a voting member of the Apache Software Foundation, on the Accumulo PMC and works with the Apache Incubator as a mentor for several projects.
— More Information —
For more information see http://www.accumulosummit.com/
The rise of data-driven sales / Veera Virintie, Vainu.io / Next 2016Luxus Worldwide
Organized on the 19th of August on the island of Lonna, Helsinki, Next 2016 was a day-long digital marketing summit that brought together marketers, innovators and business decision-makers to discuss the challenges and opportunities of digitalization.
In her presentation, Veera shared the story of Vainu, one of Europe's fastest growing startups. Key points included the rise of data driven sales and automating lead generation so that you'll never miss a sales opportunity again.
IT due diligence, software audit and software quality standards are very important for startups that want to sell to or partner with large companies and corporates. In this invited talk the importance of quality is discussed from a startup perspective.
Hello! VEBTECH team creates solutions that develop your business. 200+ certified developers with 25 years of experience in FinTech. We are strong in:
• Website, platform, and mobile application development
• BPM (PEGA, Zendesk)
• Blockchain solutions
• Cybersecurity (Red Team)
• IT - Monitoring
Platform governance, gestire un ecosistema di microservizi a livello enterpriseGiulio Roggero
A livello enterprise, le moderne architetture distribuite coinvolgono molti team differenti, centinaia di sviluppatori e operations e migliaia microservizi ed API in produzione. Come si può gestire questa
e o
un'esplosione di costi e preservando il time-to-market?
How to Enable, Monitor, and Secure Your Remote WorkforceSolarWinds
In the wake of COVID-19, many businesses, educational institutions, and government entities have had to change their models to accommodate a remote workforce. In this new norm, it’s important for employees to be able to work in secure, remote environments. Your organization and its stakeholders depend on your applications, and remote users can put new strains on application and network capabilities. This makes an IT professional’s job more challenging than ever.
Please join Sterling Computers and SolarWinds for an interactive webinar. We’ll review some examples of how to best support your remote users with various tools, including the following:
• SolarWinds® Server & Application Monitor (SAM)
• Network Performance Monitor (NPM)
• NetFlow Traffic Analyzer (NTA)
• Security Event Manager (SEM)
• Access Rights Manager™ (ARM)
We’ll present and demonstrate these SolarWinds products and review how they can address the key remote workforce use-cases faced by organizations and end users. Additionally, we’ll demonstrate how these modules can give you deep insight into your applications and networks and how they affect remote users.
Register and attend this webinar to learn how Sterling Computers and SolarWinds can help you do the following:
• Monitor and troubleshoot your video conferencing and communication applications, such as Zoom, Webex, Microsoft® Teams, Citrix, etc.
• Ensure the availability and performance of your site-to-site VPNs and your remote VPN user connections
• Monitor logs to help improve compliance and reduce threats
• Manage and audit user access to Active Directory, Azure® AD, Exchange, SharePoint, OneDrive, and file servers
• Monitor the performance, capacity and health of Linux® and Windows® applications across data centers, remote offices, and the cloud
Securing Manufacturing: How we can improve speed and efficiency while protect...Conor Bronsdon
in 2019 Microsoft Services started a series of eBooks examining key opportunities in Cybersecurity & Identity transformation within different industries.
This paper (which I contributed to) examines industrial manufacturing security within the context of today's digital transformation.
Barcelona Digital Festival 28th Nov 2019 - Data Analytics in eSports. UbeatCa...CIO Edge
Taken from our BCN Digital Festival last week, for info on attending, speaking or sponsoring our next event on the 29/30th April 2020 email enquiry@digitalenterprisefest.com
Data Analytics in eSports. UbeatCase Study
Building AI & Automate services need a solid base on Data Management, but the current environment is volatile, uncertain, complex and ambiguous so you never know what data will be important in the following months.
The Data Management Platform in an extremely dynamic market like eSports where everything is currently being created, in reinvention and is to be validated is even more challenging.
UBEAT is the leading streaming platform of eSports related content. Created in November 2018, it still hasn’t 12 months of existence but a lot of learnings in its rear mirror and a lot of future to come in its high beam. Especially regarding Data Management.
To apply to speak or sponsor our 2020 events goto www.digitalenterprisefest.com
Give ‘Em What They Want! Self-Service Middleware Monitoring in a Shared Servi...SL Corporation
Self-service monitoring dashboards enhance cross-department productivity and reduce information-reporting burden on middleware operations teams.
In this presentation, Intuit shares their professional best practices for providing real-time and historic health and performance information on their shared middleware platform to different groups across the enterprise using RTView® self-service dashboards.
For more information on SL and RTView® Enterprise Monitor™, End-to-End Monitoring and Middleware Monitoring, please visit us at http://www.sl.com.
Chaos Engineering: Why the World Needs More Resilient SystemsC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2luk9iS.
Tammy Butow shares her experiences using chaos engineering to build resilient systems, when they couldn’t build their systems from scratch. Filmed at qconlondon.com.
Tammy Butow is a Principal SRE at Gremlin where she works on Chaos Engineering, the facilitation of controlled experiments to identify systemic weaknesses. Previously, she led SRE teams at Dropbox responsible for Databases and Storage systems used by over 500 million customers.
How to add security in dataops and devopsUlf Mattsson
The emerging DataOps is not Just DevOps for Data. According to Gartner, DataOps is a collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and consumers across an organization.
The goal of DataOps is to create predictable delivery and change management of data, data models and related artifacts. DataOps uses technology to automate data delivery with the appropriate levels of security, quality and metadata to improve the use and value of data in a dynamic environment.
This session will discuss how to add Security in DataOps and DevOps.
Application Security Testing for Software Engineers: An approach to build sof...Michael Hidalgo
This talk was presented at the 7th WCSQ World Congress for Software Quality in Lima, Perú on Wednesday, 22nd March 2017.
Writing secure code certainly is not an easy endeavor. In the book titled “Writing Secure Code: Practical Strategies and Proven Techniques for Building Secure Applications in a Networked World (Developer Best Practices)” authors Howard and LeBlanc talk about the so called attacker’s advantage and the defenders dilemma and they put into perspective the fact that developers (identified as defenders) must build better quality software because attackers have the advantage.
In this dilemma, software applications must be on a state of defense because attackers are out there taking advantage of any minor mistake, whereas the defender must be always vigilant, adding new features to the code, fixing issues, adding new engineers to the team. All this conditions are important when it comes to software security.
Sadly, strong understanding of software security principles is not always a characteristic of most software engineers but we can’t blame them. Writing code is a complex task per se, the abstraction level required, along with choosing and/or writing the accurate algorithm and dealing with tight schedules seems to be always a common denominator and the outcome when talking to developers.
This talk also includes techniques, tools and guidance that software engineers can use to perform Application Security testing during the development stage, enabling them to catch vulnerabilities at the time they are created.
Hello! VEBTECH team creates solutions that develop your business. 200+ certified developers with 25 years of experience in FinTech. We are strong in:
• Website, platform, and mobile application development
• BPM (PEGA, Zendesk)
• Blockchain solutions
• Cybersecurity (Red Team)
• IT - Monitoring
Platform governance, gestire un ecosistema di microservizi a livello enterpriseGiulio Roggero
A livello enterprise, le moderne architetture distribuite coinvolgono molti team differenti, centinaia di sviluppatori e operations e migliaia microservizi ed API in produzione. Come si può gestire questa
e o
un'esplosione di costi e preservando il time-to-market?
How to Enable, Monitor, and Secure Your Remote WorkforceSolarWinds
In the wake of COVID-19, many businesses, educational institutions, and government entities have had to change their models to accommodate a remote workforce. In this new norm, it’s important for employees to be able to work in secure, remote environments. Your organization and its stakeholders depend on your applications, and remote users can put new strains on application and network capabilities. This makes an IT professional’s job more challenging than ever.
Please join Sterling Computers and SolarWinds for an interactive webinar. We’ll review some examples of how to best support your remote users with various tools, including the following:
• SolarWinds® Server & Application Monitor (SAM)
• Network Performance Monitor (NPM)
• NetFlow Traffic Analyzer (NTA)
• Security Event Manager (SEM)
• Access Rights Manager™ (ARM)
We’ll present and demonstrate these SolarWinds products and review how they can address the key remote workforce use-cases faced by organizations and end users. Additionally, we’ll demonstrate how these modules can give you deep insight into your applications and networks and how they affect remote users.
Register and attend this webinar to learn how Sterling Computers and SolarWinds can help you do the following:
• Monitor and troubleshoot your video conferencing and communication applications, such as Zoom, Webex, Microsoft® Teams, Citrix, etc.
• Ensure the availability and performance of your site-to-site VPNs and your remote VPN user connections
• Monitor logs to help improve compliance and reduce threats
• Manage and audit user access to Active Directory, Azure® AD, Exchange, SharePoint, OneDrive, and file servers
• Monitor the performance, capacity and health of Linux® and Windows® applications across data centers, remote offices, and the cloud
Securing Manufacturing: How we can improve speed and efficiency while protect...Conor Bronsdon
in 2019 Microsoft Services started a series of eBooks examining key opportunities in Cybersecurity & Identity transformation within different industries.
This paper (which I contributed to) examines industrial manufacturing security within the context of today's digital transformation.
Barcelona Digital Festival 28th Nov 2019 - Data Analytics in eSports. UbeatCa...CIO Edge
Taken from our BCN Digital Festival last week, for info on attending, speaking or sponsoring our next event on the 29/30th April 2020 email enquiry@digitalenterprisefest.com
Data Analytics in eSports. UbeatCase Study
Building AI & Automate services need a solid base on Data Management, but the current environment is volatile, uncertain, complex and ambiguous so you never know what data will be important in the following months.
The Data Management Platform in an extremely dynamic market like eSports where everything is currently being created, in reinvention and is to be validated is even more challenging.
UBEAT is the leading streaming platform of eSports related content. Created in November 2018, it still hasn’t 12 months of existence but a lot of learnings in its rear mirror and a lot of future to come in its high beam. Especially regarding Data Management.
To apply to speak or sponsor our 2020 events goto www.digitalenterprisefest.com
Give ‘Em What They Want! Self-Service Middleware Monitoring in a Shared Servi...SL Corporation
Self-service monitoring dashboards enhance cross-department productivity and reduce information-reporting burden on middleware operations teams.
In this presentation, Intuit shares their professional best practices for providing real-time and historic health and performance information on their shared middleware platform to different groups across the enterprise using RTView® self-service dashboards.
For more information on SL and RTView® Enterprise Monitor™, End-to-End Monitoring and Middleware Monitoring, please visit us at http://www.sl.com.
Chaos Engineering: Why the World Needs More Resilient SystemsC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2luk9iS.
Tammy Butow shares her experiences using chaos engineering to build resilient systems, when they couldn’t build their systems from scratch. Filmed at qconlondon.com.
Tammy Butow is a Principal SRE at Gremlin where she works on Chaos Engineering, the facilitation of controlled experiments to identify systemic weaknesses. Previously, she led SRE teams at Dropbox responsible for Databases and Storage systems used by over 500 million customers.
How to add security in dataops and devopsUlf Mattsson
The emerging DataOps is not Just DevOps for Data. According to Gartner, DataOps is a collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and consumers across an organization.
The goal of DataOps is to create predictable delivery and change management of data, data models and related artifacts. DataOps uses technology to automate data delivery with the appropriate levels of security, quality and metadata to improve the use and value of data in a dynamic environment.
This session will discuss how to add Security in DataOps and DevOps.
Application Security Testing for Software Engineers: An approach to build sof...Michael Hidalgo
This talk was presented at the 7th WCSQ World Congress for Software Quality in Lima, Perú on Wednesday, 22nd March 2017.
Writing secure code certainly is not an easy endeavor. In the book titled “Writing Secure Code: Practical Strategies and Proven Techniques for Building Secure Applications in a Networked World (Developer Best Practices)” authors Howard and LeBlanc talk about the so called attacker’s advantage and the defenders dilemma and they put into perspective the fact that developers (identified as defenders) must build better quality software because attackers have the advantage.
In this dilemma, software applications must be on a state of defense because attackers are out there taking advantage of any minor mistake, whereas the defender must be always vigilant, adding new features to the code, fixing issues, adding new engineers to the team. All this conditions are important when it comes to software security.
Sadly, strong understanding of software security principles is not always a characteristic of most software engineers but we can’t blame them. Writing code is a complex task per se, the abstraction level required, along with choosing and/or writing the accurate algorithm and dealing with tight schedules seems to be always a common denominator and the outcome when talking to developers.
This talk also includes techniques, tools and guidance that software engineers can use to perform Application Security testing during the development stage, enabling them to catch vulnerabilities at the time they are created.
2. Bloomberg
• Bloomberg technology helps drive the world’s financial markets
– We build our own software, digital platforms, mobile applications and state of the
art hardware
– We run one of the world’s largest private networks with over 20,000 routers across
our network
– We have the largest server side JavaScript deployment in the world – 22 million
lines of JavaScript code
– We developed “cloud computing” and deployed “software as a service” well ahead
of the general marketplace
– Our technology, has brought transparency to the global financial markets
• Bloomberg technologists
– More than 3,000 software developers and designers located around the world
(London, NYC, SF “tech hubs”)
– BloombergLabs.com (@BloombergLabs) is our platform for dialogue between our
experts and the broader tech community
• Our clients
– Over 320,000 subscribers
– Primarily financial professionals including investment bankers, CFOs, investor
relations, hedge funds managers, foreign exchange, etc.
6. Compliance
Platform
and
Processing
Pipeline
Chat
Reference
Data
Trade
Data
Customer
Data
Product
Data
Market
Data
Counterparty
Email
Social Media Voice
Human-‐
and
Machine-‐
generated
Data
Surveillance
Pipeline
Communication
Data
Transactional
Data
User
Data
Case
Management
Compliance
Platform
Compliance
Storage
Compliance
Officers
Search,
Review,
Analyze
8. Need
for
a
robust,
scalable,
high
performance,
geo-‐distributed
data
storage
and
retrieval
system
❑ More
than
3
Peta
Bytes
of
archived
data
❑ 80+
Billion
indexed
objects
❑ Real-‐time
scanning
of
35
million
objects
per
day
100’s
Gigabytes/year
Communication
Data
Growth Cumulative
Data
Growth
Over
3
Petabytes
today
$0.00
$0.75
$1.50
$2.25
$3.00
List Price Replication DR Isolation
$2.31
$1.15
$0.58
$0.19
Storing 1GB of Data
Storage
Cost
2000 2002 2004 2006 2008 2010 2012
9. Need
for
Low
Level
Security
Primitives
Document Level Security
Lorem
ipsum
dolor
sit
amet,
consectetur
adipiscing
elit,
sed
do
eiusmod
tempor
incididunt
ut
labore
et
dolore
magna
aliqua.
Ut
enim
ad
minim
veniam,
quis
nostrud
exercitation
ullamco
laboris
nisi
ut
aliquip
ex
ea
commodo
consequat.
Duis
aute
irure
dolor
in
reprehenderit
in
voluptate
velit
esse
cillum
dolore
eu
fugiat
nulla
pariatur.
Excepteur
sint
occaecat
cupidatat
non
proident,
sunt
in
culpa
qui
officia
deserunt
mollit
anim
id
est
laborum
Company Level Security
Data StoreData Pipe Application
User Level Security
Data Store
10. Security
Solutions
• Post-process the queries
– Too slow
– Nasty bugs
• Generate unique document for each view
– Exponential growth in number of documents
• Use application specific features
– Solr dynamic fields, Mangled Fields
• Accumulo Visibility
– Fast, Clean, Generic
11. Data
Model
Row ID Value
CompanyA_userX_20150426 <bytes>
CompanyA_userX_20150426 <bytes>
CompanyA_userX_20150427 <bytes>
CompanyA_userX_20150428 <bytes>
CompanyA_userY_20150427 <bytes>
CompanyB_userX_20150428 <bytes>
CompanyB_userX_20150428 <bytes>
CompanyB_userX_20150428 <bytes>
12. Find
all
Communications
for
a
Set
of
Users
for
a
Date
Range
Row ID Value
CompanyA_userX_20150426 <bytes>
CompanyA_userX_20150426 <bytes>
CompanyA_userX_20150427 <bytes>
CompanyA_userX_20150428 <bytes>
CompanyA_userY_20150427 <bytes>
CompanyB_userX_20150428 <bytes>
CompanyB_userX_20150428 <bytes>
CompanyB_userX_20150428 <bytes>
Batch
Scanner
Application
13. Find
all
Records
with
“Libor”
Filter
Row ID Value
CompanyA_userX_20150426 <bytes>
CompanyA_userX_20150426 <bytes>
CompanyA_userX_20150427 <bytes>
CompanyA_userX_20150428 <bytes>
CompanyA_userY_20150427 <bytes>
CompanyB_userX_20150428 <bytes>
CompanyB_userX_20150428 <bytes>
CompanyB_userX_20150428 <bytes>
Batch
Scanner
Application
14. Count
Number
of
Objects
that
Match
a
Filter
Counting
Iterator
Filter
Row ID Value
CompanyA_userX_20150426 <bytes>
CompanyA_userX_20150426 <bytes>
CompanyA_userX_20150427 <bytes>
CompanyA_userX_20150428 <bytes>
CompanyA_userY_20150427 <bytes>
CompanyB_userX_20150428 <bytes>
CompanyB_userX_20150428 <bytes>
CompanyB_userX_20150428 <bytes>
Batch
Scanner
Application