This presentation describes how NOSQL solutions such as the Neo4j graph database and Lucene/Solr index was used in a classic middleware stack in Telenor to solve perfomance and scalability issues.
Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor...Sebastian Verheughe
Learn how Telenor uses Neo4j to protect data in business critical services running in production. Sebastian will discuss lessons learned both with technology and our experience after running it in production for half a year, backing many of our mission critical services.
Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor...Sebastian Verheughe
Learn how Telenor uses Neo4j to protect data in business critical services running in production. Sebastian will discuss lessons learned both with technology and our experience after running it in production for half a year, backing many of our mission critical services.
this is the slides from the talk i gave at DevGeekWeek2014
further details are in my blog: http://blogs.microsoft.co.il/iblogger/2014/06/25/devgeekweek-2014-slides-and-demos/
Enterprise Application Architectures by Dr. Indika KumaraThejan Wijesinghe
Enterprise Applications/Computing
Architecture Styles for Enterprise Applications
Method-oriented
Message-oriented
Resource-oriented
REST (representational state transfer)
Event-oriented
SOA (service-oriented architecture)
Basic and extended SOA
Implementing SOA
RESTful
WS-* (web services stack)
ESB (enterprise service bus)
Business processes and service compositions
Building Cloud-Native App Series - Part 4 of 11
Microservices Architecture Series
NoSQL vs SQL
Redis, MongoDB, AWS DynamoDB
Big Data Design Patterns
Sharding, Partitions
Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor...Sebastian Verheughe
Learn how Telenor uses Neo4j to protect data in business critical services running in production. Sebastian will discuss lessons learned both with technology and our experience after running it in production for half a year, backing many of our mission critical services.
Using Graph Databases in Real-Time to Solve Resource Authorization at Telenor...Sebastian Verheughe
Learn how Telenor uses Neo4j to protect data in business critical services running in production. Sebastian will discuss lessons learned both with technology and our experience after running it in production for half a year, backing many of our mission critical services.
this is the slides from the talk i gave at DevGeekWeek2014
further details are in my blog: http://blogs.microsoft.co.il/iblogger/2014/06/25/devgeekweek-2014-slides-and-demos/
Enterprise Application Architectures by Dr. Indika KumaraThejan Wijesinghe
Enterprise Applications/Computing
Architecture Styles for Enterprise Applications
Method-oriented
Message-oriented
Resource-oriented
REST (representational state transfer)
Event-oriented
SOA (service-oriented architecture)
Basic and extended SOA
Implementing SOA
RESTful
WS-* (web services stack)
ESB (enterprise service bus)
Business processes and service compositions
Building Cloud-Native App Series - Part 4 of 11
Microservices Architecture Series
NoSQL vs SQL
Redis, MongoDB, AWS DynamoDB
Big Data Design Patterns
Sharding, Partitions
Building Cloud-Native App Series - Part 1 of 11
Microservices Architecture Series
Design Thinking, Lean Startup, Agile (Kanban, Scrum),
User Stories, Domain-Driven Design
Increase Profits with Better Vehicle Listing DataConnotate
Auto dealers and providers of dealer support systems face big challenges obtaining vehicle listing data at an affordable price. Licensed data feeds are very expensive, and using internal IT resources to “do it yourself” creates operational headaches. Connotate’s Vehicle Listings webinar presents an innovative solution.
Oracle GoldenGate is the leading real-time data integration software provider in the industry - customers include 3 of the top 5 commercial banks, 3 of the top 3 busiest ATM networks, and 4 of the top 5 telecommunications providers.
Oracle GoldenGate moves transactional data in real-time across heterogeneous database, hardware and operating systems with minimal impact. The software platform captures, routes, and delivers data in real time, enabling organizations to maintain continuous uptime for critical applications during planned and unplanned outages.
Additionally, it moves data from transaction processing environments to read-only reporting databases and analytical applications for accurate, timely reporting and improved business intelligence for the enterprise.
Lessons from Building Large-Scale, Multi-Cloud, SaaS Software at DatabricksDatabricks
The cloud has become one of the most attractive ways for enterprises to purchase software, but it requires building products in a very different way from traditional software
In this webinar by Jonas Bonér, creator of Akka and CTO/Co-Founder of Lightbend, we take a look at Cloudstate, an OSS tool built on Akka, gRPC, Knative, GraalVM, and Kubernetes. Cloudstate lets you model, manage, and scale stateful services while preserving responsiveness by designing for resilience and elasticity.
Unlocking the Power of Salesforce Integrations with ConfluentAaronLieberman5
Salesforce currently has 150,000 customers across the world who use Salesforce in some capacity. If you are one of those customers, you've likely had to work through how to integrate it with your other back office systems: ERP, Marketing Automation, BI systems, etc. Or perhaps you're a brand new Salesforce customer and are just now trying to understand what options exist for integration.
It is undeniable that the rate of integrating with Salesforce is increasing, and extracting the valuable data that is in Salesforce is not always an easy feat when you have to consider how to do this best in your own unique environment.
In this webinar, Big Compass and Confluent will talk about the various techniques for getting data out of Salesforce, and how Confluent and Kafka can play an integral role in not only brokering these messages at an incredibly fast and scalable rate, but to also make it very easy to exchange data with Salesforce.
YOU WILL LEARN:
What integration capabilities exist within Salesforce
How Confluent can be used to integrate with Salesforce
Techniques in Confluent for pub/sub, streaming, and building business logic using KSQL and Kafka Streams
Patterns of Salesforce integration in general and specifically with Confluent
Strengths and weaknesses of each pattern and scenarios where they work best
WHO SHOULD ATTEND:
IT leaders who are looking for the most efficient methods for integration with Salesforce
Developers/System Integrators who are interested in seeing Salesforce integration techniques
Anyone in the Salesforce ecosystem who is interested in integration
REASONS TO ATTEND:
Learn about methods of Salesforce integration and explore Confluent’s built-in capabilities if you're considering an off-the-shelf solution
CSI – IT2020, IIT Mumbai, October 6th 2017
Computer Society of India, Mumbai Chapter
The presentation focuses on Microservices architecture and the comparison between MicroService with Standard Monolithic Apps and SOA based Apps. It also gives a quick outline of Domain Driven Design, Event Sourcing and CQRS, Functional Reactive Programming and comparison of SAGA pattern with 2 Phase Commit.
http://www.csimumbai.org/it2020-17/index.html
Microservices, Kafka Streams and KafkaEsqueconfluent
Speakers: Patrick Schuh, Bearing Point + Patrik Kleindl, Bearing Point
Abstract:
- Managing topic configurations and dependencies in a microservice deployment
- Managing Kafka Streams configurations
- KafkaEsque: an open source support tool for Apache Kafka® development (https://github.com/patschuh/KafkaEsque)
Webinar: Large Scale Graph Processing with IBM Power Systems & Neo4jNeo4j
We live in a profoundly connected world. From supply chains to payment networks to digital business and complex portfolios, our ability to understand and navigate not just data, but relationships inside the data, play an increasingly important role in all aspects of business. Highly connected value chains that generate massive volumes of connected data create an opportunity for graph analysis, which Gartner describes as "the single most single most effective competitive differentiator for organizations pursuing data-driven operations and decisions." This talk will introduce the power of graph databases and share how the latest IBM Power Systems offerings featuring the POWER8 processor and CAPI-attached Flash enable unique scaling, performance and price-performance advantages for Neo4j workloads.
50 Shades of Data - how, when and why Big, Fast, Relational, NoSQL, Elastic, ...Lucas Jellema
Data has been and will be the key ingredient to enterprise IT. What is changing is the nature, scope and volume of data and the place of data in the IT architecture. BigData, unstructured data and non-relational data stored on Hadoop, in NoSQL databases and held in Elastic Search, Caches and Message Queues complements data in the enterprise RDBMS. Trends such as microservices that contain their own data, BASE, CQRS and Event Sourcing have changed the way we store, share and govern data. This session introduces patterns, technologies and hypes around storing, processing and retrieving data using products such as Oracle Database, Cassandra, MySQL, Neo4J, Kafka, Redis, Elastic Search and Hadoop/Spark -locally,in containers and on the cloud. Key take away: what an application architect and a developer should know about the various types of data in enterprise IT and how to store/manage/query/manipulate them. What products and technologies are at your disposal. How can you make these work together – for a consistent (enough) overall data presentation.
Lessons from Large-Scale Cloud Software at DatabricksMatei Zaharia
Keynote by Matei Zaharia at SOCC 2019
Abstract:
The cloud has become one of the most attractive ways for enterprises to purchase software, but it requires building products in a very different way from traditional software, which has not been heavily studied in research. I will explain some of these challenges based on my experience at Databricks, a startup that provides a data analytics platform as a service on AWS and Azure. Databricks manages millions of VMs per day to run data engineering and machine learning workloads using Apache Spark, TensorFlow, Python and other software for thousands of customers. Two main challenges arise in this context: (1) building a reliable, scalable control plane that can manage thousands of customers at once and (2) adapting the data processing software itself (e.g. Apache Spark) for an elastic cloud environment (for instance, autoscaling instead of assuming static clusters). These challenges are especially significant for data analytics workloads whose users constantly push boundaries in terms of scale (e.g. number of VMs used, data size, metadata size, number of concurrent users, etc). I’ll describe some of the common challenges that our new services face and some of the main ways that Databricks has extended and modified open source analytics software for the cloud environment (e.g., designing an autoscaling engine for Apache Spark and creating a transactional storage layer on top of S3 in the Delta Lake open source project).
Bio:
Matei Zaharia is an Assistant Professor of Computer Science at Stanford University and Chief Technologist at Databricks. He started the Apache Spark project during his PhD at UC Berkeley in 2009, and has worked broadly on datacenter systems, co-starting the Apache Mesos project and contributing as a committer on Apache Hadoop. Today, Matei tech-leads the MLflow open source machine learning platform at Databricks and is a PI in the DAWN Lab focusing on systems for ML at Stanford. Matei’s research was recognized through the 2014 ACM Doctoral Dissertation Award for the best PhD dissertation in computer science, an NSF CAREER Award, and the US Presidential Early Career Award for Scientists and Engineers (PECASE).
IBM Cloud Direct Link 2.0 is the NextGen offering on Direct Link. This presentation provide details on the new DL 2.0 offering and difference between DL 1.0 and 2.0
Oracle Service Bus and Oracle SOA Suite in the Mobile WorldGuido Schmutz
Starting from a unified mobile enterprise architecture, this session will dive into how features in Oracle SOA Suite and especially Oracle Service Bus can be used support mobile architectures. We will show how Oracle Service Bus plays a crucial role in the mobile enablement of existing enterprise services, with a particular focus on how REST binding, introduced in 12c, supports the exposure of any service on the Service Bus in standard REST/JSON. The session will also cover how OSB Result Caching and Service can be used to enable Enterprise Mobility, and how Oracle SOA Suite BPEL and OEP components can support a mobile strategy.
Managing Genetic Ancestry at Scale with Neo4j and Kafka - StampedeCon 2015StampedeCon
At the StampedeCon 2015 Big Data Conference: The global Monsanto R&D pipeline produces millions of new plant populations every year; each which contributes to a dataset of genetic ancestry spanning several decades. Historically the constraints of modeling and processing this data within an RDBMS has made drawing inferences from this dataset complex and computationally infeasible at large scale. Fortunately, the genetic history of any plant population forms a naturally occurring directed acyclic graph, a property that has allowed us to utilize graph theory to re-imagine how ancestral lineage data is modeled, stored, and queried.
In this talk we present our solutions to these problems, as realized using a graph-based approach within Neo4j. We will discuss our learnings around using Neo4j in a production setting that includes transactional and high-throughput computation, including how we transitioned from recursive JOIN queries to using Cypher and the Neo4j traversal framework to take full advantage of index-free adjacency. Our approach to polyglot persistence will be discussed via our use of a distributed commit log, Apache Kafka, to feed our graph store from sources of live transactional data. Finally, we will touch upon how we are using these technologies to annotate our genetic ancestry dataset with molecular genomics data in order to build an pipeline-scale genotype imputation platform with core algorithms built using Apache Spark.
Building Cloud-Native App Series - Part 1 of 11
Microservices Architecture Series
Design Thinking, Lean Startup, Agile (Kanban, Scrum),
User Stories, Domain-Driven Design
Increase Profits with Better Vehicle Listing DataConnotate
Auto dealers and providers of dealer support systems face big challenges obtaining vehicle listing data at an affordable price. Licensed data feeds are very expensive, and using internal IT resources to “do it yourself” creates operational headaches. Connotate’s Vehicle Listings webinar presents an innovative solution.
Oracle GoldenGate is the leading real-time data integration software provider in the industry - customers include 3 of the top 5 commercial banks, 3 of the top 3 busiest ATM networks, and 4 of the top 5 telecommunications providers.
Oracle GoldenGate moves transactional data in real-time across heterogeneous database, hardware and operating systems with minimal impact. The software platform captures, routes, and delivers data in real time, enabling organizations to maintain continuous uptime for critical applications during planned and unplanned outages.
Additionally, it moves data from transaction processing environments to read-only reporting databases and analytical applications for accurate, timely reporting and improved business intelligence for the enterprise.
Lessons from Building Large-Scale, Multi-Cloud, SaaS Software at DatabricksDatabricks
The cloud has become one of the most attractive ways for enterprises to purchase software, but it requires building products in a very different way from traditional software
In this webinar by Jonas Bonér, creator of Akka and CTO/Co-Founder of Lightbend, we take a look at Cloudstate, an OSS tool built on Akka, gRPC, Knative, GraalVM, and Kubernetes. Cloudstate lets you model, manage, and scale stateful services while preserving responsiveness by designing for resilience and elasticity.
Unlocking the Power of Salesforce Integrations with ConfluentAaronLieberman5
Salesforce currently has 150,000 customers across the world who use Salesforce in some capacity. If you are one of those customers, you've likely had to work through how to integrate it with your other back office systems: ERP, Marketing Automation, BI systems, etc. Or perhaps you're a brand new Salesforce customer and are just now trying to understand what options exist for integration.
It is undeniable that the rate of integrating with Salesforce is increasing, and extracting the valuable data that is in Salesforce is not always an easy feat when you have to consider how to do this best in your own unique environment.
In this webinar, Big Compass and Confluent will talk about the various techniques for getting data out of Salesforce, and how Confluent and Kafka can play an integral role in not only brokering these messages at an incredibly fast and scalable rate, but to also make it very easy to exchange data with Salesforce.
YOU WILL LEARN:
What integration capabilities exist within Salesforce
How Confluent can be used to integrate with Salesforce
Techniques in Confluent for pub/sub, streaming, and building business logic using KSQL and Kafka Streams
Patterns of Salesforce integration in general and specifically with Confluent
Strengths and weaknesses of each pattern and scenarios where they work best
WHO SHOULD ATTEND:
IT leaders who are looking for the most efficient methods for integration with Salesforce
Developers/System Integrators who are interested in seeing Salesforce integration techniques
Anyone in the Salesforce ecosystem who is interested in integration
REASONS TO ATTEND:
Learn about methods of Salesforce integration and explore Confluent’s built-in capabilities if you're considering an off-the-shelf solution
CSI – IT2020, IIT Mumbai, October 6th 2017
Computer Society of India, Mumbai Chapter
The presentation focuses on Microservices architecture and the comparison between MicroService with Standard Monolithic Apps and SOA based Apps. It also gives a quick outline of Domain Driven Design, Event Sourcing and CQRS, Functional Reactive Programming and comparison of SAGA pattern with 2 Phase Commit.
http://www.csimumbai.org/it2020-17/index.html
Microservices, Kafka Streams and KafkaEsqueconfluent
Speakers: Patrick Schuh, Bearing Point + Patrik Kleindl, Bearing Point
Abstract:
- Managing topic configurations and dependencies in a microservice deployment
- Managing Kafka Streams configurations
- KafkaEsque: an open source support tool for Apache Kafka® development (https://github.com/patschuh/KafkaEsque)
Webinar: Large Scale Graph Processing with IBM Power Systems & Neo4jNeo4j
We live in a profoundly connected world. From supply chains to payment networks to digital business and complex portfolios, our ability to understand and navigate not just data, but relationships inside the data, play an increasingly important role in all aspects of business. Highly connected value chains that generate massive volumes of connected data create an opportunity for graph analysis, which Gartner describes as "the single most single most effective competitive differentiator for organizations pursuing data-driven operations and decisions." This talk will introduce the power of graph databases and share how the latest IBM Power Systems offerings featuring the POWER8 processor and CAPI-attached Flash enable unique scaling, performance and price-performance advantages for Neo4j workloads.
50 Shades of Data - how, when and why Big, Fast, Relational, NoSQL, Elastic, ...Lucas Jellema
Data has been and will be the key ingredient to enterprise IT. What is changing is the nature, scope and volume of data and the place of data in the IT architecture. BigData, unstructured data and non-relational data stored on Hadoop, in NoSQL databases and held in Elastic Search, Caches and Message Queues complements data in the enterprise RDBMS. Trends such as microservices that contain their own data, BASE, CQRS and Event Sourcing have changed the way we store, share and govern data. This session introduces patterns, technologies and hypes around storing, processing and retrieving data using products such as Oracle Database, Cassandra, MySQL, Neo4J, Kafka, Redis, Elastic Search and Hadoop/Spark -locally,in containers and on the cloud. Key take away: what an application architect and a developer should know about the various types of data in enterprise IT and how to store/manage/query/manipulate them. What products and technologies are at your disposal. How can you make these work together – for a consistent (enough) overall data presentation.
Lessons from Large-Scale Cloud Software at DatabricksMatei Zaharia
Keynote by Matei Zaharia at SOCC 2019
Abstract:
The cloud has become one of the most attractive ways for enterprises to purchase software, but it requires building products in a very different way from traditional software, which has not been heavily studied in research. I will explain some of these challenges based on my experience at Databricks, a startup that provides a data analytics platform as a service on AWS and Azure. Databricks manages millions of VMs per day to run data engineering and machine learning workloads using Apache Spark, TensorFlow, Python and other software for thousands of customers. Two main challenges arise in this context: (1) building a reliable, scalable control plane that can manage thousands of customers at once and (2) adapting the data processing software itself (e.g. Apache Spark) for an elastic cloud environment (for instance, autoscaling instead of assuming static clusters). These challenges are especially significant for data analytics workloads whose users constantly push boundaries in terms of scale (e.g. number of VMs used, data size, metadata size, number of concurrent users, etc). I’ll describe some of the common challenges that our new services face and some of the main ways that Databricks has extended and modified open source analytics software for the cloud environment (e.g., designing an autoscaling engine for Apache Spark and creating a transactional storage layer on top of S3 in the Delta Lake open source project).
Bio:
Matei Zaharia is an Assistant Professor of Computer Science at Stanford University and Chief Technologist at Databricks. He started the Apache Spark project during his PhD at UC Berkeley in 2009, and has worked broadly on datacenter systems, co-starting the Apache Mesos project and contributing as a committer on Apache Hadoop. Today, Matei tech-leads the MLflow open source machine learning platform at Databricks and is a PI in the DAWN Lab focusing on systems for ML at Stanford. Matei’s research was recognized through the 2014 ACM Doctoral Dissertation Award for the best PhD dissertation in computer science, an NSF CAREER Award, and the US Presidential Early Career Award for Scientists and Engineers (PECASE).
IBM Cloud Direct Link 2.0 is the NextGen offering on Direct Link. This presentation provide details on the new DL 2.0 offering and difference between DL 1.0 and 2.0
Oracle Service Bus and Oracle SOA Suite in the Mobile WorldGuido Schmutz
Starting from a unified mobile enterprise architecture, this session will dive into how features in Oracle SOA Suite and especially Oracle Service Bus can be used support mobile architectures. We will show how Oracle Service Bus plays a crucial role in the mobile enablement of existing enterprise services, with a particular focus on how REST binding, introduced in 12c, supports the exposure of any service on the Service Bus in standard REST/JSON. The session will also cover how OSB Result Caching and Service can be used to enable Enterprise Mobility, and how Oracle SOA Suite BPEL and OEP components can support a mobile strategy.
Managing Genetic Ancestry at Scale with Neo4j and Kafka - StampedeCon 2015StampedeCon
At the StampedeCon 2015 Big Data Conference: The global Monsanto R&D pipeline produces millions of new plant populations every year; each which contributes to a dataset of genetic ancestry spanning several decades. Historically the constraints of modeling and processing this data within an RDBMS has made drawing inferences from this dataset complex and computationally infeasible at large scale. Fortunately, the genetic history of any plant population forms a naturally occurring directed acyclic graph, a property that has allowed us to utilize graph theory to re-imagine how ancestral lineage data is modeled, stored, and queried.
In this talk we present our solutions to these problems, as realized using a graph-based approach within Neo4j. We will discuss our learnings around using Neo4j in a production setting that includes transactional and high-throughput computation, including how we transitioned from recursive JOIN queries to using Cypher and the Neo4j traversal framework to take full advantage of index-free adjacency. Our approach to polyglot persistence will be discussed via our use of a distributed commit log, Apache Kafka, to feed our graph store from sources of live transactional data. Finally, we will touch upon how we are using these technologies to annotate our genetic ancestry dataset with molecular genomics data in order to build an pipeline-scale genotype imputation platform with core algorithms built using Apache Spark.
Strategic Design by Architecture and Organisation @ FINN.no - JavaZone 2016Sebastian Verheughe
Experience talk about how architecture and organization come together to address the challenges we face at FINN.no. How we believe decentralised ownership and decision making can help improve development speed and product quality over time. Where we still see complexity in FINN after we started using micro services. And how we try to use the inverse Conway manoeuvre together with DDD to extract the strategic parts of our legacy code. The talk will also address how we currently address data flows across services, and how we are moving in the direction of using events and data streams.
Using Graph Databases in Real-time to Solve Resource Authorization at Telenor...Neo4j
Learn how Telenor uses Neo4j to protect data in business critical services running in production. Sebastian will discuss lessons learned both with technology and our experience after running it in production for half a year, backing many of our mission critical services.
The most recently updated version of the slides are available here: http://www.slideshare.net/verheughe/graph-connect-sf-4-oct-2013
Building a Distributed Reservation System with Cassandra (Andrew Baker & Jeff...DataStax
At Choice Hotels International, we are in the midst of a multi-year effort to replace our 25 year old monolithic reservation system with a cloud-based, microservice-style architecture using Cassandra. Since processing the first live reservation on the new system in December 2015, we've been shifting an increasing amount of shopping and booking traffic to the new system, with retirement of the old system scheduled for early 2017.
After a quick review of our problem space, architecture, schema design, and Cassandra deployment, we'll take a closer look several challenges we faced and discuss how they impacted our data modeling, development and deployment:
* Managing data with varying consistency requirements
* Maintaining data integrity across microservice boundaries
* Performing complex queries involving overlapping time ranges
* Relying on time-to-live (TTL) for data cleanup
* Balancing denormalization, performance and cost
About the Speakers
Andrew Baker Senior Software Engineer, Choice Hotels International
Andrew is the technical lead of the service development team responsible for storage and maintenance of rates and reservations for thousands of hotels around the world.
Jeffrey Carpenter Systems Architect, Choice Hotels International
Jeff Carpenter is a software and systems architect with experience in the hospitality and defense industries, it. Jeff is currently working on a cloud-based hotel reservation system using Cassandra and is the author of the new O'Reilly book "Cassandra: The Definitive Guide, 2nd edition".
Automotive community platform - My GarageDejan Restak
Vibe Network is seeking partners in order to expand internationally with its automotive community platform that was successfully tested in Serbia on our website MojaGaraza.rs (My Garage)
Website and our software platform have attracted great interest in SEE and were successfully sold to Infostud member of Alma Media (Finland). Infostud acquired website MojaGaraza.rs together with software license for Serbian language only.
After we validated our concept and technology with the best clients from the automotive market in local region but also with more than 230 000 unique visitors per month we have a know-how to establish successful global automotive community.
Vibe Network is seeking additional investment in order to expand internationally with its automotive community platform that was successfully tested in Serbia on our website MojaGaraza.rs (My garage)
Website and our software platform have attracted great interest in SEE and were successfully sold to Infostud member of Daily Mail and General Trust plc. Infostud acquired website MojaGaraza.rs together with software license for Serbian language only.
After we validated our concept and technology with the best clients from the automotive market in local region but also with more than 100 000 unique visitors per month we have a know-how to establish successful global automotive community.
Why paperless lab is just the first step towards a smart labOSTHUS
Life science’s main asset is its data. Data forms the basis of scientific decision making and its availability via electronic systems is a prerequisite for collaborative work and successful innovation. While more data is published as linked (open) data, huge amounts of data remain unused in internal data silos, such as various ELN’s, because of substantial integration efforts and data quality issues. Since the overwhelming amount of data is unstructured, information extraction and corresponding classification and semantic labeling of content is required. To generate value from your ELN data, a solid informatics strategy is needed to ensure data quality and streamline analytics. Semantic technologies are key enabler to overcome existing limitations.
My talk from SMI ELN conference looking at the ELN landscape:
• Business drivers
- Comparing the different sectors and disciplines
• Build or buy?
• An overview of the solution space
• Patterns of success
An overview of the current state of electronic laboratory notebooks (ELNs), pros and cons of using an ELN, and important considerations for adopting an ELN.
Lab Notebooks as Data Management (SLA Winter Virtual Conference 2012)Kristin Briney
This talk, aimed at librarians, describes the data management issues surrounding paper and electronic lab notebooks. It offers several ways for librarians to support good practices and the transition from paper to electronic.
Wanderu - Lessons from Building a Travel Site with Neo4jNeo4j
Wanderu is a consumer-focused search engine for buses and trains. In this webinar, we will recount the architectural, modeling and other technical "lessons learned" and "lessons unlearned" in implementing our geospatial and search features using Neo4j in the context of a NoSQL polyglot solution.
Speaker: Eddy Wong, CTO, Wanderu
A technologist, innovator and entrepreneur who has architected products and web sites for companies like Hasbro, Maark, Allurent, Macromedia, Allaire, Open Sesame, Philips and AT&T. He was the Chief Architect at Open Sesame where he built one of the first attribute-based personalization engines. Eddy has over 15 years of experience as a software architect and is a Boston tech-community leader in the areas of NoSQL, Big Data and Personalization. He is also the organizer of the Boston GraphDB Meetup.
The Windows Azure Platform is an internet-scale cloud services platform hosted in Microsoft data centers that provides an operating system and a set of developer services that can be used individually or together. The Azure platform can be used to build new applications to run from the cloud or enhance existing applications with cloud-based capabilities. The open and extensible nature of Azure’s architecture gives developers the choice to build web applications, applications running on connected devices, PCs, servers, or hybrid solutions offering the best of online and on-premises.
In this session, Wade Wegner will focus on two scenarios – new application development and the migration of existing applications. While walking through demos, Wade will discuss concepts such as application authentication and authorization, data synchronization between the cloud and on-premises databases, application integration, delegation of identity, and operations and management. Particular emphasis will be placed on the migration of existing internal applications to Windows Azure, securing applications through claims-based authentication and passive federation with Geneva Server, using relational databases in the cloud with SQL Azure, and the migration of data to the cloud through tools like SSIS.
Making communication across boundaries simple with Azure Service BusParticular Software
There are times when you should consider setting up secure communications between your software components across network boundaries.
Here are just a few:
* Your application is enormous (e.g., the global deployment of a marketing site targeting billions of people)
* Remoteness (e.g., your company has branch office locations around the globe)
* Your network constraints prevent communication (e.g., your machines in Azure Cloud Services are unable to talk to each other directly)
* You don't know the network conditions (e.g., IoT or mobile devices)
Yves Goeleven and Sean Feldman show how to overcome such challenges using Azure Service Bus.
Service Discovery and Registration in a Microservices ArchitecturePLUMgrid
Microservices, Service Discovery and Registration have been heading towards the peak of inflated expectations on the Gartner Hype cycle for over the last year or so, but there has often been a lack of clarity as to what these are, why are they needed or how to implement them well.
Service discovery and registration are key components of most distributed systems and service oriented architectures. In this session we will talk about what, why and how of service registration and discovery in distributed systems in general and OpenStack in particular.
We will talk about some of the technologies that address this challenge like Zookeeper, Etcd, Consul, Mesos-DNS, Minuteman, SkyDNS, SmartStack or Eureka. We will also address how these technologies as well as existing OpenStack projects can be used to solve this problem inside OpenStack environments.
Cloud Architecture Tutorial - Running in the Cloud (3of3)Adrian Cockcroft
Part 3 of the talk covers how to transition to cloud, how to bootstrap developers, how to run cloud services including Cassandra, capacity planning and workload analysis, and organizational structure
Venture capitalist Matt Ocko’s 20-year track record of success in the startup world has given him unique insight into how AWS has changed the venture financing process. In this session, you’ll learn about industries susceptible to disruption by AWS-based startups, and where VCs are willing to take new risks on those startups, including the heavily-regulated medical, government, financial, and industrial sectors. Matt will talk about how new, supercomputing startups are now possible because of AWS technologies. Hear about how using AWS technologies can actually reduce risk – and reduce time to customer penetration – from a VC perspective, and how to go from ‘AWS to Series A’ in 5 easy pieces.
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...ScyllaDB
Discover how to avoid common pitfalls when shifting to an event-driven architecture (EDA) in order to boost system recovery and scalability. We cover Kafka Schema Registry, in-broker transformations, event sourcing, and more.
Overcoming the Top Four Challenges to Real‐Time Performance in Large‐Scale, D...SL Corporation
The most critical large-scale applications today, regardless of industry, involve a demand for real-time data transfer and visualization of potentially large volumes of data. With this demand comes numerous challenges and limiting factors, especially if these applications are deployed in virtual or cloud environments. Attend this session and learn how to overcome the top four challenges to real-time application performance: database performance, network data transfer bandwidth limitations, processor performance and lack of real-time predictability. Solutions discussed will include design of the proper data model for the application data, along with design patterns that facilitate optimal and minimal data transfer across networks.
Early Draft: Service Mesh allows developers to focus on business logic while the crosscutting network data layer code is handled by the Service Mesh. This is a boon because this code can be tricky to implement and hard to test all of the edge cases. Service Mesh takes this a few steps further than AOP or Servlet Filters or custom language-specific frameworks because it works regardless of the underlying programming language being used which is great for polyglot development shops. Thus standardizing how these layers work, while allowing teams to pick the best tools or languages for the job at hand. Kubernetes and Istio Service Mesh automate best practices for DevSecOps needs like: failover, scale-out, scalability, health checks, circuit breakers, rate limiters, metrics, observability, avoiding cascading failure, disaster recovery, and traffic routing; supporting CI/CD and microservices architecture.
Istio’s ability to automate and maintaining zero trust networks is its most important feature. In the age of high-profile data breaches, security is paramount. Companies want to avoid major brand issues that impact the bottom line and shrink market capitalization in an instant. Istio allows a standard way to do mTLS and auto certificate rotation which helps prevent a breach and limits the blast radius if a breach occurs. Istio also takes the concern of mTLS from microservices deployments and makes it easy to use taking the burden off of application developers.
Transitioning to the Cloud: Implications for Reliability, Redundancy & Recove...RightScale
RightScale Webinar: July 23, 2009 – Listen to a recorded live panel discussion moderated by Dave Nielsen, founder of CloudCamp and Principal Consultant at Platform D. Hear from leading experts in cloud computing to gain real-world insight into how their customers are using cloud computing and achieving new levels of backup, recovery and reliability. Learn how you can migrate your apps and data to the cloud in a way that's affordable and reliable. Watch video at http://vimeo.com/rightscale/transitioning-to-the-cloud.
Micro services - Practicalities & things to watch out forParthiban J
What is micro services and what it takes to build a micro services application. From discovering your services to some best practices that can be followed to avoid common pitfalls
Overcoming the Top Four Challenges to Real-Time Performance in Large-Scale, D...SL Corporation
The most critical large-scale applications today, regardless of industry, involve a demand for real-time data transfer and visualization of potentially large volumes of data. With this demand comes numerous challenges and limiting factors, especially if these applications are deployed in virtual or cloud environments. In this session, SL’s CEO, Tom Lubinski, explains how to overcome the top four challenges to real-time application performance: database performance, network data transfer bandwidth limitations, processor performance and lack of real-time predictability. Solutions discussed will include design of the proper data model for the application data, along with design patterns that facilitate optimal and minimal data transfer across networks.
Join this workshop to understand the core concepts of “Cloud Computing” and how businesses around the world are running the infrastructure that supports their websites to lower costs, improve time-to-market, and enable rapid scalability matching resource to demands of users. Whether you are an enterprise looking for IT innovation, agility and resiliency or small and medium business who wants to accelerate growth without a big upfront investment in cash or time for technology, the AWS Cloud provides a complete set of services at zero upfront costs which are available with a few clicks and within minutes.
A brief 30 minute tutorial on Spring Cloud Service Discovery - Centralized, distributed service registration and discovery mechanism. Helps you quickly get started with Services discovery concept and framework provisioned by Spring Cloud Netflix module which is very well known as 'Eureka'.
Enterprise Service Delivery from the AWS Cloud (ARC208) | AWS re:Invent 2013Amazon Web Services
(Presented by Citrix)
As we move to a world where all users are mobile and apps are increasingly delivered from the cloud, security, compliance, and user experience service-level expectations are higher than ever, necessitating that IT look beyond traditional methods for delivering applications. However, there are intelligent cloud networking and provisioning solutions on AWS that can be leveraged to create a service delivery model that addresses the new paradigm. Learn how Citrix NetScaler VPX on AWS provides full application visibility and control through a combination of customer case studies and demos.
In this session, you learn how to:
-Deploy Citrix application delivery technologies (NetScaler, NetScaler Gateway, CloudBridge) into AWS
-Optimize next-gen web applications delivered from AWS, using traffic management and application acceleration capabilities
-Provide global application availability across on-premises data centers and multiple AWS regions using CloudBridge, global server load balancing, and Amazon Route 53 DNS
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofsAlex Pruden
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
Welcome to the first live UiPath Community Day Dubai! Join us for this unique occasion to meet our local and global UiPath Community and leaders. You will get a full view of the MEA region's automation landscape and the AI Powered automation technology capabilities of UiPath. Also, hosted by our local partners Marc Ellis, you will enjoy a half-day packed with industry insights and automation peers networking.
📕 Curious on our agenda? Wait no more!
10:00 Welcome note - UiPath Community in Dubai
Lovely Sinha, UiPath Community Chapter Leader, UiPath MVPx3, Hyper-automation Consultant, First Abu Dhabi Bank
10:20 A UiPath cross-region MEA overview
Ashraf El Zarka, VP and Managing Director MEA, UiPath
10:35: Customer Success Journey
Deepthi Deepak, Head of Intelligent Automation CoE, First Abu Dhabi Bank
11:15 The UiPath approach to GenAI with our three principles: improve accuracy, supercharge productivity, and automate more
Boris Krumrey, Global VP, Automation Innovation, UiPath
12:15 To discover how Marc Ellis leverages tech-driven solutions in recruitment and managed services.
Brendan Lingam, Director of Sales and Business Development, Marc Ellis
The Metaverse and AI: how can decision-makers harness the Metaverse for their...Jen Stirrup
The Metaverse is popularized in science fiction, and now it is becoming closer to being a part of our daily lives through the use of social media and shopping companies. How can businesses survive in a world where Artificial Intelligence is becoming the present as well as the future of technology, and how does the Metaverse fit into business strategy when futurist ideas are developing into reality at accelerated rates? How do we do this when our data isn't up to scratch? How can we move towards success with our data so we are set up for the Metaverse when it arrives?
How can you help your company evolve, adapt, and succeed using Artificial Intelligence and the Metaverse to stay ahead of the competition? What are the potential issues, complications, and benefits that these technologies could bring to us and our organizations? In this session, Jen Stirrup will explain how to start thinking about these technologies as an organisation.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™UiPathCommunity
In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni.
📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath:
Autopilot per Studio Web
Autopilot per Studio
Autopilot per Apps
Clipboard AI
GenAI applicata alla Document Understanding
👨🏫👨💻 Speakers:
Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant
Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath
Andrei Tasca, RPA Solutions Team Lead @NTT Data
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
How NOSQL Paid off for Telenor
1. How NoSQL Paid Off for Telenor
JavaZone
13 September 2012 - Oslo
2. Sebastian Verheughe
Architect and developer
Telenor - mobile middleware services (COS)
Katrina Sponheim
Architect and developer
Telenor – business self service solutions
3. Telenor NoSQL Experience
o The problem
o The business case
o The solution
o The challenges
o The results
o My 5 cents
5. Min Bedrift
Self service portal where Telenor's corporate
customers can manage their entire portfolio of
products.
From small businesses to large corporations
7. The Challenge With Large Corporate Customers
Customers with large portfolios presented a couple of
challenges for the self service solution Min Bedrift:
1. Middleware Services - Not Designed for Search
The middleware services were not designed for managing
large data volumes, resulting in a lot of processing in the
client, and the need for extensive caching there.
2. Resource Authorization – Long Calculation Time
User access to resources required the middleware to
calculate and cache all accesses at logon, something that
could take up to many minutes.
8. The Nightly Logon & Pre-fetch Solution
In order to achieve acceptable response times in MinBedrift,
administrators were logged on and customer data was pre-
fetched and put in a cache each night.
However, as the usage of the solution grew, it became obvious
that the time window available for pre-fetching each night was
closing fast.
0 0 0
9 3 9 3 9 3
6 6 6
2012 2013 2014
9. The Future - Unhandled
Telenor calculated that the pre-fetch time window would soon
be filled, and a increasing percentage of the customers would
experience logon response times above the acceptable x sec.
Login Not pre-fetched Pre-fetched
time
Portfolio Size
x
Customer Portfolio Size
10. The Business Impact
In the end, Telenor would risk losing corporate customers due
to deteriorated customer experience
11. Other Caching Drawbacks
o Stale data up to 24 hours old
o Refresh/login for new users still takes a lot of time
o Memory challenges in Min Bedrift
o Unwanted network/middleware/database load
13. Business Case
The business case is built on the negative consequence of NOT
addressing the problem.
Loss of customers (revenue)
Reduced sales transactions (revenue)
Increased manual support (expenses)
Other
15. Solution Requirements - High Level
The middleware search services should be designed to support
large data sets in a better way for the all clients.
Resource authorization must be fast enough to deliver real time
calculations on demand.
16. The Previous Architecture
Client Client Client Client Client Client
Middleware Services
Master
Data
RDBMS
Multiple Sources
17. The New Architecture
One Master (r/w) – Several Replicas (r)
Client Client Client Client Client Client
Middleware Services
Master
Data
Sybase RDBMS
Search Res Auth
Solr / Lucene Neo4j
Multiple Sources
18. Domain Event Messaging
Domain Domain
Event Event
Search Messaging Res Auth
Solr / Lucene Apache Camel Neo4j
Raw DB
Event
Master
Data
RDBMS
19. Putting it All Together
Min Bedrift MW Search MW Auth
search
getAuthorizedResources
filteredSearch
authorized match
21. Search Service
Today implemented in Min Bedrift
o Cached nightly
o Simple, and iterates over the nodes when searching
o With memory/GC challenges
22. New Search Service
Data stored in Solr/Lucene search engine
o New middleware module exposing WS using tomcat
o Everything indexed makes search extremely fast
o De-normalized data does not require joins
o Search by relevance, paging, sorting and much more
23. Solr Cores
Customer
Search
Client Account
Service
Subscription
24. Entity Denormalization
An entity may include data from several tables
Customer
Account also contains customer name & id
Subscription also contains account name & id
25. Solr/Lucene - Denormalized List View
has
Subscription
Customer
Account
user
Arthur | Jackson Total 555 21 1234 2341
Lisa | Simpson Youth 555 64 3634 3435
John | Brown Pro 555 25 5433 5352
26. Searching by Relevance
Search some or all rows, and return hits by relevance (or sorted)
Subscription
User Name Subscription Phone Number Account Ref. Score Rank
Jane Youth 555 21 3253 5 3253 10 15 1
Paul Premium 555 23 4365 5262
John Standard 555 95 1436 7346
Nina Standard 555 15 3263 3734
Lydia Youth 555 92 3253
5 7334 5
2
Tom Standard 555 02 6394 3212
Neil Premium 555 03 2583 3523
28. Resource Authorization Service
Stored procedure in RDBMS calculating all accesses
o Uses several minutes to calculate for large customers
o Cached for up to 24 hours
o Extremely complex to understand (1500 lines of sql)
o Tightly coupled with other services querying the database
29. New Resource Authorization Service
Customer structure stored in Neo4j graph database
o New middleware module exposing WS using tomcat
o Designed to focus on the relationships between objects
o Very fast – independent of total amount of objects stored
30. Nodes and Relationships
o Relationships with type and direction
o Nodes (with type as property)
U
USER_ACCESS (with prop inherit: true/false)
C
PART_OF
C
C
CONTROLLED_BY
A
A
A A
S S SUBSCRIBED_BY
S S S S
S S
S S
User Customer Account Subscription
31. Traversal (query)
All traversals start from a single node
The start node is often the
U
user node in our case
C
C
C
A
A A
A A
S S
S S S S S
S S S
S S
User Customer Account Subscription
32. Following the Relationships
One custom PathExpander class
o Only follow valid relationships and direction
o Only follow necessary relationships
o Check inheritance rules for current path
Just override the expand method
Iterable<Relationship> expand(Path, BranchState)
33. Picking the Nodes
Custom Evaluator
o Decide to include or exclude
o Delegate to filter that fits your search
o Filter may further evaluate neighbor nodes
Just override the evaluate method
Evaluation evaluate(Path path){
if (resourceFilter.filter(path)
return Evaluation.INCLUDE_AND_CONTINUE
return Evaluation.EXCLUDE_AND_CONTINUE
}
34. Example Access Authorization
Retrieve all subscriptions using a fan out search
U
C
C
C
A
A A
A A
S S
S S S S S
S S S
S S
User Customer Account Subscription
35. Example Access Authorization
Has access to resource using a reverse search to limit number
of nodes to evaluate. Find all paths, and validate one of them.
U
C
C
C
A
A A
A A
S S
S S S S S
S S S
S S
User Customer Account Subscription
37. Lucene/Solr
o Using a document store in a relational world – updates
o Change mindset to search by relevance, not sorting
o The time is in the small stuff – not difficult but needs learning
o What type of queries to search on this platform, and NOT
o Scaling & Distribution – Actually, not a challenge…
38. Neo4j
o Competence
o New way of thinking
o Making them really fast (profile & understand graph impl)
o Getting the classic middleware take use of the new service
o What type of queries to search on this platform, and NOT
o Scaling, not easy across servers (not needed for now)
41. Project State
o Phase 1 in production (subscription only, nightly populated)
o Phase 2 in system test (the rest + live population)
The following results are from the test environment now
42. Neo4J Runtime Environment
Initial State: Prewarmed at startup, all data in heap
Population: ~20 M nodes (all indexed)
~20 M node properties (only 1 per node)
~50 M relationships
Batchwise (50 K nodes) in 35 minutes
Base heap usage: 10 GB (of 16 GB)
Load: Minimal (not measured with heavy load)
43. Neo4J Measured Performance
Customers measured for performance:
Corporation Customer Accounts Subscriptions
X 160 1 300 147 000
Y 32 000 23 000 52 000
Z 7 18 95 000
X Y Z
48. Service Performance from Min Bedrift
“Google” search for corporation x: 120 ms
Min Bedrift
searchAllResources
120 ms Search Solr
findAuthorizedResources
55 ms Auth Graph
49. Old vs. New Resource Authorization Service
Calculate All Resources RDBMS Graph
X 12 min 18 sec < 2 sec
Y 22 min 58 sec < 2 sec
Z 3 min 15 sec < 2 sec
Cold Warm Heap
51. Summary
Scalable It allows customer growth
Fast logon On demand resource authorization
Fast search Server side search engine much faster
Reusable All clients may use new services
Fresh data Not up to 24 hours old – almost live
52. Alternatives
In-Memory Database (Sybase)
This option was discussed, but license cost and the uncertainty
if it would be enough made us go for the NoSQL option.
Other NoSQL Solutions
We chose to prototype Neo4j and Lucene/Solr because they
were popular and seemed to fit us well, and since it worked we
stuck with them.
53. How We Started Using NoSQL Technology
o Downloaded and prototyped technology very early
o Got training on site to accelerate the development startup
o At the end of development, did a review/QA of the solution
For Lucene/Solr, we got training and support from local Solr/Lucene
expert consultant Jan Høydahl
For Neo4j, we got training and excellent support from NeoTech directly
55. Think About…
New Technology
Do you have enough in-house competence, or can you easily buy the
necessary competence? Also when maintaining the code.
No Language Standard for Graph Databases
How simple (or possible) is it to change the NoSQL provider?
Working With Relationships
Graph databases are intuitive and fast to work with when interested
in how objects are related to each other.
Gentle NoSQL Introduction
Easier to start using when supporting a specific and limited services
Complexity
You introduce complexity, so make sure it is worth it!