wikimedia-architecture
9 likes5,931 views
The document describes the architecture of Wikimedia, which includes: - A global content delivery network with 3 data centers across Tampa, Amsterdam, and Seoul. - Caching layers using Squid and a memcached distributed object cache. - Around 350 servers running the MediaWiki software powered by MySQL databases. - Geographic load balancing directs users to the nearest data center for improved performance.
1 of 32
Downloaded 152 times
Ad
Recommended
NoSQL
NoSQLNuno Simaria NoSQL refers to non-relational databases that were developed to handle large volumes of data across many servers. NoSQL databases such as Redis, Memcached, MongoDB and CouchDB do not use SQL for queries and are more scalable and flexible than traditional relational databases. They often sacrifice consistency, complex queries, and transaction support in favor of availability and flexibility.
Cache-partitioning
Cache-partitioningdavidkftam This document presents a software-based technique for partitioning shared last-level caches (L2 caches) on multicore systems to improve performance. It implements page coloring to allocate physical pages for each process to distinct cache line colors. Experimental results on a Power5 system show this approach can control cache usage and improve performance for multiprogrammed workloads by up to 17% compared to an uncontrolled shared cache. The document also finds that cache stall rates provide a better performance analysis metric than miss rates for some workloads.
Rapid JCR applications development with Sling
Rapid JCR applications development with SlingBertrand Delacretaz The document presents a detailed overview of rapid Java Content Repository (JCR) application development using Apache Sling, highlighting the JCR API specifications (JSR-170/JSR-283) that manage content as hierarchical structures. It covers the benefits of integrating JCR with Sling's scripting capabilities, providing examples of code snippets and demonstrating a blog application built on Sling. The presentation also mentions known compliant repositories and applications that leverage JCR, showcasing its versatility and applicability in various enterprise environments.
Behind the Scenes at LiveJournal: Scaling Storytime
Behind the Scenes at LiveJournal: Scaling StorytimeSergeyChernyshev The document discusses the scaling history and architecture of LiveJournal, including its backend components and various software solutions developed to enhance performance and reliability. Key topics include the transition from a single server to a complex distributed system, the importance of caching mechanisms like memcached, and strategies for managing database writes and user data clustering. It emphasizes the challenges faced during scaling and the need for custom solutions when existing technologies did not meet their requirements.
JCR In 10 Minutes
JCR In 10 MinutesBertrand Delacretaz The document discusses the Java Content Repository (JCR) API, which provides a standard way to access and manage content in a content repository. It defines what JCR is, notes some examples of compliant repositories and applications that use JCR, and provides a short code example for creating and retrieving content from a JCR repository.
GlusterFS as a DFS
GlusterFS as a DFSMahmoud Shiri Varamini The document discusses GlusterFS, a scalable distributed file system designed for high-performance computing and data storage. It covers fundamental concepts of distributed file systems, types of storage, and the advantages of using object storage. GlusterFS is highlighted as an open-source solution that operates on common hardware, supporting various data access methods and scalable storage solutions.
Storage infrastructure using HBase behind LINE messages
Storage infrastructure using HBase behind LINE messagesLINE Corporation (Tech Unit) The document discusses NHN Japan's use of HBase for the LINE messaging platform's storage infrastructure. Some key points:
- HBase is used to store tens of billions of message rows per day for LINE, achieving sub-10ms response times and high availability through dual clusters.
- The presentation covers their experience migrating HBase clusters between data centers online, handling NameNode failures, and stabilizing the LINE message storage cluster.
- It describes the custom HBase replication and bulk data migration tools developed by NHN Japan to support online cluster migrations without downtime. Failure handling and cluster stabilization techniques are also discussed.
Caching in Distributed Environment
Caching in Distributed Environmentabhigad The document discusses caching in distributed environments, outlining types of caches such as temporal, spatial, primed, and demand cache, and how they impact application performance. It covers caching strategies based on data types, the architecture of caching layers, and introduces Microsoft's distributed caching platform 'Velocity'. Additionally, it addresses management practices for efficient caching, emphasizing aspects like transaction processing and shared caches.
Storage solutions for High Performance Computing
Storage solutions for High Performance Computinggmateesc This document discusses storage infrastructure for high-performance computing. It begins by introducing data-intensive science and the need for parallel storage systems. It then discusses several parallel file systems used in HPC like GPFS, Lustre, and PanFS. Key concepts covered include data striping, scale-out NAS, parallel file systems, and IO acceleration techniques. The document also discusses challenges of data growth, bottlenecks in scaling storage, and architectures of various parallel file systems.
Introduction to h base
Introduction to h baseTrendProgContest13 HBase is a distributed, scalable, big data store that provides fast lookup capabilities like Google BigTable. It uses a table-like data structure with rows indexed by a key and stores data in columns grouped by families. HBase is designed to operate on top of Hadoop HDFS for scalability and high availability. It allows for fast lookups, full table scans, and range scans across large datasets distributed across clusters of commodity servers.
Considerations for building your private cloud folsom update 041513
Considerations for building your private cloud folsom update 041513OpenStack Foundation The document discusses considerations for building a private cloud using OpenStack Folsom. It covers topics like defining a private cloud, sizing flavors and capacity planning, networking with nova-network, image management with Glance, storage options, and example architectures. The presentation aims to help architects build private clouds that can easily scale and have good performance.
Cloud computing era
Cloud computing eraTrendProgContest13 Cloud computing, big data, and mobile technologies are driving major changes in the IT world. Cloud computing provides scalable computing resources over the internet. Big data involves extremely large data sets that are analyzed to reveal business insights. Hadoop is an open-source software framework that allows distributed processing of big data across commodity hardware. It includes tools like HDFS for storage and MapReduce for distributed computing. The Hadoop ecosystem also includes additional tools for tasks like data integration, analytics, workflow management, and more. These emerging technologies are changing how businesses use and analyze data.
Hadoop HDFS Architeture and Design
Hadoop HDFS Architeture and Designsudhakara st HDFS is a distributed file system designed for storing very large data files across commodity servers or clusters. It works on a master-slave architecture with one namenode (master) and multiple datanodes (slaves). The namenode manages the file system metadata and regulates client access, while datanodes store and retrieve block data from their local file systems. Files are divided into large blocks which are replicated across datanodes for fault tolerance. The namenode monitors datanodes and replicates blocks if their replication drops below a threshold.
Cluster Computing with Dryad
Cluster Computing with Dryadbutest This document summarizes a presentation given by Mihai Budiu on DryadLINQ and cloud computing. The presentation introduced DryadLINQ as an integration of LINQ queries with Dryad, Microsoft's data-parallel execution engine. This allows expressing data-parallel algorithms like joins, aggregations, and machine learning in a high-level language like C#, and having them automatically parallelized and run on a cluster. The document provides examples of expressing algorithms like histograms, word counting, and machine learning using DryadLINQ.
ROMA User-Customizable NoSQL Database in Ruby
ROMA User-Customizable NoSQL Database in RubyRakuten Group, Inc. Rakuten, Inc. developed ROMA, a user-customizable NoSQL database in Ruby. ROMA provides high scalability, availability, and fault tolerance using techniques like consistent hashing and virtual nodes. It has a plug-in architecture that allows users to extend ROMA's functionality through custom commands. The plug-ins use a domain specific language for easily defining commands to perform operations on structured data stored in ROMA, such as lists and maps.
Coordinating Metadata Replication: Survival Strategy for Distributed Systems
Coordinating Metadata Replication: Survival Strategy for Distributed SystemsKonstantin V. Shvachko The document discusses coordinating metadata replication survival strategies for distributed systems, emphasizing the importance of a coordination engine for achieving consensus among nodes. It details the architecture of HDFS and HBase, explaining how various roles, such as proposer, learner, and acceptor, contribute to maintaining reliability and consistency in data operations. Additionally, it addresses challenges related to distributed systems, including failure scenarios and block management across geographically distributed setups.
Hadoop at Rakuten, 2011/07/06
Hadoop at Rakuten, 2011/07/06Rakuten Group, Inc. Rakuten Inc. uses Hadoop for various purposes including generating recommendation indexes, analyzing logs, and calculating metrics. Their current Hadoop system includes a cluster with 3 masters and 69 slaves, Ganglia monitoring, and HA with DRBD and Heartbeat. It provides benefits over their previous system like lower costs, improved scalability, and faster transaction times. However, they still face challenges around using up HDFS space and fully realizing their data warehouse goals with the new system.
Storage Infrastructure Behind Facebook Messages
Storage Infrastructure Behind Facebook Messagesyarapavan Facebook uses HBase running on HDFS to store messaging data and metadata. Key reasons for choosing HBase include high write throughput, horizontal scalability, and integration with HDFS. Typical clusters have multiple regions and racks for redundancy. Facebook stores small messages, metadata, and attachments in HBase, while larger messages and attachments are stored separately. The system processes billions of read and write operations daily and continues to optimize performance and reliability.
Building A Scalable Open Source Storage Solution
Building A Scalable Open Source Storage SolutionPhil Cryer This document discusses the development of a scalable, open-source storage and processing solution for biodiversity data, highlighting the challenges of current data access, copyright concerns, and redundancy risks. It outlines the plans for a new data center named the 'Darwin Repository' with significant funding and technology collaborations aimed at enhancing data management capabilities. The proposal emphasizes the use of distributed storage and processing techniques to ensure efficiency, fault tolerance, and sustainability in handling the growing volume of biodiversity data.
HDFS Trunncate: Evolving Beyond Write-Once Semantics
HDFS Trunncate: Evolving Beyond Write-Once SemanticsDataWorks Summit The document describes HDFS's implementation of file truncation, which allows reducing a file's length. It evolved HDFS's write-once semantics to support data mutation. Truncate uses the lease and block recovery framework to truncate block replicas in-place, except when snapshots exist, where it uses "copy-on-truncate" to preserve the snapshot. The truncate operation returns immediately after updating metadata, while block adjustments occur in the background.
A Content Repository for TYPO3 5.0
A Content Repository for TYPO3 5.0Karsten Dambekalns The document discusses the development of a PHP-based content repository for TYPO3 5.0, highlighting the advantages of using an object-based storage system over traditional databases. It outlines the current status of the project, including the implementation of the JSR-170 API in PHP, and emphasizes the need for defining a domain model and node types for effective content management. Key challenges mentioned include managing node type changes and developing a system for storing content efficiently.
Oracle RAC and Docker: The Why and How
Oracle RAC and Docker: The Why and HowSeth Miller The document discusses Oracle RAC and Docker, including why Oracle would be used in containers, considerations for using Oracle RAC in containers, how containers and virtual networks work, preparing storage, images, and networking for Oracle RAC containers, and how to configure Oracle Grid Infrastructure in Docker containers. Key points include reducing resources and time through containers, challenges of shared-nothing architecture and privileged access in containers, and steps to configure storage, virtual networking, and Oracle software in images before deploying Oracle RAC containers.
Hadoop HDFS by rohitkapa
Hadoop HDFS by rohitkapakapa rohit HDFS is a distributed file system designed for storing very large data sets reliably and efficiently across commodity hardware. It has three main components - the NameNode, Secondary NameNode, and DataNodes. The NameNode manages the file system namespace and regulates access to files. DataNodes store and retrieve blocks when requested by clients. HDFS provides reliable storage through replication of blocks across DataNodes and detects hardware failures to ensure data is not lost. It is highly scalable, fault-tolerant, and suitable for applications processing large datasets.
Apachecon Euro 2012: Elastic, Multi-tenant Hadoop on Demand
Apachecon Euro 2012: Elastic, Multi-tenant Hadoop on DemandRichard McDougall Elastic, Multi-tenant Hadoop on Demand! Richard McDougall, Chief Architect, Application Infrastructure and Big Data, VMware, Inc @richardmcdougll ApacheCon Europe, 2012. Broadens the application of Hadoop technology with horizontal and vertical use cases. Hadoop enables parallel processing through a programming framework for highly parallel data processing using MapReduce and the Hadoop Distributed File System (HDFS) for distributed data storage. Serengeti automates deployment of Hadoop on virtual platforms in under 30 minutes for multi-tenant elastic Hadoop as a service.
Hadoop Summit 2012 | HBase Consistency and Performance Improvements
Hadoop Summit 2012 | HBase Consistency and Performance ImprovementsCloudera, Inc. The document discusses improvements in Apache HBase's consistency and performance, detailing its architecture, data format, write path, and the implementation of ACID transactions. It highlights issues related to atomic bulk uploads and read consistency, while proposing workarounds and encouraging upgrades to newer versions for bug fixes and performance enhancements. Key areas for performance improvement identified include the read path, compactions, and HDFS integration.
Architecting the Future of Big Data & Search - Eric Baldeschwieler
Architecting the Future of Big Data & Search - Eric Baldeschwielerlucenerevolution The document discusses architecting big data and search technologies, focusing on Apache Hadoop and its integration with Lucene. It covers Hadoop's infrastructure, MapReduce programming model, case studies highlighting its application at Yahoo, and the future potential of Hadoop in enterprise data management. Key points include the scalability, reliability, and evolving ecosystem around Hadoop, alongside various challenges and developmental goals for enhancing its usability.
Session 49 - Semantic metadata management practical
Session 49 - Semantic metadata management practical ISSGC Summer School This document describes a tutorial on using semantic metadata with Grid services. The tutorial will cover:
1. Setting up a Globus container and deploying various semantic services and an operation provider for sticky notes.
2. Attaching RDF semantic bindings to sticky notes to represent their metadata.
3. Querying the semantic bindings of sticky notes using SPARQL or other query languages. The queries can exploit relationships defined in an ontology.
The hands-on exercises will guide participants in building a semantically-aware Grid service by completing the various setup and configuration steps. Participants will learn how to attach, query, and infer over semantic metadata for Grid resources.
Session9part2 Servers Detailed
Session9part2 Servers DetailedISSGC Summer School This document summarizes a presentation on the UNICORE Server Components. It discusses how UNICORE provides a web services framework for job submission and management across different computing resources. Key points include:
- UNICORE uses a gateway, service containers, and atomic services to expose target systems through standardized web service interfaces.
- Atomic services include job management, storage management, and file transfer services that provide abstract access to computing jobs and files on remote systems.
- Security is handled through XUUDB authentication, XACML authorization policies, and message signing. Configurable security handlers provide flexibility.
Yahoo Communities Architecture Unlikely Bedfellows
Yahoo Communities Architecture Unlikely BedfellowsConSanFrancisco123 This document provides an overview of Yahoo!'s architecture, infrastructure, and processes. It describes Yahoo!'s shared hardware and software platforms that properties use, along with operational tools like monitoring and configuration management. Properties have accountability for decisions while benefiting from shared resources. Standards and oversight help maintain quality and cost-effectiveness across the platform.
CRX 2 Content Application Platform
CRX 2 Content Application PlatformCédric Hüsler CRX 2 is a content application platform that leverages open standards and APIs like JCR 2.0 and CMIS for managing structured and unstructured content. It integrates with various systems including Microsoft SharePoint and provides features such as simplified versioning, lifecycle management, and improved access controls. Developers are encouraged to install CRX and participate in the application's challenge for content app development.
More Related Content
What's hot (20)
Storage solutions for High Performance Computing
Storage solutions for High Performance Computinggmateesc This document discusses storage infrastructure for high-performance computing. It begins by introducing data-intensive science and the need for parallel storage systems. It then discusses several parallel file systems used in HPC like GPFS, Lustre, and PanFS. Key concepts covered include data striping, scale-out NAS, parallel file systems, and IO acceleration techniques. The document also discusses challenges of data growth, bottlenecks in scaling storage, and architectures of various parallel file systems.
Introduction to h base
Introduction to h baseTrendProgContest13 HBase is a distributed, scalable, big data store that provides fast lookup capabilities like Google BigTable. It uses a table-like data structure with rows indexed by a key and stores data in columns grouped by families. HBase is designed to operate on top of Hadoop HDFS for scalability and high availability. It allows for fast lookups, full table scans, and range scans across large datasets distributed across clusters of commodity servers.
Considerations for building your private cloud folsom update 041513
Considerations for building your private cloud folsom update 041513OpenStack Foundation The document discusses considerations for building a private cloud using OpenStack Folsom. It covers topics like defining a private cloud, sizing flavors and capacity planning, networking with nova-network, image management with Glance, storage options, and example architectures. The presentation aims to help architects build private clouds that can easily scale and have good performance.
Cloud computing era
Cloud computing eraTrendProgContest13 Cloud computing, big data, and mobile technologies are driving major changes in the IT world. Cloud computing provides scalable computing resources over the internet. Big data involves extremely large data sets that are analyzed to reveal business insights. Hadoop is an open-source software framework that allows distributed processing of big data across commodity hardware. It includes tools like HDFS for storage and MapReduce for distributed computing. The Hadoop ecosystem also includes additional tools for tasks like data integration, analytics, workflow management, and more. These emerging technologies are changing how businesses use and analyze data.
Hadoop HDFS Architeture and Design
Hadoop HDFS Architeture and Designsudhakara st HDFS is a distributed file system designed for storing very large data files across commodity servers or clusters. It works on a master-slave architecture with one namenode (master) and multiple datanodes (slaves). The namenode manages the file system metadata and regulates client access, while datanodes store and retrieve block data from their local file systems. Files are divided into large blocks which are replicated across datanodes for fault tolerance. The namenode monitors datanodes and replicates blocks if their replication drops below a threshold.
Cluster Computing with Dryad
Cluster Computing with Dryadbutest This document summarizes a presentation given by Mihai Budiu on DryadLINQ and cloud computing. The presentation introduced DryadLINQ as an integration of LINQ queries with Dryad, Microsoft's data-parallel execution engine. This allows expressing data-parallel algorithms like joins, aggregations, and machine learning in a high-level language like C#, and having them automatically parallelized and run on a cluster. The document provides examples of expressing algorithms like histograms, word counting, and machine learning using DryadLINQ.
ROMA User-Customizable NoSQL Database in Ruby
ROMA User-Customizable NoSQL Database in RubyRakuten Group, Inc. Rakuten, Inc. developed ROMA, a user-customizable NoSQL database in Ruby. ROMA provides high scalability, availability, and fault tolerance using techniques like consistent hashing and virtual nodes. It has a plug-in architecture that allows users to extend ROMA's functionality through custom commands. The plug-ins use a domain specific language for easily defining commands to perform operations on structured data stored in ROMA, such as lists and maps.
Coordinating Metadata Replication: Survival Strategy for Distributed Systems
Coordinating Metadata Replication: Survival Strategy for Distributed SystemsKonstantin V. Shvachko The document discusses coordinating metadata replication survival strategies for distributed systems, emphasizing the importance of a coordination engine for achieving consensus among nodes. It details the architecture of HDFS and HBase, explaining how various roles, such as proposer, learner, and acceptor, contribute to maintaining reliability and consistency in data operations. Additionally, it addresses challenges related to distributed systems, including failure scenarios and block management across geographically distributed setups.
Hadoop at Rakuten, 2011/07/06
Hadoop at Rakuten, 2011/07/06Rakuten Group, Inc. Rakuten Inc. uses Hadoop for various purposes including generating recommendation indexes, analyzing logs, and calculating metrics. Their current Hadoop system includes a cluster with 3 masters and 69 slaves, Ganglia monitoring, and HA with DRBD and Heartbeat. It provides benefits over their previous system like lower costs, improved scalability, and faster transaction times. However, they still face challenges around using up HDFS space and fully realizing their data warehouse goals with the new system.
Storage Infrastructure Behind Facebook Messages
Storage Infrastructure Behind Facebook Messagesyarapavan Facebook uses HBase running on HDFS to store messaging data and metadata. Key reasons for choosing HBase include high write throughput, horizontal scalability, and integration with HDFS. Typical clusters have multiple regions and racks for redundancy. Facebook stores small messages, metadata, and attachments in HBase, while larger messages and attachments are stored separately. The system processes billions of read and write operations daily and continues to optimize performance and reliability.
Building A Scalable Open Source Storage Solution
Building A Scalable Open Source Storage SolutionPhil Cryer This document discusses the development of a scalable, open-source storage and processing solution for biodiversity data, highlighting the challenges of current data access, copyright concerns, and redundancy risks. It outlines the plans for a new data center named the 'Darwin Repository' with significant funding and technology collaborations aimed at enhancing data management capabilities. The proposal emphasizes the use of distributed storage and processing techniques to ensure efficiency, fault tolerance, and sustainability in handling the growing volume of biodiversity data.
HDFS Trunncate: Evolving Beyond Write-Once Semantics
HDFS Trunncate: Evolving Beyond Write-Once SemanticsDataWorks Summit The document describes HDFS's implementation of file truncation, which allows reducing a file's length. It evolved HDFS's write-once semantics to support data mutation. Truncate uses the lease and block recovery framework to truncate block replicas in-place, except when snapshots exist, where it uses "copy-on-truncate" to preserve the snapshot. The truncate operation returns immediately after updating metadata, while block adjustments occur in the background.
A Content Repository for TYPO3 5.0
A Content Repository for TYPO3 5.0Karsten Dambekalns The document discusses the development of a PHP-based content repository for TYPO3 5.0, highlighting the advantages of using an object-based storage system over traditional databases. It outlines the current status of the project, including the implementation of the JSR-170 API in PHP, and emphasizes the need for defining a domain model and node types for effective content management. Key challenges mentioned include managing node type changes and developing a system for storing content efficiently.
Oracle RAC and Docker: The Why and How
Oracle RAC and Docker: The Why and HowSeth Miller The document discusses Oracle RAC and Docker, including why Oracle would be used in containers, considerations for using Oracle RAC in containers, how containers and virtual networks work, preparing storage, images, and networking for Oracle RAC containers, and how to configure Oracle Grid Infrastructure in Docker containers. Key points include reducing resources and time through containers, challenges of shared-nothing architecture and privileged access in containers, and steps to configure storage, virtual networking, and Oracle software in images before deploying Oracle RAC containers.
Hadoop HDFS by rohitkapa
Hadoop HDFS by rohitkapakapa rohit HDFS is a distributed file system designed for storing very large data sets reliably and efficiently across commodity hardware. It has three main components - the NameNode, Secondary NameNode, and DataNodes. The NameNode manages the file system namespace and regulates access to files. DataNodes store and retrieve blocks when requested by clients. HDFS provides reliable storage through replication of blocks across DataNodes and detects hardware failures to ensure data is not lost. It is highly scalable, fault-tolerant, and suitable for applications processing large datasets.
Apachecon Euro 2012: Elastic, Multi-tenant Hadoop on Demand
Apachecon Euro 2012: Elastic, Multi-tenant Hadoop on DemandRichard McDougall Elastic, Multi-tenant Hadoop on Demand! Richard McDougall, Chief Architect, Application Infrastructure and Big Data, VMware, Inc @richardmcdougll ApacheCon Europe, 2012. Broadens the application of Hadoop technology with horizontal and vertical use cases. Hadoop enables parallel processing through a programming framework for highly parallel data processing using MapReduce and the Hadoop Distributed File System (HDFS) for distributed data storage. Serengeti automates deployment of Hadoop on virtual platforms in under 30 minutes for multi-tenant elastic Hadoop as a service.
Hadoop Summit 2012 | HBase Consistency and Performance Improvements
Hadoop Summit 2012 | HBase Consistency and Performance ImprovementsCloudera, Inc. The document discusses improvements in Apache HBase's consistency and performance, detailing its architecture, data format, write path, and the implementation of ACID transactions. It highlights issues related to atomic bulk uploads and read consistency, while proposing workarounds and encouraging upgrades to newer versions for bug fixes and performance enhancements. Key areas for performance improvement identified include the read path, compactions, and HDFS integration.
Architecting the Future of Big Data & Search - Eric Baldeschwieler
Architecting the Future of Big Data & Search - Eric Baldeschwielerlucenerevolution The document discusses architecting big data and search technologies, focusing on Apache Hadoop and its integration with Lucene. It covers Hadoop's infrastructure, MapReduce programming model, case studies highlighting its application at Yahoo, and the future potential of Hadoop in enterprise data management. Key points include the scalability, reliability, and evolving ecosystem around Hadoop, alongside various challenges and developmental goals for enhancing its usability.
Session 49 - Semantic metadata management practical
Session 49 - Semantic metadata management practical ISSGC Summer School This document describes a tutorial on using semantic metadata with Grid services. The tutorial will cover:
1. Setting up a Globus container and deploying various semantic services and an operation provider for sticky notes.
2. Attaching RDF semantic bindings to sticky notes to represent their metadata.
3. Querying the semantic bindings of sticky notes using SPARQL or other query languages. The queries can exploit relationships defined in an ontology.
The hands-on exercises will guide participants in building a semantically-aware Grid service by completing the various setup and configuration steps. Participants will learn how to attach, query, and infer over semantic metadata for Grid resources.
Session9part2 Servers Detailed
Session9part2 Servers DetailedISSGC Summer School This document summarizes a presentation on the UNICORE Server Components. It discusses how UNICORE provides a web services framework for job submission and management across different computing resources. Key points include:
- UNICORE uses a gateway, service containers, and atomic services to expose target systems through standardized web service interfaces.
- Atomic services include job management, storage management, and file transfer services that provide abstract access to computing jobs and files on remote systems.
- Security is handled through XUUDB authentication, XACML authorization policies, and message signing. Configurable security handlers provide flexibility.
Similar to wikimedia-architecture (20)
Yahoo Communities Architecture Unlikely Bedfellows
Yahoo Communities Architecture Unlikely BedfellowsConSanFrancisco123 This document provides an overview of Yahoo!'s architecture, infrastructure, and processes. It describes Yahoo!'s shared hardware and software platforms that properties use, along with operational tools like monitoring and configuration management. Properties have accountability for decisions while benefiting from shared resources. Standards and oversight help maintain quality and cost-effectiveness across the platform.
CRX 2 Content Application Platform
CRX 2 Content Application PlatformCédric Hüsler CRX 2 is a content application platform that leverages open standards and APIs like JCR 2.0 and CMIS for managing structured and unstructured content. It integrates with various systems including Microsoft SharePoint and provides features such as simplified versioning, lifecycle management, and improved access controls. Developers are encouraged to install CRX and participate in the application's challenge for content app development.
20080528dublinpt1
20080528dublinpt1Jeff Hammerbacher Jeff Hammerbacher manages Facebook's data team of 22 data engineers and scientists. He discusses Facebook's data infrastructure including software used like Linux, Apache, MySQL, PHP, and Memcached. Facebook serves over 10 million requests per second from over 10,000 web servers and stores session data in cookies. It uses the distributed computing framework Thrift to build most internal services. Facebook is moving from its in-house distributed log file processing system Cheetah to the open source Hadoop framework for more flexible offline batch processing and ad hoc querying of its growing trove of user data.
MySQL Cluster Scaling to a Billion Queries
MySQL Cluster Scaling to a Billion QueriesBernd Ocklin MySQL Cluster is a distributed database that provides extreme scalability, high availability, and real-time performance. It uses an auto-sharding and auto-replicating architecture to distribute data across multiple low-cost servers. Key benefits include scaling reads and writes, 99.999% availability through its shared-nothing design with no single point of failure, and real-time responsiveness. It supports both SQL and NoSQL interfaces to enable complex queries as well as high-performance key-value access.
20080611accel
20080611accelJeff Hammerbacher This document provides an overview of data management at Facebook including details about:
- The data team led by Jeff Hammerbacher which includes 25 data engineers and scientists.
- The infrastructure used to serve Facebook.com including web, memcached, and MySQL tiers.
- The Thrift service framework used to build services and APIs.
- How Hadoop is used for offline batch processing and distributed processing including applications like sentiment analysis and machine learning.
- Other data tools used like Hive for structured data management and Cassandra for structured storage over a peer-to-peer network.
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear Scalability
Beyond The Data Grid: Coherence, Normalisation, Joins and Linear ScalabilityBen Stopford The document introduces the ODC, a highly distributed, normalized in-memory data store designed to enhance scalability and performance by adapting the snowflake schema for efficient data access. It critiques traditional RDBMS architectures as legacy technologies, advocating for a new approach that emphasizes normalizing data to avoid the overhead of denormalization and connect only necessary dimensions through a 'connected replication' pattern. The ODC aims to resolve the distributed join problem, enabling fast queries without the typical complexities associated with traditional database structures.
A closer look to locaweb IaaS
A closer look to locaweb IaaSGleicon Moraes This document provides an overview of Locaweb's Infrastructure as a Service (IaaS) engineering team and capabilities. It discusses the team size and responsibilities, defines IaaS, describes Locaweb's IaaS architecture including virtual/physical servers, open source software used, provisioning processes, and resource monitoring. It also lists several open source projects developed by Locaweb to support their IaaS operations.
Living with SQL and NoSQL at craigslist, a Pragmatic Approach
Living with SQL and NoSQL at craigslist, a Pragmatic ApproachJeremy Zawodny The document discusses data storage strategies at Craigslist, highlighting their use of various technologies such as MySQL, Memcached, Redis, MongoDB, and Sphinx. It emphasizes the importance of selecting the right tools based on durability, performance, and support needs, along with the challenges of managing a combination of SQL and NoSQL databases. The document also outlines Craigslist's architecture, data flow, and the team's focus on reliability and efficiency in their self-hosted infrastructure.
RunningQuantumOnQuantumAtNicira.pdf
RunningQuantumOnQuantumAtNicira.pdfOpenStack Foundation This document discusses running OpenStack private clouds with Quantum networking and Nicira NVP. It summarizes the benefits of deploying complex enterprise applications on OpenStack clouds in under a minute compared to months previously. It also discusses how OpenStack Quantum and NVP can provide developers sandbox networks in seconds rather than days and enable cloud bursting for continuous integration testing in minutes rather than weeks. The architecture shows OpenStack Quantum integrated with the Nicira NVP plugin to dynamically provision virtual networks.
State of the Container Ecosystem
State of the Container EcosystemVinay Rao This document discusses containerization and the Docker ecosystem. It begins by describing the challenges of managing different software stacks across multiple environments. It then introduces Docker as a solution that packages applications into standardized units called containers that are portable and can run anywhere. The rest of the document covers key aspects of the Docker ecosystem like orchestration tools like Kubernetes and Docker Swarm, networking solutions like Flannel and Weave, storage solutions, and security considerations. It aims to provide an overview of the container landscape and components.
CloudStack Architecture Future
CloudStack Architecture FutureKimihiko Kitase The document summarizes CloudStack architecture plans for the future. It discusses moving to management server clusters per availability zone rather than per region. It also discusses using an object storage system for templates and snapshots rather than a separate NFS server. Finally, it discusses a possible future model where CloudStack manages existing virtualization clusters rather than deploying and managing its own system VMs.
Web Scale Applications using NeflixOSS Cloud Platform
Web Scale Applications using NeflixOSS Cloud PlatformSudhir Tonse The document discusses Netflix's cloud architecture and its strategy for building resilient web-scale applications using IaaS and PaaS. It outlines key principles for cloud architecture, including the emphasis on stateless microservices and fault tolerance, along with various open-source tools provided by Netflix to help developers create their own PaaS. Additionally, it highlights Netflix's scale, operational insights, and the significance of monitoring and scaling in cloud deployments.
Ram chinta hug-20120922-v1
Ram chinta hug-20120922-v1Ram Chinta - CloudStack is an open source cloud computing platform that was donated to the Apache Software Foundation in 2012. It provides infrastructure as a service and supports various hypervisors and physical hardware.
- CloudStack has a scalable architecture designed to support thousands of hosts and VMs across multiple availability zones. It provides rich networking and storage capabilities.
- CloudStack can support both traditional server virtualization workloads as well as "Amazon-style" workloads with software defined networks and object storage.
- The CloudStack community is growing rapidly and encourages participation through mailing lists, IRC, forums and meetup groups.
Learn OpenStack from trystack.cn ——Folsom in practice
Learn OpenStack from trystack.cn ——Folsom in practiceOpenCity Community The document discusses the Trystack community and the OpenStack platform, highlighting its open-source nature, community involvement, and the architecture for networking capabilities. It outlines the challenges and solutions in managing network connectivity, tenant isolation, and the transition to software switches. Additionally, it provides information on the OpenStack Foundation and upcoming events and releases planned for 2013.
Getting started with MariaDB with Docker
Getting started with MariaDB with DockerMariaDB plc This document discusses using MariaDB and Docker together from development to production. It begins by outlining the benefits of containers and Docker for database deployments. Requirements for databases in containers like data redundancy, self-discovery, self-healing and application tier discovery are discussed. An overview of MariaDB and how it meets these requirements with Galera cluster and MaxScale is provided. The document then demonstrates how to develop and deploy a Python/Flask app with MariaDB from development to a Docker Swarm production cluster behind HAProxy, including scaling the web tier and implementing a hardened database tier with Galera cluster and MaxScale behind secrets. Considerations around storage, networking and upgrades are discussed.
20081022cca
20081022ccaJeff Hammerbacher 1. Jeff Hammerbacher managed Facebook's data team from 2006 to 2008 and helped build systems to handle the massive growth of user data and services at Facebook.
2. The data team developed distributed data platforms like Hadoop and Cassandra to process terabytes of user data across hundreds of servers and make it available for services and insights.
3. Key challenges included simplifying complex systems, achieving scalability to handle large data volumes, improving manageability with less human intervention, and using open source technologies to build developer communities.
Cassandra at mahalo_com_scale_la_meetup_de
Cassandra at mahalo_com_scale_la_meetup_demahalomeetup This document provides information about Cassandra usage at Mahalo.com. It describes Mahalo's hardware configuration including an 8 node Cassandra cluster. It outlines two current use cases for Cassandra - an activity log requiring fast writes and curated search result pages previously in a large MySQL table. It also shares some lessons learned including the importance of data modeling, monitoring, and dev onboarding for Cassandra.
A Tale of 2 Systems
A Tale of 2 SystemsDavid Newman The document discusses the infrastructure and technologies behind Wordpress.com, highlighting its extensive use of databases, caching, and content delivery systems to ensure scalability and performance. It emphasizes the importance of tools like HyperDB for database management, advanced caching mechanisms, and a focus on security through strategies like DDoS protection and SSL implementation. The final section suggests best practices such as sharding databases, caching at the edge, and employing AI for resource planning.
MayaData Datastax webinar - Operating Cassandra on Kubernetes with the help ...
MayaData Datastax webinar - Operating Cassandra on Kubernetes with the help ...MayaData Inc The document discusses using Cassandra on Kubernetes with the support of OpenEBS and LitmusChaos, highlighting Mayadata's role in advancing container attached storage for stateful workloads. It outlines best practices for Kubernetes storage, the functionalities of the Cassandra operator, and the importance of chaos engineering in maintaining application reliability. Key topics include dynamic provisioning, data management, and the evolution of architecture to leverage new hardware technologies like NVMe.
Ad
More from Kapil Mohan (17)
97 Things Every SRE Should Know
97 Things Every SRE Should KnowKapil Mohan This document is a table of contents for the book "97 Things Every SRE Should Know" by Emil Stolarsky and Jaime Woo. It lists 55 chapters organized into three parts: New to SRE, Zero to One, and One to Ten. Each chapter is titled with an SRE-related topic and lists the author's name. The table of contents provides a high-level overview of the breadth of SRE concepts and practices covered in the book at a glance.
Why Google Stores Billions of Lines of Code in a Single Repository
Why Google Stores Billions of Lines of Code in a Single RepositoryKapil Mohan Google uses a single monolithic code repository called Piper to store its codebase, which includes over 1 billion files, 2 billion lines of code, and a history of 35 million commits from over 25,000 developers. Piper is a custom-built version control system that provides benefits like unified versioning, extensive code sharing, and simplified dependency management. It has shown that the monolithic model can successfully scale to an extremely large repository through tools like its distributed architecture, caching infrastructure, and cloud-based developer workflows supported by systems like CitC. Google employs trunk-based development where most work is done on the mainline trunk, with limited use of branches, to further maximize the advantages of the single codebase approach.
Stories on Alignment
Stories on AlignmentKapil Mohan The document outlines the journey of developing a more effective alignment process using OKRs (Objectives and Key Results) at a company, particularly highlighting the struggles and learnings experienced by a character named Ilse. It emphasizes the importance of aligning team objectives with company strategy, maintaining clear communication, and adapting agile practices to enhance performance. The narrative also reflects on the need for continuous improvement and adaptability in leadership and goal-setting processes.
Definitive guide to building a remote team
Definitive guide to building a remote teamKapil Mohan This document provides an overview of managing remote teams. It discusses establishing a clear purpose or "why" for the business that employees can rally behind. This helps ensure employees are motivated and make decisions aligned with the business goals. It also acknowledges the challenges of managing remote teams and provides examples of tools that can help with communication, time tracking, and project management. The document emphasizes developing a "no excuses" approach to management by clearly defining expectations and deadlines upfront so there is no room for excuses from employees.
What ya smokin'?
What ya smokin'?Kapil Mohan The document lists and provides alternative names for several commonly smoked drugs, including tobacco, cannabis, charas, bhang, opium, heroin, crack, and meth. It asks if the reader will smoke tobacco, cannabis, charas, opium, heroin, crack, or meth, or if they want to ask something else.
SlideShare's Lean Startup Journey: Lessons Learnt
SlideShare's Lean Startup Journey: Lessons LearntKapil Mohan This document outlines SlideShare's transition to a lean startup approach. It discusses how SlideShare adapted from focusing on being a platform for sharing PowerPoint presentations in 2006 to becoming a social media and content sharing platform by 2010-2012. It emphasizes the importance of being able to adapt quickly to changes, developing minimum viable products, embracing technical debt, automating processes, and monitoring metrics. The document stresses that becoming a lean startup requires shifting company culture and processes, and that transformations to lean methodologies can be difficult.
Devops at SlideShare: Talk at Devopsdays Bangalore 2011
Devops at SlideShare: Talk at Devopsdays Bangalore 2011Kapil Mohan The document discusses the challenges faced by SlideShare in deploying to production, including tedious manual deployment processes, lack of automation and visibility. This led to long downtimes, wasted time troubleshooting issues and arguments between teams. To address these problems, SlideShare implemented DevOps practices like automated deployments, continuous integration, monitoring and alerting tools. This improved deployment frequency from zero to 5 times a day, increased visibility, shared ownership and helped scale their engineering processes.
Awakening India - Jago Party
Awakening India - Jago PartyKapil Mohan The manifesto outlines key points for a political agenda including job creation through education, stringent actions against crime, and financial support for voters. It emphasizes community engagement to promote the party and increase awareness of rights and governmental responsibilities. The document also proposes various initiatives for fundraising and expanding the party's reach through media and local events.
Actionscript Standards
Actionscript StandardsKapil Mohan This document provides guidelines for coding ActionScript applications, including:
1) Code should be organized, commented, and consistently named according to conventions.
2) Variables should be scoped relative to their path rather than using absolute addressing like _root.
3) Code should be modularized using #include files and external ActionScript files.
4) Timelines should use layers and keyframes to represent application states, with minimal code directly on layers.
Tashan - the movie in pics
Tashan - the movie in pics Kapil Mohan Tashan (English: Style) is a forthcoming Indian Bollywood film starring Akshay Kumar, Saif Ali Khan, Kareena Kapoor and Anil Kapoor in the lead roles. Produced by Aditya Chopra and Yash Chopra under Yash Raj Films, the film marks the directorial debut of Vijay Krishna Acharya who wrote the screenplay and dialogues for hits like Dhoom (2004) and Dhoom 2 (2006). Tashan has completed filming and is scheduled to release on April 25, 2008.
Amazon S3 storage engine plugin for MySQL
Amazon S3 storage engine plugin for MySQLKapil Mohan This document describes a storage engine for Amazon S3 that allows storing large amounts of data in S3 without needing to purchase disks. It works by translating SQL queries to S3 operations and storing S3 items as rows in MySQL tables with the S3 item key as the primary key and item contents as the column value. This allows using inexpensive S3 storage through a traditional SQL interface.
Larry's Free Culture
Larry's Free CultureKapil Mohan The document discusses the history of copyright law and how it has evolved over time from initially limiting copyright terms to just 14 years, to increasingly longer terms up to life plus 70 years now. It argues that society has become less free as the past seeks to exert more control over creativity that builds upon it through increasingly restrictive copyright laws and technologies. The solution proposed is to embrace free and open source ideas to promote transparent and collaborative creativity that can help ensure future innovation is not limited by the past.
Wrestle Mania 23
Wrestle Mania 23Kapil Mohan The document discusses the benefits of exercise for mental health. Regular physical activity can help reduce anxiety and depression and improve mood and cognitive functioning. Exercise causes chemical changes in the brain that may help protect against mental illness and improve symptoms.
Honeymoon Travels Movie Preview
Honeymoon Travels Movie PreviewKapil Mohan Honeymoon is considered to be an exciting times for all couples who have just stepped into marital hood. While it is a paradise for most, it is also a time when one gets to know each other better...or for worse!
Made of several sweet and a little bitter moments, debutant director Reema Kagti's 'Honeymoon Travels Pvt. Ltd' is a quirky comedy that tells the tale of 6 newly wed couples who are out on a bus journey from Mumbai to Goa, which is easily going to one ride of their lifetime.
Coming together for a 4 day package honeymoon tour, these couples interact with each other, have fun, share some great moments and also explore life and its beauty. While they are similar in some ways, they are different too in a lot of ways. And this is not just between the fellow couples but also within spouses!
Wanna meet these couples? Here you go!
Cast: Abhay Deol, Minissha Lamba, Shabana Azmi, Boman Irani, Kay Kay Menon, Raima Sen, Amisha Patel, Karan Khanna, Sandhya Mridul, Vikram Chatwal, Ranvir Shorey, Dia Mirza, Arjun Rampal, Suzanne Bernert, Naseruddin Shah
Direction: Reema Kagti
Production: Farhan Akhtar, Ritesh Sidhwani
Music: Vishal Dadlani, Shekhar Ravjiani
Parzania Movie Preview
Parzania Movie PreviewKapil Mohan There is a world where we all live in. And then there is a world that children build for themselves. A world that is selfless, happy and doesn't comprise of any evils whatsoever.
This was the world of Parzan [Master Farzan Dastoor], a 10 year old kid, and he named it plain and simple - PARZANIA!
Cast: Naseruddin Shah, Sarika, Master Farzan Dastoor
Direction: Rahul Dholakia
Production: K.B.Sareen, Kamal Patel, Rahul Dholakia
Music: Zakir Hussain
Source: IndiaGlitz
Traffic Signal Movie Preview
Traffic Signal Movie PreviewKapil Mohan Have you ever been witness to the day to day activities that happen at a traffic signal and how lives of people around it are affected? You would have caught a glimpse of it while stopping your vehicle at a traffic signal or walking past it on the pavements/footpaths but the reality is only for those who live their life at these signals. Those innumerous vendors, beggars, eunuchs, lepers, street kids, drug addicts, prostitutes, vendors etc. for whom an entire world is restricted to these signals.
'Traffic Signal' tells that tale of about 60 odd characters who have their world centered on this place. Each of them have their own life and how they make it happen on the road is what the film is all about.
Banner: Perfect Picture Company
Cast: Neetu Chandra, Konkona Sen Sharma, Ranveer Shourey, Sudhir Mishra
Direction: Madhur Bhandarkar
Source: IndiaGlitz
Ad
Recently uploaded (20)
UserCon Belgium: Honey, VMware increased my bill
UserCon Belgium: Honey, VMware increased my billstijn40 VMware’s pricing changes have forced organizations to rethink their datacenter cost management strategies. While FinOps is commonly associated with cloud environments, the FinOps Foundation has recently expanded its framework to include Scopes—and Datacenter is now officially part of the equation. In this session, we’ll map the FinOps Framework to a VMware-based datacenter, focusing on cost visibility, optimization, and automation. You’ll learn how to track costs more effectively, rightsize workloads, optimize licensing, and drive efficiency—all without migrating to the cloud. We’ll also explore how to align IT teams, finance, and leadership around cost-aware decision-making for on-prem environments. If your VMware bill keeps increasing and you need a new approach to cost management, this session is for you!
Using the SQLExecutor for Data Quality Management: aka One man's love for the...
Using the SQLExecutor for Data Quality Management: aka One man's love for the...Safe Software The SQLExecutor is one of FME’s most powerful and flexible transformers. Pivvot maintains a robust internal metadata hierarchy used to support ingestion and curation of thousands of external data sources that must be managed for quality before entering our platform. By using the SQLExecutor, Pivvot can efficiently detect problems and perform analysis before data is extracted from our staging environment, removing the need for rollbacks or cycles waisted on a failed job. This presentation will walk through three distinct examples of how Pivvot uses the SQLExecutor to engage its metadata hierarchy and integrate with its Data Quality Management workflows efficiently and within the source postgres database. Spatial Validation –Validating spatial prerequisites before entering a production environment. Reference Data Validation - Dynamically validate domain-ed columns across any table and multiple columns per table. Practical De-duplication - Removing identical or near-identical well point locations from two distinct source datasets in the same table.
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdfcaoyixuan2019 A presentation at Internetware 2025.
Raman Bhaumik - Passionate Tech Enthusiast
Raman Bhaumik - Passionate Tech EnthusiastRaman Bhaumik A Junior Software Developer with a flair for innovation, Raman Bhaumik excels in delivering scalable web solutions. With three years of experience and a solid foundation in Java, Python, JavaScript, and SQL, she has streamlined task tracking by 20% and improved application stability.
10 Key Challenges for AI within the EU Data Protection Framework.pdf
10 Key Challenges for AI within the EU Data Protection Framework.pdfPriyanka Aash 10 Key Challenges for AI within the EU Data Protection Framework
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025pcprocore 👉𝗡𝗼𝘁𝗲:𝗖𝗼𝗽𝘆 𝗹𝗶𝗻𝗸 & 𝗽𝗮𝘀𝘁𝗲 𝗶𝗻𝘁𝗼 𝗚𝗼𝗼𝗴𝗹𝗲 𝗻𝗲𝘄 𝘁𝗮𝗯> https://pcprocore.com/ 👈◀
CapCut Pro Crack is a powerful tool that has taken the digital world by storm, offering users a fully unlocked experience that unleashes their creativity. With its user-friendly interface and advanced features, it’s no wonder why aspiring videographers are turning to this software for their projects.
AI Agents and FME: A How-to Guide on Generating Synthetic Metadata
AI Agents and FME: A How-to Guide on Generating Synthetic MetadataSafe Software In the world of AI agents, semantics is king. Good metadata is thus essential in an organization's AI readiness checklist. But how do we keep up with the massive influx of new data? In this talk we go over the tips and tricks in generating synthetic metadata for the consumption of human users and AI agents alike.
Quantum AI Discoveries: Fractal Patterns Consciousness and Cyclical Universes
Quantum AI Discoveries: Fractal Patterns Consciousness and Cyclical UniversesSaikat Basu Embark on a cosmic journey exploring the intersection of quantum
computing, consciousness, and ancient wisdom. Together we'll uncover the
recursive patterns that bind our reality.
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...Fwdays Let's talk about our history. How we started the project with a small vector database of less than 2 million records. Later, we received a request for +100 million records, then another +100... And so gradually we reached almost 1 billion. Standard tools were quickly running out of steam - we were running into performance, index size, and very limited resources. After a long series of trials and errors, we built our own low-cost cluster, which today stably processes thousands of queries to more than 1B vectors.
Security Tips for Enterprise Azure Solutions
Security Tips for Enterprise Azure SolutionsMichele Leroux Bustamante Delivering solutions to Azure may involve a variety of architecture patterns involving your applications, APIs data and associated Azure resources that comprise the solution. This session will use reference architectures to illustrate the security considerations to protect your Azure resources and data, how to achieve Zero Trust, and why it matters. Topics covered will include specific security recommendations for types Azure resources and related network security practices. The goal is to give you a breadth of understanding as to typical security requirements to meet compliance and security controls in an enterprise solution.
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdf
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdfPriyanka Aash GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now
"Scaling in space and time with Temporal", Andriy Lupa.pdf
"Scaling in space and time with Temporal", Andriy Lupa.pdfFwdays Design patterns like Event Sourcing and Event Streaming have long become standards for building real-time analytics systems. However, when the system load becomes nonlinear with fast and often unpredictable spikes, it's crucial to respond quickly in order not to lose real-time operating itself.
In this talk, I’ll share my experience implementing and using a tool like Temporal.io. We'll explore the evolution of our system for maintaining real-time report generation and discuss how we use Temporal both for short-lived pipelines and long-running background tasks.
2025_06_18 - OpenMetadata Community Meeting.pdf
2025_06_18 - OpenMetadata Community Meeting.pdfOpenMetadata The community meetup was held Wednesday June 18, 2025 @ 9:00 AM PST.
Catch the next OpenMetadata Community Meetup @ https://www.meetup.com/openmetadata-meetup-group/
In this month's OpenMetadata Community Meetup, "Enforcing Quality & SLAs with OpenMetadata Data Contracts," we covered data contracts, why they matter, and how to implement them in OpenMetadata to increase the quality of your data assets!
Agenda Highlights:
👋 Introducing Data Contracts: An agreement between data producers and consumers
📝 Data Contracts key components: Understanding a contract and its purpose
🧑🎨 Writing your first contract: How to create your own contracts in OpenMetadata
🦾 An OpenMetadata MCP Server update!
➕ And More!
"Database isolation: how we deal with hundreds of direct connections to the d...
"Database isolation: how we deal with hundreds of direct connections to the d...Fwdays What can go wrong if you allow each service to access the database directly? In a startup, this seems like a quick and easy solution, but as the system scales, problems appear that no one could have guessed.
In my talk, I'll share Solidgate's experience in transforming its architecture: from the chaos of direct connections to a service-based data access model. I will talk about the transition stages, bottlenecks, and how isolation affected infrastructure support. I will honestly show what worked and what didn't. In short, we will analyze the controversy of this talk.
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdf
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdfEarley Information Science As AI reshapes expectations in retail and B2B commerce, organizations are recognizing a critical reality: meaningful AI outcomes depend on well-structured, adaptable infrastructure. In this session, Seth Earley is joined by Phil Ryan - AI strategist, search technologist, and founder of Glass Leopard Technologies - for a candid conversation on what it truly means to engineer systems for scale, agility, and intelligence.
Phil draws on more than two decades of experience leading search and AI initiatives for enterprise organizations. Together, he and Seth explore the challenges businesses face when legacy architectures limit personalization, agility, and real-time decisioning - and what needs to change to support agentic technologies and next-best-action capabilities.
Key themes from the webinar include:
Composability as a prerequisite for AI - Why modular, loosely coupled systems are essential for adapting to rapid innovation and evolving business needs
Search and relevance as foundational to AI - How techniques honed-in enterprise search have laid the groundwork for more responsive and intelligent customer experiences
From MDM and CDP to agentic systems - How data platforms are evolving to support richer customer context and dynamic orchestration
Engineering for business alignment - Why successful AI programs require architectural decisions grounded in measurable outcomes
The conversation is practical and forward-looking, connecting deep technical understanding with real-world business needs. Whether you’re modernizing your commerce stack or exploring how AI can enhance product discovery, personalization, or customer journeys, this session provides a clear-eyed view of the capabilities, constraints, and priorities that matter most.
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC and Open Hackathons Monthly Highlights June 2025OpenACC The OpenACC organization focuses on enhancing parallel computing skills and advancing interoperability in scientific applications through hackathons and training. The upcoming 2025 Open Accelerated Computing Summit (OACS) aims to explore the convergence of AI and HPC in scientific computing and foster knowledge sharing. This year's OACS welcomes talk submissions from a variety of topics, from Using Standard Language Parallelism to Computer Vision Applications. The document also highlights several open hackathons, a call to apply for NVIDIA Academic Grant Program and resources for optimizing scientific applications using OpenACC directives.
From Manual to Auto Searching- FME in the Driver's Seat
From Manual to Auto Searching- FME in the Driver's SeatSafe Software Finding a specific car online can be a time-consuming task, especially when checking multiple dealer websites. A few years ago, I faced this exact problem while searching for a particular vehicle in New Zealand. The local classified platform, Trade Me (similar to eBay), wasn’t yielding any results, so I expanded my search to second-hand dealer sites—only to realise that periodically checking each one was going to be tedious. That’s when I noticed something interesting: many of these websites used the same platform to manage their inventories. Recognising this, I reverse-engineered the platform’s structure and built an FME workspace that automated the search process for me. By integrating API calls and setting up periodic checks, I received real-time email alerts when matching cars were listed. In this presentation, I’ll walk through how I used FME to save hours of manual searching by creating a custom car-finding automation system. While FME can’t buy a car for you—yet—it can certainly help you find the one you’re after!
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Coordinated Disclosure for ML - What's Different and What's the Same.pdfPriyanka Aash Coordinated Disclosure for ML - What's Different and What's the Same
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...Josef Weingand IBM LTO10
Oh, the Possibilities - Balancing Innovation and Risk with Generative AI.pdf
Oh, the Possibilities - Balancing Innovation and Risk with Generative AI.pdfPriyanka Aash Oh, the Possibilities - Balancing Innovation and Risk with Generative AI