With the size and time required for creating game content on the rise, the human scalability factor is an increasing issue in delivering larger more detailed worlds. A prime issue is single user editing tools lead to bottlenecks in interactions between users. One of the ways to alleviate these bottlenecks is with synchronized multiuser editing tools, which allow users to see each others work in real-time. One option to implement multiuser editing is a revision database. This talk covers the basics of revision databases as implemented at CCP: how they work, creating a programmer friendly API, branching, and backwards compatibility.
This document is part of OBIEE11g Certification course offered by Adiva Consulting Inc.
Start a great career as Oracle Business Intelligence Expert, Join Adiva Online BI Certification course. contact info@adivaconsulting.com
Apache Iceberg: An Architectural Look Under the CoversScyllaDB
Data Lakes have been built with a desire to democratize data - to allow more and more people, tools, and applications to make use of data. A key capability needed to achieve it is hiding the complexity of underlying data structures and physical data storage from users. The de-facto standard has been the Hive table format addresses some of these problems but falls short at data, user, and application scale. So what is the answer? Apache Iceberg.
Apache Iceberg table format is now in use and contributed to by many leading tech companies like Netflix, Apple, Airbnb, LinkedIn, Dremio, Expedia, and AWS.
Watch Alex Merced, Developer Advocate at Dremio, as he describes the open architecture and performance-oriented capabilities of Apache Iceberg.
You will learn:
• The issues that arise when using the Hive table format at scale, and why we need a new table format
• How a straightforward, elegant change in table format structure has enormous positive effects
• The underlying architecture of an Apache Iceberg table, how a query against an Iceberg table works, and how the table’s underlying structure changes as CRUD operations are done on it
• The resulting benefits of this architectural design
From Windows to Linux: Converting a Distributed Perforce Helix InfrastructurePerforce
There are many advantages to running Perforce Helix on Linux servers. See the process and pitfalls encountered when converting a distributed Perforce infrastructure from Windows to Linux.
Compuware product managers Irene Ford, Bill Mackey and Jonathan Manley discuss and demo some of File-AID’s new and notable enhancements, including:
- File-AID for MVS: the use of 64-Bit storage when working with larger datasets; multi-dataset Search/Update functionality; and new Compare functionality designed to allow users to consolidate their Compare needs onto File-AID for MVS.
- File-AID for IMS and DB2: IBM Health Checker for z/OS support; customer requested enhancements that have been implemented in File-AID for IMS and File-AID for DB2; and enhancements to product architecture that address performance and usage of z/OS Unix.
- Test Data Privacy and File-AID/EX: list variable support; new functions in rule logic; disguise of CLOB and XML columns; improved handling of DISTINCT data types; and more.
Easier, Faster, and More Powerful – Notes Document Properties Reimaginedpanagenda
Webinar Recording: https://www.panagenda.com/webinars/easier-faster-and-more-powerful-notes-document-properties-reimagined/
Have you ever felt frustrated by the small properties dialog in Notes? Had to create an agent or button to quickly change a field? Searched endlessly for the field you wanted to compare each time you selected a new document? Wished you could just make the damned thing bigger? Luckily, there is a solution – and you probably already have it installed! With the free panagenda Document Properties (Pro) you get the properties dialog you always needed. Big, resizable, full-text searchable. View multiple documents at once or compare them with a diff viewer. Modify any field, and finally have an easy way to handle profile documents for all users. Join HCL Lifetime Ambassador Julian Robichaux to discover how Document Properties can simplify your work and assist you daily when using Domino applications – in the client or the designer. You will never look back!
Key takeaways from this session
- What Document Properties is, which editions there are, and how you can find it in Notes and Domino Designer
- How you can search for and edit any field, compare documents, or CSV export all data
- How to find, edit, and even delete profile documents
- Which configuration settings are available to customize feature
SQL Server 2014 Memory Optimised Tables - AdvancedTony Rogerson
Hekaton is large piece of kit, this session will focus on the internals of how in-memory tables and native stored procedures work and interact – Database structure: use of File Stream, backup/restore considerations in HA and DR as well as Database Durability, in-memory table make up: hash and range indexes, row chains, Multi-Version Concurrency Control (MVCC). Design considerations and gottcha’s to watch out for.
The session will be demo led.
Note: the session will assume the basics of Hekaton are known, so it is recommended you attend the Basics session.
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...panagenda
Webinar Recording: https://www.panagenda.com/webinars/alles-neu-macht-der-mai-wir-durchleuchten-den-verbesserten-notes-eigenschaftendialog/
Haben Sie sich schon einmal über den zu kleinen Eigenschaftendialog in Notes geärgert? Mussten Sie einen Agenten oder eine Aktion erstellen, um schnell mal ein Feld zu ändern? Haben Sie jedes mal endlos nach dem zu vergleichenden Feld gesucht, nachdem Sie ein neues Dokument ausgewählt haben? Wollten Sie das verdammte Ding einfach nur größer machen? Zum Glück gibt es dafür eine Lösung – und sie ist wahrscheinlich bereits installiert! Mit dem kostenlosen panagenda Document Properties (Pro) erhalten Sie den Eigenschaftendialog, den Sie schon immer haben wollten. Größer, anpassbar, und im Volltext durchsuchbar. Sehen Sie mehrere Dokumente gleichzeitig oder vergleichen Sie mit einem Diff-Viewer. Ändern Sie beliebige Felder und haben Sie endlich eine einfache Möglichkeit, Profildokumente für alle Benutzer zu verwalten. Entdecken Sie mit HCL Ambassador Marc Thomas, wie Document Properties Ihre Arbeit vereinfachen und Sie bei der täglichen Verwendung
von Domino-Anwendungen unterstützen kann – im Client oder im Designer.
Sie werden es nicht bereuen!
Für Sie in diesem Webinar
- Was Document Properties ist, welche Editionen es gibt und wo es in Notes
und Domino Designer zu finden ist
- Wie Sie nach einem beliebigen Feld suchen und es bearbeiten,
Dokumente vergleichen oder alle Daten per CSV exportieren können
- Suchen, Bearbeiten und auch Löschen von Profildokumenten
- Welche Konfigurationseinstellungen verfügbar sind, um Funktionen anzupassen
- Wie Ihre Endbenutzer davon profitieren
- Sehen Sie alles in einer Live-Demo
Using probabilistic data structures in sessions to power personalization and customization in real-time. Examples in Redis and Node.js
Demo code at: https://github.com/stockholmux/qcon-redis-session-store-demo
Presented at QCon SF 2017.
Stream Analytics with SQL on Apache FlinkFabian Hueske
Apache Flink's DataStream API is very expressive and gives users precise control over time and state. However, many applications do not require this level of expressiveness and can be implemented more concisely and easily with a domain-specific API.
SQL is undoubtedly the most widely used language for data processing but usually applied in the domain of batch processing. Apache Flink features two relational APIs for unified stream and batch processing, the Table API, a language-integrated relational query API for Scala and Java, and SQL. A Table API or SQL query computes the same result regardless whether it is evaluated on a static file or on a Kafka topic. While Flink evaluates queries on batch input like a conventional query engine, queries on streaming input are continuously processed and their results constantly updated and refined.
In this talk we present Flink’s unified relational APIs, show how streaming SQL queries are processed, and discuss exciting new use-cases.
This document is part of OBIEE11g Certification course offered by Adiva Consulting Inc.
Start a great career as Oracle Business Intelligence Expert, Join Adiva Online BI Certification course. contact info@adivaconsulting.com
Apache Iceberg: An Architectural Look Under the CoversScyllaDB
Data Lakes have been built with a desire to democratize data - to allow more and more people, tools, and applications to make use of data. A key capability needed to achieve it is hiding the complexity of underlying data structures and physical data storage from users. The de-facto standard has been the Hive table format addresses some of these problems but falls short at data, user, and application scale. So what is the answer? Apache Iceberg.
Apache Iceberg table format is now in use and contributed to by many leading tech companies like Netflix, Apple, Airbnb, LinkedIn, Dremio, Expedia, and AWS.
Watch Alex Merced, Developer Advocate at Dremio, as he describes the open architecture and performance-oriented capabilities of Apache Iceberg.
You will learn:
• The issues that arise when using the Hive table format at scale, and why we need a new table format
• How a straightforward, elegant change in table format structure has enormous positive effects
• The underlying architecture of an Apache Iceberg table, how a query against an Iceberg table works, and how the table’s underlying structure changes as CRUD operations are done on it
• The resulting benefits of this architectural design
From Windows to Linux: Converting a Distributed Perforce Helix InfrastructurePerforce
There are many advantages to running Perforce Helix on Linux servers. See the process and pitfalls encountered when converting a distributed Perforce infrastructure from Windows to Linux.
Compuware product managers Irene Ford, Bill Mackey and Jonathan Manley discuss and demo some of File-AID’s new and notable enhancements, including:
- File-AID for MVS: the use of 64-Bit storage when working with larger datasets; multi-dataset Search/Update functionality; and new Compare functionality designed to allow users to consolidate their Compare needs onto File-AID for MVS.
- File-AID for IMS and DB2: IBM Health Checker for z/OS support; customer requested enhancements that have been implemented in File-AID for IMS and File-AID for DB2; and enhancements to product architecture that address performance and usage of z/OS Unix.
- Test Data Privacy and File-AID/EX: list variable support; new functions in rule logic; disguise of CLOB and XML columns; improved handling of DISTINCT data types; and more.
Easier, Faster, and More Powerful – Notes Document Properties Reimaginedpanagenda
Webinar Recording: https://www.panagenda.com/webinars/easier-faster-and-more-powerful-notes-document-properties-reimagined/
Have you ever felt frustrated by the small properties dialog in Notes? Had to create an agent or button to quickly change a field? Searched endlessly for the field you wanted to compare each time you selected a new document? Wished you could just make the damned thing bigger? Luckily, there is a solution – and you probably already have it installed! With the free panagenda Document Properties (Pro) you get the properties dialog you always needed. Big, resizable, full-text searchable. View multiple documents at once or compare them with a diff viewer. Modify any field, and finally have an easy way to handle profile documents for all users. Join HCL Lifetime Ambassador Julian Robichaux to discover how Document Properties can simplify your work and assist you daily when using Domino applications – in the client or the designer. You will never look back!
Key takeaways from this session
- What Document Properties is, which editions there are, and how you can find it in Notes and Domino Designer
- How you can search for and edit any field, compare documents, or CSV export all data
- How to find, edit, and even delete profile documents
- Which configuration settings are available to customize feature
SQL Server 2014 Memory Optimised Tables - AdvancedTony Rogerson
Hekaton is large piece of kit, this session will focus on the internals of how in-memory tables and native stored procedures work and interact – Database structure: use of File Stream, backup/restore considerations in HA and DR as well as Database Durability, in-memory table make up: hash and range indexes, row chains, Multi-Version Concurrency Control (MVCC). Design considerations and gottcha’s to watch out for.
The session will be demo led.
Note: the session will assume the basics of Hekaton are known, so it is recommended you attend the Basics session.
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...panagenda
Webinar Recording: https://www.panagenda.com/webinars/alles-neu-macht-der-mai-wir-durchleuchten-den-verbesserten-notes-eigenschaftendialog/
Haben Sie sich schon einmal über den zu kleinen Eigenschaftendialog in Notes geärgert? Mussten Sie einen Agenten oder eine Aktion erstellen, um schnell mal ein Feld zu ändern? Haben Sie jedes mal endlos nach dem zu vergleichenden Feld gesucht, nachdem Sie ein neues Dokument ausgewählt haben? Wollten Sie das verdammte Ding einfach nur größer machen? Zum Glück gibt es dafür eine Lösung – und sie ist wahrscheinlich bereits installiert! Mit dem kostenlosen panagenda Document Properties (Pro) erhalten Sie den Eigenschaftendialog, den Sie schon immer haben wollten. Größer, anpassbar, und im Volltext durchsuchbar. Sehen Sie mehrere Dokumente gleichzeitig oder vergleichen Sie mit einem Diff-Viewer. Ändern Sie beliebige Felder und haben Sie endlich eine einfache Möglichkeit, Profildokumente für alle Benutzer zu verwalten. Entdecken Sie mit HCL Ambassador Marc Thomas, wie Document Properties Ihre Arbeit vereinfachen und Sie bei der täglichen Verwendung
von Domino-Anwendungen unterstützen kann – im Client oder im Designer.
Sie werden es nicht bereuen!
Für Sie in diesem Webinar
- Was Document Properties ist, welche Editionen es gibt und wo es in Notes
und Domino Designer zu finden ist
- Wie Sie nach einem beliebigen Feld suchen und es bearbeiten,
Dokumente vergleichen oder alle Daten per CSV exportieren können
- Suchen, Bearbeiten und auch Löschen von Profildokumenten
- Welche Konfigurationseinstellungen verfügbar sind, um Funktionen anzupassen
- Wie Ihre Endbenutzer davon profitieren
- Sehen Sie alles in einer Live-Demo
Using probabilistic data structures in sessions to power personalization and customization in real-time. Examples in Redis and Node.js
Demo code at: https://github.com/stockholmux/qcon-redis-session-store-demo
Presented at QCon SF 2017.
Stream Analytics with SQL on Apache FlinkFabian Hueske
Apache Flink's DataStream API is very expressive and gives users precise control over time and state. However, many applications do not require this level of expressiveness and can be implemented more concisely and easily with a domain-specific API.
SQL is undoubtedly the most widely used language for data processing but usually applied in the domain of batch processing. Apache Flink features two relational APIs for unified stream and batch processing, the Table API, a language-integrated relational query API for Scala and Java, and SQL. A Table API or SQL query computes the same result regardless whether it is evaluated on a static file or on a Kafka topic. While Flink evaluates queries on batch input like a conventional query engine, queries on streaming input are continuously processed and their results constantly updated and refined.
In this talk we present Flink’s unified relational APIs, show how streaming SQL queries are processed, and discuss exciting new use-cases.
Similar to John Rittenhouse - Programming Addressing Human Scalability (20)
This BMW M5 is a high performance version of the BMW 5-Series executive car built by the Motorsport division of BMW. Beginning production in 1985, the first incarnation of the M5 was hand-built utilizing the 535i chassis and a modified BMW M1 engine, being the fastest production sedan in the world at the time of its introduction. Subsequent iterations of the M5 have been built from each generation of the 5-Series platform, including the E34, E39, E60/E61, and most recently, F10, with the first orders delivered in late 2011.
John Rittenhouse - Programming Addressing Human Scalability
1. Addressing Human Scalability Through
Multi-User Editing Using Revision Databases
John Rittenhouse
johnr@ccpgames.com
2. Overview
• Introduction
• The Problem
• The Solution
• Revisioned Database Overview
• System Implementation Overview
• Examples
• Performance Hotspots
• Issues
• Summary
3. Who is CCP?
• Independent MMO Company with 600 employees
• Three MMO Projects
– EVE – Sandbox space ship game with ~370k subscribers, 65k PCU
– Dust 514 – FPS MMO set in the EVE Universe on the PS3
– World of Darkness – MMO based on the White Wolf Property
• We need robust game editing solutions
4. Why address content scalability?
• Customers are seeking larger and more detailed gaming
worlds
• Content teams are becoming larger
– Generalist approach to content creation
• General level designers focusing on smaller and smaller areas
– Specialist approach to content creation
• Level designers, lighters, scripters, etc.
• Results in user clashing
– Locked out files
– Unable to see what others are doing
5. Possible Solutions
• Mergable File Formats
– YAML or other text based file formats
– Issue: Will often require a programmer to help merge
• Layered Levels
– Different parts of the levels
– Issue: Often different users will work on the same part and can’t see
the changes that other users have made but have not submitted or
synced to latest
• Realtime Multiuser Editing
– Changes by users are visible to other users in real time
– Issue: More complex to implement than other options
8. Problems to Tackle
• Synchronization of data amongst users
• How to lock the data
• Alerting systems of other user changes
9. CCP’s Problems to Tackle
• Minimal server reboots (Live Editing)
• Multiple users editing the same area
• Backwards compatible with existing database tables
• Support potentially up to 100 content developers
• Ease of use for programmers
– Transparent
– Efficient
• System tolerable of high latency
– Has to handle Trans-Atlantic latency
• Needs to be implemented in Stackless Python
10. Possible Implementations of MultiUser Editing
• Server Authoritative Editing
– Server would know immediately when changes occurred
– Relies on the server always being up for content developers
• Revisioned Database
– Allows multiple users to see the same set of changes
– Have to design an easy API to use
11. What are Revisioned Databases?
• Think of them as version control for databases
• Similar concepts as version control
– Submitting/Reverting
– Changelists
– Locking
• Rows are your atomic unit
12. Revisioned Database Tables
• Revision Table
– Tracks all revisions and in which table and key did they occur with
– Each of them has a changelist id they are in
• Changelist Table
– Tracks all the changelists and whether they have been submitted
• User Table
– Who can edit the data
• Recent Changes Table
– Tracks all recent changes for polling purposes
• Data Tables
– Table with all the changes
13. Data Table Example
• Lets say we are labeling fruit
• So our columns for this table will be fruitID and fruitName
• Need also revisionID and changeType
14. Data Table Example
revisionID fruitID fruitName changeType
0 0 Apple Add
Table View
revisionID fruitID fruitName changeType
0 0 Apple Add
15. Data Table Example
revisionID fruitID fruitName changeType
0 0 Apple Add
1 0 Fuji Apple Edit
Table View
revisionID fruitID fruitName changeType
1 0 Fuji Apple Edit
16. Data Table Example
revisionID fruitID fruitName changeType
0 0 Apple Add
1 0 Fuji Apple Edit
2 1 Grape Add
Table View
revisionID fruitID fruitName changeType
1 0 Fuji Apple Edit
2 1 Grape Add
17. Data Table Example
revisionID fruitID fruitName changeType
0 0 Apple Add
1 0 Fuji Apple Edit
2 1 Grape Add
3 0 Red Delicious Edit
Table View
revisionID fruitID fruitName changeType
3 0 Red Delicious Edit
2 1 Grape Add
18. Data Table Example
revisionID fruitID fruitName changeType
0 0 Apple Add
1 0 Fuji Apple Edit
2 1 Grape Add
3 0 Red Delicious Edit
4 2 Raspberry Add
Table View
revisionID fruitID fruitName changeType
3 0 Red Delicious Edit
2 1 Grape Add
4 2 Raspberry Add
19. Data Table Example
revisionID fruitID fruitName changeType
0 0 Apple Add
1 0 Fuji Apple Edit
2 1 Grape Add
3 0 Red Delicious Edit
4 2 Raspberry Add
5 1 Grape Delete
Table View
revisionID fruitID fruitName changeType
3 0 Red Delicious Edit
4 2 Raspberry Add
20. Locking
• Locks occur on a row level
– Only one user is allowed to change a row at a time
• If a row is not in a submitted change list then it is a locked
row.
• Database should reject any changes on locked rows by other
users
21. Syncing
• Copies data from one DB to another DB
• Filtered on
– Submitted/Unsubmitted Changelists
– Revision Number
– Branch
22. Branching
• Works on the same database instead of trying to merge two
databases together
• Change lists can be assigned to a particular branch
• Uses a promotion branch model
23. Brief Revisioned Database Summary
• Row changes
– Tracked and stored with revision number and change list
– Can be submitted or reverted through a change list
– Rows are locked to other users till their associated change list are
submitted or reverted
• Change Lists
• Syncing
• Branching
24. Handling Updates
• Remote Updates - Poll Recent Revisions Table
– Retrieves table and keys of rows since last check
– Tables informed to update those rows
• Scatter Update Events on Local/Remote Changes
– Alerts systems that are listening to the table, rows, and columns that
changed with previous and new values
– Originally just alerted about tables and rows but not the actual data
values that were changed (made it hard for systems to minimize
processing)
25. CCP’s Revisioned Database System
• Called Branched Static Data (BSD)
– Developed originally by Jörundur Matthíasson
• Beyond the Database we added layers to Python to ease
usage by other programmers
26. Layer Overview
DB Tables
Database
BSD Layer
BSD Table Service
Python BSDTable
BSDRow
27. Branched Static Data (BSD)
• Internal CCP Revisioned Database
DB Tables – Uses views to remain backwards compatible
– Made of SQL tables, views, and stored procedures
BSD Layer
• Per Row Operations
– Add/Edit/Delete
– Rows Locked to Single User
BSD Table • Submit/Revert
Service
• History of Changes
– Table with all Changes
BSDTable – View with most recent
• Branching/Syncing
BSDRow • Recent Revisions Table
28. BSD Table Service
• Holds references to each table
DB Tables
– Each table is a Python Class
– GetTable function
BSD Layer • Responsible for Table Updates
– Polls recent revisions table
BSD Table
– Alerts tables
Service – Handles update tasklets
BSDTable
BSDRow
29. BSDTable Class
• Loads the Table Data
DB Tables
– Loads row data into BSDRows
– Responsible for Indexing
BSD Layer • Holds references to the Rows
– GetRowByKey
BSD Table
– GetRows (Filtering)
Service • Handles Adding/Deleting of Rows
– AddRow
BSDTable – DeleteRow(s)
BSDRow
30. BSDRow Class
• Handles the data at the row level
DB Tables
• Columns accessed via Properties
– print row.columnName
BSD Layer – row.columnName = 2
• Responsible for data editing
BSD Table – Threading
Service – Merging edits (bucketing)
BSDTable
BSDRow
31. Example – Table Definition
Schema Table Name Label Key ID 1 Key ID 2
32. Example – Python Code
Output
Setting 'Ball' to the origin
Found 302 objects with the name 'chair_wood_windsorRes'
Added an object to worldSpaceID 8 with objectID 33 , but deleting it now
33. Performance Hotspot - Filtering
• Indexed on Columns
– Initially indexed into lists but swapped to sets
• Relational databases uses set theory to be fast so lets do the same
– Choosing columns to index on
• Used to use the DB to filter if our data isn’t fully loaded
– Resulted in a short term boost for a longer
• Future Changes
– Case insensitive & delete/nondeleted Indexing
– Caching/filtering using SQLite
34. Performance Hotspot - Transactions
• Operations often involve multiple rows
– Adds often depend on the keys of previous adds
– Occurs when there is a main table plus additional optionable tables
– DB Latency makes waiting on responses to slow
• Allows merging of all BSD operations in a tasklet into a single
DB query
• Provides traditional benefits of entire operations written at
once to DB
• Easy to Use
– TransactionStart()/TransactionEnd(transactionName)
– with bsd.Transaction(transactionName):
35. Common Pitfalls for Programmers
• Writing Cacheless Algorithms
– Relies on BSD Table Services for caches
– Makes creating live systems easier
– Often results in higher order algorithms
• Assuming Data is Loaded Transaction Based
– Data written as a transaction isn’t guaranteed to be loaded in a single
operation
– Possible Solutions
• Allow checking to see if a full Recent Revision Update is completed
• Actually Setup Loading to be transaction based
36. Issues – Promotion Branching
• We have moved from a promotion to mainline branch model
Promotion Branching Mainline Branching
Branch A Main
Branch B
Branch 1 Branch 2
Branch C
Branch D Branch 1.1 Branch 2.1 Branch 2.2
37. Issues – Promotion Branching
• Causes issues with non-backwards compatible changes
necessary in other branches
– Handled with scripts to move the static data between branches
• Interlinked data needs to be set to the same branch
– Hard to determine the issues without validation
– Weird Testing flag which makes the issue even worse
38. Replacing with a Mainline Branch Model
• Databases
– Main Database
• Responsible for all table ID’s
– All other DB’s derive from the Main DB
– Databases will correspond to the equivalent Perforce Branch
• Code and Databases will be integrated together
– Databases track the change list number they have been integrated
from and to
– Table merges occur automatically except for row conflicts
– Conflicts at the row resolved by choosing one side or the other
– Does not handle all potential issues
39. Summary
• Revisioned Databases work well for multiuser editing
– Locking is often handled automatically within them
– Easy to determine changes
• Ease of use is important for programmers
• For a large project assume the programmer won’t know what
you consider to be the “normal” use case
– Optimize based on use cases (programmers learn by example)
• DB branching model needs to match the code branching
40. CCP Is Hiring
Apply online at http://www.ccpgames.com/en/jobs.aspx
Jobs available in
Atlanta, USA
Reykjavik, Iceland
Shanghai, China