co:op-READ-Convention Marburg - Jesper Zedlitz

•

2 likes•885 views

ICARUS - International Centre for Archival Research

Jesper Zedlitz (German Society for Computer Genealogy, DE): Innovative Technologies for Making Crowdsourcing most Effective in Online Indexing co:op-READ-Convention Marburg Technology meets Scholarship, or how Handwritten Text Recognition will Revolutionize Access to Archival Collections. With a special focus on biographical data in archives Hessian State Archives Marburg Friedrichsplatz 15, D - 35037 Marburg 19-21 January 2016

Science

Innovative Technologies for Making
Crowdsourcing most Effective in
Online Indexing
Dr. Jesper Zedlitz
Christian-Albrechts-Universität Kiel
Verein für Computergenealogie = German Society for Computer Genealogy

2
CompGen · CAU Kiel
Agenda
● Short introduction
● Examples
● “Data Entry System” (DES)
● Transcription Speed
● Transcription Quality
● Future development

3
CompGen · CAU Kiel
Short Introduction
● Verein für Computergenealogie
= German Society for Computer Genealogy
– since 1989
– largest German Genealogical Society
– 3.600 members
– no public funding

5
CompGen · CAU Kiel
Examples
Many quite simple
entries on one page

6
CompGen · CAU Kiel
Examples
One or a few complex
entries on one page

7
CompGen · CAU Kiel
DES – The Project
● DES = Data Entry System
● objective: transcription of structured data
from printed sources
Family name: Kühnel
Birth name: Ziltz
Given names: Hertha Antonia
Date: 22.2.
Record number: 100

8
CompGen · CAU Kiel
DES – The Project
● start beginning of 2012
● first project: German WW1 casualty lists
– 31,000 pages

9
CompGen · CAU Kiel
DES – The Project
● meanwhile variety of other sources
– city directories,
– civil records,
– war grave lists, …

10
CompGen · CAU Kiel
DES – Technology
● web-based solution
+ no DVD shipping necessary
+ does not require special software
+ quick assistance possible
● work “on the image“
● search during work
+ voluntary transcribers want to see “their” entries online
● two-tiered transcription
1. single entries
2. groups of entries

11
CompGen · CAU Kiel
DES Web Application

12
CompGen · CAU Kiel
Web App – Data Entry

13
CompGen · CAU Kiel
Web App – Data Entry

14
CompGen · CAU Kiel
Web App – Data Entry

15
CompGen · CAU Kiel
Web App – Data Entry

19
CompGen · CAU Kiel
Web App – Area Data

20
CompGen · CAU Kiel
Web App – Area Data

21
CompGen · CAU Kiel
Web App – Area Data

22
CompGen · CAU Kiel
Web App – Area Data

23
CompGen · CAU Kiel
Web App – Area Data

24
CompGen · CAU Kiel
Web App – Area Data

26
CompGen · CAU Kiel
Transcription Speed
● Transcribed entries and number of registered users

27
CompGen · CAU Kiel
Transcription Speed
● Entries per day

29
CompGen · CAU Kiel
Approaches
● single transcription
● proofreading
transcriber
proofreader

30
CompGen · CAU Kiel
Approaches
● single transcription + double proofreading
– e.g. Wiki-Source
● proofreading with arbitration
expert

31
CompGen · CAU Kiel
Approaches
● double-keying with arbitration
● double-keying with N-keyed run-off votes
A B A, B or C
…
A B C A

32
CompGen · CAU Kiel
Approaches
● totally inefficient?
…
A B C A
? →

33
CompGen · CAU Kiel
Quality Improvement
● German WW1 casualty lists
– Gothic print
● small sample of 5,000 entries
● on average 15.6% errors for single keying
● varies from 6% to 78% for different users

34
CompGen · CAU Kiel
Quality Improvement
15.6% difference
1.1% difference
only 2.4% errors spotted
85% of the error
remained undetected!

35
CompGen · CAU Kiel
Future Development

36
CompGen · CAU Kiel
Future Development
● Entry of relations between entries

37
CompGen · CAU Kiel
Future Development
● Entry of relations between entries
● Data Entry on mobile devices

38
CompGen · CAU Kiel
Future Development
● Entry of relations between entries
● Data Entry on mobile devices
● Captchas

39
CompGen · CAU Kiel
Future Development
● Entry of relations between entries
● Data Entry on mobile devices
● Captchas
● Combination OCR + manual transcription

40
CompGen · CAU Kiel
Thank you for your attention!
Questions?
des.genealogy.net

This document discusses modern trends for C++ development. While C++ is still widely used in areas like operating systems, large services, and games, its use has declined relative to other languages. This is due to factors like its steep learning curve and slow development speed. However, C++ remains important in specialized domains like aerospace, science/health technologies, and embedded systems. The document provides advice for C++ developers to stay relevant, such as pursuing freelance work, joining product companies, or specializing in niche areas. It also highlights GlobalLogic's C++ expertise and projects involving medical devices, automotive software, and network infrastructure.

Test Driven Interviewing in C++

GlobalLogic Ukraine

This presentation was delivered by Vitalii Lysenko (Associate Manager, Consultant, GlobalLogic, Kyiv) during his master-class at GlobalLogic Kharkiv C++ TechTalk #1 on May 16, 2018. Master class topics: - Software development lifecycle - Modern lightweight IDEs for C++ - TDD in C++ via Google C++ Testing Framework - Continuous integration with Jenkins pipelines - Secure Git repositories - Disassembly hacking

Plotter

Aang Herie

Final Presentation-DATNYC

Karen Zellner

The document discusses using data on room use and class schedules from 2239 courses over 9 semesters to develop models for optimizing space allocation on campus. It analyzes current and ideal room utilization, converts excess space to flexible areas, and calculates costs for conversion scenarios. The results show how many and what types of rooms will be needed now and in the future to accommodate growth. Next steps include improving the scoring metrics, expanding the model to other buildings, visualizing updates on floor plans, and integrating with BIM software.

Kjetel Digre

Even Isachsen

OpenStack Berlin 2018 Recap

Vietnam Open Infrastructure User Group

This document recaps the OpenStack Summit held in Berlin in 2018. Over 2,700 attendees from 63 countries participated in the summit, more than 50% coming from outside Germany. The Rocky release of OpenStack was highlighted, and the 2018 user survey provided insights into typical OpenStack deployments, which most commonly use KVM as the hypervisor, Ceph for block storage, SQL for identity, CentOS for the operating system, and Ansible for configuration management.

FlinkForward Berlin 2019 - Scotty: Efficient Window Aggregation with General ...

Jonas Traub

Window aggregation is a core operation in data stream processing. Existing aggregation techniques focus on reducing latency, eliminating redundant computations, and minimizing memory usage. However, each technique operates under different assumptions with respect to workload characteristics such as properties of aggregation functions (e.g., invertible, associative), window types (e.g., sliding, sessions), windowing measures (e.g., time- or countbased), and stream (dis)order. Violating the assumptions of a technique can deem it unusable or drastically reduce its performance. In this talk, we present Scotty an implementation of a general stream slicing technique for window aggregation. This technique automatically adapts to workload characteristics to improve performance without sacrificing its general applicability. Our experiments show that Scotty outperforms alternative implementations, like the default window operator in Flink, by up to one order of magnitude. Furthermore, we present how to use Scotty as a library in Flink, Storm, or Beam without changing the underlying Stream Processing System. General stream slicing was first published at EDBT 2019 (http://www.user.tu-berlin.de/powibol/assets/publications/traub-efficient-window-aggregation-with-general-stream-slicing-edbt-2019.pdf) where it received the Best Paper Award. The Scotty library and its connectors are available as open-source (https://github.com/TU-Berlin-DIMA/scotty-window-processor) and contributions are highly welcome.

This document references several sources related to climate change and decarbonization efforts, including the EFH European Climate Leadership Report from 2017, National Geographic from 2017, the movie BladeRunner 2049, ElectricityMap.org from 2018, and the Nuclear Reimagined series by Third Way which outlines scenarios for using nuclear energy in industry applications, clean transit hubs, and remote arctic communities. Urban growth in China from Shenzhen in 1980 to 2013 is also referenced.

SC4 Hangout - Luigi Selmi, Transport pilot architecture

BigData_Europe

The document describes the SC4 Pilot Online Hangout which will test a new big data system for collecting and processing multisource transport data in Thessaloniki, Greece. The pilot will use Kafka and Flink to ingest taxi and Bluetooth data, integrate the data, process it using R and PostGIS, store it in Elasticsearch, and present it in Kibana. The components are containerized and the pilot will test deploying them on a 3 node Docker Swarm for fault tolerance and scalability. Current work involves setting up the swarm and future work will test the system.

Atmosphere 2018: Wojciech Krysmann- INFRA AS CODE - TERRAFORM DEEP DIVE AND B...

PROIDEA

This document summarizes a presentation about Terraform best practices and a deep dive into how it works. The presentation covers what Terraform is, how it can be used to implement infrastructure as code from manual processes to collaborative workflows, and why automating infrastructure provides benefits like faster deployments, increased control and predictability. It discusses best practices for Terraform configuration, implementation patterns like separating infrastructure from application code, and sample workflows for deploying infrastructure and platform services.

Things to do with OpenStreetMap

Servei de SIG i Teledetecció (SIGTE), Universitat de Girona

The document outlines various things that can be done with OpenStreetMap (OSM) data, including downloading OSM data in different formats, editing and tagging OSM data, creating maps on the web using APIs, importing OSM data into databases, rendering maps from the data, routing, and using libraries and frameworks. It provides information on popular tools for each task and highlights the benefits of the OSM ecosystem beyond just downloading and editing map data.

code.talks 2019 - Scotty: Efficient Window Aggregation for your Stream Proces...

Jonas Traub

This presentation was held at Code.Talks 2019 in Hamburg. A video is available at: https://www.youtube.com/watch?v=K1y5dJvP1jM Window aggregation is a core operation in data stream processing. Stream Processing Systems, like Flink or Storm, implement general aggregation techniques which perform poorly under specific workloads (e.g. Sliding Windows). To this end, we present Scotty, a new highly-efficient window operator. Scotty exploits specific workload properties such as the type of aggregation functions (e.g., invertible, associative), window types (e.g., sliding, sessions), windowing measures (e.g., time- or countbased), and stream (dis)order. This allows Scotty to outperform systems like Flink by up to one order of magnitude. The structure of this talk is threefold: First, we give an introduction to the semantics and implementations of window aggregations in modern Stream Processing Systems. Second, we discuss the design of Scotty and show why Scotty is able to outperform the default window operators of many stream processing systems. Third, we give a hands-on introduction to Scotty and demonstrate how it can be integrated into standard Flink, Storm, or Beam stream processing pipelines. Scotty and its connectors are available as open-source (https://github.com/TU-Berlin-DIMA/scotty-window-processor) and contributions are highly welcome.

6. 2x2 matrixaddmulttouchpad

Media4math

R in gvSIG

Óscar Martínez Olmos

This document discusses integrating the open source statistical programming language R into the geographic information system gvSIG. It provides an overview of R, its uses, and available tools. It then examines different options for integrating R into gvSIG - using the Java implementation Renjin, including an internal R installation, or linking to an external R installation. The main objective of integration is to extend gvSIG's capabilities by adding new geoprocesses, functions, graphs, and reports using R. Documentation on integrating R into gvSIG is available in an online workshop.

Concourse ci container based ci for the cloud

Johannes Rudolph

Concourse is a CI/CD tool that uses containers to provide repeatable builds. It models pipelines as a series of jobs that are connected through their inputs and outputs. Resources are checked for new versions and pulled into tasks where the work is done. Custom resources and caches can be created with containers to extend Concourse. Complex pipelines can be built with techniques like parallelization, aggregation, artifact storage, and grouping of jobs.

APPUiO Quick Start (OpenShift > DevOps > App Dev)

Peter Bittner

Linked data in the swiss federal data infra

geoportal of the federal authorities of the Swiss Confederation

- swisstopo is the Federal Office of Topography of Switzerland which is legally mandated to develop and manage the country's spatial data infrastructure. - It maintains a popular geospatial portal, map.geo.admin.ch, which serves over 500 layers to 2 million annual visitors. - swisstopo began a project in July 2016 to publish key geospatial datasets as linked open data using semantic web standards in order to improve discoverability and reuse of the data by non-experts. - The first dataset being published is the Administrative Units of Switzerland, with addresses to follow in 2017, available at ld.geo.admin.ch.

X fare forecasting

Lviv Data Science Summer School

Future Proof

Viasys VDC

The document discusses future-proofing and provides examples of items and technologies that have remained useful over long periods of time despite advances, such as steam irons, toasters, laundry dryers, Tupperware, dishwashers, and contact lenses. It also discusses challenges related to aging infrastructure, population growth, and financing infrastructure projects to meet future demand. Case studies are presented on projects that utilized 3D machine control, total stations, unmanned aerial vehicles, and modeling and planning software to overcome challenges and achieve savings.

Truck truck is a rust cad kernel

YoshinoriTanimura

1. Yoshinori Tanimura is a researcher working on truck, a Rust CAD kernel, for 3 years. 2. Truck is a Rust CAD kernel that reimplements B-rep geometry with NURBS. 3. Truck is designed as a collection of small crates, taking inspiration from Theseus' ship, with crates for tasks like 3D modeling, visualization, and WebGPU utilities, building upon the work of senior projects like OCCT, VTK, and three.js.

OpenHistoricalMap tim waters - Topomancy / NYPL Lightning Talk

chippy

Introduction to Fluvio Data Engineer.pdf

Sehz1

What is Big O in JavaScript - Part-1

Ideas2IT Technologies

Mapping french open data actors on the web with common crawl

data publica

This document describes a project to map French open data actors on the web using the Common Crawl dataset. The author used different crawling techniques like focused crawling and prospective crawling on Common Crawl to identify French websites and open data websites. Websites were given scores to determine how French and open data-focused they were. A graph was built from the links between websites and visualized in Gephi and Sigma.js to show interactions between open data actors. The final graph provided insights into the open data ecosystem in France and who the key actors are. Issues with the methodology and Common Crawl data are also discussed.

BDE_SC4_WS3_6_Luigi Selmi - Pilot SC4

BigData_Europe

The document describes a pilot project to build a scalable and fault-tolerant platform to process unlimited data sets using open source frameworks. It will use Apache Kafka as a messaging system, Apache Flink for stream and batch processing, PostgreSQL with PostGIS for road network storage and Elasticsearch for result storage. The platform will include a Docker swarm cluster to ingest and map-match real-time floating car data and classify roads by traffic level. It will also develop a short-term traffic forecasting algorithm using a feedforward neural network. The SANSA stack will enable additional use cases by linking semantic technologies and linked data.

Apache Flink Training Workshop @ HadoopCon2016 - #2 DataSet API Hands-On

Apache Flink Taiwan User Group

VIRUS INFORMATICOS

Laura_aguilar33

Este documento describe diferentes tipos de virus informáticos, incluyendo gusanos, caballos de Troya, bombas lógicas, hoaxes, virus de enlace y residentes. Los virus son programas maliciosos que infectan sistemas y realizan acciones no deseadas como borrar o dañar información. Algunos tipos consumen muchos recursos, esconden su daño dentro de programas legítimos, o se activan en fechas específicas.

Prezentácia1Roman Ládi

Diplomka

Hagiu Andrei Marian

What's hot

Clean energy-roundtable-2018-09

Kirsty Gogan Alexander

SC4 Hangout - Luigi Selmi, Transport pilot architecture

BigData_Europe

Atmosphere 2018: Wojciech Krysmann- INFRA AS CODE - TERRAFORM DEEP DIVE AND B...

PROIDEA

Things to do with OpenStreetMap

Servei de SIG i Teledetecció (SIGTE), Universitat de Girona

code.talks 2019 - Scotty: Efficient Window Aggregation for your Stream Proces...

Jonas Traub

6. 2x2 matrixaddmulttouchpad

Media4math

R in gvSIG

Óscar Martínez Olmos

Concourse ci container based ci for the cloud

Johannes Rudolph

APPUiO Quick Start (OpenShift > DevOps > App Dev)

Peter Bittner

Linked data in the swiss federal data infra

geoportal of the federal authorities of the Swiss Confederation

X fare forecasting

Lviv Data Science Summer School

Future Proof

Viasys VDC

Truck truck is a rust cad kernel

YoshinoriTanimura

OpenHistoricalMap tim waters - Topomancy / NYPL Lightning Talk

chippy

Introduction to Fluvio Data Engineer.pdf

Sehz1

What is Big O in JavaScript - Part-1

Ideas2IT Technologies

Mapping french open data actors on the web with common crawl

data publica

BDE_SC4_WS3_6_Luigi Selmi - Pilot SC4

BigData_Europe

Apache Flink Training Workshop @ HadoopCon2016 - #2 DataSet API Hands-On

Apache Flink Taiwan User Group

What's hot (19)

Clean energy-roundtable-2018-09

SC4 Hangout - Luigi Selmi, Transport pilot architecture

Atmosphere 2018: Wojciech Krysmann- INFRA AS CODE - TERRAFORM DEEP DIVE AND B...

Things to do with OpenStreetMap

code.talks 2019 - Scotty: Efficient Window Aggregation for your Stream Proces...

6. 2x2 matrixaddmulttouchpad

R in gvSIG

Concourse ci container based ci for the cloud

APPUiO Quick Start (OpenShift > DevOps > App Dev)

Linked data in the swiss federal data infra

X fare forecasting

Future Proof

Truck truck is a rust cad kernel

OpenHistoricalMap tim waters - Topomancy / NYPL Lightning Talk

Introduction to Fluvio Data Engineer.pdf

What is Big O in JavaScript - Part-1

Mapping french open data actors on the web with common crawl

BDE_SC4_WS3_6_Luigi Selmi - Pilot SC4

Apache Flink Training Workshop @ HadoopCon2016 - #2 DataSet API Hands-On

Viewers also liked

VIRUS INFORMATICOS

Laura_aguilar33

Prezentácia1Roman Ládi

Diplomka

Hagiu Andrei Marian

ICARUS-Lecture 1: Die tschechischen Archive im Web

ICARUS - International Centre for Archival Research

AV ČR na VS AVO 2015

AVOprezentace

Antikoncepce

ZŠ Hustopeče

Mikrobiologie

nagyfilip

Derecho ecleacistico............. gp

parka200

El documento habla sobre el derecho procesal eclesiástico. Explica que este derecho garantiza los derechos reconocidos en el derecho objetivo canónico y que los tribunales eclesiásticos se encargan de resolver controversias surgidas en las relaciones jurídicas entre personas. También describe los tipos de procesos, como los procesos matrimoniales, y las características de los órganos judiciales eclesiásticos como sus jueces. Por último, hace referencia al derecho penal canónico, explicando conceptos

SMZ_Poster_042015

Amber Von Harten

The document discusses Special Management Zones (SMZs), which have historically been used to manage artificial reefs. It proposes a new use of SMZs to designate spawning areas for certain species by selecting sites, establishing size guidelines, and expanding the current SMZ procedure. The new approach would consider creating artificial reefs and fish attraction devices to increase biological production and fishing opportunities, while restricting incompatible gear. Transit through and fishing for some pelagic species would be allowed in the spawning SMZs, but anchoring and fishing for snapper grouper species would be prohibited.

co:op-READ-Convention Marburg - Herbert Wurster

ICARUS - International Centre for Archival Research

cover letter

Lance Bankston

Lance Anthony Bankston is seeking an accounting position. He recently graduated summa cum laude from Southeastern Louisiana University with a 4.0 GPA in accounting. He held leadership roles as Vice President of his chapter of Beta Alpha Psi and Vice President of the Society of Human Resource Management. His previous work experience includes scanning invoices and performing data entry as a student worker for the East Baton Rouge City Parish Accounts Payable division. He looks forward to discussing how he can contribute to the company.

Jaguares te lo pido por favor.-1

Eduardo Rolas

bestuurders- verkeer mer den bosch-veghel

son279

Corporación unificada nacional de educación superior

Junior torres

Las publicaciones digitales representan uno de los arquetipos ideales de la informática actual, permitiendo que las personas accedan e interactúen con información de manera más fácil. Algunas plataformas en línea como Tikatok, Letterpop y Myebook permiten crear libros, boletines y publicaciones digitales de manera sencilla, compartiéndolas o guardándolas en línea. Estas herramientas han cambiado la forma en que se publica información actualmente.

co:op-READ-Convention Marburg - Andreas Kuczera

ICARUS - International Centre for Archival Research

Andreas Kuczera (Acadamy of Science and Literature Mainz, DE): Encoding and Presenting Historical Biographical Data with Graph Data Bases co:op-READ-Convention Marburg Technology meets Scholarship, or how Handwritten Text Recognition will Revolutionize Access to Archival Collections. With a special focus on biographical data in archives Hessian State Archives Marburg Friedrichsplatz 15, D - 35037 Marburg 19-21 January 2016

The Hungarian National Archives’ online portal on medieval charters

ICARUS - International Centre for Archival Research

The Hungarian National Archives launched an online portal called Medieval Hungary Online to digitize and provide access to medieval charters and documents. Over 300,000 records from the Archives of the Medieval Hungarian Kingdom dating before 1526 have been digitized, including 108,000 original charters and 94,000 photographic records. The database allows free public access to search and view digital images and transcripts of these important historical sources.

Síntesis de contenido del diplomado de educación superior

rafael cassis jimenez

Aula 01

guest1d9f6f

1) Há um modelo de desenvolvimento predatório sobre a natureza e cidades crescendo de forma desordenada, consumindo mais do que o planeta pode suportar. 2) Isso ocorre porque se estimula o consumo excessivo e descartável e um estilo de vida baseado no consumo e ostentação. 3) Gestão ambiental é necessária porque ultrapassamos os limites do planeta e ameaçamos a sobrevivência de espécies e o clima global.

Ronald Fisher - Progetto 2Massimiliano Grosso

การบริหารเชิงคุณภาพ ชุดที่ 7

Atit Patumvan

This document discusses principles of customer-supplier relationships in total quality management, including partnering, sourcing, supplier selection, supplier rating, and relationship development. The key points are that customers and suppliers should have long-term commitments based on trust and shared visions, methods for evaluating quality and supplier performance are important, and close collaboration through inspection, training, and team approaches helps develop strong relationships.

Viewers also liked (20)

VIRUS INFORMATICOS

Prezentácia1

Diplomka

ICARUS-Lecture 1: Die tschechischen Archive im Web

AV ČR na VS AVO 2015

Antikoncepce

Mikrobiologie

Derecho ecleacistico............. gp

SMZ_Poster_042015

co:op-READ-Convention Marburg - Herbert Wurster

cover letter

Jaguares te lo pido por favor.-1

bestuurders- verkeer mer den bosch-veghel

Corporación unificada nacional de educación superior

co:op-READ-Convention Marburg - Andreas Kuczera

The Hungarian National Archives’ online portal on medieval charters

Síntesis de contenido del diplomado de educación superior

Aula 01

Ronald Fisher - Progetto 2

การบริหารเชิงคุณภาพ ชุดที่ 7

Similar to co:op-READ-Convention Marburg - Jesper Zedlitz

Holistic data application quality

Lars Albertsson

Delivery Pipelines as a First Class Citizen @deliverAgile2019

ciberkleid

In this talk, we will cover important elements for successful CI and CD. We will discuss how these elements make CI and CD much simpler, and hence more attainable. We will cover some best practices / recommendations to include in your application pipelines. We will look at a sample implementation of a pipeline leveraging modern tools. Finally, we will discuss some forthcoming ideas for making it even easier to declaratively enable CI and CD for applications.

Secure software supply chain on a shoestring budget

Lars Albertsson

The NotPetya, SolarWinds, and Kaseya cybersecurity attacks were all executed by injection of malicious code in software shipped by vendors to thousands of companies. These attacks have made the public more aware of the importance of secure software supply chains. But the path from awareness to ensuring a secure supply chain is long. Developers have gotten used to the convenience of easily downloading third party software into containers, and it is challenging to tighten supply chain security in a company with a sprawl of open source components. Scling is a small data engineering startup, and since we ask our customers to entrust us with their data, we must take security seriously. We have been securing our software supply chain since the company was founded. We have no venture capital, and our customers expect quick development iteration cycles, so we have solved supply chain security with minimal effort and minimal impact on developer productivity. In this presentation, we share how we have addressed the different supply chain attack vectors, e.g. Python and JVM packages, with technical solutions. We will present how we automate third party software upgrades to stay up to date with security upgrades while minimising the risk of downloading rogue code.

Mortal analytics - Covid-19 and the problem of data quality

Lars Albertsson

Social media are full of Covid-19 graphs, each pointing to an "obvious" conclusion that fits the author's agenda. Unfortunately, even the official sources publish analytics that point at incorrect conclusions. Bad data quality has become a matter of life and death. We look at the quality problems with official Covid-19 data presentations. The problems are common in all domains, and solutions are known, but not widespread. We describe tools and patterns that data mature companies use to assess and improve data quality in similar situations. Mastering data quality and data operations is a prerequisite for building sustainable AI solutions, and we will explain how these patterns fit into machine learning product development.

India Analytics and Big Data Summit 2015

Kanwal Prakash Singh

Kanwal Prakash Singh gave a presentation at the India Analytics and Big Data Summit 2015 in Mumbai on February 3rd. He discussed how Housing uses data science and analytics to optimize operations like reducing flat duplication, forecasting supply and demand, and route optimization. Specifically, he described how Housing analyzed data to identify issues with low data collector utilization and many travelers between jobs. This led to developing a new branch location identification model and scheduling algorithm that optimally allocated listing requests to data collectors, reducing operational costs by 30% while making the solutions transferable to other delivery systems.

India Analytics and Big Data Summit 2015

Kanwal Prakash Singh

Evolution of AWS infrastructure for ML: from Zero to Hero

Taras Slipets

Building for success on the capable web - t3imd 2020

Andrey Lipattsev

This document discusses building for success on the capable web. It emphasizes acquiring users, engaging them through the conversion funnel, and retaining them. It highlights the importance of core web vitals like LCP, CLS and FID for performance. It also discusses monitoring success through Google Site Kit and bringing new content formats like Stories to the web. The overall message is on optimizing the user experience at each stage of the funnel through performance, engagement and content.

Free GitOps Workshop + Intro to Kubernetes & GitOps

Weaveworks

Follow along in this free workshop and experience GitOps! AGENDA:  Welcome - Tamao Nakahara, Head of DX (Weaveworks)  Introduction to Kubernetes & GitOps - Mark Emeis, Principal Engineer (Weaveworks)  Weave Gitops Overview - Tamao Nakahara  Free Gitops Workshop - David Harris, Product Manager (Weaveworks) If you're new to Kubernetes and GitOps, we'll give you a brief introduction to both and how GitOps is the natural evolution of Kubernetes. Weave GitOps Core is a continuous delivery product to run apps in any Kubernetes. It is free and open source, and you can get started today! https://www.weave.works/product/gitops-core If you’re stuck, also come talk to us at our Slack channel! #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)

Automated Testing Environments With Kubernetes & GitLab

Vladislav Supalov

Let’s see what it takes to build an automated testing/QA setup with Kubernetes and GitLab from scratch, using off-the-shelf components. The goal is to automatically deploy any feature-x or hotfix-x branch to individual environments for automated tests and to make it easier to share intermediate results within the team. This talk will touch on abstract topics like continuous delivery, the value of developer tooling and well-defined workflows, before diving deep into implementation details, learnings, gotchas and stuff-which-went-wrong.

Data Science in the Cloud @StitchFix

C4Media

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2lGNybu. Stefan Krawczyk discusses how his team at StitchFix use the cloud to enable over 80 data scientists to be productive. He also talks about prototyping ideas, algorithms and analyses, how they set up & keep schemas in sync between Hive, Presto, Redshift & Spark and make access easy for their data scientists, etc. Filmed at qconsf.com.. Stefan Krawczyk is Algo Dev Platform Lead at StitchFix, where he’s leading development of the algorithm development platform. He spent formative years at Stanford, LinkedIn, Nextdoor & Idibon, working on everything from growth engineering, product engineering, data engineering, to recommendation systems, NLP, data science and business intelligence.

Engineering data quality

Lars Albertsson

Garbage in, garbage out - we have all heard about the importance of data quality. Having high quality data is essential for all types of use cases, whether it is reporting, anomaly detection, or for avoiding bias in machine learning applications. But where does high quality data come from? How can one assess data quality, improve quality if necessary, and prevent bad quality from slipping in? Obtaining good data quality involves several engineering challenges. In this presentation, we will go through tools and strategies that help us measure, monitor, and improve data quality. We will enumerate factors that can cause data collection and data processing to cause data quality issues, and we will show how to use engineering to detect and mitigate data quality problems.

Building Pinterest Real-Time Ads Platform Using Kafka Streams

confluent

Building Pinterest Real-Time Ads Platform Using Kafka Streams (Liquan Pei + Boyang Chen, Pinterest) Kafka Summit SF 2018 In this talk, we are sharing the experience of building Pinterest’s real-time Ads Platform utilizing Kafka Streams. The real-time budgeting system is the most mission-critical component of the Ads Platform as it controls how each ad is delivered to maximize user, advertiser and Pinterest value. The system needs to handle over 50,000 queries per section (QPS) impressions, requires less than five seconds of end-to-end latency and recovers within five minutes during outages. It also needs to be scalable to handle the fast growth of Pinterest’s ads business. The real-time budgeting system is composed of real-time stream-stream joiner, real-time spend aggregator and a spend predictor. At Pinterest’s scale, we need to overcome quite a few challenges to make each component work. For example, the stream-stream joiner needs to maintain terabyte size state while supporting fast recovery, and the real-time spend aggregator needs to publish to thousands of ads servers while supporting over one million read QPS. We choose Kafka Streams as it provides milliseconds latency guarantee, scalable event-based processing and easy-to-use APIs. In the process of building the system, we performed tons of tuning to RocksDB, Kafka Producer and Consumer, and pushed several open source contributions to Apache Kafka. We are also working on adding a remote checkpoint for Kafka Streams state to reduce the time of code start when adding more machines to the application. We believe that our experience can be beneficial to people who want to build real-time streaming solutions at large scale and deeply understand Kafka Streams.

Building Information Modeling Using Constraint Logic Programming

Universidad Rey Juan Carlos

Building Information Modeling (BIM) produces three-dimensional object-oriented models of buildings combining the geometrical information with a wide range of properties about materials, products, safety and so on. BIM is slowly but inevitably revolutionizing the architecture, engineering, and construction (AEC) industry. Buildings need to be compliant with regulations about stability, safety, and environmental impact. Manual compliance checking is tedious and error-prone, and amending flaws discovered only at construction time causes huge additional costs and delays. Several tools can check BIM models for conformance with rules/guidelines. For example, Singapore’s CORENET e-Submission System checks fire safety. But since the current BIM exchange format only contains basic information of building objects, a separate, ad-hoc model pre-processing is required to determine, e.g., evacuation routes. Moreover, they face difficulties in adapting existing built-in rules and/or adding new ones (to cater for building regulations, that can vary not only among countries but also among parts of the same city), if at all possible.<br>We propose the use of logic-based executable formalisms (CLP and Constraint ASP) to couple BIM models with advanced knowledge representation and reasoning capabilities. Previous experience shows that such formalisms can be used to uniformly capture and reason with knowledge (including ambiguity) in a large variety of domains. Additionally, incorporating checking within design tools makes it possible to ensure that models are rule-compliant at every step. This also prevents erroneous designs from having to be (partially) redone, which is also costly and burdensome. To validate our proposal, we implemented a preliminary reasoner under CLP(Q/R) and ASP with constraints and evaluated it with several BIM models.

PowerGraph

Igor Shevchenko

PowerGraph is a distributed graph processing system that is well-suited for analyzing large natural graphs. It introduces a vertex-cut partitioning approach that distributes vertices across machines rather than edges. This addresses limitations of previous systems when processing graphs with power-law degree distributions. PowerGraph uses a Gather-Apply-Scatter decomposition that allows vertex programs to be parallelized across machines. It also employs techniques like delta caching to optimize performance. Evaluation on real-world graphs demonstrated reduced communication, runtime, and storage compared to previous systems.

How to build an event driven architecture with kafka and kafka connect

Loi Nguyen

How to build an event driven architecture with Kafka & Kafka Connect? Bài talk chia sẻ về quá trình 2 năm ứng dụng Kafka, Kafka Connect để chuyển đổi mô hình hệ thống của Vexere từ Monolithic thành Microservice, event driven - Event driven architecture là gì? - Làm thế nào để xây dựng 1 hệ thống event driven architecture một cách hiệu qủa bằng Kafka và Kafka Connect onnect - Các usercase hữu ích với Kafka & Kafka Connect - Kinh nghiệm thực tế và các bài học rút ra

Data ops in practice - Swedish style

Lars Albertsson

DataOps requires a cultural shift that brings the principles of lean manufacturing and DevOps to data analytics. It breaks down silos between developers, data scientists, and operators, resulting in rapid cycle times and low error rates. At Spotify in 2013, the concept of DataOps did not exist but the Swedish company needed a way to align the people, processes, and technologies of the data organization to accelerate the development of high-quality analytics. The result was a Swedish-style DataOps, influenced by Scandinavian culture and agile principles, that enabled the company to become a true data-driven leader.

Monolith to serverless service based architectures in the enterprise

Sameh Deabes

Crossing the data divide

Lars Albertsson

Did you know that the tech elite does not work at all like you do? Most people don't, and don't want to know. The State of DevOps report concluded a span of 1000x in delivery time and reliability between the elite and low performers. There is a similar gap for delivery time of data or ML pipelines to production. The gap in ability to compute datasets is higher, somewhere around a million times. We call this the data divide or the AI divide. It is widening over time, since most companies are not aware of its width. We will share the principles we applied in the most successful Scandinavian crossing of the data divide. We never explicitly shared or described, nor fully understood the principles at the time, but it is long due to explicitly enumerate them. The presentation will likely be uncomfortable and surprising, because it does not match what you do and what your vendors say. You will have no practical use of the information, since you cannot apply the principles, because they contradict many contemporary trends and popular technologies on the market, and you would be unable to overcome the forces of trends, popularity, and messages from vendors. They worked beautifully for us at the time.

Afterwork big data et data viz - du lac à votre écran

Joseph Glorieux

This document discusses a data visualization workshop hosted by OCTOSuisse on exploring and visualizing big data from a data lake. It provides an overview of OCTO's big data capabilities and projects. It then uses a case study of Swiss public transportation data to demonstrate data exploration, analysis, and visualization techniques using tools like Tableau. The goal is to understand data, identify insights, and effectively communicate findings to others.

Similar to co:op-READ-Convention Marburg - Jesper Zedlitz (20)

Holistic data application quality

Delivery Pipelines as a First Class Citizen @deliverAgile2019

Secure software supply chain on a shoestring budget

Mortal analytics - Covid-19 and the problem of data quality

India Analytics and Big Data Summit 2015

Evolution of AWS infrastructure for ML: from Zero to Hero

Building for success on the capable web - t3imd 2020

Free GitOps Workshop + Intro to Kubernetes & GitOps

Automated Testing Environments With Kubernetes & GitLab

Data Science in the Cloud @StitchFix

Engineering data quality

Building Pinterest Real-Time Ads Platform Using Kafka Streams

Building Information Modeling Using Constraint Logic Programming

PowerGraph

How to build an event driven architecture with kafka and kafka connect

Data ops in practice - Swedish style

Monolith to serverless service based architectures in the enterprise

Crossing the data divide

Afterwork big data et data viz - du lac à votre écran

More from ICARUS - International Centre for Archival Research

ICARUS-Meeting #20 | The Age of Digital Technology: Documents, Archives and S...

ICARUS - International Centre for Archival Research

ICARUS-Meeting #20 | The Age of Digital Technology: Documents, Archives and S...

ICARUS - International Centre for Archival Research

ICARUS-Meeting #20 | The Age of Digital Technology: Documents, Archives and S...

ICARUS - International Centre for Archival Research

ICARUS-Meeting #20 | The Age of Digital Technology: Documents, Archives and S...

ICARUS - International Centre for Archival Research

ICARUS-Meeting #20 | The Age of Digital Technology: Documents, Archives and S...

ICARUS - International Centre for Archival Research

ICARUS-Meeting #20 | The Age of Digital Technology: Documents, Archives and S...

ICARUS - International Centre for Archival Research

ICARUS-Meeting #20 | The Age of Digital Technology: Documents, Archives and S...

ICARUS - International Centre for Archival Research

ICARUS-Meeting #20 | The Age of Digital Technology: Documents, Archives and S...

ICARUS - International Centre for Archival Research

Presenting Archives Portal Europe, the largest online archives catalogue in the world. It provides access to archival descriptions from over 1,300 institutions across Europe. The portal uses international metadata standards like EAD and EAC-CPF and sees 500,000 users annually. The Archives Portal Europe Foundation governs the portal and has a strategy to improve its API, processing of EAC records, and additional finding aids.

ICARUS-Meeting #20 | The Age of Digital Technology: Documents, Archives and S...

ICARUS - International Centre for Archival Research