Apache Kafka is a distributed streaming platform used for building real-time data pipelines and streaming apps. It provides a unified, scalable, and durable platform for handling real-time data feeds. Kafka works by accepting streams of records from one or more producers and organizing them into topics. It allows both storing and forwarding of these streams to consumers. Producers write data to topics which are replicated across clusters for fault tolerance. Consumers can then read the data from the topics in the order it was produced. Major companies like LinkedIn, Yahoo, Twitter, and Netflix use Kafka for applications like metrics, logging, stream processing and more.
Troubleshooting Kafka's socket server: from incident to resolutionJoel Koshy
LinkedIn’s Kafka deployment is nearing 1300 brokers that move close to 1.3 trillion messages a day. While operating Kafka smoothly even at this scale is testament to both Kafka’s scalability and the operational expertise of LinkedIn SREs we occasionally run into some very interesting bugs at this scale. In this talk I will dive into a production issue that we recently encountered as an example of how even a subtle bug can suddenly manifest at scale and cause a near meltdown of the cluster. We will go over how we detected and responded to the situation, investigated it after the fact and summarize some lessons learned and best-practices from this incident.
101 mistakes FINN.no has made with Kafka (Baksida meetup)Henning Spjelkavik
1. The document summarizes 101 mistakes that Finn.no has made with Kafka. It discusses various configuration mistakes and operational mistakes made when initially adopting and using Kafka, and the consequences of those mistakes.
2. Common mistakes included not considering backwards compatibility of Kafka versions, treating Kafka like a database, not properly defining schemas, and not understanding client-side rebalancing.
3. Finn.no's solutions to address the mistakes included running multiple Kafka clusters during a migration, using fewer Kafka partitions as a default, and adopting better configuration practices tailored to their use cases.
Apache Kafka is a distributed streaming platform used for building real-time data pipelines and streaming apps. It provides a unified, scalable, and durable platform for handling real-time data feeds. Kafka works by accepting streams of records from one or more producers and organizing them into topics. It allows both storing and forwarding of these streams to consumers. Producers write data to topics which are replicated across clusters for fault tolerance. Consumers can then read the data from the topics in the order it was produced. Major companies like LinkedIn, Yahoo, Twitter, and Netflix use Kafka for applications like metrics, logging, stream processing and more.
Troubleshooting Kafka's socket server: from incident to resolutionJoel Koshy
LinkedIn’s Kafka deployment is nearing 1300 brokers that move close to 1.3 trillion messages a day. While operating Kafka smoothly even at this scale is testament to both Kafka’s scalability and the operational expertise of LinkedIn SREs we occasionally run into some very interesting bugs at this scale. In this talk I will dive into a production issue that we recently encountered as an example of how even a subtle bug can suddenly manifest at scale and cause a near meltdown of the cluster. We will go over how we detected and responded to the situation, investigated it after the fact and summarize some lessons learned and best-practices from this incident.
101 mistakes FINN.no has made with Kafka (Baksida meetup)Henning Spjelkavik
1. The document summarizes 101 mistakes that Finn.no has made with Kafka. It discusses various configuration mistakes and operational mistakes made when initially adopting and using Kafka, and the consequences of those mistakes.
2. Common mistakes included not considering backwards compatibility of Kafka versions, treating Kafka like a database, not properly defining schemas, and not understanding client-side rebalancing.
3. Finn.no's solutions to address the mistakes included running multiple Kafka clusters during a migration, using fewer Kafka partitions as a default, and adopting better configuration practices tailored to their use cases.
Geografiske analyser i ArcGIS - Esri norsk BK 2014Geodata AS
Minikurset gir deg tips om hvilke betrakninger man bør gjøre i forkant av analyser i ArcGIS for Desktop, hvordan man kan automatisere analyseprosessen med ModelBuilder og ikke minst presentere og dele resultatet med ArcGIS Online. I tillegg ser vi på oppsett og bruk av analysetjenester med ArcGIS for Server. Gled deg til mange praktiske eksempler og demonstrasjoner.
2013-05-30_Samferdselsskolen Bergen_BIM for Infrastruktur gjennom PlanfaseneÅge Langedrag
Foredrag på Samferdselsskolen.
Veg- og bane-Norge forener krefter: Utdanner neste generasjon ledere
Nå blir ledere og prosjektledere innen veg og bane oppfordret til å søke bransjens nye utviklingssatsing – Samferdselsskolen.
https://www.banenor.no/Nyheter/Nyhetsarkiv/2010/Veg--og-bane-Norge-forener-krefter-Utdanner-neste-generasjon-ledere/
Bruk av gatebilder i ArcGIS til profesjonelt bruk - Esri norsk BK 2014Geodata AS
Bruk av gatebilder har etter hvert blitt vanlig i forskjellige geo-relaterte prosesser og profesjonelle tjenester basert på slik bildeteknologi er utviklet for en rekke markeder. Brukernes krav til kvalitet og pålitelighet har drevet utviklingen og brukertilpassede løsninger basert på gatebilde-teknologi blir mere og mere brukt av det profesjonelle markedet. Bilder og programvare er tilgjengelig som webtjenester fra skyen og kan enkelt integreres i ArcMap ved hjelp av en ArcMap plug-in og online tilgang til bildene. På denne måten kan en se både kart og flybilder sammen med gatebilder over det samme område hvilket gir brukerne en mere detaljert oversikt for riktig vurderinger, samt at en kan gjøre målinger i gatebildene og integrere kartdata som shapefiler inn i bildene.
Hles 2021 Digital transformation - How to use digital tools to improve our ev...Henning Spjelkavik
This document discusses various tools and strategies for remote collaboration, communication, and document sharing. It provides recommendations for online meeting platforms like Zoom, Google Meet, and Microsoft Teams. It also reviews options for cloud-based document storage and sharing, including Google Drive, Dropbox, and Box. Finally, it considers virtual solutions for remote participation in events like races and trainings.
Geografiske analyser i ArcGIS - Esri norsk BK 2014Geodata AS
Minikurset gir deg tips om hvilke betrakninger man bør gjøre i forkant av analyser i ArcGIS for Desktop, hvordan man kan automatisere analyseprosessen med ModelBuilder og ikke minst presentere og dele resultatet med ArcGIS Online. I tillegg ser vi på oppsett og bruk av analysetjenester med ArcGIS for Server. Gled deg til mange praktiske eksempler og demonstrasjoner.
2013-05-30_Samferdselsskolen Bergen_BIM for Infrastruktur gjennom PlanfaseneÅge Langedrag
Foredrag på Samferdselsskolen.
Veg- og bane-Norge forener krefter: Utdanner neste generasjon ledere
Nå blir ledere og prosjektledere innen veg og bane oppfordret til å søke bransjens nye utviklingssatsing – Samferdselsskolen.
https://www.banenor.no/Nyheter/Nyhetsarkiv/2010/Veg--og-bane-Norge-forener-krefter-Utdanner-neste-generasjon-ledere/
Bruk av gatebilder i ArcGIS til profesjonelt bruk - Esri norsk BK 2014Geodata AS
Bruk av gatebilder har etter hvert blitt vanlig i forskjellige geo-relaterte prosesser og profesjonelle tjenester basert på slik bildeteknologi er utviklet for en rekke markeder. Brukernes krav til kvalitet og pålitelighet har drevet utviklingen og brukertilpassede løsninger basert på gatebilde-teknologi blir mere og mere brukt av det profesjonelle markedet. Bilder og programvare er tilgjengelig som webtjenester fra skyen og kan enkelt integreres i ArcMap ved hjelp av en ArcMap plug-in og online tilgang til bildene. På denne måten kan en se både kart og flybilder sammen med gatebilder over det samme område hvilket gir brukerne en mere detaljert oversikt for riktig vurderinger, samt at en kan gjøre målinger i gatebildene og integrere kartdata som shapefiler inn i bildene.
Similar to Geomatikkdagene 2016 - Kart på FINN.no (20)
Hles 2021 Digital transformation - How to use digital tools to improve our ev...Henning Spjelkavik
This document discusses various tools and strategies for remote collaboration, communication, and document sharing. It provides recommendations for online meeting platforms like Zoom, Google Meet, and Microsoft Teams. It also reviews options for cloud-based document storage and sharing, including Google Drive, Dropbox, and Box. Finally, it considers virtual solutions for remote participation in events like races and trainings.
10 years of microservices at finn.no - why is that dragon still here (ndc o...Henning Spjelkavik
Over the past 10 years, FINN transitioned their monolithic Java application built on a Sybase database to a microservices architecture running on PostgreSQL, Kafka and Kubernetes. They broke the monolith into independent services but still struggled with dependencies on the legacy Sybase database. They learned it takes a long time to fully migrate and should focus on delivering value incrementally. While the transition is still ongoing, they have made major progress deploying new services and migrating data and are now 93% in Kubernetes with the goal of moving completely to the cloud.
How FINN became somewhat search engine friendly @ Oslo SEO meetup 2018Henning Spjelkavik
Last week FINN.no received 1 million visits from Google searches, generating 15 million pageviews. This was mostly from branded searches for "FINN" and keywords. The site wanted to improve its search engine optimization to attract more long-tail traffic. Issues identified included wasted pages, missing titles and descriptions, pages blocked from crawling, and missing structured data. Fixes such as allowing crawling of category pages, adding titles and descriptions, and correcting redirect issues led to an immediate 50-300% increase in clicks from branded and unbranded searches.
An approach to it in a high level event - IOF HLES 2017Henning Spjelkavik
This document discusses IT considerations for organizing a high-level orienteering event like the World Orienteering Championships (WOC). It recommends defining IT requirements upfront in a contract to ensure quality delivery. Previous WOCs are reviewed to show how different organizers sourced IT services like timing, results, GPS tracking, and TV graphics. Options include buying a full-service package, renting equipment with consultants, or splitting tasks between in-house and outsourced teams. Proper project management is key - appointing a director, timeline, testing, and clear responsibilities. Live coverage adds complexity versus regular races and requires real-time data delivery to spectators. Fair competition for athletes must take priority in system design.
This document discusses 101 mistakes that FINN.no learned from in running Apache Kafka. It begins with an introduction to Kafka and why FINN.no chose to use it. It then discusses FINN.no's Kafka architecture and usage over time as their implementation grew. The document outlines several common mistakes made including not distinguishing between internal and external data, lack of external schema definition, using a single configuration for all topics, defaulting to 128 partitions, and running Zookeeper on overloaded nodes. Each mistake is explained, potential consequences are given, better solutions are proposed, and what FINN.no has done to address them.
The document discusses 101 mistakes that can be made when configuring Kafka. Some of the key mistakes discussed include:
1) Not distinguishing between internal and external data when publishing to Kafka topics
2) Not defining schemas for Kafka data externally such as in a schema registry
3) Using the same Kafka configuration for all topics and clients instead of tailoring configurations based on individual needs
4) Defaulting all topics to 128 partitions without considering actual throughput needs
5) Deploying Kafka in production on overloaded nodes that were intended for proof-of-concept use only
Hystrix is a latency and fault tolerance library used to isolate points of access to remote systems, services and 3rd party libraries, stop failures from cascading, and enable resilience in complex distributed systems. The presentation covered how to create Hystrix commands to wrap calls to external dependencies, use monitoring and dashboards to monitor circuit breaker states and metrics, and examples of how Finn has used Hystrix including request collapsing to batch calls. Key learnings included that graceful degradation is an important mindset change, nested commands can work but require care, and errors detected client-side with Hystrix may not always require application restarts.
The document provides an overview of key considerations and best practices for IT systems at a high-level orienteering event. It emphasizes the importance of thorough planning, testing, and backup solutions. The checklist highlights questions for event organizers to ask about organization of the IT team, general systems, punching systems, timing systems, and tracking to help ensure all elements are properly managed and integrated for a successful event. Having the right expertise to advise on these areas and addressing issues well in advance is also stressed.
Arena and TV-production - at IOF Open Technical Meeting in Lavarone 2014Henning Spjelkavik
This document discusses options for producing live audio and video streams of orienteering events to engage spectators watching from home. It presents several cases ranging from low-cost audio-only streaming to full television broadcast production. Case 1 involves using a webcam and streaming platform for a very low-cost video option. Case 2 adds additional cameras and video mixing software for a small multi-camera production. Case 3 describes using professional video switchers and cameras for an in-arena video production. Case 4 involves using a broadcast television production vehicle and pre-produced graphics for a true broadcast-quality live TV show. Examples of previous successful orienteering event live streams and broadcasts are also provided.
3. Historien om en “En ledende norsk karttjeneste”
2016 – flere unike brukere enn det er innbyggere
Traue keiner Statistik, die Du nicht selber gefälscht hast
4. FINN.no
• Agenda
– Idag! Norges fantastiske kartdata
– Historien - fra statisk til levende
– Integrerte rubrikkannonser med
morsomme algoritmer
– Litt om fremtiden
Henning Spjelkavik
@spjelkavik
henning.spjelkavik@finn.no
18. Skissen – ca 1999/2000
• Billig
• Oversikt
• Ikke egnet til å booke
strandhotell
19. Kart på www.finn.no – del 1 (1. generasjon)
2000
Flexim
CGI
2003
ArcIMS
CGI
2006
ArcGIS 9.2
Javascript
2007
Norkart
20. Kart på www.finn.no og mobil
2007
Norkart WMS
OpenLayers 2
2008
3D-kart
2009
Gatebilder
2011
Responsive
2012
Kart i native
app
2014
Retina/HD-tiles
21. State of the art
2003
Dårlig responstid
Gammeldags Javascript
Skrekkelig skalerbarhet
Proprietær kartserver
Lisens og kartlisenskostnader
25. State of the art - 2004
map.search.ch is completely
Javascript driven, there are no
Java or Flash components.
http://www.bernhardseefeld.ch/archives/000099.html
9th October 2004
27. FINN.no – Målsetninger – Kart 2006/2007-
• Den beste karttjenesten for våre brukere => integrasjon
– Få et overblikk over hvor objektene finnes
– Mulighet til å se gode detaljer rundt objektene
• Må absolutt ikke knekke den ordinære tjenesten
– Beta; egne servere (devops :) og webapp, deployment når vi ønsket
(continuous deployment :)
– Søk (tekst) begrenset antall pr sekund (QPS) (Fast...)
– Løsning: Gjør det i minnet
28. Krav til kartmotor
• Skulle ikke kreve plugin (ikke activex, flash, silverlight,
macromedia)
• => Javascript
• OpenLayers 2, ka-map eller eget
29. FINN.no – Kart – Søkemotor
• På hvert flytt
– R-tre
– Filter
– Cluster
– custom json (neste gang - geojson)
• Idag: SOLR søk etter objekter (inkludert filtre), R-tre for POIer
30.
31. 3D og gatebilder (2008-2009)
• Gatebilder
– FINN foto (i dag Making View)
– Etter hvert C3
• 3D – modell, generert fra bilder
– C3, Agency 9
– Apple
• Sesam – i praksis fra
markedsføringsbudsjett
• I dag ingen egen satsing
33. Hvordan vise?
• Bruke <img> tag?
• 2007 & IE: Mer enn 100 objekter => lås eller BSOD
• Transparent bilde
• onMouseOver
– Google and maps.ch brukte mus x,y sniffing
– Image maps! Skalerte utmerket på IE6
• Google gjør fortsatt dette for IE8
37. Kart på mobil-app
• MyVR SDK med 3D-støtte (2012)
• Mapbox (2013 iOS)
• iOS SDK (2014 iOS)
• Android SDK (2013 Android)
• http://kart.finn.no skal fungere på moderne telefoner som iOS, Android
og brukbart på Windows Phone.
• HD/retina tiles (2014)
38. Tydelige trender
• Mapbox, Leaflet eller OpenLayers 3
• Raskere oppdatering (Geosynkronisering, daily OSM)
• Nokia Here – WebGL 3D
• Google Maps 2014 (WebGL)
• Vektor - generelt
• Lokasjon i brukeropplevelsen
– Zillow
– Trulia http://on.trulia.com/21kKETF
– AirBnB