BeakerX - Tiezheng Li

•Download as PPTX, PDF•

2 likes•1,240 views

BeakerX is a collection of kernels and extensions to the Jupyter interactive computing platform. Its major features are: 1) JVM kernel support including Java, Scala, Groovy, Clojure, Kotlin, and SQL. The kernels are built from a shared base kernel that includes magics and classpath support. 2) a collection of interactive widgets for time-series plots, tables, and forms. There are APIs for our JVM languages plus Python and JavaScript. 3) prototype autotranslation for polyglot programming 4) One-click publication including interactive widgets, and 5) a data browser with drag-and-drop into the notebook. The presentation will include a demo of BeakerX and discussion of its history and relationship to its predecessor the Beaker Notebook.

Data & Analytics

Tiezheng Li
Tiezheng.Li@twosigma.com
Two Sigma Investments, LLC
April 27, 2018
The Beaker Extensions for Jupyter:

Agenda
Beaker Notebook
From Beaker Notebook to BeakerX
BeakerX Live Demo

DISCOVERABLE
DATA
DATA
ANALYSIS
+ MODELING
SCALABLE
+ DISTRIBUTED COMPUTE
PUBLICATION
+ COLLABORATION
OUR VISION FOR DATA SCIENCE

Oct 2013
Internal GA
Mar 2015
R, Scala, Java, Python2/3 support
Jun 2015
PySpark, SparkR, Clojure, Kdb
support
Nov 2016
BeakerX Pivot
Apr 2016
External Beaker Lab Alpha
LIFE OF BEAKER
May 2014
Open Source Beta
Aug 2017
BeakerX RC1

nbconvert
nbviewer
nbpresent
nbgrader
Jupyter Hub nbdime
nbmanager
binder
Jupyter Lab

● Time Series Visualizations
● JVM Kernels
● Interactive Tables
● Collaborative Publication
● True Polyglot Analysis (in progress)
● Data Discovery (in progress)
BeakerX: A unique addition to the Jupyter Ecosystem

Future Work:
● Migration to Jupyter Lab
● Spark deep integration
● True Polyglot Analysis
● Data Discovery
● and more … !

What's hot

Integrating Google APIs into Your Applications

Chris Schalk

Lessons Learned: Spring Cloud -> Docker -> Kubernetes

Mauricio (Salaboy) Salatino

Seventh openCypher Implementers Group Meeting: Status Update

openCypher

PixieDust

Margriet Groenendijk

This week in Neo4j -13th January 2018

Neo4j

There are many tools and possibilities available in the Icinga monitoring ecosystem. Setup Icinga 2 within a distributed architecture, put Icinga Web 2 on top and visualise alerts. Further you’ve already setup your preferred graphing solution (Graphite, InfluxDB, etc.) and Grafana greets you with a shiny metrics dashboard. Log events are processed and Elastic Stack or Graylog add more possibilities to correlate them with monitoring alerts. Having so many tools requires you to know how to connect them, or simply integrate an existing data source or frontend into your all-in-one user dashboard. Be it Graphs in your detail view, additional log events on a critical service view or yet a global map for location based alerts. Maybe you’ll also want to provide as much details in your notifications as possible, think of fancy Grafana graphs. This talk dives into existing and possible integrations, and explains why you sometimes just write your own integration. Carsten is the author of the Icinga Web 2 Grafana module while Michael focusses on Dashing and log processing. We’ll catch up with war stories (“meh, nothing exists”) to hero stories (“hooray, users love my integration”) and hope to motivate everyone out there doing the same. We will continue with practical development on Friday during the OSMC hackathon.

OSMC 2017 | Ops and dev stories- Integrate everything into your monitoring st...

NETWAYS

Big Data Analytics London - Data Science in the Cloud

Margriet Groenendijk

Presented at FOSDEM 2021: https://fosdem.org/2021/schedule/event/k8sconfigmgmtlandscape/ “We are all YAML engineers now” as Bob Walker said in 2018 in Ghent. So we now need something to manage these millions of lines of YAML. There’s the “Kubernetes application management tools” list by Bryan Grant with over 120 tool aiming at config management for Kubernetes. This area is evolving fast, and as k8s users we need some guidance to make informed decisions on what to use. In this talk I’ll describe the problem I need to solve and take a look into what we can learn from the previous generation of cfg mgmt tools. Then, will show some of the new tools and methods. Will not go into details of each solution, but rather compare different approaches and discuss which is good for specific needs.

Kubernetes Config Management Landscape

Tomasz Tarczyński

Streaming Sensor Data with Grafana and InfluxDB | Ryan Mckinley | Grafana

InfluxData

The Property Graph Query Language Landscape: openCypher and Property Graph Ex...

openCypher

In this episode, we will focus on open sourcing how we run Netflix's open source program. Netflix has been using and contributing to open source for several years. Over the years, Netflix has released over one hundred Netflix Open Source (aka NetflixOSS) libraries, servers, and technologies. Netflix engineers benefit by accepting contributions and gathering feedback with key collaborators around the world. Users of NetflixOSS from many industries benefit from our solutions including Big Data, Build and Delivery Tools, Runtime Services and Libraries, Data Persistence, Insight, Reliability and Performance, Security and User Interface. With such a large and mature open source program, Netflix has worked on approaches and tools that help manage and improve the NetflixOSS source offerings and communities. Netflix has taken a different approach to building support for open source as compared to other Internet scale companies. Come to this session to learn about the unique approaches Netflix has taken to both distribute and automate the responsibilities of building a world-class open source program.

Netflix OSS Meetup Season 4 Episode 4

aspyker

Spark + i python

Guillermo Blasco Jiménez

Full Stack Development with Neo4j and GraphQL

Neo4j

Neo4j is a high performance, open-source graph database written in Java and Scala. But of course, you can use and extend it from other JVM Languages, like Kotlin. I've been using Kotlin on and off since 2012, that's when I wrote my first article about this pragmatic and clean programming language. Today I want to demonstrate in 4 examples how you can use Neo4j with Kotlin. In preparation for KotlinConf 2017 we started to gather community activity of the Kotlin community in a Neo4j graph. You can see tweets, GitHub projects, StackOverflow questions and answers and meetup events in this database of developer activity. After showing some examples we will explore how to query that database using a Kotlin driver for Neo4j. Then we'll look at the Kotlin + Spring (Data) demo app, that shows community members who get a lot of praise on a Twitter wall. My main Kotlin project over the last year has been the GraphQL extension for Neo4j. So we'll have a look at how we can extend Neo4j with a custom HTTP API and how we integrate the GraphQL-Java library using Kotlin. This extension manages a GraphQL schema to translate GraphQL queries to Neo4js native query language Cypher. Another way to extend Neo4j is with user-defined procedures and functions which (with a small trick) are really easy to implement with Kotlin too. We'll take a look at the procedures and functions we expose as part of the neo4j-graphql extension.

Building Community APIs using GraphQL, Neo4j, and Kotlin

Neo4j

This week in Neo4j - 7th October 2017

Neo4j

Lessons Learned: From Java EE to Spring Cloud in the context of Activiti OSS

Mauricio (Salaboy) Salatino

Data Science covers the complete workflow from defining a question, finding the most suitable data source, identifying the right tools and finally presenting the best possible answer in a clear, engaging manner. But it all starts with having access to the data. In these slides I will walk your through some examples of how to collect, store and access data in the Cloud with the use of different APIs.

Cloud architectures for data science

Margriet Groenendijk

Elastic at Procter & Gamble: A Network Story

Elasticsearch

SW360 Update Tooling Telco

Shane Coughlan

12th Meeting OpenChain Reference Tooling Work Group - 25th March - Slides

Shane Coughlan

What's hot (20)

Integrating Google APIs into Your Applications

Lessons Learned: Spring Cloud -> Docker -> Kubernetes

Seventh openCypher Implementers Group Meeting: Status Update

PixieDust

This week in Neo4j -13th January 2018

OSMC 2017 | Ops and dev stories- Integrate everything into your monitoring st...

Big Data Analytics London - Data Science in the Cloud

Kubernetes Config Management Landscape

Streaming Sensor Data with Grafana and InfluxDB | Ryan Mckinley | Grafana

The Property Graph Query Language Landscape: openCypher and Property Graph Ex...

Netflix OSS Meetup Season 4 Episode 4

Spark + i python

Full Stack Development with Neo4j and GraphQL

Building Community APIs using GraphQL, Neo4j, and Kotlin

This week in Neo4j - 7th October 2017

Lessons Learned: From Java EE to Spring Cloud in the context of Activiti OSS

Cloud architectures for data science

Elastic at Procter & Gamble: A Network Story

SW360 Update Tooling Telco

12th Meeting OpenChain Reference Tooling Work Group - 25th March - Slides

Similar to BeakerX - Tiezheng Li

Voxxed Athens 2018 - Java EE is dead Long live jakarta EE!

Voxxed Athens

Containers at Netflx - An Evolving Story QConSF2015

Sangeeta Narayanan

Apache spark 2.4 and beyond

Xiao Li

Data has always been used in every company irrespective of its domain to improve the operational efficiency and the products themselves. However, analyzing and extracting information from “Big Data” is the next revolution in technology, since previously unknown nuggets of information are now made visible. In fact, over 90% of the data available in the world has been generated in the last two years. “Big Data” analytics has become the next hot topic for most companies - from financial institutions to technology companies to service providers. Likewise in software engineering, data collected about the development of software, the operation of the software in the field, and the users feedback on software have been used before. However, collecting and analyzing this information across hundreds of thousands or millions of software projects gives us the unique ability to reason about the ecosystem at large, and software in general. At no time in history has there been easier access to extremely powerful computational resources as it is today, thanks to the advances in cloud computing, both from the technology and business perspectives. Therefore, it is easier today than ever before to analyze big data. In this technical briefing, we will present the state-of-the-art with respect to the research carried out in the area of big data analytics in software engineering research. We will present the research along three dimensions: 1) What are the software engineering problems being solved? Examples of problems include: How much source code is newly written and how much is reused from past projects? Can we recommend best practices to developers by observing the development of software among hundreds of thousands of software projects? 2) What are the datasets that are being used? Examples of my datasets include: all the mobile apps in the Google Play store, all of the world's Open Source projects, and hundreds of gigabytes of execution logs. Such large datasets provide us with a unique view into the SE field. 3) What are the tools and techniques available to analyze the large datasets? We intend to present generic software solutions that have been applied to big datasets in other areas of research, and the tools and techniques created by software engineering researchers. In the end we will present the challenges inherently present in large datasets - volume, variety, velocity, and veracity. Such challenges often complicate the analysis of the data and can invalidate the interpretation of the results. We will conclude with the future opportunities that are present in big data analytics for software engineering research.

Big(ger) Data in Software Engineering

Mehdi Mirakhorli

Google Cloud at GraphSummit London 14 Nov 2023.pptx

Neo4j

From stream to recommendation using apache beam with cloud pubsub and cloud d...

Neville Li

Microsoft has been a leader in the enterprise analytics space for years. In 2014, Microsoft had already created R language functionality within Azure Machine Learning. On April 6, 2015, Microsoft and closed on a deal to acquire Revolution Analytics, a company focusing on scalable processing solutions initiated by the well-known R language. Many data science projects and initial demos do not need high-volume solutions: however, having a high-volume answer for the R language allows for planning or working toward the largest data science solutions. This presentation describes the add-value for the Revolution Analytics acquisition. The talk covers 1) an overview of current data science technologies from Microsoft; 2) a description of the R language; 3) a brief review of the add-value for R with Azure Machine Learning, and 4) a description of the performance architecture and demo of the language constructs developed by Revolution Analytics. Most of the presentation will be focused on sections two and four. It is anticipated that these technologies will be partially if not fully integrated into SQL Server 2016.

Microsoft and Revolution Analytics -- what's the add-value? 20150629

Mark Tabladillo

2020.04.22 aws deepracer log analysis

TomaszPtak4

Delivering Agile Data Science on Openshift - Red Hat Summit 2019

John Archer

Migrating PostgreSQL to the Cloud

Mike Fowler

Apache Spark 2.0 set the architectural foundations of structure in Spark, unified high-level APIs, structured streaming, and the underlying performant components like Catalyst Optimizer and Tungsten Engine. Since then the Spark community has continued to build new features and fix numerous issues in releases Spark 2.1 and 2.2. Continuing forward in that spirit, the upcoming release of Apache Spark 2.3 has made similar strides too, introducing new features and resolving over 1300 JIRA issues. In this talk, we want to share with the community some salient aspects of soon-to-be-released Spark 2.3 features: • New deployment mode: Kubernetes scheduler backend • PySpark performance and enhancements • New structured streaming execution engine: continuous processing • Data source v2 APIs for both structured streaming and Spark SQL • ML on structured streaming • Image reader • Stable codegen engine • Spark History Server V2 • Native ORC support • Vectorized ORC and SQL cache readers • Stream-stream Join • UDF enhancements • Various SQL enhancements Speakers Xiao Li, Software Engineer, Databricks Wenchen Fan, Software Engineer, Databricks

What’s new in Apache Spark 2.3

DataWorks Summit

On May 3rd 2017, Engineering released the new version of SpagoBI, the OW2 flagship project on Business Intelligence. Starting from this version, the project is now branded as Knowage, and is available in two editions: - the Community Edition, entirely open source, released and down-loadable from the OW2 forge, supported by the open source community - the Entreprise Edition, fully supported by Engineering under a subscription model Knowage suite is composed of several modules, each one conceived for a specific analytical domain. They can be used individually as a complete solution for a certain task, or combined with one another to ensure full coverage of users’ requirements, allowing to build a tailored product. The presentation focus on the Community Edition and provides examples, demos and use cases on the most important functional evolutions of the product: - the brand new and responsive user interface based on Angular.js - the extended set of graphical widgets to perform advanced data visualization - the self-service capabilities to perform data discovery - the new web-based user interface to define the metadata layer by exploiting the data federation principle - the possibility to define and import advanced analytic algorithms, written in R or Python, into a shared function catalog

SpagoBI version 6 rebranded as Knowage offers unpaired analytical experience,...

OW2

Session 8 - Creating Data Processing Services | Train the Trainers Program

FIWARE

Javantura v3 - The story of Java & HUJAK

HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association

Delivering Developer Tools at Scale

Oracle Developers

Announcing Databricks Cloud (Spark Summit 2014)

Databricks

Drupal 8 and 9, Backwards Compatibility, and Drupal 8.5 update

Angela Byron

Docker in Production, die Mobiliar

Philipp Grossenbacher

From http://www.csdn.net/article/2015-12-17/2826501 《Databricks公司联合创始人、Spark首席架构师辛湜：Spark发展：回顾2015，展望2016 》辛湜介绍了Spark的目标是“Unified engine across data workloads and platforms”。在谈到Spark在2015年最大的改变时，他感觉应该是增加了DataFrames API。对于Spark的生态圈，他表示主要侧重三个不同的方向，一个是上层的应用，二是下层的环境，还有最重要的是连接到的数据源。

BDTC2015 databricks-辛湜-state of spark

Jerry Wen

Building data pipelines for modern data warehouse with Apache® Spark™ and .NE...

Michael Rys

Similar to BeakerX - Tiezheng Li (20)

Voxxed Athens 2018 - Java EE is dead Long live jakarta EE!

Containers at Netflx - An Evolving Story QConSF2015

Apache spark 2.4 and beyond

Big(ger) Data in Software Engineering

Google Cloud at GraphSummit London 14 Nov 2023.pptx

From stream to recommendation using apache beam with cloud pubsub and cloud d...

Microsoft and Revolution Analytics -- what's the add-value? 20150629

2020.04.22 aws deepracer log analysis

Delivering Agile Data Science on Openshift - Red Hat Summit 2019

Migrating PostgreSQL to the Cloud

What’s new in Apache Spark 2.3

SpagoBI version 6 rebranded as Knowage offers unpaired analytical experience,...

Session 8 - Creating Data Processing Services | Train the Trainers Program

Javantura v3 - The story of Java & HUJAK

Delivering Developer Tools at Scale

Announcing Databricks Cloud (Spark Summit 2014)

Drupal 8 and 9, Backwards Compatibility, and Drupal 8.5 update

Docker in Production, die Mobiliar

BDTC2015 databricks-辛湜-state of spark

Building data pipelines for modern data warehouse with Apache® Spark™ and .NE...

More from Two Sigma

How much of a problem is school bullying in NYC? The answer depends on who you ask. Data Clinic volunteers compared local surveys (where many students say bullying is happening) with federal data (where a majority of schools report zero incidents), to analyze these disparities for the 2013-14 school year. To present this work, the Data Clinic hosted an event as part of NYC’s Open Data Week, featuring a presentation of the analysis and a panel discussion with researchers, advocates, and journalists to better understand this important student safety issue.

The State of Open Data on School Bullying

BeakerX - Tiezheng Li

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to BeakerX - Tiezheng Li

Similar to BeakerX - Tiezheng Li (20)

More from Two Sigma

More from Two Sigma (20)

Recently uploaded

Recently uploaded (20)

BeakerX - Tiezheng Li

Editor's Notes