SlideShare a Scribd company logo
The road to sbt 1.0
Paved with Server
Who are we?
@eed3si9n
(Eugene Yokota)
● sbt core dev
● scalaxb
● treehugger.scala
● "learning Scalaz"
● sbt-assembly etc
Josh Suereth
● Big Nerd
● Leader of tools
team @ Typesafe
● Author
o Scala in Depth
o sbt in Action
Agenda
● A brief (incomplete and mostly wrong) history of Build Tools
● A roadmap to sbt 1.0
o Modularization
● sbt-server
A brief history of build tools
Incomplete and mostly wrong
#!/bin/sh
$ make -j 2 hello
# Makefile
OBJ = test.o test2.o
%.o : %.c
$(CC) -c -o $@ < $<
hello : $(OBJ)
gcc -o $@ $^
Make (1976)
● new! Dependency-based programming
● Uses system command to carry out tasks
● problem! Machine-dependent
Q: How do I build this program?
Autoconf/Automake (1991/1996)
● Generate configure script and Makefile templates
o Configure generates header scripts and real makefile
o Configure discovers what the machine has
● Use an even MORE arcane syntax
o m4 macros
o magic strings
Q: How do I build on this machine?
Internet
$ ant dist
<project name="MyProject" default="dist" basedir=".">
<description>example build file</description>
<property name="src" location="src"/>
<property name="build" location="build"/>
<property name="dist" location="dist"/>
<target name="compile" description="compile the source " >
<javac srcdir="${src}" destdir="${build}"/>
</target>
<target name="dist" depends="compile"
description="generate the distribution" >
<mkdir dir="${dist}/lib"/>
<jar jarfile="${dist}/lib/MyProject-${DSTAMP}.jar"
basedir="${build}"/>
</target>
</project>
Ant (2000)
Q: How do I build on this machine?
● new! Built-in tasks like mkdir, javac, and jar
o Self-contained and platform-independent
● task plugins (like ant-contrib)
● problem! Hard to reuse build logic
Package managers
Q: How do I find library source/binary?
● new! Metadata to track library dependencies
● new! Repository to host source and binary
● CTAN (1993), Port (1994), RPM (1997), APT (1998)
Maven (2004)
Q: How do I find library source/binary?
Q: How do I enforce consistent builds?
● Metadata to track library dependencies
● Repository to host source and binary
● new! Default build (Convention over Configuration)
● new! Plugins that allowed reuse of build logic
● problem! Difficult to customize
Ivy (2004)
Q: How do I find library source/binary?
● Metadata to track library dependencies
● Repository to host source and binary
● Maven integration on top of Ant
Rake (2003)
rule '.o' => ['.c'] do |t|
sh "cc #{t.source} -c -o #{t.name}"
end
task :name, [:first_name, :last_name] do |t, args|
puts "First name is #{args.first_name}"
puts "Last name is #{args.last_name}"
end
Rake (2004)
Q: How do I build on this machine?
Q: How do I customize my build?
● new! Internal DSL
o Leverage Ruby libraries rather than shell programs
o Provide cross-platform ruby libraries for supported language tools
● Real programming language to write tasks
O NOES XML!
SAX parsing is declared uncool
(immediately after inception)
REST/JSON attempt to displace SOAP
Buildr (2008)/Gradle (2009)/ Leningen(2009)
● Maven integration
Q: How do I find library source/binary?
Q: How do I customize my build?
● Use an internal DSL and a real program language
sbt (2008)
Q: How do I find library source/binary?
Q: How do I customize my build?
Q: How can I develop faster?
● new! Incremental compiler tracking source deps
● new! Interactive shell + library-aware REPL
● new! Test framework abstraction
● new! Parallel by default
● shabby-chic! ANSI colors
● Internal DSL + Maven integration
Google Blaze (2009)
Q: How can I develop faster?
Q: How do I avoid version conflicts?
● new! Incremental build
● new! Cache builds remotely
● new! Clustered Building
● Abandon Maven. Check in all source to version control.
Force everyone on the same version.
● 1 SCM repo for all projects
● Assume homogeneous-ish environment
Pants (2014)/Buck (2014)
Q: How can I develop faster?
Q: How do I avoid version conflicts?
● Incremental build
● Cache builds remotely
● Force all dependencies to the same version for all
projects
● Put everything in a mono-repository
History
make (1977)
automake (1996)/
autoconf (1991)
Rake (2003)
Maven (2004)
Ivy (2004)
sbt (2008)
Blaze (2009)
Gradle (2009)
Leiningen
(2009)
Pants (2014)
Buck (2014)
RPM (1997)
Ant (2000)
Buildr (2008)
Jon Pretty's shell scripts (2004-2015)
History Recap
● sbt has inherited:
o dependency based programming (Make)
o internal build DSL (Rake)
o Package/Library Management (Maven)
o Convention over configuration (Maven)
o Re-usable build flow (Maven)
● sbt brings:
o interactivity (on the shell)
o parallel by default
A roadmap to sbt 1.0
stability + sbt server => modularization
Stability
1. Conceptual stability
2. Binary compatibility of sbt plugins
3. Source compatibility of build files
Concepts (mostly stable)
1. Scala incremental compiler
2. Dependency manager (Scala-aware)
3. Task and plugin system (using Scala)
4. Test framework abstraction
5. Text-based interactive shell
5. sbt server + client(s)
Plugin binary compatibility
● sbt 0.13 maintained 18 months of bincompat
o Lots of hacks and effort. Unable to remove cruft.
● sbt 1.x.y should be bincompat with 1.0.0
● Need to minimize surface API
o able to add small features when requested
Build source compatibility
● Source compatibility of your build.sbt
● sbt 1.x.y should be stable
● sbt 1.0 gives us opportunity to break DSL
o Deprecate project/build.scala?
o Unify sbt shell key syntax w/ Scala DSL syntax
Modularization
Componentize stable features, innovate new ideas
1. Pull out cohesive subprojects
2. Distinguish public API and internal details
3. Document usages
4. Clean up historical code
5. Cross publish for latest scala versions
(if applicable)
6. Publish to Maven Central (or JCenter)
Module candidates
● Launcher API
● Serialization API
● Compiler/REPL API
● Test framework API
● Dependency Management API
● Network API
● IO API
● Task DSL
● Completion API
● sbt client (sbt-server)
sbt-server
Architecture and Design
The problem
● Many things want access to the build
● We need to centrally control build-related
tasks to avoid breakage.
o intellij auto-import + "sbt ~ test"
o activator + "play run"
sbt-server design
sbt-server
disk (target/)
sbt client
commands /
watches
changes
Events
(logs, status, custom)
Before server - Command Engine
CMD
(compile)
Engine
(sequential)
Task Engine
(parallel)
Read next
command from
stdin
Next
command
Read previous
log file Reload the build
Terminal
Client
After server - Execution Queue
CMD
(compile)
Engine
TasksRead server
Queue
Next
command
previous
log file
Reload
the build
Server Event
Loop
Request Queue
CommandQueue
Client #1 Client #2
LatestState
Next State
Problem: Server discovery
Who starts the sbt-server, how do we know it is
running, how do we connect?
sbt-launcher
● sbt 0.13.2 launcher had support for "servers"
o launch config specifies "lock" file which prevents
more than one service being started
o "lock" file contains location (ip / port) of the service
o server is tested (ping) for liveness by the launcher.
Problem: Disconnects
● Client may not be the one to start sbt
● Client may disconnect from server, or server
may crash
Connect as an Event
val connector = SbtConnector(
"terminal", "Command Line Terminal",
configuration.baseDirectory)
def onConnect(client: SbtClient): Unit = {
client handleEvents { … }
client watch ...
}
connector.open(onConnect, onError)(<execution context>)
Connect as an Event
val connector = SbtConnector(
"terminal", "Command Line Terminal",
configuration.baseDirectory)
def onConnect(client: SbtClient): Unit = {
client handleEvents { … }
client watch ...
}
connector.open(onConnect, onError)(<execution context>)
Clients
reconstruct their
watches
and restore their
view of the build
on any reconnect
Problem: protocol
We needed a good mechanism for client +
server to communicate.
● Something that can evolve and maintain
compatibility
● Something that is easy for plugin authors to
extend.
sbt-serialization
● New library based on Scala Pickling
o JSON - currently supported format.
● sbt-remote-control has versioned protocol
and tests.
● (coming) plugin *Keys as protocol
o remote clients can watch/get values
o require pickler/unpickler when creating *Key
o ??? release plugin "client" jar separate from impl
Problem: Input
sbt tasks may want the user to type input
Example Client (input)
client.requestExecution(
"run",
Some(TerminalInteraction -> TerminalThread)
)
client server
ExecutionRequest
Example task (input)
readInput := {
val context = (interactionService in Global).value
context.readLine("> ", mask=false)
}
Problem: commands block
When running the `run` task, all other build
commands are blocked.
Background Jobs
● tasks can "fork" background jobs in server
● clients can discover/connect/stop with
background jobs.
server
background
job service
Forked "run"
of application
scala REPL
ensime
server?
not implemented
Input/Interaction Design
● Commands
o The requesting client provides the means for
terminal interaction
o Logs and stderr/stdout are sent to all clients.
● Background Jobs (Not Completed Yet)
o Any client can take terminal interaction for a
background task
o opt-in for getting stdout/stderr events
Problem: Existing plugins
Current plugins + sbt should be able to try out
sbt server, without breaking anything
sbt-core-next
A new, optional, plugin for sbt 0.13.x series
which provides the new services for sbt-server
● BackgroundRunService
● InteractionServicePlugin
● SendEventServicePlugin
● SerializersService
In use in Play 2.4
What's next?
sbt-server TODOS
● Interaction Improvements
o readline abstraction (a.k.a. Scala REPL support)
o Background Job hooks
● meta-project as first class citizen
o replace `reload plugins` command for first-class
support
o ~ as first-class citizen in server
● kill-on-bad-state watchdog
How can I help?
http://www.scala-sbt.org/community.html#how-can-I-help
● Follow @scala_sbt
● Contribute to StackOverflow sbt tag
● Report bugs
● Create plugins
o Try/migrate your plugins to remote APIs: https://github.com/sbt/sbt-
core-next
● Patch the core
o Github Community labeled issues
o Subscribe to sbt-dev list
o Join the discussion on Gitter sbt/sbt

More Related Content

What's hot

Flyway: The agile database migration framework for Java
Flyway: The agile database migration framework for JavaFlyway: The agile database migration framework for Java
Flyway: The agile database migration framework for Java
Axel Fontaine
 

What's hot (20)

Microservices in Scala: Play Framework
Microservices in Scala: Play FrameworkMicroservices in Scala: Play Framework
Microservices in Scala: Play Framework
 
Ansible, integration testing, and you.
Ansible, integration testing, and you.Ansible, integration testing, and you.
Ansible, integration testing, and you.
 
Empowering developers to deploy their own data stores
Empowering developers to deploy their own data storesEmpowering developers to deploy their own data stores
Empowering developers to deploy their own data stores
 
Node.js Development with Apache NetBeans
Node.js Development with Apache NetBeansNode.js Development with Apache NetBeans
Node.js Development with Apache NetBeans
 
De Java 8 a Java 11 y 14
De Java 8 a Java 11 y 14De Java 8 a Java 11 y 14
De Java 8 a Java 11 y 14
 
Apache Camel in the belly of the Docker whale
Apache Camel in the belly of the Docker whaleApache Camel in the belly of the Docker whale
Apache Camel in the belly of the Docker whale
 
Nashorn: JavaScript that doesn't suck - Tomer Gabel, Wix
Nashorn: JavaScript that doesn't suck - Tomer Gabel, WixNashorn: JavaScript that doesn't suck - Tomer Gabel, Wix
Nashorn: JavaScript that doesn't suck - Tomer Gabel, Wix
 
Flyway: The agile database migration framework for Java
Flyway: The agile database migration framework for JavaFlyway: The agile database migration framework for Java
Flyway: The agile database migration framework for Java
 
Exploring Java Heap Dumps (Oracle Code One 2018)
Exploring Java Heap Dumps (Oracle Code One 2018)Exploring Java Heap Dumps (Oracle Code One 2018)
Exploring Java Heap Dumps (Oracle Code One 2018)
 
Polygot Java EE on the GraalVM
Polygot Java EE on the GraalVMPolygot Java EE on the GraalVM
Polygot Java EE on the GraalVM
 
Utilizing the OpenNTF Domino API
Utilizing the OpenNTF Domino APIUtilizing the OpenNTF Domino API
Utilizing the OpenNTF Domino API
 
A Tour of the Modern Java Platform
A Tour of the Modern Java PlatformA Tour of the Modern Java Platform
A Tour of the Modern Java Platform
 
Building Distributed Systems in Scala
Building Distributed Systems in ScalaBuilding Distributed Systems in Scala
Building Distributed Systems in Scala
 
Javantura v3 - ES6 – Future Is Now – Nenad Pečanac
Javantura v3 - ES6 – Future Is Now – Nenad PečanacJavantura v3 - ES6 – Future Is Now – Nenad Pečanac
Javantura v3 - ES6 – Future Is Now – Nenad Pečanac
 
Faster Java EE Builds with Gradle
Faster Java EE Builds with GradleFaster Java EE Builds with Gradle
Faster Java EE Builds with Gradle
 
[오픈소스컨설팅] 쿠버네티스와 쿠버네티스 on 오픈스택 비교 및 구축 방법
[오픈소스컨설팅] 쿠버네티스와 쿠버네티스 on 오픈스택 비교  및 구축 방법[오픈소스컨설팅] 쿠버네티스와 쿠버네티스 on 오픈스택 비교  및 구축 방법
[오픈소스컨설팅] 쿠버네티스와 쿠버네티스 on 오픈스택 비교 및 구축 방법
 
Getting started with agile database migrations for java flywaydb
Getting started with agile database migrations for java flywaydbGetting started with agile database migrations for java flywaydb
Getting started with agile database migrations for java flywaydb
 
Modular Java applications with OSGi on Apache Karaf
Modular Java applications with OSGi on Apache KarafModular Java applications with OSGi on Apache Karaf
Modular Java applications with OSGi on Apache Karaf
 
De Java 8 a Java 17
De Java 8 a Java 17De Java 8 a Java 17
De Java 8 a Java 17
 
Developing distributed applications with Akka and Akka Cluster
Developing distributed applications with Akka and Akka ClusterDeveloping distributed applications with Akka and Akka Cluster
Developing distributed applications with Akka and Akka Cluster
 

Viewers also liked

Viewers also liked (9)

The state of sbt 0.13, sbt server, and sbt 1.0 (ScalaMatsuri ver)
The state of sbt 0.13, sbt server, and sbt 1.0 (ScalaMatsuri ver)The state of sbt 0.13, sbt server, and sbt 1.0 (ScalaMatsuri ver)
The state of sbt 0.13, sbt server, and sbt 1.0 (ScalaMatsuri ver)
 
SBT Concepts, part 2
SBT Concepts, part 2SBT Concepts, part 2
SBT Concepts, part 2
 
Road to sbt 1.0: Paved with server (2015 Amsterdam)
Road to sbt 1.0: Paved with server (2015 Amsterdam)Road to sbt 1.0: Paved with server (2015 Amsterdam)
Road to sbt 1.0: Paved with server (2015 Amsterdam)
 
SBT Crash Course
SBT Crash CourseSBT Crash Course
SBT Crash Course
 
Innovating faster with SBT, Continuous Delivery, and LXC
Innovating faster with SBT, Continuous Delivery, and LXCInnovating faster with SBT, Continuous Delivery, and LXC
Innovating faster with SBT, Continuous Delivery, and LXC
 
Hadoop Summit 2013 : Continuous Integration on top of hadoop
Hadoop Summit 2013 : Continuous Integration on top of hadoopHadoop Summit 2013 : Continuous Integration on top of hadoop
Hadoop Summit 2013 : Continuous Integration on top of hadoop
 
The state of sbt 0.13, sbt server, and sbt 1.0 (ScalaSphere ver)
The state of sbt 0.13, sbt server, and sbt 1.0 (ScalaSphere ver)The state of sbt 0.13, sbt server, and sbt 1.0 (ScalaSphere ver)
The state of sbt 0.13, sbt server, and sbt 1.0 (ScalaSphere ver)
 
Continuous Integration for Spark Apps by Sean McIntyre
Continuous Integration for Spark Apps by Sean McIntyreContinuous Integration for Spark Apps by Sean McIntyre
Continuous Integration for Spark Apps by Sean McIntyre
 
Build application using sbt
Build application using sbtBuild application using sbt
Build application using sbt
 

Similar to Road to sbt 1.0 paved with server

Why kernelspace sucks?
Why kernelspace sucks?Why kernelspace sucks?
Why kernelspace sucks?
OpenFest team
 
Counterclockwise past present future
Counterclockwise  past present futureCounterclockwise  past present future
Counterclockwise past present future
lolopetit
 

Similar to Road to sbt 1.0 paved with server (20)

차세대컴파일러, VM의미래: 애플 오픈소스 LLVM
차세대컴파일러, VM의미래: 애플 오픈소스 LLVM차세대컴파일러, VM의미래: 애플 오픈소스 LLVM
차세대컴파일러, VM의미래: 애플 오픈소스 LLVM
 
Nodejs
NodejsNodejs
Nodejs
 
Why kernelspace sucks?
Why kernelspace sucks?Why kernelspace sucks?
Why kernelspace sucks?
 
.NET Applications & Cloud Meetup at 7 Peaks
.NET Applications & Cloud Meetup at 7 Peaks.NET Applications & Cloud Meetup at 7 Peaks
.NET Applications & Cloud Meetup at 7 Peaks
 
Scala, docker and testing, oh my! mario camou
Scala, docker and testing, oh my! mario camouScala, docker and testing, oh my! mario camou
Scala, docker and testing, oh my! mario camou
 
Spring Boot 3 And Beyond
Spring Boot 3 And BeyondSpring Boot 3 And Beyond
Spring Boot 3 And Beyond
 
DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
DevOpsDays Taipei 2019 - Mastering IaC the DevOps WayDevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
DevOpsDays Taipei 2019 - Mastering IaC the DevOps Way
 
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orches...
 
OSDC 2016 | rkt and Kubernetes: What’s new with Container Runtimes and Orches...
OSDC 2016 | rkt and Kubernetes: What’s new with Container Runtimes and Orches...OSDC 2016 | rkt and Kubernetes: What’s new with Container Runtimes and Orches...
OSDC 2016 | rkt and Kubernetes: What’s new with Container Runtimes and Orches...
 
Toolbox of a Ruby Team
Toolbox of a Ruby TeamToolbox of a Ruby Team
Toolbox of a Ruby Team
 
Makefile for python projects
Makefile for python projectsMakefile for python projects
Makefile for python projects
 
Dart the Better JavaScript
Dart the Better JavaScriptDart the Better JavaScript
Dart the Better JavaScript
 
[KubeCon NA 2018] Telepresence Deep Dive Session - Rafael Schloming & Luke Sh...
[KubeCon NA 2018] Telepresence Deep Dive Session - Rafael Schloming & Luke Sh...[KubeCon NA 2018] Telepresence Deep Dive Session - Rafael Schloming & Luke Sh...
[KubeCon NA 2018] Telepresence Deep Dive Session - Rafael Schloming & Luke Sh...
 
(1) c sharp introduction_basics_dot_net
(1) c sharp introduction_basics_dot_net(1) c sharp introduction_basics_dot_net
(1) c sharp introduction_basics_dot_net
 
Настройка окружения для кросскомпиляции проектов на основе docker'a
Настройка окружения для кросскомпиляции проектов на основе docker'aНастройка окружения для кросскомпиляции проектов на основе docker'a
Настройка окружения для кросскомпиляции проектов на основе docker'a
 
Counterclockwise past present future
Counterclockwise  past present futureCounterclockwise  past present future
Counterclockwise past present future
 
Modern Web-site Development Pipeline
Modern Web-site Development PipelineModern Web-site Development Pipeline
Modern Web-site Development Pipeline
 
MoldCamp - multidimentional testing workflow. CIBox.
MoldCamp  - multidimentional testing workflow. CIBox.MoldCamp  - multidimentional testing workflow. CIBox.
MoldCamp - multidimentional testing workflow. CIBox.
 
(phpconftw2012) PHP as a Middleware in Embedded Systems
(phpconftw2012) PHP as a Middleware in Embedded Systems(phpconftw2012) PHP as a Middleware in Embedded Systems
(phpconftw2012) PHP as a Middleware in Embedded Systems
 
DrupalCon Los Angeles - Continuous Integration Toolbox
DrupalCon Los Angeles - Continuous Integration ToolboxDrupalCon Los Angeles - Continuous Integration Toolbox
DrupalCon Los Angeles - Continuous Integration Toolbox
 

More from Eugene Yokota (9)

Equality in Scala (ScalaMatsuri 2020)
Equality in Scala (ScalaMatsuri 2020)Equality in Scala (ScalaMatsuri 2020)
Equality in Scala (ScalaMatsuri 2020)
 
sbt: core concepts and updates (Scala Love 2020)
sbt: core concepts and updates (Scala Love 2020)sbt: core concepts and updates (Scala Love 2020)
sbt: core concepts and updates (Scala Love 2020)
 
Analysis of Zinc (nescala 2020)
Analysis of Zinc (nescala 2020)Analysis of Zinc (nescala 2020)
Analysis of Zinc (nescala 2020)
 
Analysis of Zinc (ScalaSphere 2019)
Analysis of Zinc (ScalaSphere 2019)Analysis of Zinc (ScalaSphere 2019)
Analysis of Zinc (ScalaSphere 2019)
 
sbt core concepts (ScalaMatsuri 2019)
sbt core concepts (ScalaMatsuri 2019)sbt core concepts (ScalaMatsuri 2019)
sbt core concepts (ScalaMatsuri 2019)
 
pull requests I sent to scala/scala (ny-scala 2019)
pull requests I sent to scala/scala (ny-scala 2019)pull requests I sent to scala/scala (ny-scala 2019)
pull requests I sent to scala/scala (ny-scala 2019)
 
sbt 1
sbt 1sbt 1
sbt 1
 
sbt server (LSP discussion, 2018 Jan)
sbt server (LSP discussion, 2018 Jan)sbt server (LSP discussion, 2018 Jan)
sbt server (LSP discussion, 2018 Jan)
 
Thinking in Cats
Thinking in CatsThinking in Cats
Thinking in Cats
 

Recently uploaded

JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
Max Lee
 

Recently uploaded (20)

Breaking the Code : A Guide to WhatsApp Business API.pdf
Breaking the Code : A Guide to WhatsApp Business API.pdfBreaking the Code : A Guide to WhatsApp Business API.pdf
Breaking the Code : A Guide to WhatsApp Business API.pdf
 
AI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in MichelangeloAI/ML Infra Meetup | ML explainability in Michelangelo
AI/ML Infra Meetup | ML explainability in Michelangelo
 
IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024IT Software Development Resume, Vaibhav jha 2024
IT Software Development Resume, Vaibhav jha 2024
 
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
 
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)
 
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1
 
How to install and activate eGrabber JobGrabber
How to install and activate eGrabber JobGrabberHow to install and activate eGrabber JobGrabber
How to install and activate eGrabber JobGrabber
 
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purityAPVP,apvp apvp High quality supplier safe spot transport, 98% purity
APVP,apvp apvp High quality supplier safe spot transport, 98% purity
 
The Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion ProductionThe Impact of PLM Software on Fashion Production
The Impact of PLM Software on Fashion Production
 
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG
 
INGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by DesignINGKA DIGITAL: Linked Metadata by Design
INGKA DIGITAL: Linked Metadata by Design
 
5 Reasons Driving Warehouse Management Systems Demand
5 Reasons Driving Warehouse Management Systems Demand5 Reasons Driving Warehouse Management Systems Demand
5 Reasons Driving Warehouse Management Systems Demand
 
A Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data MigrationA Guideline to Zendesk to Re:amaze Data Migration
A Guideline to Zendesk to Re:amaze Data Migration
 
Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024Secure Software Ecosystem Teqnation 2024
Secure Software Ecosystem Teqnation 2024
 
JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)JustNaik Solution Deck (stage bus sector)
JustNaik Solution Deck (stage bus sector)
 
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdfA Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf
 
Designing for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web ServicesDesigning for Privacy in Amazon Web Services
Designing for Privacy in Amazon Web Services
 
CompTIA Security+ (Study Notes) for cs.pdf
CompTIA Security+ (Study Notes) for cs.pdfCompTIA Security+ (Study Notes) for cs.pdf
CompTIA Security+ (Study Notes) for cs.pdf
 
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfMicrosoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
 
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfImplementing KPIs and Right Metrics for Agile Delivery Teams.pdf
Implementing KPIs and Right Metrics for Agile Delivery Teams.pdf
 

Road to sbt 1.0 paved with server

  • 1. The road to sbt 1.0 Paved with Server
  • 2. Who are we? @eed3si9n (Eugene Yokota) ● sbt core dev ● scalaxb ● treehugger.scala ● "learning Scalaz" ● sbt-assembly etc Josh Suereth ● Big Nerd ● Leader of tools team @ Typesafe ● Author o Scala in Depth o sbt in Action
  • 3. Agenda ● A brief (incomplete and mostly wrong) history of Build Tools ● A roadmap to sbt 1.0 o Modularization ● sbt-server
  • 4. A brief history of build tools Incomplete and mostly wrong
  • 6. $ make -j 2 hello # Makefile OBJ = test.o test2.o %.o : %.c $(CC) -c -o $@ < $< hello : $(OBJ) gcc -o $@ $^
  • 7. Make (1976) ● new! Dependency-based programming ● Uses system command to carry out tasks ● problem! Machine-dependent Q: How do I build this program?
  • 8. Autoconf/Automake (1991/1996) ● Generate configure script and Makefile templates o Configure generates header scripts and real makefile o Configure discovers what the machine has ● Use an even MORE arcane syntax o m4 macros o magic strings Q: How do I build on this machine?
  • 10. $ ant dist <project name="MyProject" default="dist" basedir="."> <description>example build file</description> <property name="src" location="src"/> <property name="build" location="build"/> <property name="dist" location="dist"/> <target name="compile" description="compile the source " > <javac srcdir="${src}" destdir="${build}"/> </target> <target name="dist" depends="compile" description="generate the distribution" > <mkdir dir="${dist}/lib"/> <jar jarfile="${dist}/lib/MyProject-${DSTAMP}.jar" basedir="${build}"/> </target> </project>
  • 11. Ant (2000) Q: How do I build on this machine? ● new! Built-in tasks like mkdir, javac, and jar o Self-contained and platform-independent ● task plugins (like ant-contrib) ● problem! Hard to reuse build logic
  • 12. Package managers Q: How do I find library source/binary? ● new! Metadata to track library dependencies ● new! Repository to host source and binary ● CTAN (1993), Port (1994), RPM (1997), APT (1998)
  • 13. Maven (2004) Q: How do I find library source/binary? Q: How do I enforce consistent builds? ● Metadata to track library dependencies ● Repository to host source and binary ● new! Default build (Convention over Configuration) ● new! Plugins that allowed reuse of build logic ● problem! Difficult to customize
  • 14. Ivy (2004) Q: How do I find library source/binary? ● Metadata to track library dependencies ● Repository to host source and binary ● Maven integration on top of Ant
  • 15. Rake (2003) rule '.o' => ['.c'] do |t| sh "cc #{t.source} -c -o #{t.name}" end task :name, [:first_name, :last_name] do |t, args| puts "First name is #{args.first_name}" puts "Last name is #{args.last_name}" end
  • 16. Rake (2004) Q: How do I build on this machine? Q: How do I customize my build? ● new! Internal DSL o Leverage Ruby libraries rather than shell programs o Provide cross-platform ruby libraries for supported language tools ● Real programming language to write tasks
  • 17. O NOES XML! SAX parsing is declared uncool (immediately after inception) REST/JSON attempt to displace SOAP
  • 18. Buildr (2008)/Gradle (2009)/ Leningen(2009) ● Maven integration Q: How do I find library source/binary? Q: How do I customize my build? ● Use an internal DSL and a real program language
  • 19. sbt (2008) Q: How do I find library source/binary? Q: How do I customize my build? Q: How can I develop faster? ● new! Incremental compiler tracking source deps ● new! Interactive shell + library-aware REPL ● new! Test framework abstraction ● new! Parallel by default ● shabby-chic! ANSI colors ● Internal DSL + Maven integration
  • 20. Google Blaze (2009) Q: How can I develop faster? Q: How do I avoid version conflicts? ● new! Incremental build ● new! Cache builds remotely ● new! Clustered Building ● Abandon Maven. Check in all source to version control. Force everyone on the same version. ● 1 SCM repo for all projects ● Assume homogeneous-ish environment
  • 21. Pants (2014)/Buck (2014) Q: How can I develop faster? Q: How do I avoid version conflicts? ● Incremental build ● Cache builds remotely ● Force all dependencies to the same version for all projects ● Put everything in a mono-repository
  • 22. History make (1977) automake (1996)/ autoconf (1991) Rake (2003) Maven (2004) Ivy (2004) sbt (2008) Blaze (2009) Gradle (2009) Leiningen (2009) Pants (2014) Buck (2014) RPM (1997) Ant (2000) Buildr (2008) Jon Pretty's shell scripts (2004-2015)
  • 23. History Recap ● sbt has inherited: o dependency based programming (Make) o internal build DSL (Rake) o Package/Library Management (Maven) o Convention over configuration (Maven) o Re-usable build flow (Maven) ● sbt brings: o interactivity (on the shell) o parallel by default
  • 24. A roadmap to sbt 1.0 stability + sbt server => modularization
  • 25. Stability 1. Conceptual stability 2. Binary compatibility of sbt plugins 3. Source compatibility of build files
  • 26. Concepts (mostly stable) 1. Scala incremental compiler 2. Dependency manager (Scala-aware) 3. Task and plugin system (using Scala) 4. Test framework abstraction 5. Text-based interactive shell 5. sbt server + client(s)
  • 27. Plugin binary compatibility ● sbt 0.13 maintained 18 months of bincompat o Lots of hacks and effort. Unable to remove cruft. ● sbt 1.x.y should be bincompat with 1.0.0 ● Need to minimize surface API o able to add small features when requested
  • 28. Build source compatibility ● Source compatibility of your build.sbt ● sbt 1.x.y should be stable ● sbt 1.0 gives us opportunity to break DSL o Deprecate project/build.scala? o Unify sbt shell key syntax w/ Scala DSL syntax
  • 29. Modularization Componentize stable features, innovate new ideas 1. Pull out cohesive subprojects 2. Distinguish public API and internal details 3. Document usages 4. Clean up historical code 5. Cross publish for latest scala versions (if applicable) 6. Publish to Maven Central (or JCenter)
  • 30. Module candidates ● Launcher API ● Serialization API ● Compiler/REPL API ● Test framework API ● Dependency Management API ● Network API ● IO API ● Task DSL ● Completion API ● sbt client (sbt-server)
  • 32. The problem ● Many things want access to the build ● We need to centrally control build-related tasks to avoid breakage. o intellij auto-import + "sbt ~ test" o activator + "play run"
  • 33. sbt-server design sbt-server disk (target/) sbt client commands / watches changes Events (logs, status, custom)
  • 34. Before server - Command Engine CMD (compile) Engine (sequential) Task Engine (parallel) Read next command from stdin Next command Read previous log file Reload the build Terminal Client
  • 35. After server - Execution Queue CMD (compile) Engine TasksRead server Queue Next command previous log file Reload the build Server Event Loop Request Queue CommandQueue Client #1 Client #2 LatestState Next State
  • 36. Problem: Server discovery Who starts the sbt-server, how do we know it is running, how do we connect?
  • 37. sbt-launcher ● sbt 0.13.2 launcher had support for "servers" o launch config specifies "lock" file which prevents more than one service being started o "lock" file contains location (ip / port) of the service o server is tested (ping) for liveness by the launcher.
  • 38. Problem: Disconnects ● Client may not be the one to start sbt ● Client may disconnect from server, or server may crash
  • 39. Connect as an Event val connector = SbtConnector( "terminal", "Command Line Terminal", configuration.baseDirectory) def onConnect(client: SbtClient): Unit = { client handleEvents { … } client watch ... } connector.open(onConnect, onError)(<execution context>)
  • 40. Connect as an Event val connector = SbtConnector( "terminal", "Command Line Terminal", configuration.baseDirectory) def onConnect(client: SbtClient): Unit = { client handleEvents { … } client watch ... } connector.open(onConnect, onError)(<execution context>) Clients reconstruct their watches and restore their view of the build on any reconnect
  • 41. Problem: protocol We needed a good mechanism for client + server to communicate. ● Something that can evolve and maintain compatibility ● Something that is easy for plugin authors to extend.
  • 42. sbt-serialization ● New library based on Scala Pickling o JSON - currently supported format. ● sbt-remote-control has versioned protocol and tests. ● (coming) plugin *Keys as protocol o remote clients can watch/get values o require pickler/unpickler when creating *Key o ??? release plugin "client" jar separate from impl
  • 43. Problem: Input sbt tasks may want the user to type input
  • 44. Example Client (input) client.requestExecution( "run", Some(TerminalInteraction -> TerminalThread) ) client server ExecutionRequest
  • 45. Example task (input) readInput := { val context = (interactionService in Global).value context.readLine("> ", mask=false) }
  • 46. Problem: commands block When running the `run` task, all other build commands are blocked.
  • 47. Background Jobs ● tasks can "fork" background jobs in server ● clients can discover/connect/stop with background jobs. server background job service Forked "run" of application scala REPL ensime server? not implemented
  • 48. Input/Interaction Design ● Commands o The requesting client provides the means for terminal interaction o Logs and stderr/stdout are sent to all clients. ● Background Jobs (Not Completed Yet) o Any client can take terminal interaction for a background task o opt-in for getting stdout/stderr events
  • 49. Problem: Existing plugins Current plugins + sbt should be able to try out sbt server, without breaking anything
  • 50. sbt-core-next A new, optional, plugin for sbt 0.13.x series which provides the new services for sbt-server ● BackgroundRunService ● InteractionServicePlugin ● SendEventServicePlugin ● SerializersService In use in Play 2.4
  • 52. sbt-server TODOS ● Interaction Improvements o readline abstraction (a.k.a. Scala REPL support) o Background Job hooks ● meta-project as first class citizen o replace `reload plugins` command for first-class support o ~ as first-class citizen in server ● kill-on-bad-state watchdog
  • 53. How can I help? http://www.scala-sbt.org/community.html#how-can-I-help ● Follow @scala_sbt ● Contribute to StackOverflow sbt tag ● Report bugs ● Create plugins o Try/migrate your plugins to remote APIs: https://github.com/sbt/sbt- core-next ● Patch the core o Github Community labeled issues o Subscribe to sbt-dev list o Join the discussion on Gitter sbt/sbt

Editor's Notes

  1. This is one of the earliest build script known to human history, which builds System V on PDP-11.
  2. The grandfather of all build tools. In April, 1976, Stuart Feldman found a precious golden ring, and the tab key. He used these to conjure forth Make, a tool of arcane syntax and great power.
  3. Internet happens. Most of the credit is given to HTML.
  4. XML naturally takes over. This is the 2nd generation build tool.
  5. Adds metadata on top of Makefile. Takes advantage of Internet. Not "build tool" in the strict sense but becomes inspiration.
  6. 2004 is a major inflection point in the history of build tools. Combines the ideas from package managers + self-contained tasks. XML.
  7. In 2003 Jim Weirich creates Rake. A Make-like build tool using Ruby.
  8. 4th generation of build tools. Rake revolution hits JVM.
  9. Internal build tool of Google.
  10. 5th generation of build tools. Inspired by Google Blaze Buck is focused on Android development and optimising hot-paths in android dexing Pants is a Twitter/Foursquare project, aimed at Java/Scala and written in python.
  11. Stability from the user experience perspective. No need to learn and re-learn.
  12. sbt plugin is a huge asset to the ecosystem. We care a lot of the binary compatibility of the plugins.
  13. Scala Process API have moved to standard lib We'll be using more external libraries like Scala Pickling and Scala ARM.