Ever since we broke apart the front and back-end of our systems, we’ve longed to partially reunite them with a shared language. The benefits of code reuse and shared tooling are compelling but is this nirvana possible? In this session we will explore building both the front (mobile and web) and back-end of an application with a shared Kotlin codebase. You will learn how to setup the build, share code, and deploy the back-end as a serverless app.

1. Kotlin
Mullets
Business on the Front End, Party in Back
Chet Haase, Developer Advocate, Android
James Ward, Developer Advocate, Google Cloud

2. Demo!

3. Track
● Track
Motion
Events
Record
● Add
Locations
to Path(s)
View
● Invalidate
ML
● Ready to
Analyze
Draw
● Draw Path
Data
User Drawings

4. UI Layout
<androidx.constraintlayout.widget.ConstraintLayout ...>
<Spinner ... />
<Button ... />
<Button ... />
<ToggleButton... />
<Button ... />
<TextView... />
<TextView ... />
<TextView ... />
<TextView ... />
<com.jamesward.airdraw.DrawingCanvas ... />
</androidx.constraintlayout.widget.ConstraintLayout>

5. UI Layout, with custom view
<androidx.constraintlayout.widget.ConstraintLayout ...>
< ... />
<com.jamesward.airdraw.DrawingCanvas
android:id="@+id/drawingCanvas"
android:layout_width="0dp"
android:layout_height="0dp"
android:layout_margin="8dp"
android:background="@android:color/black"
app:layout_constraintDimensionRatio="1:1"
app:layout_constraintEnd_toEndOf="@id/rightGuide"
app:layout_constraintHorizontal_bias="0.0"
app:layout_constraintStart_toStartOf="@id/leftGuide"
app:layout_constraintTop_toBottomOf="@+id/fourthGuess" />
</androidx.constraintlayout.widget.ConstraintLayout>

6. DrawingCanvas
class DrawingCanvas(context: Context?, attr: AttributeSet) :
View(context, attr) {
val paint: Paint = Paint(Paint.ANTI_ALIAS_FLAG)
private val drawingPath = Path()
fun getBitmap():Bitmap { ... }
fun setBitmap(bitmap: Bitmap) { ... }
override fun onTouchEvent(event: MotionEvent?): Boolean { ... }
fun clear() { ... }
override fun onDraw(canvas: Canvas?) { ... }
}

7. Track Motion Events
override fun onTouchEvent(event: MotionEvent?): Boolean {
var handled = false
if (event != null) {
when {
event.action == MotionEvent.ACTION_DOWN -> {
drawingPath.moveTo(event.x, event.y)
drawingPath.lineTo(event.x, event.y)
invalidate()
handled = true
}
event.action == MotionEvent.ACTION_MOVE -> {
drawingPath.lineTo(event.x, event.y)
invalidate()
handled = true
}
event.action == MotionEvent.ACTION_UP -> {
handled = true
}
}
}
return handled
}

8. Draw Path
override fun onDraw(canvas: Canvas?) {
canvas?.drawPath(drawingPath, paint)
}

9. Interlude: Learning About Machine Learning
Tensor FlowTraining Data Model
TFLite Model
On-Device
Model
User Data
Results

10. Interlude: ML
Results
Labels
Barcodes
Text
Objects
Faces
Translations
+ Confidence Values

11. Or…. Use Built-in Models
Tensor FlowTraining Data Model
TFLite Model
On-Device
Model
User Data
Results

12. Convert Drawing
● Extract
Bitmap
Setup Labeler
● Create
Firebase
Vision
Image
● Create On-
Device
Labeler
Send Data
● Process
Data
Receive
Results
● onSuccess
callback
receives
results
Get Image Labels
Display
Results
● Show
Labels and
Confidence

13. Get Bitmap Data
fun getBitmap():Bitmap {
val bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
val canvas = Canvas(bitmap)
draw(canvas)
return bitmap
}

14. KTX

15. Get Bitmap Data
fun getBitmap():Bitmap {
val bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
val canvas = Canvas(bitmap)
draw(canvas)
return bitmap
}

16. Get Bitmap Data
fun getBitmap() = createBitmap(width, height).applyCanvas(::draw)
fun getBitmap():Bitmap {
val bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
val canvas = Canvas(bitmap)
draw(canvas)
return bitmap
}

17. Call Image Labeler
val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap)
val labeler = FirebaseVision.getInstance().onDeviceImageLabeler
labeler.processImage(firebaseImage).addOnSuccessListener {
// . . .
}
{
{“label”, confidence},
{“label”, confidence},
…
}

18. Digital Processing: MNIST
Image: ©Josef Steppan, https://commons.wikimedia.org/wiki/File:MnistExamples.png
Modified National Institute of Standards and Technology
Hand-written digits for image-processing

19. Create Interpreter
● Load model
● Create
Interpreter
with model
Get Bitmap
● Get user
data as
bitmap
● Resize to
match
model
Run Interpreter
● Process
data
Process
Results
● Sort results
by
confidence
Get Image Labels
Display
Results
● Show
results and
confidence

20. Get Model as ByteBuffer
val model = loadModelFile()
private fun loadModelFile(): ByteBuffer {
val fileDescriptor = assets.openFd("mnist.tflite")
val inputStream = FileInputStream(fileDescriptor.fileDescriptor)
val fileChannel = inputStream.channel
val startOffset = fileDescriptor.startOffset
val declaredLength = fileDescriptor.declaredLength
return fileChannel.map(FileChannel.MapMode.READ_ONLY,
startOffset, declaredLength)
}

21. Initialize Interpreter, Sizes
val options = Interpreter.Options()
options.setUseNNAPI(true)
val interpreter = Interpreter(model, options)
val inputShape = interpreter.getInputTensor(0).shape()
inputImageWidth = inputShape[1]
inputImageHeight = inputShape[2]
modelInputSize = 4 * inputImageWidth * inputImageHeight

22. Get Results
private fun detectDigit(bitmap: Bitmap,
resultsListener: (List<LabelAnnotation>) -> Unit) {
lateinit var digitSequence: List<Pair<String,Float>>
val resizedImage = Bitmap.createScaledBitmap(bitmap,
inputImageWidth, inputImageHeight, true)
val byteBuffer = convertBitmapToByteBuffer(resizedImage)
val result = Array(1) { FloatArray(10) }
interpreter.run(byteBuffer, result)
// format and display results
}

23. Demo!

24. KTS

25. Type-safe Builds
dependencies {
implementation(kotlin("stdlib-jdk8"))
implementation("androidx.appcompat:appcompat:1.1.0")
...
}
build.gradle.kts
fun Project.dependencies(configuration: DependencyHandlerScope.() -> Unit) =
DependencyHandlerScope.of(dependencies).configuration()
fun DependencyHandler.`implementation`(dependencyNotation: Any): Dependency? =
add("implementation", dependencyNotation)

26. Declarative(ish) & Imperative
if (startParameter.taskRequests.find { it.args.contains(":server:shadowJar") } == null) {
include("common", "android", "web", "server")
} else {
include("common", "web", "server")
}
settings.gradle.kts

27. ML in the Cloud
val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap)
val labeler = FirebaseVision.getInstance().onDeviceImageLabeler
labeler.processImage(firebaseImage).addOnSuccessListener { … }
On-Device Labeling

28. ML in the Cloud
val firebaseImage = FirebaseVisionImage.fromBitmap(bitmap)
val labeler = FirebaseVision.getInstance().cloudImageLabeler
labeler.processImage(firebaseImage).addOnSuccessListener { … }
Cloud Labeling

29. Demo!

30. Coroutines

31. Get Results
private fun detectDigit(bitmap: Bitmap,
resultsListener: (List<LabelAnnotation>) -> Unit) {
lateinit var digitSequence: List<Pair<String,Float>>
val resizedImage = Bitmap.createScaledBitmap(bitmap,
inputImageWidth, inputImageHeight, true)
val byteBuffer = convertBitmapToByteBuffer(resizedImage)
val result = Array(1) { FloatArray(10) }
interpreter.run(byteBuffer, result)
// format and display results
}

32. Get Results
private fun detectDigit(bitmap: Bitmap,
resultsListener: (List<LabelAnnotation>) -> Unit) {
lateinit var digitSequence: List<Pair<String,Float>>
val resizedImage = Bitmap.createScaledBitmap(bitmap,
inputImageWidth, inputImageHeight, true)
val byteBuffer = convertBitmapToByteBuffer(resizedImage)
val result = Array(1) { FloatArray(10) }
interpreter.run(byteBuffer, result)
// format and display results
}

33. Get Results
private suspend fun detectDigit(bitmap: Bitmap,
resultsListener: (List<LabelAnnotation>) -> Unit) {
lateinit var digitSequence: List<Pair<String,Float>>
val resizedImage = Bitmap.createScaledBitmap(bitmap,
inputImageWidth, inputImageHeight, true)
val byteBuffer = convertBitmapToByteBuffer(resizedImage)
val result = Array(1) { FloatArray(10) }
withContext(Dispatchers.IO) {
interpreter.run(byteBuffer, result)
}
// format and display results
}

34. Server-Side

36. Draw
● Capture
Bitmap
Phone ML
● Predict with
Local Model
● Display
Results
REST Service
● Send
Bitmap &
Results to
REST API
Browser
● Poll d' Bus
● Display
Bitmap &
Results
Pub/Sub
● On d' Bus
Browser
Bus
Server

37. Shared Common
Root Gradle Project
Common
ServerAndroid
data class LabelAnnotation(
val description: String,
val score: Float)
data class ImageResult(
val image: ByteArray,
val labelAnnotations: List<LabelAnnotation>)
common/src/commonMain/kotlin/Data.kt

38. Multiplatform Common
plugins {
id("com.android.library")
kotlin("multiplatform")
}
java {
sourceCompatibility = JavaVersion.VERSION_1_8
}
android {
defaultConfig {
compileSdkVersion(29)
}
}
common/build.gradle.ktx

39. Multiplatform Common
kotlin {
sourceSets {
commonMain {
dependencies {
implementation(kotlin("stdlib"))
}
}
}
jvm {
val main by compilations.getting {
kotlinOptions {
jvmTarget = JavaVersion.VERSION_1_8
}
}
}
android()
}
common/build.gradle.ktx

40. Multiplatform Common Dependency
dependencies {
implementation(project(":common"))
}
android/build.gradle.ktx
server/build.gradle.kts

42. Cloud Service - Deps
plugins {
kotlin("kapt")
}
dependencies {
implementation("io.micronaut:micronaut-runtime:1.2.6")
implementation("io.micronaut:micronaut-http-server-netty:1.2.6")
implementation("io.micronaut:micronaut-views:1.2.0")
api("ch.qos.logback:logback-classic:1.2.3")
api("com.fasterxml.jackson.module:jackson-module-kotlin:2.10.1")
api("org.thymeleaf:thymeleaf:3.0.11.RELEASE")
kapt("io.micronaut:micronaut-inject-java:1.2.6")
}
server/build.gradle.ktx

43. Cloud Service
@Post("/show")
fun show(@Body imageResult: ImageResult): HttpResponse<String> {
bus.put(imageResult)
return HttpResponse.ok("")
}
@Get("/events")
fun events(): HttpResponse<ImageResult> {
val maybe = bus.take()
return if (maybe != null)
HttpResponse.ok(maybe)
else
HttpResponse.noContent()
}
server/src/main/kotlin/WebApp.kt

44. Client
@Client("${drawurl}")
interface DrawService {
@Post("/show")
fun show(@Body imageResult: ImageResult): Single<Unit>
}
@Inject
var drawService: DrawService? = null
drawService?.show(imageResult)?.subscribe()?.dispose()
android/src/main/kotlin/MainActivity.kt

45. Setup DI
class BaseApplication: Application() {
private var ctx: ApplicationContext? = null
override fun onCreate() {
super.onCreate()
val pm = applicationContext.packageManager
val ai = pm.getApplicationInfo(packageName, PackageManager.GET_META_DATA)
val propertySource = AndroidMetadataPropertySource(ai.metaData)
val appCtxBuilder = applicationContext.build(MainActivity::class.java, Environment.ANDROID)
ctx = appCtxBuilder.propertySources(propertySource).start()
registerActivityLifecycleCallbacks(object: ActivityLifecycleCallbacks {
override fun onActivityCreated(activity: Activity?, bundle: Bundle?) {
if (activity != null) ctx?.inject(activity)
}
})
}
android/src/main/kotlin/BaseApplication.kt

46. Config via AndroidManfest.xml
<meta-data android:name="drawurl" android:value="${drawurl}"/>
val drawUrl: String? by project
if (drawUrl != null) {
manifestPlaceholders = mapOf("drawurl" to drawUrl)
}
else {
// 10.0.2.2 is the IP for your machine from the Android emulator
manifestPlaceholders = mapOf("drawurl" to "http://10.0.2.2:8080")
}
android/build.gradle.ktx
android/src/main/AndroidManifest.xml
drawUrl=http://192.168.1.23:8080
gradle.properties

47. Cloud Run, Run
FROM adoptopenjdk/openjdk8 as builder
WORKDIR /app
COPY . /app
RUN ./gradlew --no-daemon --console=plain :server:shadowJar
FROM adoptopenjdk/openjdk8:jre
COPY --from=builder /app/server/build/libs/server.jar /server.jar
RUN apt-get update && apt-get install -y --no-install-recommends fontconfig
CMD ["java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/server.jar"]
Dockerfile

48. Build & Deploy
steps:
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA', '/workspace']
- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA']
- name: 'gcr.io/cloud-builders/gcloud'
args: ['beta', 'run', 'deploy', '--image=gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA',
'--platform=managed', '--project=$PROJECT_ID', '--region=us-central1', '--allow-
unauthenticated', '--memory=512Mi', '$REPO_NAME']
cloudbuild.yaml

49. Demo!

50. Jetpack Compose

51. Jetpack Compose
Reactive, declarative UI Toolkit
Kotlin-first
Developer Preview
(Pre-Alpha)

52. Demo!

53. Track
● Track
Motion
Events
Record
● Add
Locations
to Path(s)
View
● Invalidate
ML
● Ready to
Analyze
Draw
● Draw Path
Data
User Drawings
In Compose!

54. UI Layout
@Composable
fun BuildUI(guesses: Guesses = Guesses()) {
MaterialTheme() {
val invalidator = +invalidate
Column(Spacing(8.dp), crossAxisAlignment = CrossAxisAlignment.Stretch) {
FlexRow(mainAxisAlignment = MainAxisAlignment.Center) {
val radioOptions = listOf("Shape", "Digit")
val (selectedOption, onOptionSelected) = +state { radioOptions[0] }
inflexible {
RadioGroup(
options = radioOptions,
selectedOption = selectedOption,
onSelectedChange = onOptionSelected
)
}
flexible(1f) {
Column(crossAxisAlignment = CrossAxisAlignment.Stretch) {
Button(text = "Local", ...)
Button(text = "Cloud", ...)
}
}
}
Button(text = "Sensorify", ...)
HeightSpacer(8.dp)
GuessDisplay(guesses)
HeightSpacer(8.dp)
DrawingCanvas(path)
HeightSpacer(8.dp)
Button(text = "Clear", ...)
HeightSpacer(8.dp)
}
}
}

55. UI Layout
@Composable
fun BuildUI(guesses: Guesses = Guesses()) {
MaterialTheme() {
val invalidator = +invalidate
Column(Spacing(8.dp), crossAxisAlignment = CrossAxisAlignment.Stretch) {
FlexRow(mainAxisAlignment = MainAxisAlignment.Center) {
val radioOptions = listOf("Shape", "Digit")
val (selectedOption, onOptionSelected) = +state { radioOptions[0] }
inflexible {
RadioGroup(
options = radioOptions,
selectedOption = selectedOption,
onSelectedChange = onOptionSelected
)
}
flexible(1f) {
Column(crossAxisAlignment = CrossAxisAlignment.Stretch) {
Button(text = "Local", ...)
Button(text = "Cloud", ...)
}
}
}
Button(text = "Sensorify", ...)
HeightSpacer(8.dp)
GuessDisplay(guesses)
HeightSpacer(8.dp)
DrawingCanvas(path)
HeightSpacer(8.dp)
Button(text = "Clear", ...)
HeightSpacer(8.dp)
}
}
}

56. DrawingCanvas
@Composable
fun DrawingCanvas(path: Path) {
val invalidate = +invalidate
RawDragGestureDetector(dragObserver = MyDragObserver(path, invalidate)) {
Surface(color = Color.Black) {
Container(modifier = ExpandedHeight, width = 200.dp, height = 350.dp) {
Draw { canvas, parentSize ->
canvas.drawPath(path, fingerPaint)
}
}
}
}
}

57. DrawingCanvas
@Composable
fun DrawingCanvas(path: Path) {
val invalidate = +invalidate
RawDragGestureDetector(dragObserver = MyDragObserver(path, invalidate)) {
Surface(color = Color.Black) {
Container(modifier = ExpandedHeight, width = 200.dp, height = 350.dp) {
Draw { canvas, parentSize ->
canvas.drawPath(path, fingerPaint)
}
}
}
}
}

58. Track Motion Events
class MyDragObserver(val dragPath: Path, val recompose: () -> Unit): DragObserver {
override fun onStart(downPosition: PxPosition) {
dragPath.moveTo(downPosition.x.value, downPosition.y.value)
}
override fun onDrag(dragDistance: PxPosition): PxPosition {
dragPath.relativeLineTo(dragDistance.x.value, dragDistance.y.value)
recompose()
return dragDistance
}
}

59. Machine Learning Code
Button(text = "Local", onClick = {
machineLearningStuff.detectObject(true, bitmap!!,
selectedOption == "Shape") {
// display results in Text objects...
}
})

60. Web UI

61. Shared Common
Root Gradle Project
Common
ServerAndroid Web
Jar

62. Kotlin JS - Build
plugins {
kotlin("js")
}
dependencies {
implementation(kotlin("stdlib-js"))
implementation(project(":common"))
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core-js:1.3.2")
implementation("org.jetbrains.kotlinx:kotlinx-html-js:0.6.12")
}
web/build.gradle.kts

63. Kotlin JS - Assemble JsJar
tasks.withType<org.jetbrains.kotlin.gradle.tasks.Kotlin2JsCompile> {
kotlinOptions {
metaInfo = false
sourceMap = true
outputFile = "$buildDir/classes/kotlin/main/${project.name}.js"
}
}
task<Copy>("assembleJsLib") {
from(...)
into("$buildDir/classes/kotlin/main")
dependsOn("mainClasses")
}
tasks {
JsJar {
into("META-INF/resources")
dependsOn("assembleJsLib")
}
}
web/build.gradle.kts

64. Kotlin JS - Polling
fun main() {
GlobalScope.launch { poll() }
}
suspend fun poll() {
val res = window.fetch("/events").await()
when (res.status.toInt()) {
200 -> {
val json = res.json().await().asDynamic()
...
val imageResult = ImageResult(byteArray, labelAnnotations)
...
}
}
window.setTimeout({ GlobalScope.launch { poll() } }, 1000)
}
web/src/main/kotlin/Main.kt

65. Kotlin JS - Rendering
val urlImage = "url('data:image/png;base64,${imageResult.image.decodeToString()}')"
document.body?.style?.backgroundImage = urlImage
document.body?.clear()
val div = document.create.div()
imageResult.labelAnnotations.forEach { labelAnnotation ->
div.append {
p {
+"${labelAnnotation.description} = ${round(labelAnnotation.score *
100)}%"
}
}
}
document.body?.append(div)
web/src/main/kotlin/Main.kt

66. Kotlin JS - Serving Static Resources
micronaut.router.static-resources.resources.enabled=true
micronaut.router.static-resources.resources.paths=classpath:META-INF/resources
micronaut.router.static-resources.resources.mapping=/resources/**
server/src/main/resources/application.properties
dependencies {
api(files("../web/build/libs/web.jar"))
}
tasks.withType<org.jetbrains.kotlin.gradle.internal.KaptWithKotlincTask> {
dependsOn(":web:JsJar")
}
server/build.gradle.kts

67. Kotlin JS - HTML Page
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Air Draw</title>
<link rel="stylesheet" href="assets/index.css">
<script src="resources/kotlin.js"></script>
<script src="resources/kotlinx-coroutines-core.js"></script>
<script src="resources/kotlinx-html-js.js"></script>
<script src="resources/kotlinx-serialization-kotlinx-serialization-
runtime.js"></script>
<script src="resources/common.js"></script>
<script src="resources/web.js"></script>
</head>
<body>
waiting for drawings...
</body>
</html>
server/src/main/kotlin/resources/views/index.html

68. Multi-Module Project

69. One Project To Kotlin Them All
Root Gradle Project
Common
ServerAndroid Web
build.gradle.kts
common/build.gradle.kts
android/build.gradle.kts
server/build.gradle.kts
web/build.gradle.kts

70. One Project To Rule Them All
buildscript {
repositories {
mavenLocal()
mavenCentral()
jcenter()
google()
}
dependencies {
classpath(kotlin("gradle-plugin", "1.3.61"))
classpath("com.android.tools.build:gradle:4.0.0-alpha04")
}
}
build.gradle.kts (Root Gradle Project)

71. Move Data Processing to the Server

72. Demo!

73. Air Draw
● Capture
Sensor Data
REST Service
● Smoooooth
Data
● Render to
Image
Cloud ML
● Send
Bitmap to
Cloud ML
● Send
Bitmap &
results to
Phone
Browser
● Poll d' Bus
● Display
Bitmap &
Results
Pub/Sub
● On d' Bus

74. Mapping Sensor Data to an Image

75. Sensors & Vision
data class Orientation(val azimuth: Float, val pitch: Float, val timestamp: Long)
SensorManager.getRotationMatrixFromVector(rotationMatrix, e.values)
val m = FloatArray(3)
SensorManager.getOrientation(rotationMatrix, orientationAngles)
val orientation = Orientation(m[0], m[1], e.timestamp)
readings.add(orientation)
common/src/commonMain/kotlin/Data.kt

76. SEND d' DATA!
@Client("${drawurl}")
interface DrawService {
@Post("/draw")
fun draw(@Body readings: List<Orientation>): Single<ImageResult>
}
machineLearningStuff.sensorAction(on) { orientations ->
val imageResultMaybe = drawService?.draw(orientations)?.blockingGet()
imageResultMaybe?.let { imageResult ->
displayResults(imageResult)
}
}
android/src/main/kotlin/MainActivity.kt

77. RECEIVE d’ DATA!
@Post("/draw")
fun draw(@Body readingsSingle: List<Orientation>):
Single<ImageResult> {
return readingsSingle.map { readings ->
airDraw.run(readings)?.let { imageResult ->
bus.put(imageResult)
HttpResponse.ok(imageResult)
}
}
}
server/src/main/kotlin/WebApp.kt

78. Things get crazy
val xl = KrigingInterpolation1D(t, x)
...
import java.awt.image.BufferedImage
val bi = BufferedImage(canvas.width, canvas.height, BufferedImage.TYPE_INT_ARGB)
...
val imgBytes = ByteString.copyFrom(bytes)
val img = Image.newBuilder().setContent(imgBytes).build()
val feature = Feature.newBuilder().setType(Type.LABEL_DETECTION).build()
val request = AnnotateImageRequest.newBuilder().addFeatures(feature).setImage(img).build()
val r = myImageAnnotatorClient.imageAnnotatorClient.batchAnnotateImages(arrayListOf(request))
server/src/main/kotlin/WebApp.kt

79. Pitfalls
● Gradle Build Plugins
● JSON
● Externalizing the URL
● Math is hard
● ML is hard
● Compose is Young

80. github.com/GoogleCloudPlatform/air-draw-demo