GeoServer Developers Workshop

Welcome
Ian Turon
Open Source Evangelist
ian@ianturton.com
@ijturton
Astun technologies
Passionate about geography and
passionate about open (source,
standards and data).
Jody Garnett
Community Lead
jgarnett@boundlessgeo.com
@jodygarnett
Boundless
Provides geospatial tools and
services for managing data and
building applications.

● Welcome
● Development Environment
○ Exercise: checkout and Build
● GeoServer Software Stack
Developer Orientation
● GeoServer Design
○ Architecture
● Dispatch
○ Open Web Services
○ Spring Application Context
○ Exercise:
■ Hello Service
● Web Admin
○ Wicket

● GeoTools Architecture
○ Optional: Checkout and Build
● Data Model:
○ Feature
○ Geometry
○ CRS
○ Grid Coverage
GeoServer: Data Access and Visualization
● Factory SPI
● Function
○ Exercise: create a function
● Process
○ Exercise: create a process

● Catalog API
○ Data management
○ Service configuration
○ Settings
● Extensions and Community modules
○ Optional: build with community profile
GeoServer Internals
● Java JUnit Testing
● GeoServer Testing
○ GeoServerSystemTest
○ Mock Objects
● Dispatcher
○ Request/Response
○ KVP Parser

Explore Java Development
● Git/GitHub feature workflow
● Java
○ Interface/Factory/SPI
○ JUnit/TestCoverage
○ Mock Objects
● Spring
○ applicationContext
○ GeoServerExternals
Development topics to explore
Explore GeoServer Ecosystem
● Documentation Sphinx/RST
● Vector internals
○ JTS Topology Suite
○ GeoTools DataStore
● Image Processing
○ JAI / JAI-Ext / JAI Tools / Jiffle
● Raster Formats
○ World Image, GeoTIFF internals
● Renderer
○ Streaming Renderer
● REST API

GeoServer Community
● Open Source
○ License / Providence
○ Open checklist
● Accountability
○ Governance
○ Quality Control
● Explore Java Development
○ Git/GitHub feature workflow
○ Java
■ Interface/Factory/SPI
■ JUnit/TestCoverage
■ Mock Objects
○ Spring
■ applicationContext
■ GeoServerExternals

Developer Orientation
GeoServer Developer Workshop

GeoServer checkout and build
http://docs.geoserver.org/latest/en/developer/
● Developers Guide “Latest”
● Scroll down to Tools, Source Code and Quickstart sections
Q: Do I need to make a fork?
A: Yes this is a course on GeoServer development?
Q: Internet is slow, can I make a shallow local copy?
A: When checking out use a shallow copy for today
git clone --depth 3 git://github.com/<username>/geoserver.git

Optional: GeoTools checkout and build
http://docs.geotools.org
● User Guide "Latest"
● Scroll down to Building section
Q: Download maven source?
A: No download the binary
Q: Clone is taking too long!
A: When checking out use a shallow copy for today
git clone --depth 3 git://github.com/geotools/geotools.git

Development Environment
● Java 8
○ We are not in a position to use Java 10 at present
○ As a server project GeoServer is a conservative preferring long-term-support releases
● Git
○ On windows we recommend github windows client which also provides command line tools
○ On macOS github client can be a great place to start, may also consider paying Tower client
○ On linux you already love the command line
● Maven
○ Maven will download and use other tools as required
● Integrated Development Environment
○ https://www.jetbrains.com/idea/download/
○ https://www.eclipse.org/downloads/

Development Environment: Windows
Windows is not a very common development environment
● You will be "popular" as one of the few developers able to directly experience and fix windows
specific problems!
● Administrator access is useful for installing software and tools
● NTFS struggles creating/deleting thousands of files during compile and test
If you are getting "file lock" failures
● Check windows search is not indexing your development folders
(it is sure to get a file lock failure trying to index files as we delete them)
● Config your antivirus software to ignoring development folders

Environmental Variables
● M2_HOME - maven install directory
● JAVA_HOME - java install directory
● PATH - windows
○ Should include %M2_HOME%/bin - for mvn tool
○ Should include %JAVA_HOME%/bin - for java command
● PATH - linux and OS X
○ Should include $M2_HOME/bin;$JAVA_HOME/bin

> java -version
java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)
> mvn -version
Apache Maven 3.5.3
Maven home: /usr/local/Cellar/maven/3.5.3/libexec
Java version: 1.8.0_152, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/jre
Default locale: en_CA, platform encoding: UTF-8
OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
Ready to go on?

How Maven differs from Make and Ant
Make and ANT are task based
● Execution model is a series of "build targets"
● Make files are great for copy and paste, but hard to maintain
● Ant files are just as "good" but with the power of XML
Maven is the opposite of ANT
● Popular "tasks" are packaged on the internet as a "maven plugin"
● maven assumes a "standard" directory structure so they can share tasks
● Tasks are mapped to your project directories and "run"
● This allows many projects to share the same "task" steps!
(much easier to maintain, but more magic when you watch it go)

How Maven Works
● Your project directories are described using a "pom.xml" file
● pom.xml defines a "project object model"
○ documents how *your* directories are laid out (src/ or src/main/java)
○ describing everything about your project!
● jar management (ie "dependency" management)
○ a common ant task downloads a list of jars and puts them in a "lib" folder
○ Maven takes a list of "dependencies" and downloads it to a "repository" folder in your home directory
■ This "local repository" folder is shared between all projects
■ If you build a tool locally and "install" it into the "local repository" folder other projects will not be able to
tell if it came from the internet or not!

Maven Repository
Local Repository
● on your machine
● holds everything downloaded from the
internet
● both "maven plugins"
● also "dependencies"
External Repositories
● maven central is "official"
● OSGeo repo for geotools
○ geotools releases
○ third-party dependencies
● Repo.boundless.com
○ geoserver releases
○ generated snapshots
○ cache of osgeo repo

● Plugin:goal
○ Dependency:tree
○ eclipse:eclipse
● Profiles
○ -Poracle
● Variables
○ -Dall
○ -DskipTests
● Options
○ -nsu (don’t fetch snapshots)
○ -o (offline don’t download any new jars)
○ -T (threads to use)
life cycle
● validate
● compile
● test
● package
● integration-test
● verify
● install (local repository)
● deploy
Using maven

GeoServer
cd src
mvn install -DskipTests -T 2C
cd web
cd app
mvn jetty:run
http://localhost:8080/geoserver
mvn jetty:run -DconfigId=release
-DconfigDirectory=../../../data/realease
Building
GeoTools (Optional)
mvn install -Dall -DskipTests -T 2C

git setup
● upstream: this is the original geoserver
○ master: this is the latest
○ 2.13.x: this is the stable branch
○ factory-registry: this is a "feature branch
where developers were working together
● origin: this is your "fork" of geoserver
○ master: this is the latest
○ fix_me: this is a feature branch where you
are working on a fix
● local:
○ pull from upstream:master and then push
to origin:master
○ publish origin:fix_me, and do a pull request
to upstream:master
local
origin upstream
factory-registry
2.13.x
master
fix_me
master
fix_me
master
remote repositories
local repository

feature branch
Distributed Development
local
origin upstream
2.13.x
master
2.13.x
master
feature branch
master
pull request
stash
workspace
index
GitHub
Local Machine
File System
fetch/pull push
checkout commit

http://ndpsoftware.com/git-cheatsheet.html

Workshop hint: Reset the origin of your repo
Workshop internet is slow, please copy geoserver checkout from Ian:
git remote -v
git remote remove origin
git remote add origin git@github.com:<user>/geoserver.git
git remote add upstream https://github.com/geoserver/geoserver.git
git remote -v

git workflow
https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow
● feature branch for each fix
● submit pull request to master
● backport to stable branches
● CONTRINBUTING.md
Guidelines
1. Ensure your IDE is properly configured
(we now use google formatter in build)
2. Include only relevant changes
3. Fix one thing at a time
4. Always add a test
5. Reference Jira ticket in commit message
a. Used to create the release notes!
6. Be patient
a. Devs are contributing their own time
b. Writing a test showing functionality / fix

IntelliJ
● great maven integration out of the box
● Fast enough to have both geoserver and
geotools loaded at the same time

Eclipse IDE Maven
Developers Guide: Eclipse M2 Quickstart
● Maven integration provided out-of-the box
● Provides build steps and editors for
Eclipse IDE
● Teaches eclipse how to interact with
maven build and repositories
● Tips
○ To fix “errors” right click on project and run
maven->UpdateProject

Eclipse IDE Maven Glitches
● java-cc-maven-plugin
○ Build once on the command line first
(to create generated java files)
● Import trouble with some build steps!
○ Treat as warning in Preferences
○ Use “Resolve Later” during import
○ Exclude in lifecycle mappings
● fmt-maven-plugin
○ Configure google formatter in Eclipse
● Missing closed source jars? (oracle)
○ download and install into local repo
○ or simply close those projects in eclipse
cd src/wcs1_1
mvn compile
[INFO] --- javacc-maven-plugin:2.3:jjtree (jjtree) @ gs-wcs1_1 ---
Java Compiler Compiler Version 4.0 (Tree Builder)
(type "jjtree" with no arguments for help)
"/Users/jgarnett/java/geoserver/jodygarnett/src/wcs1_1/target/jjtree/org/geoserver/wcs/kvp/rangesubset/ASTFieldI
d.java" does not exist. Will create one.
...
Annotated grammar generated successfully in
/Users/jgarnett/java/geoserver/jodygarnett/src/wcs1_1/target/jjtree/org/geoserver/wcs/kvp/rangesubset/rangeset.jj
[INFO]
[INFO] --- javacc-maven-plugin:2.3:javacc (javacc) @ gs-wcs1_1 ---
Java Compiler Compiler Version 4.0 (Parser Generator)
(type "javacc" with no arguments for help)
Reading from file
/Users/jgarnett/java/geoserver/jodygarnett/src/wcs1_1/target/jjtree/org/geoserver/wcs/kvp/rangesubset/rangeset.jj
. . .
File "TokenMgrError.java" does not exist. Will create one.
File "ParseException.java" does not exist. Will create one.
...
Parser generated successfully.
[INFO]
[INFO] --- fmt-maven-plugin:2.4.0:format (default) @ gs-wcs1_1 ---
[debug] Using AOSP style
[INFO] Processed 47 files (0 reformatted).
[INFO]

Developers Guide: Maven Eclipse Plugin Quickstart
● mvn eclipse:eclipse generates:
○ .project - eclipse project description
allowing IDE to load an existing project
○ .classpath - used during compile and
deploy, and ties projects in the IDE
together
● Execute from root directory so resulting
eclipse projects reference each other
● Add optional sources and/or javadocs
-DdownloadSources
-DdownloadJavadocs
Eclipse IDE with mvn eclipse plugin

Eclipse IDE with mvn eclipse plugin Glitches
● maven eclipse plugin not really supported
○ But it works well :)
○ Faster that built-in eclipse functionality
● No sync, so if you change a pom.xml
○ run “mvn eclipse:eclipse” again
○ refresh in eclipse
● Both command line maven and eclipse
generate to the same output directory
○ clean when switching to IDE
○ Use “mvn clean” for command line
○ Alternative: -DbuildOutputDirectory=bin
Frustrating but fast, this is a “manual option” so
you will need to duck out to the command line all
the time :)

Running from IDE
The quickstart instructions show how to run:
1. Locate web-app project
2. Set up a configuration to run org.geoserver.web.Start
3. Add program arguments to specify the data directory:
-DGEOSERVER_DATA_DIR=/Users/home/java/geoserver/data/release

Key Components of Software Stack
● GeoServer - "spring" application built with the following:
● GeoTools - GIS implementation(referencing, data formats, rendering)
● JTS Topology Suite - geometry implementation
● Java Advanced Imaging (JAI) - image processing by Sun/Oracle
● JAI-Image-IO (ImageIO) - image formats by Sun/Oracle
● ImageIO-EXT - geospatial image formats such as GeoTIFF
● JAI-EXT - geospatial operations with no-data and footprint support
○ this is a new addition for GeoServer 2.8 scheduled for September release
● JAI-Tools - helpful image processing tools
○ JAI-Tools-Jiffle - really cool "raster calculator" domain specific language

How GeoServer uses Modules
The GeoServer “web application” is comprised of a series of modules:
● Modules are compiled and packaged into jars for distribution
● We will be creating a custom module for use in today’s course
There are two good ways to work on a custom module:
● Alternative 1: Custom WebApp with Snapshots
○ Approached is used by project teams to make a custom web application for their customers
● Alternative 2: Community Module
○ Approached used for collaborative RnD work with the GeoServer community.

Alternative 1: Custom WebApp with Snapshots
Approached is used by project teams to make a custom web application for their
customers
● Pros:
○ Team members only have the modules they are working on in IDE, better on slow machines
○ Great control over exactly what is being published
● Cons:
○ Depends on build.geoserver.org publishing a nightly SNAPSHOT
○ Developers working 'downstream' from community and are not in position to quickly fix issues

● project/pom.xml
○ has web-app and custom as modules project/custom/pom.xml
● project/custom/pom.xml
○ your custom function, process, and web pages
● project/web-app/pom.xml
○ depends on custom and community snapshots

Directory Structure
project/pom.xml
project/custom/pom.xml
project/app/pom.xml

Approached used for collaborative RnD work with the GeoServer community.
● Pros:
○ Developers working 'upstream' with community and in position to quickly fix issues
○ Migration path to including work in geoserver release
● Cons:
○ the entire geoserver checkout may be a stretch for slow machines
○ Small risk of breaking the build on build.geoserver.org
(and interrupting developers around the globe)
Alternative 2: Community Module

● community/custom/pom.xml
○ Your custom function,process,web page etc..
○ Module is a child of community.
● community/pom.xml
○ add profile to include your custom module (or modules) in the build
● web/app/pom.xml
○ Add profile to include dependencies on your custom module (or modules)
○ Profile used when we build a WAR or run GeoServer

Directory Structure
src/community/pom.xml
src/community/custom/pom.xml
src/extension/…
src/web/app/pom.xml - profile added to depend on custom!

src/community/custom/pom.xml
<parent>
<groupId>org.geoserver</groupId>
<artifactId>community</artifactId>
<version>2.15-SNAPSHOT</version>
</parent>
<groupId>org.geoserver.community</groupId>
<artifactId>gs-custom</artifactId>
<packaging>jar</packaging>
<version>2.15-SNAPSHOT</version>
<name>Custom Module</name>
this is the
parent module
this is our custom
module artifact
description

src/community/pom.xml
<profile>
<id>custom</id>
<modules>
<module>custom</module>
</modules>
</profile>
id of profile we will
type on the
command line
folder name of your
custom module

src/web/app/pom.xml
<profile>
<id>custom</id>
<dependencies>
<dependency>
<groupId>org.geoserver.community</groupId>
<artifactId>gs-custom</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</profile>

● mvn eclipse:eclipse -Pcustom
○ pom.xml description generates eclipse .project and .classpath files
● Eclipse File --> Import and then General --> Existing Projects
○ import your geoserver/src folder
○ new projects, including custom, will be available to be imported
● Result:
○ maven projects (including custom) are tied together
○ You can now edit files and then immediately run Start.java from IDE
○ When debugging you can "live update" your code
IDE Build with Custom Profile

GeoServer Service Implementation
Dispatch, Service and Operations

Dispatcher
● Developers Guide: OWS Services
● Dispatcher reads the Service definitionsfrom applicationContext.xml
○ maps incoming service / version / request to appropriate implementation

applicationContext.xml
<beans>
<bean id="helloService" class="org.geoserver.custom.HelloWorld"/>
<bean id="helloService-1.0.0" class="org.geoserver.platform.Service">
<constructor-arg index="0" value="hello"/>
<constructor-arg index="1" ref="helloService"/>
<constructor-arg index="2" value="1.0.0"/>
<constructor-arg index="3">
<list>
<value>sayHello</value>
</list>
</constructor-arg>
</bean>
</beans>

RequestResponseOutputFormat
● WMS, WFS and WCS services handle:
○ GET Requests defined using Request URL
○ POST Requests defined using Request XML body
● GET Requests
○ KVP Parsers and Handlers produce Request Object
○ Example: GetMapRequest
● POST Requests
○ XML "Bindings" are used with GeoTools GTXML parser to produce Request object
○ Example: GetMapRequest
● Dispatcher responsible for
○ looking up Response object and output format
○ Example: GetFeatureInfoResponse, CSVOutputFormat

Catalog
Configuration and "Data Directory" access

Catalog
● Two implementations:
○ Data Directory - stored xml files using xstream library
■ Extended with hz-cluster-plugin and jms-cluster-plugin
○ JDBCConfig (community) - stored in database using jdbc
■ Extended with jdbcstore-plugin
● Configuration Model:
○ Catalog - data access object supporting lookup
○ WorkspaceInfo
○ StoreInfo (ie FeatureStoreInfo and CoverageStoreInfo)
○ LayerInfo / ResourceInfo
○ StyleInfo
● Internally details:
○ ResourcePool - internal cache for "datastore" connections
○ ResourceStore - "file" access to sld, template and icons

Catalog Design
● Configuration Proposal
○ original design documentation
○ Key feature for GeoServer 2.x
● GSIP 52
○ Refactor out DAO for Catalog and Configuration
● GSIP 69
○ Catalog scalability enhancements
● GSIP 106
○ ResourceStore to replace file use
● GSIP 155
○ Core improvements for large catalogs

Testing
Server testing and quality assurance

Test Libraries and Tools
● JUnit
○ provides @Test annotations, good integration with Eclipse and Maven
● XMLUnit
○ asserts for DOMs
● easymock (we use an older version)
○ Used to create fake "mock" objects
● Mockrunner
○ provides a mock of ServletRequest and ServletRespose
● jacoco
○ measure test coverage

Testing
● GeoServerSystemTestSupport - 7 seconds
○ starts up a complete GeoServer based on test data
● GeoServerMockTestSupport - 1.5 seconds
○ provides a "complete" mocked GeoServer Catalog!
● Unit Test - 0.5 seconds
○ short, sweet, minimal setup
○ use a fake "mock" object to stub out complicated classes
● See Setup Instructions and Discussion:
○ http://docs.geoserver.org/latest/en/developer/programming-guide/testing/index.html

GeoServerSystemTestSupport
@TestSetup(run = TestSetupFrequency.REPEAT)
public class HelloIntegrationTest extends GeoServerSystemTestSupport {
@Test
public void testSayService() throws Exception {
String result = getAsString(
"/ows?request=sayHello&service=hello&version=1.0.0"
);
assertNotNull("Result expected", result);
assertTrue("Expected messsage", result.contains("Hello"));
assertTrue("Expect 5 Workspaces", result.contains("5"));
}
}

GeoServerMockTestSupport
public class MockTest extends GeoServerMockTestSupport {
@Test
public void requestTest() throws Exception {
HelloWorld service = new HelloWorld();
Catalog catalog = getCatalog();
service.setCatalog(catalog);
HttpServletRequest request = new MockHttpServletRequest();
MockHttpServletResponse response = new MockHttpServletResponse();
service.sayHello( request, response );
String output = response.getOutputStreamContent();
assertNotNull("Result expected", output);
assertTrue("Expected messsage", output.contains("Hello"));
assertTrue("Expect 5 Workspaces", output.contains("5"));
}
}

UnitTest
public class UnitTest {
@Test
public void requestTest() throws Exception {
HelloWorld service = new HelloWorld();
Catalog catalog = createMock( "catalog", Catalog.class );
List<WorkspaceInfo> list = new ArrayList<WorkspaceInfo>();
for( int i=0; i<5;i++)
list.add( createMock( "workspace "+i,WorkspaceInfo.class ));
expects( catalog ).andReturn( list );
replay(catalog)
service.setCatalog(catalog);
HttpServletRequest request = new MockHttpServletRequest();
MockHttpServletResponse response = new MockHttpServletResponse();
service.sayHello( request, response );
String output = response.getOutputStreamContent();
assertNotNull("Result expected", output);
assertTrue("Expected messsage", output.contains("Hello"));
assertTrue("Expect 5 Workspaces", output.contains("5"));
}
}

Code Coverage
● Run jacoco report using provided
profile:
mvn test -Pjacoco
● The resulting report is in:
target/site/jacoco/index.html
● Modules are required to have 40%
coverage (and documentation) to be
before being proposed as an extension

GeoServer Community
Open source, open development

GeoServer Developers Guide
● Committing
● Submitting Patches
● Code Review
● Community Process
● GeoServer Improvement Proposals
● Community Modules
● Project Steering Committee

Committing
● OSGeo corporate and individual contributor agreements
● Community commit access
○ send an email the the developer list and ask
○ "play area" not subject to IP review, or included in the release
● Core commit access
○ nominated (after submitting several patches)
● Commit Guidelines
○ headers, code formatting
○ don't commit large binary data or jars (use maven)

Submitting Patches / Pull Requests
● GitHub Pull Requests (automated way to manage patches)
○ "Fork" GeoServer git repository
○ Create a "feature branch" to work on your fix
○ When ready publish your feature branch to your git repository
○ visit github page and create pull request
● Guidelines
○ Make sure you run mvn to fix formatting before making your commit!
○ Include only relevant changes
○ one thing at time
○ include a test case!
○ Tips: create a JIRA issue for the release notes! If not we can’t thank you!

Pull request:
● Everything needs a code review!
● Formatting is wrong? Update: We fixed it, code formater runs as part of the build :)
● The test case should show your fix
● Don’t forget the documentation
○ Changes the the user interface …. should also update the user guide!
Code Reviews

Community Planning
● Previously we had roadmap (short term, long term, ….)
○ This was a terrible idea!
○ Release would always get delayed for “one more thing”
○ Disappointment - proposing an idea for a roadmap is not the same as having funding
● Time Boxed Release
○ new release release every 6 months
○ releases have 6 months support and 6 months maintenance
● Adding Features
○ GeoServer Improvement Process (GSIP) - ie change control request

GeoServer Improvement Process
● Change control for developers
● Create wiki page
○ there is a template
○ Include and design documents or timeframe concerns

Project Steering Committee
● Responsible for
○ voting on GSIP proposals
○ care and feeding of GeoServer
● Nominated based on community involvement / service
○ Ideally a mix of developers, management and user representation

Last friday of the month
● Coordinate on gitter!
○ Ask for code reviews!
○ Ask for help, suggestions,
● Use the bug tracker
○ Review new tickets
○ tickets labeled “sprint” are
■ good candidate for fix
■ has clear steps to reproduce
○ Comment on tickets that are incomplete
http://blog.geoserver.org/2017/04/27/geoserver-monthly-bug-stomp/
Monthly bug stomp

Community Modules
● Easy way to get involved with GeoServer
○ Email to geoserver-devel to request community module(Project Steering Committee member will approve)
○ You are provided with commit access to work on your module
○ community modules can be "published" in the nightly build(But are NOT included in the official release)
● When your module is ready to "graduate"
○ review of headers / license
○ documentation and test "coverage" of 40%
○ When complete your module will move to extension/ folder
(and be included in the official release)

Creating a Function
● GeoTools
○ Factory Tutorial
○ Function Tutorial
● Key concepts
○ Factory Service Provider Interface (SPI)
○ Factory Design Pattern
○ Function
● How to use
○ WFS Query
○ SLD Geometry Transformation
○ WPS

Creating a Process
● GeoTools
○ Process Tutorial
● GeoServer
○ WPS Services
● Key concepts
○ Factory Service Provider Interface (SPI)
○ Factory Design Pattern
○ Process
● How to use
○ WPS
○ Geometry Transformation
○ Rendering Transformation

Web Administration Application / Wicket UI
● GeoServer
○ Web User Interface
○ Wicket Development in GeoServer
● Key concepts
○ Apache Wicket
● How to use
○ Create admin page for service

REST Service
● GeoServer
○ Implementing a RESTful Service
● Key concepts
○ Spring, Spring Annotations
● How to use
○ Create REST endpoint for service configuration

Security
● GeoServer
○ Security
● Key concepts
○ GeoServerSecurityManager
● How to use
○ Check authentication and authorization

GeoServer Developers Workshop

GeoServer Developers Workshop

Recommended

More Related Content

What's hot

What's hot(20)

Similar to GeoServer Developers Workshop

Similar to GeoServer Developers Workshop(20)

More from Jody Garnett

More from Jody Garnett(20)

Recently uploaded

Recently uploaded(20)

GeoServer Developers Workshop