In both business and academia, reproducibility is the root of trust in computational results. Software engineering is critical to consistently support the work. Basic practices including systematic build, automatic tests, version control, and documentation will be discussed. The practices and a mix of programming languages enable developers to address the challenges in scientific computing by creating a flexible and high-performance platform. The fundamental idea is to generalize and segregate common constructs so that they can be reinforced during the whole cycles of code development, to enhance the confidence in the produced results. It is also necessary to engage in software development communities, as a means to improve productivity and give back.

1. Craftsmanship in
Computational Work
2017/9/26
at NTU CE CAE Group
Yung-Yu Chen
https://www.linkedin.com/in/yungyuc/

2. Computational Science
❖ Virtualize the world, govern it like Mother Nature does.
❖ Mechanical, electrical, engineering design and analysis.
❖ … and any problem should be solved by computers.

3. Black hole simulation https://go.nasa.gov/2xhd5xD
Optical proximity correction for semiconductor manufacturing
https://commons.wikimedia.org/wiki/File:Optical_proximity_correction_structures.svg
Solve problems that can only be solved by computers

4. Ensure consistency everywhere and in every version

5. Outline
Basic Software Engineering
Platform-Centric Architecture
Meet the Craftsmen

6. “It worked on my computer yesterday”
an innocent programmer

7. https://twitter.com/esconfs/status/568724582368198657

8. Reproducibility
❖ Reproducibility is the gold standard for trust in
scientific work.
❖ Replication is a prerequisite to engineering.
❖ Engineers may accept an inaccurate result, but it must
be consistently inaccurate.
❖ Scientific work shouldn’t have a lower standard.

9. Write Readable Code
❖ Delegate anything else to computers.
❖ Use vim (http://www.vim.org/), a really good editor.
“Code is read much more often than it is written, so plan
accordingly”
Raymond Chen

10. Automation Is the Key
Build system
Test
Version control
Documentation

11. Why Use a Build System?
❖ Make software consistent
across environments and
platforms.
❖ Basic automation tool.
❖ Dependency processing.
❖ Examples: make, cmake.
CXX = clang++
CXXFLAGS =-O0 -g -fPIC -std=c++11 -stdlib=libc++
CXXFLAGS_PY = $(shell python3-config --includes)
LDFLAGS = -fPIC -stdlib=libc++ -lstdc++
LDFLAGS_PY = -lboost_python -lboost_system
WTYPE ?= bpy
ifeq ($(WTYPE), "bpy")
CXXFLAGS += -DWTYPE_BPY=1
endif
ifndef NOUSE_BOOST_SHARED_PTR
CXXFLAGS += -DUSE_BOOST_SHARED_PTR
endif
ifdef USE_ENABLE_SHARED_FROM_THIS
CXXFLAGS += -DUSE_ENABLE_SHARED_FROM_THIS
endif
DEPEND=Makefile header.hpp
default: run
pymod.o: $(WTYPE).cpp $(DEPEND)
$(CXX) -c $< -o $@ $(CXXFLAGS) $(CXXFLAGS_PY)
pymod.so: pymod.o
$(CXX) -shared $< -o $@ $(LDFLAGS) $(LDFLAGS_PY)
run: pymod.so
python3 pydrive.py

12. Automate What?
❖ Source /
library /
executable
❖ 3rdparty
❖ Scripts
❖ Tests
❖ Doc
axiom (http://www.axiom-developer.org)
Andrew Neitsch (2012). Build System Issues in Multilanguage Software (Master’s Thesis, University of Alberta).
Retrieved from https://andrew.neitsch.ca/publications/msc-20120906.pdf

13. Dependency among Source Files
❖ Not to compile a million lines of code every time a file
changes.
❖ The dependency shouldn’t be handled manually.
source: https://www.vtk.org/doc/release/7.0/html/vtkObjectBase_8h.html

14. Build Multiple Flavors
❖ Fencing macros are
commonplace in
production code.
❖ Debugging code.
❖ Ad hoc / specific
optimization.
#ifdef DEBUG_FEATURE_X
{
printf("vertex coordinates of all elems:n");
for (int i=0; i<nelem; ++i) {
printf("vertex coordinates of all elems:");
for (int j=0; j<elems[i].nvertex; ++j) {
printf(" (");
for (int idm=0; idm<NDIM; ++idm) {
if (idm != NDIM-1) {
printf("%g, ", elems[i].vertices[j][idm]);
} else {
printf("%g" , elems[i].vertices[j][idm]);
}
}
if (j != elems[i].nvertex-1) {
printf("),");
} else {
printf(")");
}
}
printf("n");
}
}
#endif // DEBUG_FEATURE_X

15. Two requirements:
Tests are automated
Test failures are treated as anomalies

16. What Are Tests
❖ Basic tests make sure the software does what it did.
❖ At least 2 levels are needed:
❖ Unit tests.
❖ Integration tests, interface tests, system tests, etc.
❖ Unit tests test for the most fine-grain constructs in a
software system.

17. Test for Development
❖ Unit tests are a great tool for quality and productivity.
❖ Some testing is almost always needed while developing code.
Why not doing it in an organized way?
❖ Unit-testing frameworks like Google Test (C++), Python
unittest standard module, or JUnit (Java) are created for this.
❖ Testing should be taken into account in the code
implementation: design for testing.
❖ (Unit) tests may be developed before features: test-driven
development (TDD).

18. Test Always
❖ Testing gives developers confidence to write code (that
they aren’t breaking things).
❖ Hit button and tests run.
❖ Tests must pass before and after code changes.
❖ Testing is part of development:
❖ Always add regression tests for new code.

19. What to Test
❖ Input and output of functions.
❖ Object states and/or lifecycles. Especially data buffers
for scientific applications.
❖ Environment dependency.
❖ Be just rightly sensitive to code errors.
❖ Many more testing types beyond regression are not
discussed here.

20. Version control: a fancy name for
systematically tracking code changes

21. What to Do with an Error
❖ Build system and testing are for reducing the chance to
have errors. Version control is for finding out how an
error crept in or what it is.
❖ Only when version control is sanely done, you can
bisect.
build / test passes build / test fails
which change introduces the error?

22. We All Make Mistakes
❖ Common debugging strategies:
❖ Inspect from the hints from test results.
❖ Use a debugger.
❖ A bug happens with the bleeding-edge version: just
change the code.
❖ When it’s with a previous version: how to fix both the
old and the new versions? Or even versions in between?

23. Branching
❖ Different “streaks” of
development need to be
traced separately.
❖ Temporal differences are
tracked with the
“streaks”.
❖ Nowadays the common
practice is to use DAG:
git, hg, etc.
“git flow”: http://nvie.com/posts/a-successful-git-branching-model/

24. Version Control 101
❖ Version control is for source code. Archive assets (“blobs”:
binary large objects) elsewhere.
❖ For small-to-medium-sized projects (almost all scientific /
research works), just use git. The decentralized system
works efficiently and securely and has a large community
for support.
❖ Each check-in should be organized logically and locally.
❖ Treat the version control history like code. It adds more
dimensions to the source code.

25. Documentation
❖ Why write non-source texts?
❖ In-source comments: supplement source code.
❖ Off-source, manual-like texts: concept build-up.
❖ API references.
❖ Specification: idea communication.
❖ Documentation is part of software.
❖ It can have bugs and should be automated.

26. Manual-style
API-style

27. Comment Blocks Are the King
❖ ASCII-art may look dumb, but it’s indispensable.
❖ Practicality beats beauty.
struct UnstructuredBlockFixture {
/**
* This is how the block looks like. It composes of only triangles. "N"
* denotes node, and "C" denotes cell.
*
* (0,1)N3
* *
* /|
* / |
* / |
* / |
* / |
* / |
* / C2 * C1
* / /^
* / / N0
* / / (0,0)
* / /
* / /
* / / C0
* //
* *-----------------------------*
* (-1,-1)N1 (1,-1)N2
*/

28. Document Generation
❖ Two ways to produce documentation: (i) extract from
source code and docstrings and (ii) compile dedicated
document files.
❖ Many documenting systems support both modes:
doxygen, sphinx, etc.
❖ Cross-referencing and hyperlinks are essential.

29. Platform-Centric
❖ The problem at hand is complexity.
❖ High-performance computing (HPC).
❖ Scripting for dynamic management.
❖ House-keeping, analysis, and visualization.
❖ Graphical user interface.
❖ A “platform” segregates everything in layers.
❖ Modular design for millions of lines of code.

30. WWI. source: https://www.youtube.com/watch?v=K0Wp7Y3Tbiw
HPC is hard. Physics is harder.
Don’t rely on the Schlieffen Plan.

31. HPC: Time to Results
❖ Engineers’ time is more valuable than computers’.
Machines don’t get paid or sued.
❖ Both less and more lead to shorter time to results.
❖ Less CPU cycles.
❖ More CPUs.

32. Optimization
❖ Memory access is expensive.
❖ Branching may be expensive too.
❖ Think like a machine: see through high-level source
code all the way to assembly.
❖ If you don’t write your own compiler, learn C++.
“There are only two hard things in Computer Science:
cache invalidation and naming things.”
Phil Karlton

33. HPC Architecture
❖ Scientific computing takes tremendous computing
power. We are interested in big problems.
❖ Some may be divided to smaller, self-contained sub-
problems, e.g., data analytics.
❖ Some are unavoidably big. A reasonably big problem
may use thousands of CPU cores for days.
❖ Accelerators like GPGPU sometimes speed up, but at a
cost of complicated code.

34. Python is the second-best
language for everything and
sometimes the best
Use it everywhere else than performance hotspot
Scripting is Python

35. Quick Iteration
❖ It's impossible to get it right the first time.
❖ Architecture design takes many iterations.
❖ Python allows quick prototyping.
❖ There is almost always a package for you.

36. NumPy
❖ The core is N-dimensional array (ndarray).
❖ ndarray is typed and offers very fast speed. Oftentimes
faster than naive C code.
❖ Efficient data storage and flexible access to memory.
❖ Linear algebra (MKL is supported), FFT, etc.
❖ SciPy: application-specific toolbox.

37. Python Is Designed for C/C++
❖ Everything may be replaced by C/C++. Python is a C
library for a dynamically-typed runtime.
❖ Python is slow, but using Python makes the whole HPC
system faster.
❖ C++ use cases:
❖ Performance hotspots.
❖ High-level abstraction in low-level code.

38. Interface to C/C++
❖ C: vanilla Python C API or Cython.
❖ C++: pybind11 (C++11) or boost.python (pre-C++11).
❖ Fortran: f2py (part of numpy).

39. pybind11
❖ https://github.com/pybind/pybind11
❖ Active development. Seamless operability between C+
+11(/14/17) and Python.
❖ Work with both Python 2 and 3.
❖ Header-only. Compact code base (~4k). Easy to use.

40. .def("__getstate__", [](wrapped_type & blk) {
py::dict pickled;
// shapes.
pickled["nnode" ] = py::cast(blk.nnode());
pickled["nface" ] = py::cast(blk.nface());
pickled["ncell" ] = py::cast(blk.ncell());
pickled["nbound" ] = py::cast(blk.nbound());
pickled["ngstnode"] = py::cast(blk.ngstnode());
pickled["ngstface"] = py::cast(blk.ngstface());
pickled["ngstcell"] = py::cast(blk.ngstcell());
pickled["use_incenter"] = py::cast(blk.use_incenter());
// arrays.
pickled["ndcrd"] = Table(blk.ndcrd()).full();
pickled["fccnd"] = Table(blk.fccnd()).full();
pickled["fcnml"] = Table(blk.fcnml()).full();
pickled["fcara"] = Table(blk.fcara()).full();
pickled["clcnd"] = Table(blk.clcnd()).full();
pickled["clvol"] = Table(blk.clvol()).full();
pickled["fctpn"] = Table(blk.fctpn()).full();
pickled["cltpn"] = Table(blk.cltpn()).full();
pickled["clgrp"] = Table(blk.clgrp()).full();
pickled["fcnds"] = Table(blk.fcnds()).full();
pickled["fccls"] = Table(blk.fccls()).full();
pickled["clnds"] = Table(blk.clnds()).full();
pickled["clfcs"] = Table(blk.clfcs()).full();
pickled["bndfcs"] = Table(blk.bndfcs()).full();
// bndvec.
py::list bndlist;
for (auto & bnd : blk.bndvec()) {
bndlist.append(WrapBoundaryData::getstate(bnd));
}
pickled["bndvec"] = bndlist;
return pickled;
})
Example: pickle (Python standard serialization) support

41. Two Types of Platform
❖ Top-down: lay out everything in Python and replace hotspots
using C/C++
❖ Pro: Fast development. Reach results early.
❖ Con: Python dynamicity weakens robustness.
❖ Bottom-up: lay out core parts in C++ and glue in Python
❖ Pro: Highly robust (if coded right.)
❖ Con: Hard to get it right and take long time to code.
❖ Equally high-performance. Python scripts work as input files.

42. Python Tools
❖ No one escapes from routine work, but Python crushes it.
❖ Data preparation and processing.
❖ Workflow automation.
❖ Distributed processing and parallel computing.
❖ Interactive analysis and visualization.
❖ Having these capabilities and the computing kernel, it’s a
fully-grown computing platform at your fingertip.

43. Data Manipulation
❖ “csv” standard module.
❖ http://www.pytables.org: HDF5 hierarchical data
access.
❖ http://unidata.github.io/netcdf4-python/: netCDF, yet
another data storage based on HDF5.
❖ http://pandas.pydata.org: de facto tool for data
analytics

44. Workflow
❖ When you want more flexibility than existing tools like
make.
❖ https://docs.python.org/3/library/argparse.html:
standard command-line argument processing
❖ https://github.com/saltstack/salt: cloud-oriented
automation for management and configuration
❖ AWS, GCE, Azure all offer SDK for Python.
❖ Lead to system admin and/or devop.

45. Concurrency
❖ https://docs.python.org/3/library/asyncio.html: support
native asynchronous constructs
❖ https://docs.python.org/3/library/multiprocessing.html:
parallel computing and distributed processing using multiple
processes
❖ Threads can’t simultaneously use multiple CPU cores
because GIL (global interpreter lock).
❖ http://zeromq.org/bindings:python: socket communication
❖ http://pythonhosted.org/mpi4py/: use MPI in Python

46. Interactive
❖ http://jupyter.org: run Python everywhere and code it
through browser.
❖ https://notebooks.azure.com: Azure sets it up for you
already
❖ https://matplotlib.org: de facto 2D plotting library
❖ https://www.vtk.org: versatile 3D visualization toolbox
❖ https://www.paraview.org: if you only want a
frontend

47. Everything Upon Python
❖ Python can hardly be a wrong choice. When it may be wrong, the answer is C/C++,
and the road has been paved.
❖ Exception: web browser frontend; only JavaScript works on it.
Quick Iteration Easy Extension
Rich Support
(free as beer)
Ideal Foundation to Build a Platform

48. Work Smart
❖ Software is eating everything.
❖ Coding is the craftsmanship everyone needs to practice.
Everyone needs the skills to command computers.
❖ Understandings to computer science is indispensable:
computer architecture, data structure, algorithms,
programming language, etc.
❖ Make friends.

49. Commercial Code Development
❖ Keep business in mind.
❖ Reorient the pursuit of knowledge to profitability.
❖ Teamwork.
❖ Be the best at what you are doing
❖ Help teammates to be the best at what they are doing
❖ Be honest. Seek help not too early and not too late
❖ If no one knows the right way to do it, work around.

50. Open-Source Code Development
❖ Technical excellency.
❖ Do whatever you want.
❖ Business may or may not be in the equation.
❖ Novelty may or may not be in the equation.
❖ Beauty must be in the equation.
❖ The world is your team.
“Talk is Cheap. Show me the Code.”
Linus Torvalds

51. Developer Communities
❖ Advancing science requires critical discussions. So does
Software.
❖ Find a team that allows you to challenge the status quo
and help you layout realistic plans.
❖ Go outside your team and meet other programmers face
to face.
❖ Learn to make friends in open source by patches.

52. What to Expect
❖ Language communities: Python, Go, Ruby, JavaScript
❖ Application communities: machine learning, data
analytics
❖ Methodology communities: agile, devops
❖ Regional general communities

53. Interesting work is a reward
for those who get work well done