The document summarizes Samuel Lampa's talk on using flow-based programming for scientific computing. It provides biographical information on Samuel Lampa, including his background in pharmaceutical bioinformatics and current work. It then gives an overview of flow-based programming, describing it as using black box processes connected by data flows, with connections specified separately from processes. Benefits mentioned include easy testing, monitoring, and changing connections without rewriting components. Examples of using FBP in Go are also presented.
EuroPython 2016 - Do I Need To Switch To GolangMax Tepkeev
Nowadays, there is a lot of buzz about Go. It happened so that for the last 6 months I’ve been mostly programming Go, and frankly speaking I fell in love with this language.
We’ll first do a quick review of the language. Go doesn’t have some language constructs, for example classes and exceptions and at first it may seem hard to write proper Go code, but in practice the language is so easy that I will try to teach you the basics and most important concepts of the language. We’ll further discuss differences and similarities in Go and Python and dive into the cool features of Go.
Finally we’ll talk about why popularity of Go is raising so fast and try to answer the most important question: Do I need to switch to Go ?
An introduction into Googles programming language Go, which had its first release in March 2012. The talk has been held at the regulars' table of the GTUG Bremen.
Given at GopherFest 2015. This is an updated version of the talk I gave in NYC Nov 14 at GothamGo.
“We need to think about failure differently. Most people think mistakes are a necessary evil. Mistakes aren't a necessary evil, they aren't evil at all. They are an inevitable consequence of doing something new and as such should be seen as valuable. “ - Ed Catmull
As Go is a "new" programming language we are all experimenting and learning how to write better Go. While most presentations focus on the destination, this presentation focuses on the journey of learning Go and the mistakes I personally made while developing Hugo, Cobra, Viper, Afero & Docker.
7 Common mistakes in Go and when to avoid themSteven Francia
I've spent the past two years developing some of the most popular libraries and applications written in Go. I've also made a lot of mistakes along the way. Recognizing that "The only real mistake is the one from which we learn nothing. -John Powell", I would like to share with you the mistakes that I have made over my journey with Go and how you can avoid them.
Java/Scala Lab: Руслан Шевченко - Implementation of CSP (Communication Sequen...GeeksLab Odessa
CSP (i.e. 'go-like' channels and 'Occam-like' transputers) can be useful programming technique for structuring asynchronous blocking tasks. Author own implementation of CSP [scala-gopher] will be presented.
The document discusses Transmogrifier, a tool for migrating content in Plone. It describes what Transmogrifier is, provides a simple example of its configuration, and shows how to create custom blueprints for processing data. It also discusses lessons learned from large migrations and ideas for future improvements.
EuroPython 2016 - Do I Need To Switch To GolangMax Tepkeev
Nowadays, there is a lot of buzz about Go. It happened so that for the last 6 months I’ve been mostly programming Go, and frankly speaking I fell in love with this language.
We’ll first do a quick review of the language. Go doesn’t have some language constructs, for example classes and exceptions and at first it may seem hard to write proper Go code, but in practice the language is so easy that I will try to teach you the basics and most important concepts of the language. We’ll further discuss differences and similarities in Go and Python and dive into the cool features of Go.
Finally we’ll talk about why popularity of Go is raising so fast and try to answer the most important question: Do I need to switch to Go ?
An introduction into Googles programming language Go, which had its first release in March 2012. The talk has been held at the regulars' table of the GTUG Bremen.
Given at GopherFest 2015. This is an updated version of the talk I gave in NYC Nov 14 at GothamGo.
“We need to think about failure differently. Most people think mistakes are a necessary evil. Mistakes aren't a necessary evil, they aren't evil at all. They are an inevitable consequence of doing something new and as such should be seen as valuable. “ - Ed Catmull
As Go is a "new" programming language we are all experimenting and learning how to write better Go. While most presentations focus on the destination, this presentation focuses on the journey of learning Go and the mistakes I personally made while developing Hugo, Cobra, Viper, Afero & Docker.
7 Common mistakes in Go and when to avoid themSteven Francia
I've spent the past two years developing some of the most popular libraries and applications written in Go. I've also made a lot of mistakes along the way. Recognizing that "The only real mistake is the one from which we learn nothing. -John Powell", I would like to share with you the mistakes that I have made over my journey with Go and how you can avoid them.
Java/Scala Lab: Руслан Шевченко - Implementation of CSP (Communication Sequen...GeeksLab Odessa
CSP (i.e. 'go-like' channels and 'Occam-like' transputers) can be useful programming technique for structuring asynchronous blocking tasks. Author own implementation of CSP [scala-gopher] will be presented.
The document discusses Transmogrifier, a tool for migrating content in Plone. It describes what Transmogrifier is, provides a simple example of its configuration, and shows how to create custom blueprints for processing data. It also discusses lessons learned from large migrations and ideas for future improvements.
Migrating from drupal to plone with transmogrifierClayton Parker
Transmogrifier is a migration framework that can help you easily migrate from one platform to another. It has been written in a way that allows re-use of migration code through blueprints. In this talk we will walk through the steps necessary to migrate from Drupal, a popular CMS written in PHP, into Plone. We will see how to use the various blueprints available to build a pipeline that prepares and imports the content into Plone
This document provides an overview of the Go programming language including:
- A brief history of Go from its origins at Google in 2007 to current widespread usage.
- Key features of Go like functions, structs, interfaces, methods, slices, pointers, go-routines and channels.
- Examples of basic Go code demonstrating functions, structs, methods, interfaces, concurrency with go-routines and channels.
- Discussion of things missing from Go like classes, generics and exceptions.
- Quotes from developers commenting both positively and negatively on Go's simplicity and tradeoffs.
- Links to further resources like the Go tour to learn more.
Transmogrifier is a tool that gives you the ability to create a configurable pipeline to import and export from various systems. Each section of the pipe can be turned into a re-usable blueprint that can be given back to the community. This gives us the ability to quickly create migration paths for existing systems and maintain them as Python packages.
Euro python2011 High Performance PythonIan Ozsvald
I ran this as a 4 hour tutorial at EuroPython 2011 to teach High Performance Python coding.
Techniques covered include bottleneck analysis by profiling, bytecode analysis, converting to C using Cython and ShedSkin, use of the numerical numpy library and numexpr, multi-core and multi-machine parallelisation and using CUDA GPUs.
Write-up with 49 page PDF report: http://ianozsvald.com/2011/06/29/high-performance-python-tutorial-v0-1-from-my-4-hour-tutorial-at-europython-2011/
Transmogrifier: Migrating to Plone with less painLennart Regebro
Transmogrifier is a migration framework written by Martijn Pieters, Jarn, that makes migrating to Plone fun again. This talk gives an introduction to Transmogrifier, an explanation of how it works and some hints for usage.
In which Richard will tell you about some things you should never (probably ever) do to or in Python. Warranties may be voided. The recording of this talk is online at http://www.youtube.com/watch?v=H2yfXnUb1S4
This document discusses user defined functions (UDFs) in Apache Pig. It provides examples of different types of UDFs including EvalFunc, FilterFunc, and LoadFunc. For EvalFunc, it shows how to write a simple function to uppercase text and how to return complex types. For FilterFunc, it demonstrates an IsEmpty function. For LoadFunc, it outlines the key interfaces and methods needed to implement a custom loader using a regular expression example.
Handling input languages like XML, YAML, or JSON is simple enough, just grab your favorite module and hammer it out. But what should one do when facing a more obscure markup language, configuration file or another artifact of a developer with acute NIH syndrome? In this talk we’ll (quickly) revisit formal languages basics and what can and cannot be parsed. We will then introduce parsimonious, a simple, fast, and economic Parsing Expression Grammar parser written by Eric Rose.
Conférence des Geeks Anonymes sur " le langage Go ", par Thomas Hayen le 23 septembre 2020.
Cette conférence est disponible en vidéo sur Youtube : https://youtu.be/AlGGneVGTJk
Reversing the dropbox client on windowsextremecoders
The document discusses reversing an obfuscated Dropbox Python client executable on Windows. It begins by unpacking the executable using py2exe to extract the embedded Python bytecode files (.pyc). The extracted .pyc files are encrypted, so the document explores dumping them from memory after the Dropbox Python interpreter decrypts them. This fails due to encryption of the bytecode. The document then leverages the PyPy Python implementation to successfully dump and decrypt one of the .pyc files. Further exploration of the PyCodeObject structure in a debugger reveals Dropbox has modified the layout to hinder reversing. A small program is used to find the actual offset of the encrypted bytecode within the PyCodeObject.
This document provides tips and tricks for installing Kotlin programming tools and covers fundamental Kotlin concepts like data types, variables, functions, control flow structures, and loops. It discusses installing Kotlin and resolving common errors. It introduces basic Kotlin syntax for strings, numbers, Booleans, arrays, and more. It also covers if/else expressions, when expressions, enums, loops like while, for, and for-each, and break/continue functionality. The document encourages practicing exercises to continue learning Kotlin.
The document discusses Go's concurrency features including goroutines, channels, and synchronization tools. It explains that goroutines are lightweight threads managed by Go, and channels provide a means of communication between goroutines. The document also covers potential concurrency issues like deadlocks and provides best practices to avoid anti-patterns when using goroutines and channels.
Go for the paranoid network programmer, 3rd editionEleanor McHugh
Draft third edition of my #golang network programming and cryptography talk given to the Belfast Gophers Meetup. Now with an introduction to websockets.
The document discusses using MongoDB and Go together for painless data storage. Some key points:
- Go is a good language for working with MongoDB as it is fast, friendly to use, and supports concurrency well out of the box.
- MongoDB is also fast and scales well. It uses a flexible document model that maps well to Go's data types.
- The mgo driver makes it easy to connect to MongoDB from Go, perform CRUD operations, and work with data concurrently. Data can be serialized and deserialized to BSON format efficiently.
- Features like GridFS allow storing files within MongoDB, providing replicated file storage.
This document discusses Go concurrency fundamentals including goroutines, channels, and synchronization primitives. It provides examples of using goroutines and channels for signaling between processes, timeouts, heartbeats, and implementing a load balancer. Key points covered include goroutines being lightweight processes, channel-based communication between goroutines, and using select to handle multiple channel operations.
This document discusses file processing and input/output (I/O) in C++. It covers opening and reading from input files, processing the data, writing output to output files, and closing the files. Key points include:
1) The fstream library is used for file I/O, with ifstream for input and ofstream for output. Open is used to connect the file streams to external files.
2) A while loop processes records from the input file by reading values, calculating output, and writing to the output file until the end of file is reached.
3) Constructors initialize file stream objects, and can be overloaded. Do-while loops are not suitable for file processing due
JavaOne 2015 - Having fun with JavassistAnton Arhipov
The document provides examples of using Javassist, an open-source bytecode engineering library, to dynamically generate, instrument, reflect and modify Java bytecode. It demonstrates how to use Javassist to generate proxy classes, insert logging before methods, and modify existing classes at runtime without recompilation. Key classes and methods discussed include ClassPool, CtClass, CtMethod and insertBefore.
scala-gopher: async implementation of CSP for scalaRuslan Shevchenko
This document describes scala-gopher, a Scala library that implements Communicating Sequential Processes (CSP) concepts like Go channels. It allows asynchronous programming using constructs like select.forever that handle input/output between channels. The library builds on Akka and SIP 22 for asynchrony. It includes channels, transputers that connect via ports, and replication capabilities for running processes in parallel. The goal is to provide CSP functionality within the Scala ecosystem as a complementary approach to streams and actors.
Migrating from drupal to plone with transmogrifierClayton Parker
Transmogrifier is a migration framework that can help you easily migrate from one platform to another. It has been written in a way that allows re-use of migration code through blueprints. In this talk we will walk through the steps necessary to migrate from Drupal, a popular CMS written in PHP, into Plone. We will see how to use the various blueprints available to build a pipeline that prepares and imports the content into Plone
This document provides an overview of the Go programming language including:
- A brief history of Go from its origins at Google in 2007 to current widespread usage.
- Key features of Go like functions, structs, interfaces, methods, slices, pointers, go-routines and channels.
- Examples of basic Go code demonstrating functions, structs, methods, interfaces, concurrency with go-routines and channels.
- Discussion of things missing from Go like classes, generics and exceptions.
- Quotes from developers commenting both positively and negatively on Go's simplicity and tradeoffs.
- Links to further resources like the Go tour to learn more.
Transmogrifier is a tool that gives you the ability to create a configurable pipeline to import and export from various systems. Each section of the pipe can be turned into a re-usable blueprint that can be given back to the community. This gives us the ability to quickly create migration paths for existing systems and maintain them as Python packages.
Euro python2011 High Performance PythonIan Ozsvald
I ran this as a 4 hour tutorial at EuroPython 2011 to teach High Performance Python coding.
Techniques covered include bottleneck analysis by profiling, bytecode analysis, converting to C using Cython and ShedSkin, use of the numerical numpy library and numexpr, multi-core and multi-machine parallelisation and using CUDA GPUs.
Write-up with 49 page PDF report: http://ianozsvald.com/2011/06/29/high-performance-python-tutorial-v0-1-from-my-4-hour-tutorial-at-europython-2011/
Transmogrifier: Migrating to Plone with less painLennart Regebro
Transmogrifier is a migration framework written by Martijn Pieters, Jarn, that makes migrating to Plone fun again. This talk gives an introduction to Transmogrifier, an explanation of how it works and some hints for usage.
In which Richard will tell you about some things you should never (probably ever) do to or in Python. Warranties may be voided. The recording of this talk is online at http://www.youtube.com/watch?v=H2yfXnUb1S4
This document discusses user defined functions (UDFs) in Apache Pig. It provides examples of different types of UDFs including EvalFunc, FilterFunc, and LoadFunc. For EvalFunc, it shows how to write a simple function to uppercase text and how to return complex types. For FilterFunc, it demonstrates an IsEmpty function. For LoadFunc, it outlines the key interfaces and methods needed to implement a custom loader using a regular expression example.
Handling input languages like XML, YAML, or JSON is simple enough, just grab your favorite module and hammer it out. But what should one do when facing a more obscure markup language, configuration file or another artifact of a developer with acute NIH syndrome? In this talk we’ll (quickly) revisit formal languages basics and what can and cannot be parsed. We will then introduce parsimonious, a simple, fast, and economic Parsing Expression Grammar parser written by Eric Rose.
Conférence des Geeks Anonymes sur " le langage Go ", par Thomas Hayen le 23 septembre 2020.
Cette conférence est disponible en vidéo sur Youtube : https://youtu.be/AlGGneVGTJk
Reversing the dropbox client on windowsextremecoders
The document discusses reversing an obfuscated Dropbox Python client executable on Windows. It begins by unpacking the executable using py2exe to extract the embedded Python bytecode files (.pyc). The extracted .pyc files are encrypted, so the document explores dumping them from memory after the Dropbox Python interpreter decrypts them. This fails due to encryption of the bytecode. The document then leverages the PyPy Python implementation to successfully dump and decrypt one of the .pyc files. Further exploration of the PyCodeObject structure in a debugger reveals Dropbox has modified the layout to hinder reversing. A small program is used to find the actual offset of the encrypted bytecode within the PyCodeObject.
This document provides tips and tricks for installing Kotlin programming tools and covers fundamental Kotlin concepts like data types, variables, functions, control flow structures, and loops. It discusses installing Kotlin and resolving common errors. It introduces basic Kotlin syntax for strings, numbers, Booleans, arrays, and more. It also covers if/else expressions, when expressions, enums, loops like while, for, and for-each, and break/continue functionality. The document encourages practicing exercises to continue learning Kotlin.
The document discusses Go's concurrency features including goroutines, channels, and synchronization tools. It explains that goroutines are lightweight threads managed by Go, and channels provide a means of communication between goroutines. The document also covers potential concurrency issues like deadlocks and provides best practices to avoid anti-patterns when using goroutines and channels.
Go for the paranoid network programmer, 3rd editionEleanor McHugh
Draft third edition of my #golang network programming and cryptography talk given to the Belfast Gophers Meetup. Now with an introduction to websockets.
The document discusses using MongoDB and Go together for painless data storage. Some key points:
- Go is a good language for working with MongoDB as it is fast, friendly to use, and supports concurrency well out of the box.
- MongoDB is also fast and scales well. It uses a flexible document model that maps well to Go's data types.
- The mgo driver makes it easy to connect to MongoDB from Go, perform CRUD operations, and work with data concurrently. Data can be serialized and deserialized to BSON format efficiently.
- Features like GridFS allow storing files within MongoDB, providing replicated file storage.
This document discusses Go concurrency fundamentals including goroutines, channels, and synchronization primitives. It provides examples of using goroutines and channels for signaling between processes, timeouts, heartbeats, and implementing a load balancer. Key points covered include goroutines being lightweight processes, channel-based communication between goroutines, and using select to handle multiple channel operations.
This document discusses file processing and input/output (I/O) in C++. It covers opening and reading from input files, processing the data, writing output to output files, and closing the files. Key points include:
1) The fstream library is used for file I/O, with ifstream for input and ofstream for output. Open is used to connect the file streams to external files.
2) A while loop processes records from the input file by reading values, calculating output, and writing to the output file until the end of file is reached.
3) Constructors initialize file stream objects, and can be overloaded. Do-while loops are not suitable for file processing due
JavaOne 2015 - Having fun with JavassistAnton Arhipov
The document provides examples of using Javassist, an open-source bytecode engineering library, to dynamically generate, instrument, reflect and modify Java bytecode. It demonstrates how to use Javassist to generate proxy classes, insert logging before methods, and modify existing classes at runtime without recompilation. Key classes and methods discussed include ClassPool, CtClass, CtMethod and insertBefore.
scala-gopher: async implementation of CSP for scalaRuslan Shevchenko
This document describes scala-gopher, a Scala library that implements Communicating Sequential Processes (CSP) concepts like Go channels. It allows asynchronous programming using constructs like select.forever that handle input/output between channels. The library builds on Akka and SIP 22 for asynchrony. It includes channels, transputers that connect via ports, and replication capabilities for running processes in parallel. The goal is to provide CSP functionality within the Scala ecosystem as a complementary approach to streams and actors.
Golang basics for Java developers - Part 1Robert Stern
This document provides an overview of Golang basics for Java developers. It covers Golang's history, features, syntax, data types, flow control, functions and interfaces, concurrency, and differences from Java. Key points include Golang being a compiled, statically typed language created at Google in 2007, its use of packages and imports, basic types like strings and integers, slices for dynamic arrays, maps for key-value pairs, functions with receivers, errors instead of exceptions, and goroutines for concurrency with channels.
The document outlines the schedule and objectives for an operating systems lab course over 10 weeks. The first few weeks focus on writing programs using Unix system calls like fork, exec, wait. Later weeks involve implementing I/O system calls, simulating commands like ls and grep, and scheduling algorithms like FCFS, SJF, priority and round robin. Students are asked to display Gantt charts, compute waiting times and turnaround times for each algorithm. The final weeks cover inter-process communication, the producer-consumer problem, and memory management techniques.
asyncio - is young library for asynchronous network programming. Lack of information on proper use and writing code in asyncio as well as production war stories slows down asyncio adoption.
In this talk I'll cover my experience with asyncio in production, best practices for writing reliable and testable asynchronous code as well as bunch of asyncio tips and tricks. We will discuss how run asyncio application in standalone mode, execute blocking code in event loop and moreover how embed asyncio in your synchronous application.
Pipe2Py is a tool that converts Yahoo Pipes workflows into Python code so they can still be run if Yahoo Pipes shuts down. Each Yahoo Pipes module is coded as a separate Python module. Pipe2Py translates Yahoo Pipes into pipelines of Python generators to closely match the original data flow. This allows users to prototype workflows visually in Yahoo Pipes and then generate Python code to host themselves independently of Yahoo Pipes.
Gore is a Go REPL that uses "go run" to evaluate code. It reads user input, generates a .go file appending the input to a main function, fixes errors using go-quickfix, and runs the code. This avoids reimplementing the Go compiler. Features include code completion using gocode, and commands like :import, :print, and :doc.
NetPonto - The Future Of C# - NetConf EditionPaulo Morgado
The document discusses various new features in C# and .NET, including:
- Async Main method allowing async entry points
- Inferred tuple element names for cleaner tuple syntax
- Default literal expressions avoiding unnecessary casts
- Non-trailing named arguments for more flexible calling syntax
- Pattern matching and switch expressions for more control flows
- Reference semantics for value types for improved performance
- And more proposals still in development like nullable reference types, ranges, and patterns.
It also provides links to documentation and code repositories for further reading.
scala.concurrent.Future is familiar to nearly all Scala devs.
This presentation first talks about referential transparency and the IO Monad in general. (Monix Task is an impl of the IO Monad.)
Then it compares Future Monix 3.x Task with their Pros and Cons.
Interop with Future: As Scala's Future is used in many environments and libraries, we look at the conversion from Task to Future and - vice versa - from Future to Task.
I will also take a look at Task evaluation, cancelation and memoization as well as tail recursive loops and asynchronous boundaries.
The presentation will include a comparative discussion on ExecutionContext (required for Future) and Scheduler (required for Task, but only to run it).
Often recurring on the valuable Monix Task doumentation at https://monix.io/docs/3x/eval/task.html the presentation can also be seen as an introduction to Monix Task.
Incredible Machine with Pipelines and Generatorsdantleech
The document discusses using generators and pipelines in PHP to build a performance testing tool called J-Meter. It begins by explaining generators in PHP and how they allow yielding control and passing values between functions. This enables building asynchronous pipelines where stages can be generators. Various PHP frameworks and patterns for asynchronous programming with generators are mentioned. The document concludes by outlining how generators and pipelines could be used to build the major components of a J-Meter-like performance testing tool in PHP.
Spark is a fast and general engine for large-scale data processing. It was developed at UC Berkley in 2009 and can run programs up to 100x faster than Hadoop MapReduce in memory or 10x faster on disk. Spark uses Resilient Distributed Datasets (RDDs) as its basic abstraction, which allow data to be operated on in parallel. The document provides examples of using Spark for word count, SQL queries, and notebooks.
Puppi is a Puppet modules that drives Puppet's knowledge of the Systems to a command line tool that you can use to check services availability, gather info on the system and deploy application with a single command.
A Recovering Java Developer Learns to GoMatt Stine
As presented at OSCON 2014.
The Go programming language has emerged as a favorite tool of DevOps and cloud practitioners alike. In many ways, Go is more famous for what it doesn’t include than what it does, and co-author Rob Pike has said that Go represents a “less is more” approach to language design.
The Cloud Foundry engineering teams have steadily increased their use of Go for building components, starting with the Router, and progressing through Loggregator, the CLI, and more recently the Health Manager. As a “recovering-Java-developer-turned-DevOps-junkie” focused on helping our customers and community succeed with Cloud Foundry, it became very clear to me that I needed to add Go to my knowledge portfolio.
This talk will introduce Go and its distinctives to Java developers looking to add Go to their toolkits. We’ll cover Go vs. Java in terms of:
* type systems
* modularity
* programming idioms
* object-oriented constructs
* concurrency
This document discusses the design and development of APIs. It covers topics such as REST APIs, GraphQL, API documentation standards like JSON, HTTP methods, versioning, error handling, and more. The goal is to explain best practices for building well-designed and developed APIs. Diagrams and examples are provided to illustrate concepts like resources, collections, requests and responses, and type systems.
Hong developed a server for proprietary GPS trackers using Go as their first experience with the language. The talk outlines Go's characteristics, how Hong built the server app including communicating over TCP, encoding/decoding data, and reading a config file. Hong recommends getting a team to adopt Go by starting small, sharing learnings, embracing experimentation, and using Go appropriately for the task.
The document discusses node.js basics including setting up a development environment, using npm commands, executing JavaScript files, using external files, function scopes, and closures. It provides code examples and explanations of key concepts like requiring modules, exports vs module.exports, and how variable scoping works differently inside and outside of functions in JavaScript.
어느덧 스무살이 된 자바. 좋든싫든 프로그래밍의 세계에 몸담고 있는 이상 부딪히지 않을 수 없는 언어인데요, 이 자바에 대한 올바른 모습의 이해화 앞으로 나아가야 할 방향을 모색해본다는 의미에서 "모던 자바의 역습"이라는 타이틀로 실시한 온라인 세미나의 슬라이드 자료입니다.
진행자: 김대우(http://lekdw.blogspot.kr/)
정도현(http://moreagile.net)
이번 세미나는 게임개발자이신 김대우님을 모시고 진행하게 되었는데요 자바 언어와 관련 하여 다음과 같은 주제로 두시간에 걸쳐 발표를 진행합니다.
- 간단히 살펴보는 자바20년의 발자취
- 자바를 둘러싼 진실 혹은 거짓(성능,생산성,품질)
- SI영역 이외에서 자바의 가치
- 모던자바의 역습: 자바8과 함수형 프로그래밍
- 자바 개발의 새로운 패러다임들
CP3108B (Mozilla) Sharing Session on Add-on SDKMifeng
The document provides an overview of how to build Firefox add-ons using the Add-On SDK. It discusses key aspects like using JavaScript, HTML and CSS with no XUL needed. It outlines the packaging and testing tools as well as CommonJS module format. It provides guidance on getting started including reading documentation, tutorials and API references. It also covers topics like using the cfx tool, Package.json, the console, modules, JavaScript best practices and asynchronous programming.
Similar to Using Flow-based programming to write tools and workflows for Scientific Computing in Go (20)
Linked Data for improved organization of research dataSamuel Lampa
Slides for a talk at a Farmbio BioScience Seminar May 18, 2018, at http://farmbio.uu.se introducing Linked Data as a way to manage research data in a way that can better keep track of provenance, make its semantics more explicit, and make it more easily integrated with other data, and consumed by others, both humans and machines.
How to document computational research projectsSamuel Lampa
These slides are from an internal meeting at pharmb.io where we discussed ways to improve documentation of our internal computational research projects. The winning solutions turns out to be markdown files, versioned with git. The slides explains a little bit about why.
Reproducibility in Scientific Data Analysis - BioScience SeminarSamuel Lampa
Slides for a talk held at BioScience Seminar at Dept. of Pharmaceutical BioSciences at Uppsala University on December 16, 2016.
The event webpage: http://www.farmbio.uu.se/calendar/kalendarium-detaljsida/?eventId=22496
Structure of the talk:
Reproducibility in Scientific Data Analysis ...
● What is it?
● Why is it important?
● Why is it a problem?
● What can we do about it?
● What does pharmb.io do about it?
SciPipe - A light-weight workflow library inspired by flow-based programmingSamuel Lampa
A presentation of the SciPipe workflow library, written in Go (Golang), inspired by Flow-based programming, at an internal workshop at Uppsala University, Department of Pharmaceutical Biosciences.
Vagrant, Ansible and Docker - How they fit together for productive flexible d...Samuel Lampa
A very quick overview of how Vagrant, Ansible and Docker fits nicely together as a very productive and flexible solution for creating automated development environments.
This document provides a cheat sheet overview of key concepts in the IRODS rule language, including numeric and string literals, arithmetic and comparison operators, functions for strings, lists, tuples, if/else statements, foreach loops, defining functions and rules, handling errors, and inductive data types. It describes syntax for defining data types using constructors, and using pattern matching to define functions over data types.
AddisDev Meetup ii: Golang and Flow-based ProgrammingSamuel Lampa
The document discusses flow-based programming (FBP), its history and concepts. FBP defines applications as networks of processes that exchange data through message passing over predefined connections. This allows the processes to be reconnected without changing their code. The document provides examples of FBP networks and components implemented in various languages like Go, Java and JavaScript. It also discusses the benefits of FBP and its growing popularity with implementations like NoFlo.
First encounter with Elixir - Some random thingsSamuel Lampa
The document discusses Samuel Lampa's first encounter with the programming language Elixir. It covers calculating GC ratios in DNA sequences, provides a DNA sequence example file, and compares Elixir processes to Go channels, noting that Elixir processes are named mailboxes tied to a process while Go channels are anonymous and separate from routines. The document is authored by Samuel Lampa from Uppsala University.
Profiling go code a beginners tutorialSamuel Lampa
This document summarizes a presentation on profiling Go code. It introduces pprof, a tool created by Dave Cheney that makes profiling Go code easier. It demonstrates pprof by profiling a string processing program and showing the performance improvements from various optimizations. It recommends resources for learning more about profiling Go programs with pprof and high performance Go programming.
This document provides an overview of flow-based programming (FBP). FBP is a programming paradigm where applications are defined as networks of black box processes that exchange data through predefined connections. These connections can be redefined without changing the internal processes, allowing for endless reconfiguration. FBP was invented in the 1960s and has seen a resurgence of interest with tools like NoFlo that allow building distributed applications as connected processes. The document discusses several open source FBP implementations and frameworks and provides examples of how FBP has been used to build applications and bioinformatics libraries.
RDFIO is an RDF import and query extension for MediaWiki. It allows users to import RDF triples into MediaWiki and query the triples using SPARQL. The architecture includes an in-memory RDF store to hold the triples and a SPARQL endpoint for querying. Future plans include enhancing editing capabilities via templates and importing triples on a per-page basis. Samuel Lampa presented RDFIO and is looking for additional ideas to improve the extension.
My lightning talk at Go Stockholm meetup Aug 6th 2013Samuel Lampa
This document discusses flow-based programming, an approach to programming invented in the 1970s where the flow of data between components is emphasized. It was successfully used in several domains including data analysis, banking software, and digital signal processing. New implementations of flow-based programming include NoFlo for Node.js and GoFlow, an open-source implementation in Go. More information on flow-based programming can be found on the listed websites.
Hooking up Semantic MediaWiki with external tools via SPARQLSamuel Lampa
This document discusses integrating Semantic MediaWiki (SMW) with external tools using the RDFIO extension. It describes the motivation for RDFIO as allowing manual schema exploration, automated data generation, and community collaboration. RDFIO solves problems with SMW by allowing the choice of wiki page titles for RDF entities and exporting RDF in the original import format. Real-world uses of RDFIO include visualizing data on SMW pages and pulling data from R into SMW using SPARQL queries. The integration of SMW and Bioclipse is also discussed.
Samuel Lampa presented his MSc thesis on integrating SWI-Prolog as a semantic querying tool in Bioclipse. He demonstrated [1] how SWI-Prolog can be used for semantic querying of biological data in RDF format within Bioclipse, [2] examples of SPARQL and Prolog code used to perform semantic queries, and [3] benchmarking of Prolog's performance as a semantic querying tool. The work adds new semantic querying functionality to Bioclipse using SWI-Prolog and demonstrates its ability to efficiently query biological data.
3rd Proj. Update: Integrating SWI-Prolog for Semantic Reasoning in BioclipseSamuel Lampa
This document summarizes Samuel Lampa's 2010 degree project on integrating SWI-Prolog for semantic reasoning in Bioclipse. It compares SWI-Prolog to other semantic tools like Jena and Pellet in terms of speed and expressiveness when querying biochemical data. Prolog code is presented for querying NMR spectrum data that finds molecules with peak values near a search value. SPARQL queries for the same use case are also shown. Observations indicate Prolog is fastest while SPARQL is easier to understand but Prolog allows easier parameter changes and logic reuse. A final presentation was planned for April 28, 2010.
2nd Proj. Update: Integrating SWI-Prolog for Semantic Reasoning in BioclipseSamuel Lampa
Contains a small background on the semantic web, and shows how Prolog is thought to be used from inside Bioclipse research software for RDF data handling.
Height and depth gauge linear metrology.pdfq30122000
Height gauges may also be used to measure the height of an object by using the underside of the scriber as the datum. The datum may be permanently fixed or the height gauge may have provision to adjust the scale, this is done by sliding the scale vertically along the body of the height gauge by turning a fine feed screw at the top of the gauge; then with the scriber set to the same level as the base, the scale can be matched to it. This adjustment allows different scribers or probes to be used, as well as adjusting for any errors in a damaged or resharpened probe.
AI in customer support Use cases solutions development and implementation.pdfmahaffeycheryld
AI in customer support will integrate with emerging technologies such as augmented reality (AR) and virtual reality (VR) to enhance service delivery. AR-enabled smart glasses or VR environments will provide immersive support experiences, allowing customers to visualize solutions, receive step-by-step guidance, and interact with virtual support agents in real-time. These technologies will bridge the gap between physical and digital experiences, offering innovative ways to resolve issues, demonstrate products, and deliver personalized training and support.
https://www.leewayhertz.com/ai-in-customer-support/#How-does-AI-work-in-customer-support
This study Examines the Effectiveness of Talent Procurement through the Imple...DharmaBanothu
In the world with high technology and fast
forward mindset recruiters are walking/showing interest
towards E-Recruitment. Present most of the HRs of
many companies are choosing E-Recruitment as the best
choice for recruitment. E-Recruitment is being done
through many online platforms like Linkedin, Naukri,
Instagram , Facebook etc. Now with high technology E-
Recruitment has gone through next level by using
Artificial Intelligence too.
Key Words : Talent Management, Talent Acquisition , E-
Recruitment , Artificial Intelligence Introduction
Effectiveness of Talent Acquisition through E-
Recruitment in this topic we will discuss about 4important
and interlinked topics which are
Open Channel Flow: fluid flow with a free surfaceIndrajeet sahu
Open Channel Flow: This topic focuses on fluid flow with a free surface, such as in rivers, canals, and drainage ditches. Key concepts include the classification of flow types (steady vs. unsteady, uniform vs. non-uniform), hydraulic radius, flow resistance, Manning's equation, critical flow conditions, and energy and momentum principles. It also covers flow measurement techniques, gradually varied flow analysis, and the design of open channels. Understanding these principles is vital for effective water resource management and engineering applications.
This presentation is about Food Delivery Systems and how they are developed using the Software Development Life Cycle (SDLC) and other methods. It explains the steps involved in creating a food delivery app, from planning and designing to testing and launching. The slide also covers different tools and technologies used to make these systems work efficiently.
Prediction of Electrical Energy Efficiency Using Information on Consumer's Ac...PriyankaKilaniya
Energy efficiency has been important since the latter part of the last century. The main object of this survey is to determine the energy efficiency knowledge among consumers. Two separate districts in Bangladesh are selected to conduct the survey on households and showrooms about the energy and seller also. The survey uses the data to find some regression equations from which it is easy to predict energy efficiency knowledge. The data is analyzed and calculated based on five important criteria. The initial target was to find some factors that help predict a person's energy efficiency knowledge. From the survey, it is found that the energy efficiency awareness among the people of our country is very low. Relationships between household energy use behaviors are estimated using a unique dataset of about 40 households and 20 showrooms in Bangladesh's Chapainawabganj and Bagerhat districts. Knowledge of energy consumption and energy efficiency technology options is found to be associated with household use of energy conservation practices. Household characteristics also influence household energy use behavior. Younger household cohorts are more likely to adopt energy-efficient technologies and energy conservation practices and place primary importance on energy saving for environmental reasons. Education also influences attitudes toward energy conservation in Bangladesh. Low-education households indicate they primarily save electricity for the environment while high-education households indicate they are motivated by environmental concerns.
Determination of Equivalent Circuit parameters and performance characteristic...pvpriya2
Includes the testing of induction motor to draw the circle diagram of induction motor with step wise procedure and calculation for the same. Also explains the working and application of Induction generator
Null Bangalore | Pentesters Approach to AWS IAMDivyanshu
#Abstract:
- Learn more about the real-world methods for auditing AWS IAM (Identity and Access Management) as a pentester. So let us proceed with a brief discussion of IAM as well as some typical misconfigurations and their potential exploits in order to reinforce the understanding of IAM security best practices.
- Gain actionable insights into AWS IAM policies and roles, using hands on approach.
#Prerequisites:
- Basic understanding of AWS services and architecture
- Familiarity with cloud security concepts
- Experience using the AWS Management Console or AWS CLI.
- For hands on lab create account on [killercoda.com](https://killercoda.com/cloudsecurity-scenario/)
# Scenario Covered:
- Basics of IAM in AWS
- Implementing IAM Policies with Least Privilege to Manage S3 Bucket
- Objective: Create an S3 bucket with least privilege IAM policy and validate access.
- Steps:
- Create S3 bucket.
- Attach least privilege policy to IAM user.
- Validate access.
- Exploiting IAM PassRole Misconfiguration
-Allows a user to pass a specific IAM role to an AWS service (ec2), typically used for service access delegation. Then exploit PassRole Misconfiguration granting unauthorized access to sensitive resources.
- Objective: Demonstrate how a PassRole misconfiguration can grant unauthorized access.
- Steps:
- Allow user to pass IAM role to EC2.
- Exploit misconfiguration for unauthorized access.
- Access sensitive resources.
- Exploiting IAM AssumeRole Misconfiguration with Overly Permissive Role
- An overly permissive IAM role configuration can lead to privilege escalation by creating a role with administrative privileges and allow a user to assume this role.
- Objective: Show how overly permissive IAM roles can lead to privilege escalation.
- Steps:
- Create role with administrative privileges.
- Allow user to assume the role.
- Perform administrative actions.
- Differentiation between PassRole vs AssumeRole
Try at [killercoda.com](https://killercoda.com/cloudsecurity-scenario/)
Digital Twins Computer Networking Paper Presentation.pptxaryanpankaj78
A Digital Twin in computer networking is a virtual representation of a physical network, used to simulate, analyze, and optimize network performance and reliability. It leverages real-time data to enhance network management, predict issues, and improve decision-making processes.
We have designed & manufacture the Lubi Valves LBF series type of Butterfly Valves for General Utility Water applications as well as for HVAC applications.
Using Flow-based programming to write tools and workflows for Scientific Computing in Go
1. Using Flow-based programming
... to write Tools and Workflows
for Scientific Computing
Go Stockholm Conference Oct 6, 2018
Samuel Lampa | bionics.it | @saml (slack) | @smllmp (twitter)
Ex - Dept. of Pharm. Biosci, Uppsala University | www.farmbio.uu.se | pharmb.io
Savantic AB savantic.se | RIL Partner AB rilpartner.com
2. About the speaker
● Name: Samuel Lampa
● PhD in Pharm. Bioinformatics from UU / pharmb.io (since 1 week)
● Researched: Flow-based programming-based workflow tools to build
predictive models for drug discovery
● Previously: HPC sysadmin & developer,Web developer,etc,
M.Sc. in molecular biotechnology engineering
● Next week: R&D Engineer at Savantic AB (savanticab.com)
● (Also:AfricArxiv (africarxiv.org) and RIL Partner AB (rilpartner.com))
3. Read more about my research
bit.ly/samlthesis →
(bionics.it/posts/phdthesis)
10. ● Black box, asynchronously running processes
● Data exchange across predefined connections
between named ports (with bounded buffers) by
message passing only
● Connections specified separately from processes
● Processes can be reconnected endlessly to form
different applications without having being changed
internally
FBP in brief
14. The Central Dogma of Biology …
… from DNA to RNA to Proteins
DNA
mRNA
Protein
Image credits: Nicolle Rager, National Science Foundation. License: Public domain
Amino acids
Ribosome
RNA
polymerase
Cell nucleus
Cell
15. “FBP is a particular form of dataflow
programming based on bounded buffers,
information packets with defined lifetimes,
named ports, and separate definition
of connections”
FBP vs Dataflow
16. ● Change of connection wiring
without rewriting components
● Inherently concurrent - suited
for the multi-core CPU world
● Testing, monitoring and logging very
easy: Just plug in a mock-, logging-
or debugging component.
● Etc etc ...
Benefits abound
20. Generator functions
Adapted from Rob Pike’s slides: talks.golang.org/2012/concurrency.slide#25
func main() {
c := generateInts(10) // Call function to get a channel
for v := range c { // … and loop over it
fmt.Println(v)
}
}
func generateInts(max int) <-chan int { // Return a channel of ints
c := make(chan int)
go func() { // Init go-routine inside function
defer close(c)
for i := 0; i <= max; i++ {
c <- i
}
}()
return c // Return the channel
}
21. Chaining generator functions 1/2
func reverse(cin chan string) chan string {
cout := make(chan string)
go func() {
defer close(cout)
for s := range cin { // Loop over in-chan
cout <- reverse(s) // Send on out-chan
}
}()
return cout
}
22. Chaining generator functions 2/2
// Chain the generator functions
dna := generateDNA() // Generator func of strings
rev := reverse(dna)
compl := complement(rev)
// Drive the chain by reading from last channel
for dnaString := range compl {
fmt.Println(dnaString)
}
23. Chaining generator functions 2/2
// Chain the generator functions
dna := generateDNA() // Generator func of strings
rev := reverse(dna)
compl := complement(rev)
// Drive the chain by reading from last channel
for dnaString := range compl {
fmt.Println(dnaString)
}
24. Problems with the generator approach
● Inputs not named in connection code (no keyword arguments)
● Multiple return values depend on positional arguments:
leftPart, rightPart := splitInHalves(chanOfStrings)
25. Could we emulate named ports?
type P struct {
in chan string // Channels as struct fields, to act as “named ports”
out chan string
}
func NewP() *P { // Initialize a new component
return &P{
in: make(chan string, 16),
out: make(chan string, 16),
}
}
func (p *P) Run() {
defer close(p.out)
for s := range p.in { // Refer to struct fields when reading ...
p.out <- s // ... and writing
}
}
26. Could we emulate named ports?
func main() {
p1 := NewP()
p2 := NewP()
p2.in = p1.out // Connect dependencies here, by assigning to same chan
go p1.Run()
go p2.Run()
go func() { // Feed the input of the network
defer close(p1.in)
for i := 0; i <= 10; i++ {
p1.in <- "Hej"
}
}()
for s := range p2.out { // Drive the chain from the main go-routine
fmt.Println(s)
}
}
27. Add almost no additional code, and get:
flowbase.org
28. Real-world use of FlowBase
● RDF (Semantic) MediaWiki XML→
● Import via MediaWiki XML import
● Code: github.com/rdfio/rdf2smw
● Paper: bit.ly/rdfiopub
30. Taking it further: Port structs
ttlFileRead.OutTriple().To(aggregator.In())
aggregator.Out().To(indexCreator.In())
indexCreator.Out().To(indexToAggr.In())
indexCreator.Out().To(triplesToWikiConverter.InIndex())
indexToAggr.Out().To(triplesToWikiConverter.InAggregate())
triplesToWikiConverter.OutPage().To(xmlCreator.InWikiPage())
xmlCreator.OutTemplates().To(templateWriter.In())
xmlCreator.OutProperties().To(propertyWriter.In())
xmlCreator.OutPages().To(pageWriter.In())
(So far only used in SciPipe, not yet FlowBase)
32. SciPipe
● Workflow
● Keeps track of dependency graph
● Process
● Added to workflows
● Long-running
● Typically one per operation
● Task
● Spawned by processes
● Executes just one shell command or custom Go function
● Typically one task spawned per operation on a set of input files
● Information Packet (IP)
● Most common data type passed between processes
Workflow
Process
File IP
Task
Task
Task
33. “Hello World” in SciPipe
package main
import (
// Import the SciPipe package, aliased to 'sp'
sp "github.com/scipipe/scipipe"
)
func main() {
// Init workflow with a name, and max concurrent tasks
wf := sp.NewWorkflow("hello_world", 4)
// Initialize processes and set output file paths
hello := wf.NewProc("hello", "echo 'Hello ' > {o:out}")
hello.SetOut("out", "hello.txt")
world := wf.NewProc("world", "echo $(cat {i:in}) World >> {o:out}")
world.SetOut("out", "{i:in|%.txt}_world.txt")
// Connect network
world.In("in").From(hello.Out("out"))
// Run workflow
wf.Run()
}
34. “Hello World” in SciPipe
package main
import (
// Import the SciPipe package, aliased to 'sp'
sp "github.com/scipipe/scipipe"
)
func main() {
// Init workflow with a name, and max concurrent tasks
wf := sp.NewWorkflow("hello_world", 4)
// Initialize processes and set output file paths
hello := wf.NewProc("hello", "echo 'Hello ' > {o:out}")
hello.SetOut("out", "hello.txt")
world := wf.NewProc("world", "echo $(cat {i:in}) World >> {o:out}")
world.SetOut("out", "{i:in|%.txt}_world.txt")
// Connect network
world.In("in").From(hello.Out("out"))
// Run workflow
wf.Run()
}
● Import SciPipe
● Set up any default variables
or data, handle flags etc
● Initiate workflow
● Create processes
● Define outputs and paths
● Connect outputs to inputs
(dependencies / data flow)
● Run the workflow
35. “Hello World” in SciPipe
package main
import (
// Import the SciPipe package, aliased to 'sp'
sp "github.com/scipipe/scipipe"
)
func main() {
// Init workflow with a name, and max concurrent tasks
wf := sp.NewWorkflow("hello_world", 4)
// Initialize processes and set output file paths
hello := wf.NewProc("hello", "echo 'Hello ' > {o:out}")
hello.SetOut("out", "hello.txt")
world := wf.NewProc("world", "echo $(cat {i:in}) World >> {o:out}")
world.SetOut("out", "{i:in|%.txt}_world.txt")
// Connect network
world.In("in").From(hello.Out("out"))
// Run workflow
wf.Run()
}
● Import SciPipe
● Set up any default variables
or data, handle flags etc
● Initiate workflow
● Create processes
● Define outputs and paths
● Connect outputs to inputs
(dependencies / data flow)
● Run the workflow
36. “Hello World” in SciPipe
package main
import (
// Import the SciPipe package, aliased to 'sp'
sp "github.com/scipipe/scipipe"
)
func main() {
// Init workflow with a name, and max concurrent tasks
wf := sp.NewWorkflow("hello_world", 4)
// Initialize processes and set output file paths
hello := wf.NewProc("hello", "echo 'Hello ' > {o:out}")
hello.SetOut("out", "hello.txt")
world := wf.NewProc("world", "echo $(cat {i:in}) World >> {o:out}")
world.SetOut("out", "{i:in|%.txt}_world.txt")
// Connect network
world.In("in").From(hello.Out("out"))
// Run workflow
wf.Run()
}
● Import SciPipe
● Set up any default variables
or data, handle flags etc
● Initiate workflow
● Create processes
● Define outputs and paths
● Connect outputs to inputs
(dependencies / data flow)
● Run the workflow
37. “Hello World” in SciPipe
package main
import (
// Import the SciPipe package, aliased to 'sp'
sp "github.com/scipipe/scipipe"
)
func main() {
// Init workflow with a name, and max concurrent tasks
wf := sp.NewWorkflow("hello_world", 4)
// Initialize processes and set output file paths
hello := wf.NewProc("hello", "echo 'Hello ' > {o:out}")
hello.SetOut("out", "hello.txt")
world := wf.NewProc("world", "echo $(cat {i:in}) World >> {o:out}")
world.SetOut("out", "{i:in|%.txt}_world.txt")
// Connect network
world.In("in").From(hello.Out("out"))
// Run workflow
wf.Run()
}
● Import SciPipe
● Set up any default variables
or data, handle flags etc
● Initiate workflow
● Create processes
● Define outputs and paths
● Connect outputs to inputs
(dependencies / data flow)
● Run the workflow
38. “Hello World” in SciPipe
package main
import (
// Import the SciPipe package, aliased to 'sp'
sp "github.com/scipipe/scipipe"
)
func main() {
// Init workflow with a name, and max concurrent tasks
wf := sp.NewWorkflow("hello_world", 4)
// Initialize processes and set output file paths
hello := wf.NewProc("hello", "echo 'Hello ' > {o:out}")
hello.SetOut("out", "hello.txt")
world := wf.NewProc("world", "echo $(cat {i:in}) World >> {o:out}")
world.SetOut("out", "{i:in|%.txt}_world.txt")
// Connect network
world.In("in").From(hello.Out("out"))
// Run workflow
wf.Run()
}
● Import SciPipe
● Set up any default variables
or data, handle flags etc
● Initiate workflow
● Create processes
● Define outputs and paths
● Connect outputs to inputs
(dependencies / data flow)
● Run the workflow
39. “Hello World” in SciPipe
package main
import (
// Import the SciPipe package, aliased to 'sp'
sp "github.com/scipipe/scipipe"
)
func main() {
// Init workflow with a name, and max concurrent tasks
wf := sp.NewWorkflow("hello_world", 4)
// Initialize processes and set output file paths
hello := wf.NewProc("hello", "echo 'Hello ' > {o:out}")
hello.SetOut("out", "hello.txt")
world := wf.NewProc("world", "echo $(cat {i:in}) World >> {o:out}")
world.SetOut("out", "{i:in|%.txt}_world.txt")
// Connect network
world.In("in").From(hello.Out("out"))
// Run workflow
wf.Run()
}
● Import SciPipe
● Set up any default variables
or data, handle flags etc
● Initiate workflow
● Create processes
● Define outputs and paths
● Connect outputs & inputs
(dependencies / data flow)
● Run the workflow
40. “Hello World” in SciPipe
package main
import (
// Import the SciPipe package, aliased to 'sp'
sp "github.com/scipipe/scipipe"
)
func main() {
// Init workflow with a name, and max concurrent tasks
wf := sp.NewWorkflow("hello_world", 4)
// Initialize processes and set output file paths
hello := wf.NewProc("hello", "echo 'Hello ' > {o:out}")
hello.SetOut("out", "hello.txt")
world := wf.NewProc("world", "echo $(cat {i:in}) World >> {o:out}")
world.SetOut("out", "{i:in|%.txt}_world.txt")
// Connect network
world.In("in").From(hello.Out("out"))
// Run workflow
wf.Run()
}
● Import SciPipe
● Set up any default variables
or data, handle flags etc
● Initiate workflow
● Create processes
● Define outputs and paths
● Connect outputs to inputs
(dependencies / data flow)
● Run the workflow
41. Writing SciPipe workflows
package main
import (
"github.com/scipipe/scipipe"
)
const dna = "AAAGCCCGTGGGGGACCTGTTC"
func main() {
wf := scipipe.NewWorkflow("DNA Base Complement Workflow", 4)
makeDNA := wf.NewProc("Make DNA", "echo "+dna+" > {o:dna}")
makeDNA.SetOut("dna", "dna.txt")
complmt := wf.NewProc("Base Complement", "cat {i:in} | tr ATCG TAGC > {o:compl}")
complmt.SetOut("compl", "{i:in|%.txt}.compl.txt")
reverse := wf.NewProc("Reverse", "cat {i:in} | rev > {o:rev}")
reverse.SetOut("rev", "{i:in|%.txt}.rev.txt")
complmt.In("in").From(makeDNA.Out("dna"))
reverse.In("in").From(complmt.Out("compl"))
wf.Run()
}
● Import SciPipe
● Set up any default variables
or data, handle flags etc
● Initiate workflow
● Create processes
● Define outputs and paths
● Connect outputs to inputs
(dependencies / data flow)
● Run the workflow
42. Writing SciPipe workflows
package main
import (
"github.com/scipipe/scipipe"
)
const dna = "AAAGCCCGTGGGGGACCTGTTC"
func main() {
wf := scipipe.NewWorkflow("DNA Base Complement Workflow", 4)
makeDNA := wf.NewProc("Make DNA", "echo "+dna+" > {o:dna}")
makeDNA.SetOut("dna", "dna.txt")
complmt := wf.NewProc("Base Complement", "cat {i:in} | tr ATCG TAGC > {o:compl}")
complmt.SetOut("compl", "{i:in|%.txt}.compl.txt")
reverse := wf.NewProc("Reverse", "cat {i:in} | rev > {o:rev}")
reverse.SetOut("rev", "{i:in|%.txt}.rev.txt")
complmt.In("in").From(makeDNA.Out("dna"))
reverse.In("in").From(complmt.Out("compl"))
wf.Run()
}
● Import SciPipe
● Set up any default variables
or data, handle flags etc
● Initiate workflow
● Create processes
● Define outputs and paths
● Connect outputs to inputs
(dependencies / data flow)
● Run the workflow
43. Writing SciPipe workflows
package main
import (
"github.com/scipipe/scipipe"
)
const dna = "AAAGCCCGTGGGGGACCTGTTC"
func main() {
wf := scipipe.NewWorkflow("DNA Base Complement Workflow", 4)
makeDNA := wf.NewProc("Make DNA", "echo "+dna+" > {o:dna}")
makeDNA.SetOut("dna", "dna.txt")
complmt := wf.NewProc("Base Complement", "cat {i:in} | tr ATCG TAGC > {o:compl}")
complmt.SetOut("compl", "{i:in|%.txt}.compl.txt")
reverse := wf.NewProc("Reverse", "cat {i:in} | rev > {o:rev}")
reverse.SetOut("rev", "{i:in|%.txt}.rev.txt")
complmt.In("in").From(makeDNA.Out("dna"))
reverse.In("in").From(complmt.Out("compl"))
wf.Run()
}
● Import SciPipe
● Set up any default variables
or data, handle flags etc
● Initiate workflow
● Create processes
● Define outputs and paths
● Connect outputs to inputs
(dependencies / data flow)
● Run the workflow
44. Writing SciPipe workflows
package main
import (
"github.com/scipipe/scipipe"
)
const dna = "AAAGCCCGTGGGGGACCTGTTC"
func main() {
wf := scipipe.NewWorkflow("DNA Base Complement Workflow", 4)
makeDNA := wf.NewProc("Make DNA", "echo "+dna+" > {o:dna}")
makeDNA.SetOut("dna", "dna.txt")
complmt := wf.NewProc("Base Complement", "cat {i:in} | tr ATCG TAGC > {o:compl}")
complmt.SetOut("compl", "{i:in|%.txt}.compl.txt")
reverse := wf.NewProc("Reverse", "cat {i:in} | rev > {o:rev}")
reverse.SetOut("rev", "{i:in|%.txt}.rev.txt")
complmt.In("in").From(makeDNA.Out("dna"))
reverse.In("in").From(complmt.Out("compl"))
wf.Run()
}
● Import SciPipe
● Set up any default variables
or data, handle flags etc
● Initiate workflow
● Create processes
● Define outputs and paths
● Connect outputs to inputs
(dependencies / data flow)
● Run the workflow
45. Writing SciPipe workflows
package main
import (
"github.com/scipipe/scipipe"
)
const dna = "AAAGCCCGTGGGGGACCTGTTC"
func main() {
wf := scipipe.NewWorkflow("DNA Base Complement Workflow", 4)
makeDNA := wf.NewProc("Make DNA", "echo "+dna+" > {o:dna}")
makeDNA.SetOut("dna", "dna.txt")
complmt := wf.NewProc("Base Complement", "cat {i:in} | tr ATCG TAGC > {o:compl}")
complmt.SetOut("compl", "{i:in|%.txt}.compl.txt")
reverse := wf.NewProc("Reverse", "cat {i:in} | rev > {o:rev}")
reverse.SetOut("rev", "{i:in|%.txt}.rev.txt")
complmt.In("in").From(makeDNA.Out("dna"))
reverse.In("in").From(complmt.Out("compl"))
wf.Run()
}
● Import SciPipe
● Set up any default variables
or data, handle flags etc
● Initiate workflow
● Create processes
● Define outputs and paths
● Connect outputs to inputs
(dependencies / data flow)
● Run the workflow
46. Writing SciPipe workflows
package main
import (
"github.com/scipipe/scipipe"
)
const dna = "AAAGCCCGTGGGGGACCTGTTC"
func main() {
wf := scipipe.NewWorkflow("DNA Base Complement Workflow", 4)
makeDNA := wf.NewProc("Make DNA", "echo "+dna+" > {o:dna}")
makeDNA.SetOut("dna", "dna.txt")
complmt := wf.NewProc("Base Complement", "cat {i:in} | tr ATCG TAGC > {o:compl}")
complmt.SetOut("compl", "{i:in|%.txt}.compl.txt")
reverse := wf.NewProc("Reverse", "cat {i:in} | rev > {o:rev}")
reverse.SetOut("rev", "{i:in|%.txt}.rev.txt")
complmt.In("in").From(makeDNA.Out("dna"))
reverse.In("in").From(complmt.Out("compl"))
wf.Run()
}
● Import SciPipe
● Set up any default variables
or data, handle flags etc
● Initiate workflow
● Create processes
● Define outputs and paths
● Connect outputs to inputs
(dependencies / data flow)
● Run the workflow
47. Writing SciPipe workflows
package main
import (
"github.com/scipipe/scipipe"
)
const dna = "AAAGCCCGTGGGGGACCTGTTC"
func main() {
wf := scipipe.NewWorkflow("DNA Base Complement Workflow", 4)
makeDNA := wf.NewProc("Make DNA", "echo "+dna+" > {o:dna}")
makeDNA.SetOut("dna", "dna.txt")
complmt := wf.NewProc("Base Complement", "cat {i:in} | tr ATCG TAGC > {o:compl}")
complmt.SetOut("compl", "{i:in|%.txt}.compl.txt")
reverse := wf.NewProc("Reverse", "cat {i:in} | rev > {o:rev}")
reverse.SetOut("rev", "{i:in|%.txt}.rev.txt")
complmt.In("in").From(makeDNA.Out("dna"))
reverse.In("in").From(complmt.Out("compl"))
wf.Run()
}
● Import SciPipe
● Set up any default variables
or data, handle flags etc
● Initiate workflow
● Create processes
● Define outputs and paths
● Connect outputs to inputs
(dependencies / data flow)
● Run the workflow
51. Turn Audit log into TeX/PDF report
TeX template by Jonathan Alvarsson @jonalv
52. ● Intuitive behaviour: Like conveyor belts & stations in a factory.
● Flexible: Combine command-line programs with Go components
● Custom file naming: Easy to manually browse output files
● Portable: Distribute as Go code or as compiled executable files
● Easy to debug: Use any Go debugging tools or even just println()
● Powerful audit logging: Stream outputs via UNIX FIFO files
● Efficient & Parallel: Fast code + Efficient use of multi-core CPU
Benefits of SciPipe - Thanks to Go + FBP
54. Thank you for your time!
Using Flow-based programming
... to write Tools and Workflows for Scientific Computing
Talk at Go Stockholm Conference Oct 6, 2018
Samuel Lampa | bionics.it | @saml (slack) | @smllmp (twitter)
Dept. of Pharm. Biosci, Uppsala University | www.farmbio.uu.se | pharmb.io