DataScienceLab2017_Блиц-доклад

•

1 like•177 views

DataScience Lab, 13 мая 2017 Распределенные вычисления: использование BOINC в Data Science Виталий Кошура (Software Developer at Lohika) BOINC - это открытое программное обеспечение для распределенных вычислений. Данный доклад освещает использование приложения BOINC в различных областях науки, которые связаны с обработкой огромных массивов данных, на примере текущих активных исследовательских проектов. Все материалы доступны по ссылке: http://datascience.in.ua/report2017

Distributed calculations: Use
of BOINC in Data Science
Vitalii Koshura, Software Developer @ Lohika
<lestat.de.lionkur@gmail.com>

BOINC -
Berkeley Open Infrastructure for Network Computing
From: Wikipedia

Supported devices, platforms and technologies

Power comparison
Sunway TaihuLight
93 PFLOPS
BOINC
158 PFLOPS

https://github.com/BOINC/boinc
BOINC on GitHub

Launch new project
● Server
○ Install
■ VirtualBox image: https://boinc.berkeley.edu/trac/wiki/VmServer
■ Docker container: https://github.com/marius311/boinc-server-docker
■ make_project script: https://boinc.berkeley.edu/trac/wiki/MakeProject
○ Configure
■ config.xml
■ project.inc
● Task Generator: https://boinc.berkeley.edu/trac/wiki/WorkGeneration
● Validator: https://boinc.berkeley.edu/trac/wiki/ValidationSimple

Launch new project
● Client application
○ CPU
■ Singlethread app: https://github.com/BOINC/boinc/tree/master/samples/example_app
■ Mutithread app: https://github.com/BOINC/boinc/tree/master/samples/multi_thread
○ GPU
■ OpenCL (Intel, AMD, Nvidia): https://github.com/BOINC/boinc/tree/master/samples/openclapp
■ CUDA (Nvidia): https://github.com/BOINC/boinc/tree/master/samples/nvcuda
○ Android: https://github.com/BOINC/boinc/tree/master/samples/example_app
○ VirtualBox: https://boinc.berkeley.edu/trac/wiki/VmApps

● http://boinc.berkeley.edu/dev/forum_forum.php?id=11
● https://vk.com/boinc
● ...
Share info about project

Growing toward Criteo scale means that sometimes we do need to re-invent the wheel. We will share with you real life examples of what we have done in our C# stack to achieve that scale. For instance what happens when traditional load balancing is too costly to scale ? What about migrating monolithic application to service oriented when you have got spaghetti code ? You should monitor everything right ? But can you and should you measure and monitor your code down to the task level on your production machines. Apache Kafka looks good and you want to use it but is there any good enough implementation in C# ?

Jenkins x gitops

Jose Luis Sánchez Rebollo

This year, Geko has been invited by Dataops Barcelona 2019 to participate as a speaker in one of the several sessions available this year. Because Geko is a specialized company in CI/CD and Devops, the session offered by Geko was dedicated to Gitops and Jenkins X. In this session we talked about the importance of Gitops, IaC and how Jenkins X can help us with the CD part with a Kubernetes cluster. With Jankins X we can deploy an entire Kubernetes cluster just executing one command. Jenkins X helps us to start to develop and deploy our applications in Kubernetes in an easy way and without any strong knowledge about Docker and Kubernetes. But not only that, with Jenkins X we can define any kind of complex CI/CD pipeline in order to boost our application deployment and reliability.

C# development workflow @ criteo

Ibrahim Abubakari

When you have more than one hundred developers working on a large C# codebase that spans among many git repositories, it is hard to keep an agile delivery pipeline. Criteo solved this by using a trunk based development approach. Trunk based development is powerful but comes at cost and must be implemented carefully to work properly. In this talk, we will present the workflow and tools we have put in place to allow our developers to efficiently and safely interact with our C# codebase.

(Open Hack Night 2014) GitHub Tutorial

James Griffin

GitLab London Meetup: How Containerized Pipelines and Kubernetes Can Boost Yo...

Nico Meisenzahl

Learn how to eliminate any dependencies on your CI/CD build nodes and don’t bother yourself with multiple versions of your toolchain and any corresponding constraints. Walk away with knowledge and best practices that will help you to optimize your builds and deployments with containerized pipelines! Use containerized GitLab CI/CD pipelines and Kaniko to move your build and deployment workloads into your Kubernetes cluster. Build your apps and infrastructure without any external dependencies and constraints. If you are building a Go project, deploying an app to Kubernetes or just building your infrastructure. It doesn’t matter, anything is possible! Join Nico on a deep dive into the secrets of building hassle-free containerized build and deployment pipelines with GitLab CI/CD, Kubernetes and Kaniko.

Introduction à l’intégration continue avec Jenkins

Eric Hogue

Talk given at PHP Québec on August 6 2015 L'intégration continue permet d'exécuter automatiquement les tests à chaque fois que du code est poussé. Nous allons voir les types de tests qu'un serveur d'intégration continue peut exécuter. Comment interpréter les résultats de ces outils. Comment créer le script de ‘build'. Et comment configurer Jenkins pour qu'il exécute le build et affiche les rapport produits.

Introduction to Continuous Integration with Jenkins

Eric Hogue

Talk on Continuous Integration given at Sunshine PHP 2015 on February 7. Continuous Integration allows developers to run tests automatically every time code is pushed to source control. We will look at the different tests that a Continuous Integration server can run. How to create a build script so it runs the tests and make sure it produces the needed reports. Finally, how to set up Jenkins to run the build and display the reports.

Know the Science behind WorkFlows using Git & GitHhub

Edureka!

Hot deploy

Arief Warazuhudien

WSO2Con ASIA 2016: Automate and Orchestrate DevOps

WSO2

DevOps has enabled organizations to extend traditional development to continous IT service delivery. The advantages of adopting DevOps include establishing trust and collaboration between development and IT operations and ensuring that software releases are repeatable and automated. When adopting devops, automation and orchestration are quite important. During this session, Chamith will discuss the advantages, technologies and best practices of automation and orchestration. He will also talk about the importance of establishing common standards and protocols across the organization to ensure governance and reduce lifecycle costs.

Gitlab Commit: How Containerized GitLab CI Pipelines Can Help You Streamline ...

Nico Meisenzahl

Learn how to eliminate any dependencies between tool chains and local machines, and never again ask if Terraform CLI can be upgraded without breaking all projects. In this session, Nico will share best practices that will help attendees to optimize their Infrastructure deployment with GitLab CI pipelines. With GitLab CI, Nico will demonstrate how to build containerized pipelines to deploy infrastructure without any external dependencies and constraints. Join Nico on a deep dive into the secrets of building hassle-free infrastructure pipelines with GitLab CI.

Cloud Native CI/CD with Jenkins X and Knative Pipelines

C4Media

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2Pc3H50. Christie Wilson and James Rawlings explain the CI/CD challenges in a cloud native landscape, and show how Jenkins X rises to them by leveraging open source cloud native technologies like Knative Pipelines. They demo a GitOps based Jenkins X workflow, showing how simple Jenkins X makes it for developers to stage and deploy changes on demand. Filmed at qconlondon.com. Christie Wilson is a software engineer at Google, currently leading the knative build-pipeline project. Over the past ten years she has worked in the mobile, financial and video game industries. James Rawlings is a co-creator of the open source project Jenkins X and works for CloudBees, where he aims to help developers and teams move to the cloud.

Continuous Integration with Cloud Foundry Concourse and Docker on OpenPOWER

Indrajit Poddar

CI/CD with Jenkins and Docker - DevOps Meetup Day Thailand

Troublemaker Khunpech

Volunteer Computing using BOINC

Pooyan Mehrparvar

Data Science Workflows using Docker Containers

Aly Sivji

Building a data warehouse with Pentaho and Docker

Wellington Marinho

EDW CENIPA is a opensource project designed to enable analysis of aeronautical incidentes that occured in the brazilian civil aviation. The project uses techniques and BI tools that explore innovative low-cost technologies. Historically, Business Intelligence platforms are expensive and impracticable for small projects. BI projects require specialized skills and high development costs. This work aims to break this barrier.

Trust in Every Byte - Securing Edge Workflows with Fastly Compute [Cloud Nati...

Greg Hamer

A Love Story with Kubevirt and Backstage from Cloud Native NoVA meetup Feb 2024

Cloud Native NoVA

Hyperscale SIG update

Davide Cavalca

Kubecon 2019 Recap

Aarno Aukia

Continuous Delivery w projekcie Open Source - Marcin Stachniuk - DevCrowd 2017

MarcinStachniuk

Eclipse Con US 2013 - Buildroot Eclipse Bundle : A powerful IDE for Embedded ...

melbats

Buildroot is a tool designed by embedded Linux developers to build embedded Linux systems using cross-compilation. It allows to build a cross-compilation toolchain (or re-use an existing one), a root filesystem image with applications and libraries, a kernel image and a bootloader image or any combination of these. Buildroot is typically used to create the embedded Linux systems of industrial systems based on the ARM, PowerPC or x86 architectures, by integrating multiple existing open-source components (kernel, base components, graphical libraries, networking applications, etc.) and in-house components specific to the embedded system. Many of the early embedded Linux developers are used to work with text editors such as Emacs or Vi. However, in the recent years, the major expansion of the embedded Linux platform has attracted many more developers that are more familiar with feature-full, integrated, and user-friendly development environments. To fulfill this need, Eclipse provides an excellent platform, especially with extensions such as CDT, the Autotools plugin, the multiple plugins from the Linux tools project, the Target Communication Framework, and more. As many embedded Linux developers use Buildroot to build their system, it sounded natural to provide an easy-to-use integration with Eclipse. To achieve that, we: Developed an Eclipse plugin that automatically registers the available Buildroot toolchains into Eclipse CDT. This allows application developers to transparently build their application, and then execute it and debug it on the embedded target platform directly within Eclipse. Improved the pkg-config plugin for a better integration with different toolchains. Created a specialized Eclipse version bundling all the useful existing plug-ins for embedded Linux developers: Eclipse CDT, our specialized Buildroot plugin, the latest remote debugging mechanisms, the Autotools plugin, Linux Tools plugins, etc. At Eclipse Con Europe, we presented a first initial version of those developments, with basic support for compilation, remote execution and debugging. Through this new talk, we would like to present the improvements that have been brought to the project: integration with pkg-config, improvements of the remote debugging mechanisms, integration with the Autotools plugin and the LTT-ng plugins. We will demonstrate how Eclipse has been extended to match the needs of embedded Linux developers, and we will perform a live demonstration of these capabilities using an ARM development platform for which Eclipse will help build, execute and debug applications.

IoT Prototyping using BBB and Debian

Mender.io

Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC...

Jian-Hong Pan

Magento Docker Setup.pdf

Abid Malik

Advanced Code Flow, Notes From the Field

Ariel Moskovich

What's hot

Introduction à l'intégration continue en PHP

Eric Hogue

Introduction to Continuous Integration with Jenkins

Eric Hogue

Know the Science behind WorkFlows using Git & GitHhub

Edureka!

Hot deploy

Arief Warazuhudien

WSO2Con ASIA 2016: Automate and Orchestrate DevOps

WSO2

Gitlab Commit: How Containerized GitLab CI Pipelines Can Help You Streamline ...

Nico Meisenzahl

What's hot (6)

Introduction à l'intégration continue en PHP

Introduction to Continuous Integration with Jenkins

Know the Science behind WorkFlows using Git & GitHhub

Hot deploy

WSO2Con ASIA 2016: Automate and Orchestrate DevOps

Gitlab Commit: How Containerized GitLab CI Pipelines Can Help You Streamline ...

Similar to DataScienceLab2017_Блиц-доклад

Cloud Native CI/CD with Jenkins X and Knative Pipelines

C4Media

Continuous Integration with Cloud Foundry Concourse and Docker on OpenPOWER

Indrajit Poddar

CI/CD with Jenkins and Docker - DevOps Meetup Day Thailand

Troublemaker Khunpech

Volunteer Computing using BOINC

Pooyan Mehrparvar

Data Science Workflows using Docker Containers

Aly Sivji

Building a data warehouse with Pentaho and Docker

Wellington Marinho

Trust in Every Byte - Securing Edge Workflows with Fastly Compute [Cloud Nati...

Greg Hamer

A Love Story with Kubevirt and Backstage from Cloud Native NoVA meetup Feb 2024

Cloud Native NoVA

Hyperscale SIG update

Davide Cavalca

Kubecon 2019 Recap

Aarno Aukia

Continuous Delivery w projekcie Open Source - Marcin Stachniuk - DevCrowd 2017

MarcinStachniuk

Eclipse Con US 2013 - Buildroot Eclipse Bundle : A powerful IDE for Embedded ...

melbats

IoT Prototyping using BBB and Debian

Mender.io

Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC...

Jian-Hong Pan

Magento Docker Setup.pdf

Abid Malik

Advanced Code Flow, Notes From the Field

Ariel Moskovich

Building container images - Dockerfiles, Google/Kaniko, Projectatomic/buildah...

CloudYuga

Making your app soar without a container manifest

LibbySchulze

PittsburgJUG_Cloud-Native Dev Tools: Bringing the cloud back to earth

Grace Jansen

How can we effectively develop for the cloud, when we as developers are coding back down on earth? This is where effective cloud-native developer tools can enable us to either be transported into the cloud or alternatively, to bring the cloud back down to earth. But what tools should we be using for this? In this session, we’ll explore some of the useful OSS tools and technologies that can used by developers to effectively develop, design and test cloud-native Java applications.

CI/CD with Github Actions

Md. Minhazul Haque

Similar to DataScienceLab2017_Блиц-доклад (20)

Cloud Native CI/CD with Jenkins X and Knative Pipelines

Continuous Integration with Cloud Foundry Concourse and Docker on OpenPOWER

CI/CD with Jenkins and Docker - DevOps Meetup Day Thailand

Volunteer Computing using BOINC

Data Science Workflows using Docker Containers

Building a data warehouse with Pentaho and Docker

Trust in Every Byte - Securing Edge Workflows with Fastly Compute [Cloud Nati...

A Love Story with Kubevirt and Backstage from Cloud Native NoVA meetup Feb 2024

Hyperscale SIG update

Kubecon 2019 Recap

Continuous Delivery w projekcie Open Source - Marcin Stachniuk - DevCrowd 2017

Eclipse Con US 2013 - Buildroot Eclipse Bundle : A powerful IDE for Embedded ...

IoT Prototyping using BBB and Debian

Package a PyApp as a Flatpak Package: An HTTP Server for Example @ PyCon APAC...

Magento Docker Setup.pdf

Advanced Code Flow, Notes From the Field

Building container images - Dockerfiles, Google/Kaniko, Projectatomic/buildah...

Making your app soar without a container manifest

PittsburgJUG_Cloud-Native Dev Tools: Bringing the cloud back to earth

CI/CD with Github Actions

More from GeeksLab Odessa

DataScience Lab2017_Коррекция геометрических искажений оптических спутниковых...

DataScienceLab2017_Блиц-доклад

Recommended

Recommended

More Related Content

What's hot

What's hot (6)

Similar to DataScienceLab2017_Блиц-доклад

Similar to DataScienceLab2017_Блиц-доклад (20)

More from GeeksLab Odessa

More from GeeksLab Odessa (20)

Recently uploaded

Recently uploaded (20)

DataScienceLab2017_Блиц-доклад