This document provides instructions for configuring PC routing and Squid proxy server on a Linux system. It describes how to enable IP forwarding and NAT, install and configure Squid, set up directories for caching and blocking files, input blocked site keywords, reload the configuration, and set NAT rules to redirect port 80 traffic to the Squid proxy port 3128.
1. The document summarizes the author's experience setting up Chainer environments and comparing Chainer's performance on different hardware configurations.
2. When running a MNIST training example on a ThinkPad X260 laptop, it took 714 seconds to complete 20 epochs. On an AWS p2.xlarge instance with a GPU, it took 111 seconds, around 6.4 times faster than the laptop.
3. Estimated time for a Raspberry Pi 3 was over 18 hours, over 600 times slower than the GPU instance and over 90 times slower than the laptop.
1. The document provides instructions for setting up Nginx, uWSGI, Flask, and Python virtual environments to run a Python web application on an EC2 instance.
2. It includes steps for installing dependencies, configuring a Python virtual environment, installing front-end packages, and configuring Nginx and systemd services to run the application as a daemon.
3. The instructions cover deploying the application from a GitHub repository and updating configuration files for database access, AWS S3 bucket settings, and the Nginx server configuration.
The document discusses hardening a TYPO3 website to improve security. It recommends hardening each layer of a TYPO3 application including the operating system, web server, database, PHP, and extensions. Specific techniques mentioned include regularly updating software, enabling SSL, restricting backend access, separating code from public files, removing unneeded extensions, and automating deployment processes. The goal of hardening is to reduce the number of potential vulnerabilities by minimizing exposed services and code.
The document provides instructions for setting up web blocking on an OpenWRT router using DNS redirection or host files. It involves:
1) Connecting via SSH and editing the /etc/persistent/rc.poststart file to add redirection commands
2) Editing the firewall and DNSmasq configuration files to redirect DNS requests to the local router for blocking at the DNS level
3) Rebooting the router to apply the changes and then testing blocked domains.
The document provides steps to install and configure OpenVPN on CentOS to create a VPN server. It describes installing required packages, setting up bridging and network interfaces, generating certificates using EasyRSA, configuring iptables, and writing server and client configuration files.
FlexPod is a converged infrastructure solution that combines Cisco UCS servers and fabric interconnects with NetApp storage systems. It supports the provisioning of block (iSCSI) and file (NFS) storage volumes for use with an OpenStack cloud deployment using NetApp drivers. The document provides steps for creating volume types, provisioning NFS and iSCSI volumes from a NetApp storage system, and attaching the volumes to OpenStack instances to be used as block devices or mounted filesystems.
This document discusses setting up NLTK on CentOS using Python 2.6. It recommends installing EPEL to access more recent Python packages. virtualenv is used to create a sandboxed Python environment and mod_wsgi is configured to run NLTK applications as WSGI scripts under Apache. A simple WSGI application is provided as an example.
This document provides instructions for configuring PC routing and Squid proxy server on a Linux system. It describes how to enable IP forwarding and NAT, install and configure Squid, set up directories for caching and blocking files, input blocked site keywords, reload the configuration, and set NAT rules to redirect port 80 traffic to the Squid proxy port 3128.
1. The document summarizes the author's experience setting up Chainer environments and comparing Chainer's performance on different hardware configurations.
2. When running a MNIST training example on a ThinkPad X260 laptop, it took 714 seconds to complete 20 epochs. On an AWS p2.xlarge instance with a GPU, it took 111 seconds, around 6.4 times faster than the laptop.
3. Estimated time for a Raspberry Pi 3 was over 18 hours, over 600 times slower than the GPU instance and over 90 times slower than the laptop.
1. The document provides instructions for setting up Nginx, uWSGI, Flask, and Python virtual environments to run a Python web application on an EC2 instance.
2. It includes steps for installing dependencies, configuring a Python virtual environment, installing front-end packages, and configuring Nginx and systemd services to run the application as a daemon.
3. The instructions cover deploying the application from a GitHub repository and updating configuration files for database access, AWS S3 bucket settings, and the Nginx server configuration.
The document discusses hardening a TYPO3 website to improve security. It recommends hardening each layer of a TYPO3 application including the operating system, web server, database, PHP, and extensions. Specific techniques mentioned include regularly updating software, enabling SSL, restricting backend access, separating code from public files, removing unneeded extensions, and automating deployment processes. The goal of hardening is to reduce the number of potential vulnerabilities by minimizing exposed services and code.
The document provides instructions for setting up web blocking on an OpenWRT router using DNS redirection or host files. It involves:
1) Connecting via SSH and editing the /etc/persistent/rc.poststart file to add redirection commands
2) Editing the firewall and DNSmasq configuration files to redirect DNS requests to the local router for blocking at the DNS level
3) Rebooting the router to apply the changes and then testing blocked domains.
The document provides steps to install and configure OpenVPN on CentOS to create a VPN server. It describes installing required packages, setting up bridging and network interfaces, generating certificates using EasyRSA, configuring iptables, and writing server and client configuration files.
FlexPod is a converged infrastructure solution that combines Cisco UCS servers and fabric interconnects with NetApp storage systems. It supports the provisioning of block (iSCSI) and file (NFS) storage volumes for use with an OpenStack cloud deployment using NetApp drivers. The document provides steps for creating volume types, provisioning NFS and iSCSI volumes from a NetApp storage system, and attaching the volumes to OpenStack instances to be used as block devices or mounted filesystems.
This document discusses setting up NLTK on CentOS using Python 2.6. It recommends installing EPEL to access more recent Python packages. virtualenv is used to create a sandboxed Python environment and mod_wsgi is configured to run NLTK applications as WSGI scripts under Apache. A simple WSGI application is provided as an example.
This document outlines the steps to set up a 3 node Linux cluster on AWS and install Spark for distributed processing:
1. Create 3 Linux instances on AWS and allow all network traffic; generate SSH keys.
2. Configure the instances for SSH access without passwords using the generated keys and install Java.
3. Download and install Spark, configure environment variables, and start all Spark processes and services on the master node to allow distributed processing across the cluster.
The document discusses Nouka, an open source inventory management tool for Linux. Nouka consists of three parts - Nouka data collector, Naya data store, and Yaoya data converter. Nouka data collector runs commands periodically on Linux machines and sends the results to Naya data store. Naya uses Fluentd and MongoDB to store the collected data. Yaoya then converts and outputs the data in various formats like JSON, CSV for analysis. Overall, Nouka provides an automatic and periodic way to collect and centralize inventory data from Linux machines.
A massive attack was revealed that exploited the Shellshock vulnerability in QNAP NAS devices. The attackers deployed a payload that patched the vulnerability, armed the devices for DDOS attacks, and installed a scanner to search for more vulnerable devices. Over 500 compromised devices were detected. The payload installed a backdoor that could control the armed devices for DDOS attacks through IRC commands.
The document provides steps to install and configure various monitoring tools including Cacti, MySQL, SNMP, RRDtool, and plugins on a Linux server. It includes instructions for downloading packages, compiling source code, configuring services, and testing the monitoring installations.
The document provides instructions for configuring a DNS server on a Linux system using BIND. It involves installing BIND, editing configuration files such as named.conf and named.rfc1912.zones to define the domain zone and name server, copying and editing the zone file, changing permissions on the zone file, modifying named.caching-namedserver.conf, and restarting the named service. Finally, nslookup is used to test the name resolution.
agri inventory - nouka data collector / yaoya data convertorToshiaki Baba
This document provides instructions for setting up and using an agri inventory system called nouka and yaoya. Nouka collects data from servers using commands and sends it to the naya data store, which uses fluentd and MongoDB. It explains the components, data formats, and provides steps to get the required software and set up the system.
This document provides guidance on collecting diagnostic information from a Juniper firewall for different types of issues. It outlines 7 sections for collecting data on system crashes, high CPU usage, NSRP problems, throughput and forwarding performance issues, high memory usage, data flow problems, and static routing problems. For each issue, it lists basic information and detailed commands to run to gather relevant logs, configurations, performance metrics and debug outputs.
This document discusses how to view the most recent log entries in MongoDB by tailing the oplog, which contains replication metadata. It provides steps to set up a 3-node replica set, add documents to a collection, and use the local database to view the oplog.rs collection and see replication operations across nodes in real-time. Additional resources for learning about tailable cursors in MongoDB are also listed.
This document discusses various topics related to internet privacy and security including VPNs, proxies, ad blocking, and the Surge iOS app. It provides links to Wikipedia pages on topics like SOCKS and VPNs. It also includes links to the Surge GitHub for configuration examples and a script to generate single file configs, and provides the App Store link for the Surge app.
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)Naoto MATSUMOTO
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
20-Feb-2015
SAKURA Internet Research Center.
Senior Researcher / Naoto MATSUMOTO
Buildroot (www.buildroot.net) is a tool for building small embedded linux system.
For example : Kernel + Busybox + Perl + Dancer2 ~= 60 MB
Buildroot could create also Qemu Virtual Machine image with the same footprint.
Demo with a Olimex iMX233 board (45€).
The document describes the configuration of an OpenVPN site-to-site VPN tunnel between two networks (Side A and Side B). Key steps include generating and sharing a security key between the sites, configuring firewall rules and routing on each side, and starting the OpenVPN service to establish the encrypted tunnel between the 10.0.0.1 and 10.0.0.2 addresses. Once configured, connectivity between the 192.168.1.0/24 and 192.168.2.0/24 networks can be tested using ping and traceroute.
This document outlines the steps to configure a highly available NFS server using DRBD and heartbeat on Ubuntu. Key steps include:
1. Installing DRBD, heartbeat and configuring the /etc/drbd.conf and heartbeat configuration files on both NFS servers so they are identical.
2. Making the data partition consistent between servers using drbdadm commands.
3. Installing and configuring heartbeat to monitor services and fail over the virtual IP address when either NFS server fails.
4. Testing the configuration by stopping heartbeat on each server to trigger a failover and verifying services move to the other server.
How to install OpenStack MITAKA --allinone - cheat sheet -Naoto MATSUMOTO
How to install OpenStack MITAKA --allinone - cheat sheet -
27-Jun, 2016
SAKURA Internet, Inc. / SAKURA Internet Research Center
Senior Researcher / Naoto MATSUMOTO
This document discusses Python, including installing multiple Python versions on CentOS, creating virtual environments (venv) for different projects, using pip to install packages, and basic concepts like modules, packages, and running Python files. It provides instructions for cloning a Python build script from GitHub, shows commands for installing Python 3.5.2 and listing available Python versions, and demonstrates creating and activating a venv.
Quick Start Guide using Virtuozzo 7 (β) on AWS EC2Kentaro Ebisawa
Virtuozzo 7 was open sourced and available on Amazon EC2 since October 2015.
This document aims to give you a quick overview of steps to setup Virtuozzo on Amazon EC2.
This document discusses Composer, an open source tool for dependency management in PHP. It describes what Composer is, how to install it, how to define dependencies in a composer.json file and composer.lock file, how Composer generates autoload files and installs vendor libraries, and some common Composer commands. It also provides information on joining the community and lists sources for more documentation on Composer.
This document discusses using LVS (Linux Virtual Server) and Keepalived to implement high performance and high availability load balancing. It provides instructions on installing LVS and Keepalived, configuring LVS for load balancing, and using Keepalived to realize load balancing and high availability. Key aspects covered include using LVS-DR and real server scripts, and configuring Keepalived on the master and backup servers.
redis-benchmark with AMD RYZEN 1800X Intel Kaby Lake (i7-7700K) memoNaoto MATSUMOTO
The document compares the performance of Redis benchmarks run on an AMD Ryzen 1800X system and an Intel i7-7700K system. It provides system details for each such as the number of CPU cores, operating system, and CPU governor settings before running the benchmarks to test in-memory database performance between the two systems.
To install ns-3, the user must:
1. Download the ns-3 tar file and required dependencies by running apt-get commands.
2. Extract the ns-3 tar file and configure waf by running the ./waf configure command.
3. Install netanim for animation viewing and ensure mercurial and qt4-dev-tools are installed to support it.
4. Build and run the first sample program using ./waf and ./waf --run commands.
This document outlines the steps to set up a 3 node Linux cluster on AWS and install Spark for distributed processing:
1. Create 3 Linux instances on AWS and allow all network traffic; generate SSH keys.
2. Configure the instances for SSH access without passwords using the generated keys and install Java.
3. Download and install Spark, configure environment variables, and start all Spark processes and services on the master node to allow distributed processing across the cluster.
The document discusses Nouka, an open source inventory management tool for Linux. Nouka consists of three parts - Nouka data collector, Naya data store, and Yaoya data converter. Nouka data collector runs commands periodically on Linux machines and sends the results to Naya data store. Naya uses Fluentd and MongoDB to store the collected data. Yaoya then converts and outputs the data in various formats like JSON, CSV for analysis. Overall, Nouka provides an automatic and periodic way to collect and centralize inventory data from Linux machines.
A massive attack was revealed that exploited the Shellshock vulnerability in QNAP NAS devices. The attackers deployed a payload that patched the vulnerability, armed the devices for DDOS attacks, and installed a scanner to search for more vulnerable devices. Over 500 compromised devices were detected. The payload installed a backdoor that could control the armed devices for DDOS attacks through IRC commands.
The document provides steps to install and configure various monitoring tools including Cacti, MySQL, SNMP, RRDtool, and plugins on a Linux server. It includes instructions for downloading packages, compiling source code, configuring services, and testing the monitoring installations.
The document provides instructions for configuring a DNS server on a Linux system using BIND. It involves installing BIND, editing configuration files such as named.conf and named.rfc1912.zones to define the domain zone and name server, copying and editing the zone file, changing permissions on the zone file, modifying named.caching-namedserver.conf, and restarting the named service. Finally, nslookup is used to test the name resolution.
agri inventory - nouka data collector / yaoya data convertorToshiaki Baba
This document provides instructions for setting up and using an agri inventory system called nouka and yaoya. Nouka collects data from servers using commands and sends it to the naya data store, which uses fluentd and MongoDB. It explains the components, data formats, and provides steps to get the required software and set up the system.
This document provides guidance on collecting diagnostic information from a Juniper firewall for different types of issues. It outlines 7 sections for collecting data on system crashes, high CPU usage, NSRP problems, throughput and forwarding performance issues, high memory usage, data flow problems, and static routing problems. For each issue, it lists basic information and detailed commands to run to gather relevant logs, configurations, performance metrics and debug outputs.
This document discusses how to view the most recent log entries in MongoDB by tailing the oplog, which contains replication metadata. It provides steps to set up a 3-node replica set, add documents to a collection, and use the local database to view the oplog.rs collection and see replication operations across nodes in real-time. Additional resources for learning about tailable cursors in MongoDB are also listed.
This document discusses various topics related to internet privacy and security including VPNs, proxies, ad blocking, and the Surge iOS app. It provides links to Wikipedia pages on topics like SOCKS and VPNs. It also includes links to the Surge GitHub for configuration examples and a script to generate single file configs, and provides the App Store link for the Surge app.
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)Naoto MATSUMOTO
High Availability Server Clustering without ILB(Internal Load Balancer) (MEMO)
20-Feb-2015
SAKURA Internet Research Center.
Senior Researcher / Naoto MATSUMOTO
Buildroot (www.buildroot.net) is a tool for building small embedded linux system.
For example : Kernel + Busybox + Perl + Dancer2 ~= 60 MB
Buildroot could create also Qemu Virtual Machine image with the same footprint.
Demo with a Olimex iMX233 board (45€).
The document describes the configuration of an OpenVPN site-to-site VPN tunnel between two networks (Side A and Side B). Key steps include generating and sharing a security key between the sites, configuring firewall rules and routing on each side, and starting the OpenVPN service to establish the encrypted tunnel between the 10.0.0.1 and 10.0.0.2 addresses. Once configured, connectivity between the 192.168.1.0/24 and 192.168.2.0/24 networks can be tested using ping and traceroute.
This document outlines the steps to configure a highly available NFS server using DRBD and heartbeat on Ubuntu. Key steps include:
1. Installing DRBD, heartbeat and configuring the /etc/drbd.conf and heartbeat configuration files on both NFS servers so they are identical.
2. Making the data partition consistent between servers using drbdadm commands.
3. Installing and configuring heartbeat to monitor services and fail over the virtual IP address when either NFS server fails.
4. Testing the configuration by stopping heartbeat on each server to trigger a failover and verifying services move to the other server.
How to install OpenStack MITAKA --allinone - cheat sheet -Naoto MATSUMOTO
How to install OpenStack MITAKA --allinone - cheat sheet -
27-Jun, 2016
SAKURA Internet, Inc. / SAKURA Internet Research Center
Senior Researcher / Naoto MATSUMOTO
This document discusses Python, including installing multiple Python versions on CentOS, creating virtual environments (venv) for different projects, using pip to install packages, and basic concepts like modules, packages, and running Python files. It provides instructions for cloning a Python build script from GitHub, shows commands for installing Python 3.5.2 and listing available Python versions, and demonstrates creating and activating a venv.
Quick Start Guide using Virtuozzo 7 (β) on AWS EC2Kentaro Ebisawa
Virtuozzo 7 was open sourced and available on Amazon EC2 since October 2015.
This document aims to give you a quick overview of steps to setup Virtuozzo on Amazon EC2.
This document discusses Composer, an open source tool for dependency management in PHP. It describes what Composer is, how to install it, how to define dependencies in a composer.json file and composer.lock file, how Composer generates autoload files and installs vendor libraries, and some common Composer commands. It also provides information on joining the community and lists sources for more documentation on Composer.
This document discusses using LVS (Linux Virtual Server) and Keepalived to implement high performance and high availability load balancing. It provides instructions on installing LVS and Keepalived, configuring LVS for load balancing, and using Keepalived to realize load balancing and high availability. Key aspects covered include using LVS-DR and real server scripts, and configuring Keepalived on the master and backup servers.
redis-benchmark with AMD RYZEN 1800X Intel Kaby Lake (i7-7700K) memoNaoto MATSUMOTO
The document compares the performance of Redis benchmarks run on an AMD Ryzen 1800X system and an Intel i7-7700K system. It provides system details for each such as the number of CPU cores, operating system, and CPU governor settings before running the benchmarks to test in-memory database performance between the two systems.
To install ns-3, the user must:
1. Download the ns-3 tar file and required dependencies by running apt-get commands.
2. Extract the ns-3 tar file and configure waf by running the ./waf configure command.
3. Install netanim for animation viewing and ensure mercurial and qt4-dev-tools are installed to support it.
4. Build and run the first sample program using ./waf and ./waf --run commands.
Flake8 is a Python linter that is fast, simple, and extensible. It can be configured through setup.cfg or .flake8 files to ignore certain checks or select others. The summary recommends using the flake8-bugbear plugin and avoiding all print statements with flake8-print. Linters like Flake8 help find errors, improve code quality, and avoid historical baggage, but one does not need every check and it is okay to build a long ignore list.
This document provides instructions for installing and building the Point Cloud Library (PCL) on an Ubuntu system. It outlines downloading dependencies using apt-get, adding a PCL PPA repository, cloning the PCL GitHub repository, and running cmake and make to build and install PCL. The document also lists some related topics like PostgreSQL, Kinect, deep learning frameworks, robot simulators, and motion planning software that can be used with PCL.
distribute und pip als Ersatz für setuptools und easy_install bieten im Zusammenspiel mit virtualenv viele neue Möglichkeiten bei der Entwicklung und dem Deployment von Python-Applikationen. In diesem Vortrag stelle ich alle Werkzeuge kurz vor und zeige, wie man sie zusammen einsetzen kann.
RustPython is a project that implements the Python programming language in Rust. It started in May 2018 and released version 0.3 in September 2023. The standard library has been written in Rust and pip package management works. RustPython can compile to WebAssembly to run Python code in web browsers and has an experimental just-in-time compiler. Several popular Python packages were tested with varying results, with some packages like Django and others not working due to missing Python modules or compiler issues. Packaging Python modules that use Rust is becoming increasingly common using tools like PyO3, with over 50,000 downloads per day of related projects.
The presentation from Python meetup by JettyCloud about solving a problem found in a library that uses hashlib, followed by an overview of the CPython hashlib module with implementation details.
The document provides instructions on Docker practice including prerequisites, basic Docker commands, running containers from images, committing container changes to new images, logging into Docker Hub and pushing images.
It begins with prerequisites of having Ubuntu 18.04 or higher and installing the latest Docker engine and Docker compose. It then explains that Docker runs processes in isolated containers and uses layered images.
The document demonstrates basic commands like docker version, docker images, docker pull, docker search, docker run, docker ps, docker stop, docker rm and docker rmi. It also shows how to commit container changes to a new image with docker commit, tag and push images to Docker Hub. Other topics covered include docker exec, docker save/load, docker
This document provides steps to install the ns-3 network simulator on Ubuntu 16.04. It first involves installing prerequisite packages like gcc, g++, python, and libraries. The user then clones the ns-3 source code from an online repository, downloads additional files, and runs the build process to compile the code. Finally, the user verifies the installation was successful by running example tests from the ns-3 distribution.
How to-mount-3 par-san-virtual-copy-onto-rhel-servers-by-Dusan-BaljevicCircling Cycle
The document describes how to mount a 3PAR virtual copy volume onto a RHEL server. It involves creating host definitions and exporting volumes from 3PAR to the server. The volumes are then mapped, formatted, and mounted. Finally, a virtual copy is created on 3PAR and exported to the server, where it is detected as a new volume.
The document provides instructions for installing and configuring a full-featured Linux server for hosting websites, email, and other services. It includes steps to install and configure an ISPConfig control panel along with associated software like Apache, PHP, MySQL, Postfix, Dovecot, PureFTPd, BIND, Roundcube webmail, and more. The overall process allows you to set up an full-fledged web hosting server on a Linux machine.
The document summarizes the steps taken to install and configure NGINX, PostgreSQL, Python, and uWSGI on a CentOS 7 server. Key steps include:
1) Installing NGINX using yum and enabling it to start automatically at boot;
2) Installing and initializing PostgreSQL, configuring it to allow local connections, and creating a database user;
3) Installing Python 3.7 using pyenv and setting it as the global version;
4) Installing uWSGI and using it to run a simple Python application served over HTTP on port 9090.
This article will help you fetch details about the Ubuntu based AWS EC2 instance. You need to deploy the Python (2.7) based REST Services in Apache webserver. The core of application is Python DJango framework, which uses a custom virtual environment (vitualenv). The Apache uses mod_wsgi for connecting the WSGI application and mod_sec for security purposes.
Deploying Django with Apache and mod_wsgi is a method to get Django into production. mod_wsgi is an Apache module which is supposed to host any Python WSGI application, which includes Django. Django can work with any version of Apache that supports mod_wsgi.
Read the article further, to understand the step-by-step deployment process.
Ubuntu Server is lean, fast and powerful. Its services are reliable, predictable and economical. It is the perfect base on which you can build your instances. Django is a web framework which is written in Python. One can easily guess that everything, in Django, is also done in Python. Django was developed to simplify the creation of database driven sites. The best feature in Django is that it, probably, is the fastest website framework to create a fully functioning website.
This blog is to provide steps for installation and configuration of Odoo for production environment using Git source and Python environment on an Ubuntu system. To install Odoo 17 on Ubuntu you just follow here steps. Call us!
This document provides an overview of Mininet, including getting started instructions, examples of usage, and tutorials on OpenFlow and MiniNAM. It outlines how to import a pre-made Mininet VM, configure access via SSH, test connectivity using ping and Wireshark, explore the included examples and topologies, add manual flow entries, capture and inspect OpenFlow packets, start an OpenFlow controller, and benchmark kernel-space versus user-space switches. Notes are also provided on using xterm, tcpdump, and ARP with Mininet hosts.
Slides for my talk at the Blue4IT meeting in Utrecht. It shows you how to run everything in a Docker container. You can run the DTAP environment, the build environment and the development environment (including IDE) in Docker.
This document provides steps to set up a Subversion (SVN) repository on a Linux system and configure access for users. It describes how to 1) create an SVN repository and project structure, 2) add users and set permissions, 3) import an existing project, 4) start the SVN server, 5) check out a working copy, and 6) recover an SVN repository if needed. It also provides references for more documentation on SVN and the Subclipse Eclipse plugin.
How to work with Python 3, how to create virtual environments, to install libraries, to create code skeletons and more.
Maybe an IDE for Python is right for you. If you are familiar with IntelliJ, then PyCharm is your option. There are other options such as Visual Studio Code, PyDev, Spyder, so you can choose the one you like the most.
And now you have no excuse to start with your first Python project.
How to document without dying in the attemptDatio Big Data
The document discusses the importance of documentation. It notes that documenting knowledge allows future generations to understand the past and envision the future. From a practical perspective, documentation provides advantages such as allowing others to use knowledge, standardizing processes, and producing documentation easily. The document then provides guidelines for technical writing such as using an active voice and focusing on the document's goal.
Testing is important for project quality and reliability. Unit tests check individual code units for correct functionality, are fast and independent. Behavior-driven development (BDD) applies business needs through scenarios defined in Gherkin. Test doubles replace code to independently test objects like dummies, fakes, stubs and mocks. Integration tests check connections between components while acceptance tests check overall project behavior from the user perspective, ensuring expected functionality but are slower and require more maintenance.
El documento describe las características y capacidades de Ceph, un sistema de almacenamiento de objetos distribuido de código abierto. Ceph puede escalar horizontalmente hasta el exabyte, no tiene puntos únicos de fallo, y funciona en hardware estándar. Utiliza un diseño descentralizado basado en objetos que puede exponer almacenamiento en forma de objetos, archivos o bloques.
What is Mesos? How does it works? In the following slides we make an interesting review of this open-source software project to manage computer clusters.
La infraestructura Atlantis de Datio se basa en OpenStack. Consta de 3 controladores, 16 nodos de computación y almacenamiento distribuido Ceph. A nivel físico incluye servidores, cuchillas y almacenamiento. A nivel lógico define 4 redes aisladas y despliega los servicios principales de OpenStack como Keystone, Glance, Cinder y Neutron en alta disponibilidad. Los usuarios pueden gestionar la infraestructura a través de Horizon, la API de OpenStack o Ansible.
This document discusses data integration and architectures for processing both batch and streaming data. It covers topics like data ingestion using tools like Flume, Sqoop and Kafka to move data into data lakes and warehouses. It also discusses batch processing using MapReduce on Hadoop and stream processing using real-time technologies like Kafka and architectures like lambda and kappa for serving queries on both real-time and batch-processed views of the data.
How we have designed and applied the gamification at Datio.
See more about this topic in our blog:
http://www.datio.com/corporate/gwc-2016-towards-the-high-level-engagement/
http://www.datio.com/corporate/beginning-a-gamification-project/
Pandas: High Performance Structured Data ManipulationDatio Big Data
Brief theoretical introduction to Pandas: the Python library that is used primarily for manipulation and data analysis. Here we'll explain when it is convenient to work with this one, its main characteristics and the operations that allows.
In this second part, we'll continue the Spark's review and introducing SparkSQL which allows to use data frames in Python, Java, and Scala; read and write data in a variety of structured formats; and query Big Data with SQL.
This document provides an overview comparison of SAS and Spark for analytics. SAS is a commercial software while Spark is an open source framework. SAS uses datasets that reside in memory while Spark uses resilient distributed datasets (RDDs) that can scale across clusters. Both support SQL queries but Spark SQL allows querying distributed data lazily. Spark also provides machine learning APIs through MLlib that can perform tasks like classification, clustering, and recommendation at scale.
Created at the University of Berkeley in California, Apache Spark combines a distributed computing system through computer clusters with a simple and elegant way of writing programs. Spark is considered the first open source software that makes distribution programming really accessible to data scientists. Here you can find an introduction and basic concepts.
El Quality Assurance ha experimentado una evolución equiparable a la de la especie humana. En esta presentación se explica la importancia creciente de los sistemas de gestión de la calidad en el desarollo de software.
The document discusses different types of databases including relational, column-oriented, document-oriented, and graph databases. It explains key concepts such as ACID vs BASE, CAP theorem, isolation levels, indexes, sharding, and provides descriptions and comparisons of each database type.
DC/OS: The definitive platform for modern appsDatio Big Data
DC/OS is an open source platform that provides container orchestration and management using Mesos. It allows running applications and services across data center infrastructure including bare metal, VMs, and cloud. DC/OS provides services like Marathon for container orchestration, security, monitoring, load balancing and service discovery. It has features like high resource utilization, mixed workload support, elastic scalability, high availability and zero downtime upgrades.
The real purpose of any career plan should be to improve the skills of the person owning it, to discover his/her strong points, to find out the things they need help with and eventually becoming a better professional and a more self-assured individual. Then, we should start looking for a Personal Development Plan instead.
This document discusses security and governance considerations for big data. It notes that while many businesses use big data, they may not have sufficient access controls or security practices in place. Big data breaches can be large, making security critical. It then outlines some risks like insufficiently hardened systems, uncontrolled access, and unfulfilled regulatory requirements. The document introduces GOSEC as a centralized security component that manages access control across big data services like HDFS and applications. GOSEC allows setting access policies for users and groups for resources like files and topics. It covers authentication, authorization, and auditing. The document stresses the need to integrate GOSEC with the organization's identity provider and security strategies to prevent leakage of internal and external data. It
Optimizing Gradle Builds - Gradle DPE Tour Berlin 2024Sinan KOZAK
Sinan from the Delivery Hero mobile infrastructure engineering team shares a deep dive into performance acceleration with Gradle build cache optimizations. Sinan shares their journey into solving complex build-cache problems that affect Gradle builds. By understanding the challenges and solutions found in our journey, we aim to demonstrate the possibilities for faster builds. The case study reveals how overlapping outputs and cache misconfigurations led to significant increases in build times, especially as the project scaled up with numerous modules using Paparazzi tests. The journey from diagnosing to defeating cache issues offers invaluable lessons on maintaining cache integrity without sacrificing functionality.
Introduction- e - waste – definition - sources of e-waste– hazardous substances in e-waste - effects of e-waste on environment and human health- need for e-waste management– e-waste handling rules - waste minimization techniques for managing e-waste – recycling of e-waste - disposal treatment methods of e- waste – mechanism of extraction of precious metal from leaching solution-global Scenario of E-waste – E-waste in India- case studies.
Electric vehicle and photovoltaic advanced roles in enhancing the financial p...IJECEIAES
Climate change's impact on the planet forced the United Nations and governments to promote green energies and electric transportation. The deployments of photovoltaic (PV) and electric vehicle (EV) systems gained stronger momentum due to their numerous advantages over fossil fuel types. The advantages go beyond sustainability to reach financial support and stability. The work in this paper introduces the hybrid system between PV and EV to support industrial and commercial plants. This paper covers the theoretical framework of the proposed hybrid system including the required equation to complete the cost analysis when PV and EV are present. In addition, the proposed design diagram which sets the priorities and requirements of the system is presented. The proposed approach allows setup to advance their power stability, especially during power outages. The presented information supports researchers and plant owners to complete the necessary analysis while promoting the deployment of clean energy. The result of a case study that represents a dairy milk farmer supports the theoretical works and highlights its advanced benefits to existing plants. The short return on investment of the proposed approach supports the paper's novelty approach for the sustainable electrical system. In addition, the proposed system allows for an isolated power setup without the need for a transmission line which enhances the safety of the electrical network
Understanding Inductive Bias in Machine LearningSUTEJAS
This presentation explores the concept of inductive bias in machine learning. It explains how algorithms come with built-in assumptions and preferences that guide the learning process. You'll learn about the different types of inductive bias and how they can impact the performance and generalizability of machine learning models.
The presentation also covers the positive and negative aspects of inductive bias, along with strategies for mitigating potential drawbacks. We'll explore examples of how bias manifests in algorithms like neural networks and decision trees.
By understanding inductive bias, you can gain valuable insights into how machine learning models work and make informed decisions when building and deploying them.
Comparative analysis between traditional aquaponics and reconstructed aquapon...bijceesjournal
The aquaponic system of planting is a method that does not require soil usage. It is a method that only needs water, fish, lava rocks (a substitute for soil), and plants. Aquaponic systems are sustainable and environmentally friendly. Its use not only helps to plant in small spaces but also helps reduce artificial chemical use and minimizes excess water use, as aquaponics consumes 90% less water than soil-based gardening. The study applied a descriptive and experimental design to assess and compare conventional and reconstructed aquaponic methods for reproducing tomatoes. The researchers created an observation checklist to determine the significant factors of the study. The study aims to determine the significant difference between traditional aquaponics and reconstructed aquaponics systems propagating tomatoes in terms of height, weight, girth, and number of fruits. The reconstructed aquaponics system’s higher growth yield results in a much more nourished crop than the traditional aquaponics system. It is superior in its number of fruits, height, weight, and girth measurement. Moreover, the reconstructed aquaponics system is proven to eliminate all the hindrances present in the traditional aquaponics system, which are overcrowding of fish, algae growth, pest problems, contaminated water, and dead fish.
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressionsVictor Morales
K8sGPT is a tool that analyzes and diagnoses Kubernetes clusters. This presentation was used to share the requirements and dependencies to deploy K8sGPT in a local environment.
Using recycled concrete aggregates (RCA) for pavements is crucial to achieving sustainability. Implementing RCA for new pavement can minimize carbon footprint, conserve natural resources, reduce harmful emissions, and lower life cycle costs. Compared to natural aggregate (NA), RCA pavement has fewer comprehensive studies and sustainability assessments.
18. ● Python Package Index
https://pypi.python.org/pypi
● Custom repositories (host your own libraries)
Nexus
Artifactory
● Most used libraries
Python 3 Wall of Superpowers