The document describes how to use the Batch component in Mule applications. The Batch component allows processing messages in batches by splitting messages into individual records, performing actions on each record, and reporting results. The document includes an example Mule flow that uses a Batch job to iterate over a list of maps and perform database updates on each item. It logs output showing the batch job processing the records and handling any errors.
Apache Flink Training: DataSet API BasicsFlink Forward
This document provides an overview of the Apache Flink DataSet API. It introduces key concepts such as batch processing, data types including tuples, transformations like map, filter, group, and reduce, joining datasets, data sources and sinks, and an example word count program in Java. The word count example demonstrates reading text data, tokenizing strings, grouping and counting words, and writing the results. The document contains slides with code snippets and explanations of Flink's DataSet API concepts and features.
This document discusses batch processing using Apache Flink. It provides code examples of using Flink's DataSet and Table APIs to perform batch word count jobs. It also covers iterative algorithms in Flink, including how Flink handles bulk and delta iterations more efficiently than other frameworks like Spark and MapReduce. Delta iterations are optimized by only processing changes between iterations to reduce the working data set size over time.
Apache Flink Training: DataStream API Part 2 Advanced Flink Forward
Flink can handle many data types and provides a type system to identify types for serialization and comparisons. Composite types like Tuples and POJOs can be used and fields within them can define keys. Windows provide a way to perform aggregations over finite slices of infinite streams. Connected streams allow correlating and joining multiple streams. Stateful functions have access to local and partitioned state for stateful stream processing. Kafka integration allows consuming from and producing to Kafka topics.
This document discusses different frameworks for big data processing at ResearchGate, including Hive, MapReduce, and Flink. It provides an example of using Hive to find the top 5 coauthors for each author based on publication data. Code snippets in Hive SQL and Java are included to implement the top k coauthors user defined aggregate function (UDAF) in Hive. The document evaluates different frameworks based on criteria like features, performance, and usability.
Real time and reliable processing with Apache StormAndrea Iacono
Storm is a framework for reliably processing streaming data. It allows defining topologies composed of spouts (data sources) and bolts (processing components). Spouts emit tuples that are processed by bolts which can emit additional tuples. The document describes a topology for processing tweets in real-time to identify top hashtags and display tweets on a map. It includes spouts to fetch tweets and bolts for filtering, counting hashtags, ranking them and storing results to Redis. Storm provides reliability by tracking processing of tuples through a topology using acknowledgments.
Integrate Solr with real-time stream processing applicationslucenerevolution
The document discusses integrating Apache Storm with Apache Solr for real-time stream processing applications. It provides an example of building a Storm topology that listens to click events from a URL shortener, counts the frequency of pages in a time window, ranks the top sites, and persists the results to Solr for visualization. The key points covered are using Spring to simplify building Storm topologies, integrating with Solr for indexing and search, and unit testing streaming data providers.
Building Distributed System with Celery on Docker SwarmWei Lin
This document discusses building distributed systems with Celery on Docker Swarm. It introduces Celery for asynchronous task queueing and message passing. Docker Swarm is used to deploy Celery worker containers across multiple hosts for parallel computing. Tasks can be routed to specific workers by queue or host name. This allows building distributed systems easily by sending tasks to worker containers without worrying about the underlying infrastructure.
Flink 0.10 @ Bay Area Meetup (October 2015)Stephan Ewen
Flink 0.10 focuses on operational readiness with improvements to high availability, monitoring, and integration with other systems. It provides first-class support for event time processing and refines the DataStream API to be both easy to use and powerful for stream processing tasks.
Apache Flink Training: DataSet API BasicsFlink Forward
This document provides an overview of the Apache Flink DataSet API. It introduces key concepts such as batch processing, data types including tuples, transformations like map, filter, group, and reduce, joining datasets, data sources and sinks, and an example word count program in Java. The word count example demonstrates reading text data, tokenizing strings, grouping and counting words, and writing the results. The document contains slides with code snippets and explanations of Flink's DataSet API concepts and features.
This document discusses batch processing using Apache Flink. It provides code examples of using Flink's DataSet and Table APIs to perform batch word count jobs. It also covers iterative algorithms in Flink, including how Flink handles bulk and delta iterations more efficiently than other frameworks like Spark and MapReduce. Delta iterations are optimized by only processing changes between iterations to reduce the working data set size over time.
Apache Flink Training: DataStream API Part 2 Advanced Flink Forward
Flink can handle many data types and provides a type system to identify types for serialization and comparisons. Composite types like Tuples and POJOs can be used and fields within them can define keys. Windows provide a way to perform aggregations over finite slices of infinite streams. Connected streams allow correlating and joining multiple streams. Stateful functions have access to local and partitioned state for stateful stream processing. Kafka integration allows consuming from and producing to Kafka topics.
This document discusses different frameworks for big data processing at ResearchGate, including Hive, MapReduce, and Flink. It provides an example of using Hive to find the top 5 coauthors for each author based on publication data. Code snippets in Hive SQL and Java are included to implement the top k coauthors user defined aggregate function (UDAF) in Hive. The document evaluates different frameworks based on criteria like features, performance, and usability.
Real time and reliable processing with Apache StormAndrea Iacono
Storm is a framework for reliably processing streaming data. It allows defining topologies composed of spouts (data sources) and bolts (processing components). Spouts emit tuples that are processed by bolts which can emit additional tuples. The document describes a topology for processing tweets in real-time to identify top hashtags and display tweets on a map. It includes spouts to fetch tweets and bolts for filtering, counting hashtags, ranking them and storing results to Redis. Storm provides reliability by tracking processing of tuples through a topology using acknowledgments.
Integrate Solr with real-time stream processing applicationslucenerevolution
The document discusses integrating Apache Storm with Apache Solr for real-time stream processing applications. It provides an example of building a Storm topology that listens to click events from a URL shortener, counts the frequency of pages in a time window, ranks the top sites, and persists the results to Solr for visualization. The key points covered are using Spring to simplify building Storm topologies, integrating with Solr for indexing and search, and unit testing streaming data providers.
Building Distributed System with Celery on Docker SwarmWei Lin
This document discusses building distributed systems with Celery on Docker Swarm. It introduces Celery for asynchronous task queueing and message passing. Docker Swarm is used to deploy Celery worker containers across multiple hosts for parallel computing. Tasks can be routed to specific workers by queue or host name. This allows building distributed systems easily by sending tasks to worker containers without worrying about the underlying infrastructure.
Flink 0.10 @ Bay Area Meetup (October 2015)Stephan Ewen
Flink 0.10 focuses on operational readiness with improvements to high availability, monitoring, and integration with other systems. It provides first-class support for event time processing and refines the DataStream API to be both easy to use and powerful for stream processing tasks.
Create & Execute First Hadoop MapReduce Project in.pptxvishal choudhary
The document provides a 12 step guide to create and execute a first Hadoop MapReduce project in Eclipse. The steps include installing prerequisites like Hadoop, Eclipse, and Java, creating a project in Eclipse, adding required Hadoop jar files, creating Mapper, Reducer and Driver classes, compiling the code into a jar file, and executing the MapReduce job on Hadoop by running the jar file.
This document provides an overview of how to contribute to the cPython source code. It discusses running benchmarks to understand performance differences between loops inside and outside functions. It encourages contributing to improve coding skills and help the open source community. The steps outlined are to clone the cPython source code repository, resolve any dependencies during building, review open issues on bugs.python.org, and work on resolving issues - starting with easier ones. Tips are provided such as commenting when taking ownership of an issue, reproducing bugs before working on them, writing tests for code changes, and updating documentation.
Real-time Stream Processing with Apache Flink @ Hadoop SummitGyula Fóra
Apache Flink is an open source project that offers both batch and stream processing on top of a common runtime and exposing a common API. This talk focuses on the stream processing capabilities of Flink.
Talk held at the FrOSCon 2013 on 24.08.2013 in Sankt Augustin, Germany
Agenda:
- Why Twitter Storm?
- What is Twitter Storm?
- What to do with Twitter Storm?
Distributed real time stream processing- why and howPetr Zapletal
In this talk you will discover various state-of-the-art open-source distributed streaming frameworks, their similarities and differences, implementation trade-offs, their intended use-cases, and how to choose between them. Petr will focus on the popular frameworks, including Spark Streaming, Storm, Samza and Flink. You will also explore theoretical introduction, common pitfalls, popular architectures, and much more.
The demand for stream processing is increasing. Immense amounts of data has to be processed fast from a rapidly growing set of disparate data sources. This pushes the limits of traditional data processing infrastructures. These stream-based applications, include trading, social networks, the Internet of Things, and system monitoring, are becoming more and more important. A number of powerful, easy-to-use open source platforms have emerged to address this.
Petr's goal is to provide a comprehensive overview of modern streaming solutions and to help fellow developers with picking the best possible solution for their particular use-case. Join this talk if you are thinking about, implementing, or have already deployed a streaming solution.
1) A job is first submitted to the Hadoop cluster by a client calling the Job.submit() method. This generates a unique job ID and copies the job files to HDFS.
2) The JobTracker then initializes the job by splitting it into tasks like map and reduce tasks. It assigns tasks to TaskTrackers based on data locality.
3) Each TaskTracker executes tasks by copying job files, running tasks in a child JVM, and reporting progress back to the JobTracker.
4) The JobTracker tracks overall job status and progress by collecting task status updates from TaskTrackers. It reports this information back to clients.
5) Once all tasks complete successfully, the job
This document discusses Rally, an OpenStack benchmarking tool. It provides an introduction to Rally and its components, including benchmark engines, deployment engines, server providers, and verification. The document demonstrates how to install and configure Rally, set up deployments and benchmarking scenarios, and run and analyze benchmarking tests. It also lists some supported benchmarking use cases and provides information on contributing to and getting involved with the Rally project.
Hadoop Summit Europe 2014: Apache Storm ArchitectureP. Taylor Goetz
Storm is an open-source distributed real-time computation system. It uses a distributed messaging system to reliably process streams of data. The core abstractions in Storm are spouts, which are sources of streams, and bolts, which are basic processing elements. Spouts and bolts are organized into topologies which represent the flow of data. Storm provides fault tolerance through message acknowledgments and guarantees exactly-once processing semantics. Trident is a high-level abstraction built on Storm that supports operations like aggregations, joins, and state management through its micro-batch oriented and stream-based API.
PostgreSQL is a free and open-source relational database management system that provides high performance and reliability. It supports replication through various methods including log-based asynchronous master-slave replication, which the presenter recommends as a first option. The upcoming PostgreSQL 9.4 release includes improvements to replication such as logical decoding and replication slots. Future releases may add features like logical replication consumers and SQL MERGE statements. The presenter took questions at the end and provided additional resources on PostgreSQL replication.
The document provides details about experiments to be performed in the Big Data Analytics lab course. It includes implementing various data structures like linked lists, stacks, queues, sets and maps in Java. It also describes setting up Hadoop in standalone, pseudodistributed and fully distributed modes. Other experiments involve performing file management tasks in Hadoop, running a basic word count MapReduce program, writing MapReduce programs to analyze weather data, implementing matrix multiplication in MapReduce, installing and using Pig and Hive with Hadoop, and solving some real-life big data problems.
EX-6-Implement Matrix Multiplication with Hadoop Map Reduce.pptxvishal choudhary
1. This document describes the MapReduce algorithm for matrix multiplication and provides code examples to implement it in Hadoop.
2. Key aspects include preprocessing the input matrices as key-value pairs, a map function that emits pairs of rows and columns, and a reduce function that calculates the inner products to obtain the output matrix.
3. Code examples provide the mapper, reducer, driver classes along with commands to compile a jar file, upload sample input, and execute the job to perform matrix multiplication using MapReduce on Hadoop.
Hadoop MapReduce Introduction and Deep InsightHanborq Inc.
Hadoop MapReduce introduces YARN, which separates cluster resource management from application execution. YARN introduces a global ResourceManager and per-node NodeManagers to manage resources. Applications run as ApplicationMasters and containers on the nodes. This improves scalability, fault tolerance, and allows various application paradigms beyond MapReduce. Optimization techniques for MapReduce include tuning buffer sizes, enabling sort avoidance when sorting is unnecessary, and using Netty and batch fetching to improve shuffle performance.
MRUnit is a testing library that makes it easier to test Hadoop jobs. It allows programmatically specifying test input and output, reducing the need for external test files. Tests can focus on individual map and reduce functions. MRUnit abstracts away much of the boilerplate test setup code, though it has some limitations like a lack of distributed testing. Overall though, the benefits of using MRUnit to test Hadoop jobs outweigh the problems.
Triggers are those little bits of code running in your database that gets executed when something happens that you care about. Whether you are a developer who puts all of your business logic inside of PL/pgSQL functions or someone who uses an ORM and wants to stay away from database code, you will likely end up using triggers at some point. The fact that the most recommend way of implementing table partitioning in PostgreSQL uses triggers accounts for the importance of understanding triggers.
In this talk, we will step through examples of writing various types of triggers using practical uses cases like partitioning and auditing.
The structure of a trigger
BEFORE vs AFTER triggers
Statement Level vs Row Level triggers
Conditional triggers
Event triggers
Debugging triggers
Performance overhead of triggers
All of the examples will be done using PL/pgSQL so in addition to getting an overview of triggers, you will also get a good understanding of how to code in PL/pgSQL.
You use InfluxData to monitor the performance of your infrastructure and apps—so it is equally important to keep your InfluxEnterprise instance up and running. Tim Hall, InfluxData VP of Products, will outline why and how you can monitor InfluxEnterprise with InfluxDB.
You\'ve decided to make the switch to Drupal. Fantastic! Only one problem: you have to figure out how to move your content from the old database to Drupal. Although there are many import/export modules available it\'s sometimes good to know what\'s happening behind the scenes. This session will walk you through my adventures of porting community Web sites into Drupal.
The talk will include:
* exporting usable data from your old site;
* using CCK to create the right home for your new content;
* using existing import modules (specifically: import html, node import and user import); and
* importing content \"by hand\" using MySQL command line magic.
We will also touch on some of the headaches I ran into in keeping data synchronized on very active community sites during the development phase.
This session is perfect for people who are preparing to migrate their Web site to Drupal and also people who are new to database management but want to know more about how things work behind the scenes. Of course if you\'re already a pro at data migration, please bring your stories and suggestions!
This document introduces Test Driven Development (TDD) for MapReduce jobs using the MRUnit testing framework. It discusses how TDD is difficult for Hadoop due to its distributed nature but can be achieved by abstracting business logic. It provides examples of using MRUnit to test mappers, reducers and full MapReduce jobs. It also discusses testing with real data by loading samples into the local filesystem or using a WindowsLocalFileSystem class to enable permission testing on Windows.
This presentation was part of the workshop on Materials Project Software infrastructure conducted for the Materials Virtual Lab in Nov 10 2014. It presents an introduction to the pymatgen-db database plugin for the pymatge) materials analysis library, and the custodian error recovery framework.
Pymatgen-db enables the creation of Materials Project-style MongoDB databases for management of materials data. A query engine is also provided to enable the easy translation of MongoDB docs to useful pymatgen objects for analysis purposes.
Custodian is a simple, robust and flexible just-in-time (JIT) job management framework written in Python. Using custodian, you can create wrappers that perform error checking, job management and error recovery. It has a simple plugin framework that allows you to develop specific job management workflows for different applications. Error recovery is an important aspect of many high-throughput projects that generate data on a large scale. The specific use case for custodian is for long running jobs, with potentially random errors. For example, there may be a script that takes several days to run on a server, with a 1% chance of some IO error causing the job to fail. Using custodian, one can develop a mechanism to gracefully recover from the error, and restart the job with modified parameters if necessary. The current version of Custodian also comes with sub-packages for error handling for Vienna Ab Initio Simulation Package (VASP) and QChem calculations.
Big data unit iv and v lecture notes qb model examIndhujeni
The document contains questions that could appear on a model exam for MapReduce concepts. It includes questions about the limitations of classic MapReduce, comparing classic MapReduce to YARN, Zookeeper, differences between HBase and HDFS, differences between RDBMS and Cassandra, Pig Latin, the use of Grunt, advantages of Hive, comparing relational databases to HBase, when to use HBase, anatomy of a classic MapReduce job run, YARN architecture, job scheduling, handling failures in classic MapReduce and YARN, HBase and Cassandra architectures and data models, HBase and Cassandra clients, Hive data types and file formats, Pig Latin scripts and Grunt shell, and HiveQL data definition.
The document discusses Hive, an open source data warehousing system built on Hadoop that allows users to query large datasets using SQL. It describes Hive's data model, architecture, query language features like joins and aggregations, optimizations, and provides examples of how queries are executed using MapReduce. The document also covers Hive's metastore, external tables, data types, and extensibility features.
The document summarizes a study that assessed the antimicrobial activities of secondary metabolite extracts from three soil-inhabiting fungi - Trichoderma koningii, Rhizopus stolonifer, and Fusarium oxysporum. The fungi were cultured individually in Sabouraud broth for 21 days, after which ethyl acetate was used to extract secondary metabolites from the broth. Thin layer chromatography analysis indicated the extracts contained multiple compounds. The extracts were screened for antimicrobial activity against four microorganisms using a broth microdilution assay. The extracts displayed variable but low antimicrobial activity compared to standard antimicrobial drugs. This study provides a preliminary analysis of the antimicrobial potential of extracts from these under-explored
Create & Execute First Hadoop MapReduce Project in.pptxvishal choudhary
The document provides a 12 step guide to create and execute a first Hadoop MapReduce project in Eclipse. The steps include installing prerequisites like Hadoop, Eclipse, and Java, creating a project in Eclipse, adding required Hadoop jar files, creating Mapper, Reducer and Driver classes, compiling the code into a jar file, and executing the MapReduce job on Hadoop by running the jar file.
This document provides an overview of how to contribute to the cPython source code. It discusses running benchmarks to understand performance differences between loops inside and outside functions. It encourages contributing to improve coding skills and help the open source community. The steps outlined are to clone the cPython source code repository, resolve any dependencies during building, review open issues on bugs.python.org, and work on resolving issues - starting with easier ones. Tips are provided such as commenting when taking ownership of an issue, reproducing bugs before working on them, writing tests for code changes, and updating documentation.
Real-time Stream Processing with Apache Flink @ Hadoop SummitGyula Fóra
Apache Flink is an open source project that offers both batch and stream processing on top of a common runtime and exposing a common API. This talk focuses on the stream processing capabilities of Flink.
Talk held at the FrOSCon 2013 on 24.08.2013 in Sankt Augustin, Germany
Agenda:
- Why Twitter Storm?
- What is Twitter Storm?
- What to do with Twitter Storm?
Distributed real time stream processing- why and howPetr Zapletal
In this talk you will discover various state-of-the-art open-source distributed streaming frameworks, their similarities and differences, implementation trade-offs, their intended use-cases, and how to choose between them. Petr will focus on the popular frameworks, including Spark Streaming, Storm, Samza and Flink. You will also explore theoretical introduction, common pitfalls, popular architectures, and much more.
The demand for stream processing is increasing. Immense amounts of data has to be processed fast from a rapidly growing set of disparate data sources. This pushes the limits of traditional data processing infrastructures. These stream-based applications, include trading, social networks, the Internet of Things, and system monitoring, are becoming more and more important. A number of powerful, easy-to-use open source platforms have emerged to address this.
Petr's goal is to provide a comprehensive overview of modern streaming solutions and to help fellow developers with picking the best possible solution for their particular use-case. Join this talk if you are thinking about, implementing, or have already deployed a streaming solution.
1) A job is first submitted to the Hadoop cluster by a client calling the Job.submit() method. This generates a unique job ID and copies the job files to HDFS.
2) The JobTracker then initializes the job by splitting it into tasks like map and reduce tasks. It assigns tasks to TaskTrackers based on data locality.
3) Each TaskTracker executes tasks by copying job files, running tasks in a child JVM, and reporting progress back to the JobTracker.
4) The JobTracker tracks overall job status and progress by collecting task status updates from TaskTrackers. It reports this information back to clients.
5) Once all tasks complete successfully, the job
This document discusses Rally, an OpenStack benchmarking tool. It provides an introduction to Rally and its components, including benchmark engines, deployment engines, server providers, and verification. The document demonstrates how to install and configure Rally, set up deployments and benchmarking scenarios, and run and analyze benchmarking tests. It also lists some supported benchmarking use cases and provides information on contributing to and getting involved with the Rally project.
Hadoop Summit Europe 2014: Apache Storm ArchitectureP. Taylor Goetz
Storm is an open-source distributed real-time computation system. It uses a distributed messaging system to reliably process streams of data. The core abstractions in Storm are spouts, which are sources of streams, and bolts, which are basic processing elements. Spouts and bolts are organized into topologies which represent the flow of data. Storm provides fault tolerance through message acknowledgments and guarantees exactly-once processing semantics. Trident is a high-level abstraction built on Storm that supports operations like aggregations, joins, and state management through its micro-batch oriented and stream-based API.
PostgreSQL is a free and open-source relational database management system that provides high performance and reliability. It supports replication through various methods including log-based asynchronous master-slave replication, which the presenter recommends as a first option. The upcoming PostgreSQL 9.4 release includes improvements to replication such as logical decoding and replication slots. Future releases may add features like logical replication consumers and SQL MERGE statements. The presenter took questions at the end and provided additional resources on PostgreSQL replication.
The document provides details about experiments to be performed in the Big Data Analytics lab course. It includes implementing various data structures like linked lists, stacks, queues, sets and maps in Java. It also describes setting up Hadoop in standalone, pseudodistributed and fully distributed modes. Other experiments involve performing file management tasks in Hadoop, running a basic word count MapReduce program, writing MapReduce programs to analyze weather data, implementing matrix multiplication in MapReduce, installing and using Pig and Hive with Hadoop, and solving some real-life big data problems.
EX-6-Implement Matrix Multiplication with Hadoop Map Reduce.pptxvishal choudhary
1. This document describes the MapReduce algorithm for matrix multiplication and provides code examples to implement it in Hadoop.
2. Key aspects include preprocessing the input matrices as key-value pairs, a map function that emits pairs of rows and columns, and a reduce function that calculates the inner products to obtain the output matrix.
3. Code examples provide the mapper, reducer, driver classes along with commands to compile a jar file, upload sample input, and execute the job to perform matrix multiplication using MapReduce on Hadoop.
Hadoop MapReduce Introduction and Deep InsightHanborq Inc.
Hadoop MapReduce introduces YARN, which separates cluster resource management from application execution. YARN introduces a global ResourceManager and per-node NodeManagers to manage resources. Applications run as ApplicationMasters and containers on the nodes. This improves scalability, fault tolerance, and allows various application paradigms beyond MapReduce. Optimization techniques for MapReduce include tuning buffer sizes, enabling sort avoidance when sorting is unnecessary, and using Netty and batch fetching to improve shuffle performance.
MRUnit is a testing library that makes it easier to test Hadoop jobs. It allows programmatically specifying test input and output, reducing the need for external test files. Tests can focus on individual map and reduce functions. MRUnit abstracts away much of the boilerplate test setup code, though it has some limitations like a lack of distributed testing. Overall though, the benefits of using MRUnit to test Hadoop jobs outweigh the problems.
Triggers are those little bits of code running in your database that gets executed when something happens that you care about. Whether you are a developer who puts all of your business logic inside of PL/pgSQL functions or someone who uses an ORM and wants to stay away from database code, you will likely end up using triggers at some point. The fact that the most recommend way of implementing table partitioning in PostgreSQL uses triggers accounts for the importance of understanding triggers.
In this talk, we will step through examples of writing various types of triggers using practical uses cases like partitioning and auditing.
The structure of a trigger
BEFORE vs AFTER triggers
Statement Level vs Row Level triggers
Conditional triggers
Event triggers
Debugging triggers
Performance overhead of triggers
All of the examples will be done using PL/pgSQL so in addition to getting an overview of triggers, you will also get a good understanding of how to code in PL/pgSQL.
You use InfluxData to monitor the performance of your infrastructure and apps—so it is equally important to keep your InfluxEnterprise instance up and running. Tim Hall, InfluxData VP of Products, will outline why and how you can monitor InfluxEnterprise with InfluxDB.
You\'ve decided to make the switch to Drupal. Fantastic! Only one problem: you have to figure out how to move your content from the old database to Drupal. Although there are many import/export modules available it\'s sometimes good to know what\'s happening behind the scenes. This session will walk you through my adventures of porting community Web sites into Drupal.
The talk will include:
* exporting usable data from your old site;
* using CCK to create the right home for your new content;
* using existing import modules (specifically: import html, node import and user import); and
* importing content \"by hand\" using MySQL command line magic.
We will also touch on some of the headaches I ran into in keeping data synchronized on very active community sites during the development phase.
This session is perfect for people who are preparing to migrate their Web site to Drupal and also people who are new to database management but want to know more about how things work behind the scenes. Of course if you\'re already a pro at data migration, please bring your stories and suggestions!
This document introduces Test Driven Development (TDD) for MapReduce jobs using the MRUnit testing framework. It discusses how TDD is difficult for Hadoop due to its distributed nature but can be achieved by abstracting business logic. It provides examples of using MRUnit to test mappers, reducers and full MapReduce jobs. It also discusses testing with real data by loading samples into the local filesystem or using a WindowsLocalFileSystem class to enable permission testing on Windows.
This presentation was part of the workshop on Materials Project Software infrastructure conducted for the Materials Virtual Lab in Nov 10 2014. It presents an introduction to the pymatgen-db database plugin for the pymatge) materials analysis library, and the custodian error recovery framework.
Pymatgen-db enables the creation of Materials Project-style MongoDB databases for management of materials data. A query engine is also provided to enable the easy translation of MongoDB docs to useful pymatgen objects for analysis purposes.
Custodian is a simple, robust and flexible just-in-time (JIT) job management framework written in Python. Using custodian, you can create wrappers that perform error checking, job management and error recovery. It has a simple plugin framework that allows you to develop specific job management workflows for different applications. Error recovery is an important aspect of many high-throughput projects that generate data on a large scale. The specific use case for custodian is for long running jobs, with potentially random errors. For example, there may be a script that takes several days to run on a server, with a 1% chance of some IO error causing the job to fail. Using custodian, one can develop a mechanism to gracefully recover from the error, and restart the job with modified parameters if necessary. The current version of Custodian also comes with sub-packages for error handling for Vienna Ab Initio Simulation Package (VASP) and QChem calculations.
Big data unit iv and v lecture notes qb model examIndhujeni
The document contains questions that could appear on a model exam for MapReduce concepts. It includes questions about the limitations of classic MapReduce, comparing classic MapReduce to YARN, Zookeeper, differences between HBase and HDFS, differences between RDBMS and Cassandra, Pig Latin, the use of Grunt, advantages of Hive, comparing relational databases to HBase, when to use HBase, anatomy of a classic MapReduce job run, YARN architecture, job scheduling, handling failures in classic MapReduce and YARN, HBase and Cassandra architectures and data models, HBase and Cassandra clients, Hive data types and file formats, Pig Latin scripts and Grunt shell, and HiveQL data definition.
The document discusses Hive, an open source data warehousing system built on Hadoop that allows users to query large datasets using SQL. It describes Hive's data model, architecture, query language features like joins and aggregations, optimizations, and provides examples of how queries are executed using MapReduce. The document also covers Hive's metastore, external tables, data types, and extensibility features.
The document summarizes a study that assessed the antimicrobial activities of secondary metabolite extracts from three soil-inhabiting fungi - Trichoderma koningii, Rhizopus stolonifer, and Fusarium oxysporum. The fungi were cultured individually in Sabouraud broth for 21 days, after which ethyl acetate was used to extract secondary metabolites from the broth. Thin layer chromatography analysis indicated the extracts contained multiple compounds. The extracts were screened for antimicrobial activity against four microorganisms using a broth microdilution assay. The extracts displayed variable but low antimicrobial activity compared to standard antimicrobial drugs. This study provides a preliminary analysis of the antimicrobial potential of extracts from these under-explored
Daniela Vrnoga is a real estate agent based in Silicon Valley and San Francisco who is the number one agent at the number one real estate company. She has an active online presence on LinkedIn, Twitter, and Facebook where she markets herself as the top selling agent in the area.
Kim Kjærsgaard worked as an ER MPS Lead Engineer for FMC Technologies for 1.5 years reporting to Anne Delmotte. Anne describes Kim as having excellent productivity and efficiency, very good relationships with peers and managers, and being excellent in service focus, communication, planning results, and achieving results. Anne rated Kim highly in all areas and said she would re-employ Kim if the opportunity arose. Anne noted that Kim excelled in organization and planning but could improve further in tender processes, and that Kim works best in fast-paced, hectic, and challenging environments.
MuleSoft provides an integration platform that connects applications and APIs across cloud and on-premise systems. The MuleSoft platform includes a connector for Amazon S3 that allows developers to access and store object data in S3 buckets from within their Mule applications. With the Amazon S3 connector, users can integrate S3 storage with other AWS services and build applications that leverage internet storage capabilities.
This document provides steps to connect Dropbox to Mule ESB using the Dropbox Cloud Connector and OAuth2 authentication. It involves creating a Dropbox app, configuring the Dropbox connector in Mule with the app keys and secret, and using a choice router to check if authorization was successful by looking for an OAuth access token id flow variable.
Goods and Service Tax #GST comes as India's biggest reform that will lead to the creation of a common national market, currently fragmented along state boundaries.
Este documento fornece um modelo de relatório de inspeção predial em 3 páginas, cobrindo: 1) Dados da propriedade e características, 2) Condições estruturais e danos aparentes, 3) Revisão de instalações como gás, elétrica e hidráulica, focando em segurança e manutenção.
The Ugly Duckling tells the story of an ugly duckling that is bullied by the other animals because he looks different. As he grows older, he realizes that he is not a duck at all, but instead a beautiful swan. In the end, he finds acceptance among the other swans when they see his true beauty.
Apostila engenharia civil concreto armado recomendacoesEdson D. Vizentin
Este documento fornece recomendações para a produção de estruturas de concreto armado em edifícios. Ele discute a produção de formas e escoramento, a montagem da armadura, e aspectos da produção da estrutura de concreto armado, incluindo a montagem das formas, colocação da armadura, concretagem e desforma.
This document discusses emergency call centers in Estonia. It shows maps of call centers located across Estonia and statistics on emergency call response rates. Over 90% of emergency calls in 2015 were answered quickly and call takers demonstrated professional communication skills and efficiently understood the situation. The document promotes the shared ICT network and systems used across Estonia's emergency call centers.
Cadbury entered the Indian market in 1948 and has since established manufacturing facilities and offices across the country. Its dairy milk chocolate bars are positioned as an alternative to traditional Indian sweets, targeting middle-aged consumers replacing desserts during family occasions. Cadbury uses various marketing campaigns targeting different segments like students, rural markets, and youth through mass media advertising, print media, online advertising, and public relations.
Dans le cadre de mes études universitaires, il nous a demandé de faire un projet à propos de l'une des types de question pour évaluer les compétences des élèves pour n'importe quel sujet, j’ai choisi ''les types de questions posés en classe''.
Je souhaite que vous en profitiez.
How to use database component using stored procedure callprathyusha vadla
1. The document discusses how to use the Mule Database component to call stored procedures with input and output parameters to retrieve data from a database.
2. An example Mule flow is provided that calls a stored procedure using the Database component, passing an input parameter and returning the output parameter value in the payload.
3. The flow listens for HTTP requests, calls the stored procedure to look up a state name based on a geo code parameter, and logs the state name returned in the payload.
This document discusses writing functional test cases for Mule flows using JUnit and MUnit frameworks. When using JUnit, test cases directly connect to components like SAP and Salesforce, modifying real data. MUnit allows mocking these components to avoid this issue. The document provides examples of mocking Salesforce and database components in MUnit test cases using custom payloads. Benefits of MUnit include Java/flow based unit testing, endpoint mocking, assertions, and automated testing.
MUnit is a framework for writing functional test cases in Mule that allows mocking of components like SAP, Salesforce, and databases. When writing functional tests with JUnit, the tests interact directly with the actual components. MUnit allows mocking these components to return custom payloads and avoid modifying real data during tests. The document provides examples of mocking Salesforce and database components in MUnit tests.
MUnit is a framework for writing test cases in Mule that allows mocking of components like SAP, Salesforce, and databases. When writing functional tests for Mule flows with JUnit, the tests interact with the actual components. MUnit allows mocking these components to return custom payloads and avoid modifying real data. The document provides examples of mocking Salesforce and database components in MUnit tests.
MUnit is a framework for writing functional test cases in Mule that allows mocking of components like SAP, Salesforce, and databases. When writing functional tests with JUnit, the tests interact directly with the actual components. MUnit allows mocking these components to return custom payloads and avoid modifying real data during tests. The document provides examples of mocking Salesforce and database components in MUnit tests.
MUnit is a framework for writing functional test cases in Mule that allows mocking of components like SAP, Salesforce, and databases. When writing functional tests with JUnit, the tests interact directly with the actual components. MUnit allows mocking these components to return custom payloads and avoid modifying real data during tests. The document provides examples of mocking Salesforce and database components in MUnit tests.
The primary focus of this presentation is approaching the migration of a large, legacy data store into a new schema built with Django. Includes discussion of how to structure a migration script so that it will run efficiently and scale. Learn how to recognize and evaluate trouble spots.
Also discusses some general tips and tricks for working with data and establishing a productive workflow.
The document discusses strategies for migrating large amounts of legacy data from an old database into a new Django application. Some key points:
- Migrating data in batches and minimizing database queries per row processed can improve performance for large datasets.
- Tools like SQLAlchemy and Maatkit can help optimize the migration process.
- It's important to profile queries, enable logging/debugging, and design migrations that can resume/restart after failures or pause for maintenance.
- Preserving some legacy metadata like IDs on the new models allows mapping data between the systems. Declarative and modular code helps scale the migration tasks.
The document discusses using the ForEach component in Mule applications. It allows splitting a collection into elements and processing them iteratively through embedded processors. An example flow is provided that uses ForEach to iterate over results from a database query and log each element. The flow executes a select query, passes results to ForEach, and logs the payload within the loop and after processing each element.
The document discusses using the ForEach component in Mule applications. It allows splitting a collection into elements and processing them iteratively through embedded processors. An example flow is provided that uses a ForEach component to iterate over results from a database query and log each element. The flow executes a select query, passes results to ForEach to iterate and log each record, and returns the original message.
The document discusses using the ForEach component in Mule applications. It allows splitting a collection into elements and processing them iteratively through embedded processors. An example flow is provided that uses a ForEach component to iterate over results from a database query and log each element. The flow executes a select query, passes results to ForEach to iterate and log each record, and returns the original message.
The document discusses using the ForEach component in Mule applications. It allows splitting a collection into elements and processing them iteratively through embedded processors. An example flow is provided that uses ForEach to iterate over results from a database query and log each element. The flow executes a select query, passes results to ForEach, and logs the payload within the loop and after processing each element.
The document discusses using the ForEach component in Mule applications. It allows splitting a collection into elements and processing them iteratively through embedded processors. An example flow is provided that uses ForEach to iterate over results from a database query and log each element. The flow executes a select query, passes results to ForEach, and logs the payload within the loop and after processing each element.
The document discusses using the ForEach component in Mule applications. It iterates over elements in a collection and processes them individually through embedded processors. The example shows querying a database table, storing the results in a payload, and then using ForEach to log each record. ForEach splits the payload collection and allows processing each element one by one through nested processors like Logger.
Capacity planning is a difficult challenge faced by most companies. If you have too few machines, you will not have enough compute resources available to deal with heavy loads. On the other hand, if you have too many machines, you are wasting money. This is why companies have started investing in automatically scaling services and infrastructure to minimize the amount of wasted money and resources.
In this talk, Nathan will describe how Yelp is using PaaSTA, a PaaS built on top of open source tools including Docker, Mesos, Marathon, and Chronos, to automatically and gracefully scale services and the underlying cluster. He will go into detail about how this functionality was implemented and the design designs that were made while architecting the system. He will also provide a brief comparison of how this approach differs from existing solutions.
The document discusses how to use the For Each component in Mule applications. The For Each component splits a collection into elements and processes them iteratively through embedded processors. An example Mule flow is provided that uses a For Each component to iterate over results returned from a database query and log each element. The flow executes a select query, passes the results to the For Each component, and logs the payload within the loop to output each database record individually.
The document discusses using a reference exception strategy in Mule applications. A reference exception strategy allows flows to reuse global exception handling configurations. The example shows creating a global "JsonSchemaValidatorCatch_Exception_Strategy" and applying it to a flow using <exception-strategy ref="JsonSchemaValidatorCatch_Exception_Strategy"/>. This strategy sets the HTTP status to 400 and payload to "Invalid input" when JSON validation fails.
This document discusses the Scheduler and CLI features in TYPO3. It provides an overview of what Scheduler is, how to set up the main cronjob, and how developers can create scheduler tasks using extensions. It also gives an overview of what CLI is and how users and developers can interact with it. Developers can create command controllers to expose CLI commands for extensions. The document contains code examples and links to GitHub repositories demonstrating scheduler and CLI features.
Mule Munit
1. Solution for JUnit Functional test cases By: Kiet Bui 22-Sep-2015
2. Abstract • The main motto of this white paper is what the issues to write test cases using JUnit are and how to overcome those issues.
3. Table of Contents • ABSTRACT 1. INTRODUCTION 2. PROBLEM STATEMENT 3. SOLUTION 4. BENEFITS 5. CONCLUSION 6. REFERENCES 7. ABOUT THE AUTHOR 8. ABOUT WHISHWORKS
4. Introduction • We have multiple unit test frameworks to write unit and functional test cases for our services. When we write functional test cases using JUnit we can’t mock mule components. To resolve this issues we have to use MUnit and I am going to explain what is the problem with JUnit and how to resolve using MUnit in the below.
5. Problem Statement • When we write functional test cases using JUnit, the test case will directly connect to original components like SAP, Salesforce etc. and insert/select the data. It is the issue in JUnit functional test case why because we are writing functional test cases to check whether entire functionality is working as expected or not without modifying the original components(SAP,Salesforce,Database) data, but in JUnit functional test cases it is directly connecting to original components and modifying the original data. • Examples: 1. SAP Connector • Mule flow:
The Mock component in MUnit allows you to define mocked behavior for a message processor. You can modify how a specific message processor responds when called with particular attributes. The example shows using Mock to replace the "Set Payload" processor with one that returns "Good Morning!" instead of "Hi", and asserting the payload is now "Good Morning!".
The document discusses how to use a JMS selector to filter messages on a JMS inbound endpoint in Mule. It provides an example Mule configuration that uses a JMS selector with an expression to filter for messages with a specific JMSCorrelationID property. The selector is set on the JMS inbound endpoint to filter the messages consumed from the queue.
This document provides instructions on how to use the Sftplite connector in Mulesoft. It describes that Sftplite opens and closes a connection for each call, so it is better to use for individual operations rather than persistent connections. It then provides an example Mule flow that uses Sftplite to retrieve a folder from an SFTP server on each HTTP request.
This document provides an example of how to use the Object Store Connector in Mule to store and retrieve data. It shows storing a Map containing key-value pairs in the object store using the <objectstore:store> element. It then retrieves the stored data using the <objectstore:retrieve> element and sets it as the payload to log and return the value.
This document provides instructions on how to use the SFTP connector in Mule applications. It describes that SFTP uses SSH to securely transfer files between external resources and Mule applications. An example flow is shown that uses an SFTP inbound endpoint to receive a file, logs a message, and writes the file to the local file system using an outbound file endpoint.
This document discusses how to use secure property placeholders in Mule applications. It provides instructions on installing Anypoint Enterprise Security for Anypoint Studio to enable the use of secure property placeholders. An example Mule application configuration is given that uses a secure property placeholder to securely retrieve encrypted property values from an external properties file at runtime.
The document discusses how to specify a default exception strategy in Mule applications. It provides an example of a flow that handles exceptions, showing how to configure a global catch-exception strategy that sets a 400 status code and error payload if any exceptions occur. It also shows sample output when the flow receives valid versus invalid input.
This document discusses defining global exception strategies in Mule applications. It provides an example Mule flow that references a global catch exception strategy, which sets a 400 status code and error payload when exceptions occur. The example shows log output for valid and invalid token inputs, with the invalid input triggering the global exception strategy. References are provided to the Mule documentation on error handling.
The document discusses validating JSON payloads against JSON schemas in Mule applications. It provides an example Mule flow that uses the JSON schema validator to validate incoming JSON payloads against a schema file. If validation fails, an exception is thrown detailing the errors; if it passes, a log message is displayed. The flow listens for HTTP requests and validates the payload against the schema before logging the result.
The document discusses validation in Mule applications using the Validations module. It provides an example Mule flow that uses the <validation:is-not-empty> validator to check for an empty name parameter in the HTTP request. If name is empty, a ValidationException is thrown with a custom error message. Running the flow with or without the name parameter demonstrates a success or failure case.
This document discusses how to use property placeholders in Mule applications. It provides an example Mule configuration file that defines a property placeholder for environment-specific properties files. When the flow is triggered, it will load the properties from the correct file based on the environment variable specified in mule-app.properties. The document aims to demonstrate how to configure properties using placeholders in Mule.
The document discusses using a collection aggregator in Mule applications. It provides an example Mule flow that uses a HTTP listener, splitter, collection aggregator, and logger. The flow splits an XML payload containing multiple items, aggregates them into a collection, and logs the result. It also notes that the collection aggregator will fail if items are not received within a timeout period.
CloudHub is an integration platform that allows deploying applications from Anypoint Studio directly or by exporting a mule deployable archive. There are several steps to set up an organization on Anypoint Platform and deploy an application. These include creating a project in Anypoint Studio, configuring HTTP connectors, deploying to CloudHub by signing in and selecting a domain name, runtime version and environment. The application is then packaged, uploaded and deployed to CloudHub and can be accessed via the assigned domain name. Potential issues include configuring the correct HTTP path to avoid resource not found errors.
The document describes how to implement OAuth2 authentication on an API using Anypoint Platform. It involves:
1. Enabling an OAuth2 provider (Google in this case) by registering a client ID and secret.
2. Applying an OAuth2 policy to the API specification to require valid access tokens.
3. Testing the secured API by obtaining an access token from the provider and including it in requests.
This document discusses how to build and deploy a Mule application. It describes using Anypoint Studio or Jenkins to build the application into a deployable archive file. It then explains two options for deploying the archive file - through the Mule Management Console (MMC) by adding it to the repository and deploying it to a target server, or by directly dropping the archive into the apps folder of a Mule standalone server.
The document discusses how to connect directly to an API through an organization account. It explains that API access management involves authenticating login calls to access APIs. The Apigee API management solution allows controlling access to APIs by IP address. It also notes that API products are used for authorization and access control. The document then provides steps for accessing reports APIs, including having a user account, auth token, and reporting database, as well as the four steps of creating a signing certificate, getting credentials, creating a JWT, and obtaining an access token. It gives details on generating and managing auth tokens in both browser and API modes.
The API Gateway runtime points to backend APIs and services and abstracts them into a layer managed by the Anypoint Platform. Consumer applications invoke services through APIs routed by the gateway, which enforces policies and collects analytics. The gateway acts as a dedicated orchestration layer separating orchestration from implementation concerns, and leverages API Manager governance capabilities to apply throttling, security, and other policies to APIs.
The document discusses using the Splitter component in Mule applications. The Splitter splits an incoming message into fragments based on an expression and sends each fragment to the next processor. This allows processing message parts asynchronously and then reassembling them using an Aggregator. An example flow splits an XML shipment order message by item title and logs the results.
This document discusses using an expression filter in Mule applications. It provides an example XML configuration of a flow that uses an expression filter to check if the payload is not equal to "Hello", and logs a success message if it passes the filter. The output of running the flow is also shown. It concludes by explaining that the expression filter checks if the payload is not equal to "Hello", and if so, allows the message to pass through and logs success.
The document discusses using a Bean as a data source in Mule's database connector. It shows an XML configuration that defines a Bean pointing to a SQL Server database, and uses it to configure the database connector. It then demonstrates a flow that exposes an HTTP endpoint, uses the database connector to execute a SQL select query, and logs the results.
🔥🔥🔥🔥🔥🔥🔥🔥🔥
إضغ بين إيديكم من أقوى الملازم التي صممتها
ملزمة تشريح الجهاز الهيكلي (نظري 3)
💀💀💀💀💀💀💀💀💀💀
تتميز هذهِ الملزمة بعِدة مُميزات :
1- مُترجمة ترجمة تُناسب جميع المستويات
2- تحتوي على 78 رسم توضيحي لكل كلمة موجودة بالملزمة (لكل كلمة !!!!)
#فهم_ماكو_درخ
3- دقة الكتابة والصور عالية جداً جداً جداً
4- هُنالك بعض المعلومات تم توضيحها بشكل تفصيلي جداً (تُعتبر لدى الطالب أو الطالبة بإنها معلومات مُبهمة ومع ذلك تم توضيح هذهِ المعلومات المُبهمة بشكل تفصيلي جداً
5- الملزمة تشرح نفسها ب نفسها بس تكلك تعال اقراني
6- تحتوي الملزمة في اول سلايد على خارطة تتضمن جميع تفرُعات معلومات الجهاز الهيكلي المذكورة في هذهِ الملزمة
واخيراً هذهِ الملزمة حلالٌ عليكم وإتمنى منكم إن تدعولي بالخير والصحة والعافية فقط
كل التوفيق زملائي وزميلاتي ، زميلكم محمد الذهبي 💊💊
🔥🔥🔥🔥🔥🔥🔥🔥🔥
THE SACRIFICE HOW PRO-PALESTINE PROTESTS STUDENTS ARE SACRIFICING TO CHANGE T...indexPub
The recent surge in pro-Palestine student activism has prompted significant responses from universities, ranging from negotiations and divestment commitments to increased transparency about investments in companies supporting the war on Gaza. This activism has led to the cessation of student encampments but also highlighted the substantial sacrifices made by students, including academic disruptions and personal risks. The primary drivers of these protests are poor university administration, lack of transparency, and inadequate communication between officials and students. This study examines the profound emotional, psychological, and professional impacts on students engaged in pro-Palestine protests, focusing on Generation Z's (Gen-Z) activism dynamics. This paper explores the significant sacrifices made by these students and even the professors supporting the pro-Palestine movement, with a focus on recent global movements. Through an in-depth analysis of printed and electronic media, the study examines the impacts of these sacrifices on the academic and personal lives of those involved. The paper highlights examples from various universities, demonstrating student activism's long-term and short-term effects, including disciplinary actions, social backlash, and career implications. The researchers also explore the broader implications of student sacrifices. The findings reveal that these sacrifices are driven by a profound commitment to justice and human rights, and are influenced by the increasing availability of information, peer interactions, and personal convictions. The study also discusses the broader implications of this activism, comparing it to historical precedents and assessing its potential to influence policy and public opinion. The emotional and psychological toll on student activists is significant, but their sense of purpose and community support mitigates some of these challenges. However, the researchers call for acknowledging the broader Impact of these sacrifices on the future global movement of FreePalestine.
Creative Restart 2024: Mike Martin - Finding a way around “no”Taste
Ideas that are good for business and good for the world that we live in, are what I’m passionate about.
Some ideas take a year to make, some take 8 years. I want to share two projects that best illustrate this and why it is never good to stop at “no”.
Information and Communication Technology in EducationMJDuyan
(𝐓𝐋𝐄 𝟏𝟎𝟎) (𝐋𝐞𝐬𝐬𝐨𝐧 2)-𝐏𝐫𝐞𝐥𝐢𝐦𝐬
𝐄𝐱𝐩𝐥𝐚𝐢𝐧 𝐭𝐡𝐞 𝐈𝐂𝐓 𝐢𝐧 𝐞𝐝𝐮𝐜𝐚𝐭𝐢𝐨𝐧:
Students will be able to explain the role and impact of Information and Communication Technology (ICT) in education. They will understand how ICT tools, such as computers, the internet, and educational software, enhance learning and teaching processes. By exploring various ICT applications, students will recognize how these technologies facilitate access to information, improve communication, support collaboration, and enable personalized learning experiences.
𝐃𝐢𝐬𝐜𝐮𝐬𝐬 𝐭𝐡𝐞 𝐫𝐞𝐥𝐢𝐚𝐛𝐥𝐞 𝐬𝐨𝐮𝐫𝐜𝐞𝐬 𝐨𝐧 𝐭𝐡𝐞 𝐢𝐧𝐭𝐞𝐫𝐧𝐞𝐭:
-Students will be able to discuss what constitutes reliable sources on the internet. They will learn to identify key characteristics of trustworthy information, such as credibility, accuracy, and authority. By examining different types of online sources, students will develop skills to evaluate the reliability of websites and content, ensuring they can distinguish between reputable information and misinformation.
How to Setup Default Value for a Field in Odoo 17Celine George
In Odoo, we can set a default value for a field during the creation of a record for a model. We have many methods in odoo for setting a default value to the field.
2. Abstract
• The main motto of this PPT is How to use
Batch component in our applications.
3. Introduction
• Mule possesses the ability to process messages
in batches. Within an application, you can initiate
a batch job which is a block of code that splits
messages into individual records, performs
actions upon each record, then reports on the
results and potentially pushes the processed
output to other systems or queues. This
functionality is particularly useful when working
with streaming input or when engineering "near
real-time" data integration between SaaS
applications.
8. • INFO 2016-05-28 18:51:48,882 [main] org.mule.module.launcher.MuleDeploymentService:
• ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
• + Started app 'Sample' +
• ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
• INFO 2016-05-28 18:51:48,886 [main] org.mule.module.launcher.DeploymentDirectoryWatcher:
• ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
• + Mule is up and kicking (every 5000ms) +
• ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
• INFO 2016-05-28 18:51:49,040 [main] org.mule.module.launcher.StartupSummaryDeploymentListener:
• **********************************************************************
• * - - + DOMAIN + - - * - - + STATUS + - - *
• **********************************************************************
• * default * DEPLOYED *
• **********************************************************************
• *******************************************************************************************************
• * - - + APPLICATION + - - * - - + DOMAIN + - - * - - + STATUS + - - *
• *******************************************************************************************************
• * Sample * default * DEPLOYED *
• *******************************************************************************************************
• INFO 2016-05-28 18:51:53,201 [[Sample].HTTP_Listener_Configuration.worker.01] org.mule.api.processor.LoggerMessageProcessor: --Hai
• INFO 2016-05-28 18:51:53,253 [[Sample].HTTP_Listener_Configuration.worker.01] org.mule.api.processor.LoggerMessageProcessor: --main flow--[{b=2, c=3}, {a=1}, {e=5}, {d=4}]
• INFO 2016-05-28 18:51:53,354 [[Sample].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.DefaultBatchEngine: Created instance dc959d50-24fc-11e6-b3de-
30d420524153 for batch job SampleBatch
• INFO 2016-05-28 18:51:53,359 [[Sample].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.DefaultBatchEngine: Batch job SampleBatch has no input phase.
Creating job instance
• INFO 2016-05-28 18:51:53,383 [[Sample].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.queue.BatchQueueLoader: Starting loading phase for instance
'dc959d50-24fc-11e6-b3de-30d420524153' of job 'SampleBatch'
• INFO 2016-05-28 18:51:53,548 [[Sample].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.queue.BatchQueueLoader: Finished loading phase for instance
dc959d50-24fc-11e6-b3de-30d420524153 of job SampleBatch. 4 records were loaded
• INFO 2016-05-28 18:51:53,562 [[Sample].HTTP_Listener_Configuration.worker.01] com.mulesoft.module.batch.engine.DefaultBatchEngine: Started execution of instance 'dc959d50-
24fc-11e6-b3de-30d420524153' of job 'SampleBatch'
• INFO 2016-05-28 18:51:53,711 [batch-job-SampleBatch-work-manager.01] org.mule.api.processor.LoggerMessageProcessor: --Batch step--{b=2, c=3}--
• INFO 2016-05-28 18:51:53,713 [batch-job-SampleBatch-work-manager.01] org.mule.api.processor.LoggerMessageProcessor: --for each--batch step--2
• INFO 2016-05-28 18:52:00,519 [batch-job-SampleBatch-work-manager.01] org.mule.api.processor.LoggerMessageProcessor: --for each--batch step--3
• INFO 2016-05-28 18:52:00,605 [batch-job-SampleBatch-work-manager.01] org.mule.api.processor.LoggerMessageProcessor: --Batch step--{a=1}--
• INFO 2016-05-28 18:52:00,605 [batch-job-SampleBatch-work-manager.01] org.mule.api.processor.LoggerMessageProcessor: --for each--batch step--1
• INFO 2016-05-28 18:52:00,617 [batch-job-SampleBatch-work-manager.01] org.mule.api.processor.LoggerMessageProcessor: --Batch step--{e=5}--
• INFO 2016-05-28 18:52:00,617 [batch-job-SampleBatch-work-manager.01] org.mule.api.processor.LoggerMessageProcessor: --for each--batch step--5
• INFO 2016-05-28 18:52:02,765 [batch-job-SampleBatch-work-manager.01] com.mulesoft.module.batch.DefaultBatchStep: Found exception processing record on step 'Batch_Step' for job
instance 'dc959d50-24fc-11e6-b3de-30d420524153' of job 'SampleBatch'.
• This is the first record to show this exception on this step for this job instance. Subsequent records with the same failureswill not be logged for performance and log readability reasons:
9. • ********************************************************************************
• Message : String or binary data would be truncated. (com.microsoft.sqlserver.jdbc.SQLServerException). Message payload is of type: String
• Type : org.mule.api.MessagingException
• Code : MULE_ERROR--2
• Payload : 5
• JavaDoc : http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html
• ********************************************************************************
• Exception stack is:
• 1. String or binary data would be truncated. (com.microsoft.sqlserver.jdbc.SQLServerException)
• com.microsoft.sqlserver.jdbc.SQLServerException:216 (null)
• 2. String or binary data would be truncated. (com.microsoft.sqlserver.jdbc.SQLServerException). Message payload is of type: String (org.mule.api.MessagingException)
• org.mule.module.db.internal.processor.AbstractDbMessageProcessor:93 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html)
• ********************************************************************************
• Root Exception stack trace:
• com.microsoft.sqlserver.jdbc.SQLServerException: String or binary data would be truncated.
• at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:216)
• at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1515)
• at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:404)
• at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:350)
• at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:5696)
• at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1715)
• at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:180)
• at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:155)
• at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:314)
• at org.mule.module.db.internal.domain.autogeneratedkey.NoAutoGeneratedKeyStrategy.executeUpdate(NoAutoGeneratedKeyStrategy.java:55)
• at org.mule.module.db.internal.domain.executor.UpdateExecutor.doExecuteQuery(UpdateExecutor.java:43)
• at org.mule.module.db.internal.domain.executor.AbstractSingleQueryExecutor.execute(AbstractSingleQueryExecutor.java:48)
• at org.mule.module.db.internal.processor.UpdateMessageProcessor.doExecuteQuery(UpdateMessageProcessor.java:59)
• at org.mule.module.db.internal.processor.AbstractSingleQueryDbMessageProcessor.executeQuery(AbstractSingleQueryDbMessageProcessor.java:42)
• at org.mule.module.db.internal.processor.AbstractDbMessageProcessor.process(AbstractDbMessageProcessor.java:66)
• at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
• at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:107)
• at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
• at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
• at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
• at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
• at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:107)
• at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
• at org.mule.routing.AbstractSelectiveRouter.processEventWithProcessor(AbstractSelectiveRouter.java:303)
• at org.mule.routing.AbstractSelectiveRouter.routeWithProcessors(AbstractSelectiveRouter.java:293)
10. • at org.mule.routing.AbstractSelectiveRouter.process(AbstractSelectiveRouter.java:193)
• at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
• at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:107)
• at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
• at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
• at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
• at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
• at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
• at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
• at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
• at org.mule.routing.outbound.AbstractMessageSequenceSplitter.processParts(AbstractMessageSequenceSplitter.java:129)
• at org.mule.routing.outbound.AbstractMessageSequenceSplitter.process(AbstractMessageSequenceSplitter.java:59)
• at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
• at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:107)
• at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
• at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
• at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
• at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
• at org.mule...
• ********************************************************************************
• INFO 2016-05-28 18:52:02,768 [batch-job-SampleBatch-work-manager.01] com.mulesoft.module.batch.DefaultBatchStep: Found exception processing record on step 'Batch_Step' for job
instance 'dc959d50-24fc-11e6-b3de-30d420524153' of job 'SampleBatch'.
• This is the first record to show this exception on this step for this job instance. Subsequent records with the same failureswill not be logged for performance and log readability reasons:
• ********************************************************************************
• Message : String or binary data would be truncated. (com.microsoft.sqlserver.jdbc.SQLServerException). Message payload is of type: String
• Type : org.mule.api.MessagingException
• Code : MULE_ERROR--2
• Payload : 5
• JavaDoc : http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html
• ********************************************************************************
• Exception stack is:
• 1. String or binary data would be truncated. (com.microsoft.sqlserver.jdbc.SQLServerException)
• com.microsoft.sqlserver.jdbc.SQLServerException:216 (null)
• 2. String or binary data would be truncated. (com.microsoft.sqlserver.jdbc.SQLServerException). Message payload is of type: String (org.mule.api.MessagingException)
• org.mule.module.db.internal.processor.AbstractDbMessageProcessor:93 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html)
• ********************************************************************************
• Root Exc
• com.microsoft.sqlserver.jdbc.SQLServerException: String or binary data would be truncated.
• at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:216)
• at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1515)
• at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:404)
• at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:350)
• at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:5696) eption stack trace:
11. • at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1715)
• at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:180)
• at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:155)
• at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:314)
• at org.mule.module.db.internal.domain.autogeneratedkey.NoAutoGeneratedKeyStrategy.executeUpdate(NoAutoGeneratedKeyStrategy.java:55)
• at org.mule.module.db.internal.domain.executor.UpdateExecutor.doExecuteQuery(UpdateExecutor.java:43)
• at org.mule.module.db.internal.domain.executor.AbstractSingleQueryExecutor.execute(AbstractSingleQueryExecutor.java:48)
• at org.mule.module.db.internal.processor.UpdateMessageProcessor.doExecuteQuery(UpdateMessageProcessor.java:59)
• at org.mule.module.db.internal.processor.AbstractSingleQueryDbMessageProcessor.executeQuery(AbstractSingleQueryDbMessageProcessor.java:42)
• at org.mule.module.db.internal.processor.AbstractDbMessageProcessor.process(AbstractDbMessageProcessor.java:66)
• at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
• at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:107)
• at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
• at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
• at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
• at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
• at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:107)
• at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
• at org.mule.routing.AbstractSelectiveRouter.processEventWithProcessor(AbstractSelectiveRouter.java:303)
• at org.mule.routing.AbstractSelectiveRouter.routeWithProcessors(AbstractSelectiveRouter.java:293)
• at org.mule.routing.AbstractSelectiveRouter.process(AbstractSelectiveRouter.java:193)
• at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
• at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:107)
• at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
• at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
• at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
• at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
• at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
• at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
• at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
• at org.mule.routing.outbound.AbstractMessageSequenceSplitter.processParts(AbstractMessageSequenceSplitter.java:129)
• at org.mule.routing.outbound.AbstractMessageSequenceSplitter.process(AbstractMessageSequenceSplitter.java:59)
• at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
• at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:107)
• at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
• at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
• at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
• at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
• at org.mule...
• ********************************************************************************
12. • INFO 2016-05-28 18:52:02,769 [batch-job-SampleBatch-work-manager.01] org.mule.api.processor.LoggerMessageProcessor: --Batch step--{d=4}--
• INFO 2016-05-28 18:52:02,769 [batch-job-SampleBatch-work-manager.01] org.mule.api.processor.LoggerMessageProcessor: --for each--batch step--
4
• INFO 2016-05-28 18:52:03,049 [batch-job-SampleBatch-work-manager.01] com.mulesoft.module.batch.DefaultBatchStep: Step Batch_Step finished
processing all records for instance dc959d50-24fc-11e6-b3de-30d420524153 of job SampleBatch
• INFO 2016-05-28 18:52:03,095 [batch-job-SampleBatch-work-manager.02] org.mule.api.processor.LoggerMessageProcessor: --step1--{e=5}
• INFO 2016-05-28 18:52:03,096 [batch-job-SampleBatch-work-manager.02] org.mule.api.processor.LoggerMessageProcessor: --for each--batch
step1--5--
• INFO 2016-05-28 18:52:03,488 [batch-job-SampleBatch-work-manager.02] com.mulesoft.module.batch.engine.DefaultBatchEngine: Starting
execution of onComplete phase for instance dc959d50-24fc-11e6-b3de-30d420524153 of job SampleBatch
• INFO 2016-05-28 18:52:03,508 [batch-job-SampleBatch-work-manager.02] org.mule.api.processor.LoggerMessageProcessor: --complete--
com.mulesoft.module.batch.ImmutableBatchJobResult@1fa547d1
• INFO 2016-05-28 18:52:03,509 [batch-job-SampleBatch-work-manager.02] com.mulesoft.module.batch.engine.DefaultBatchEngine: Finished
execution of onComplete phase for instance dc959d50-24fc-11e6-b3de-30d420524153 of job SampleBatch
• INFO 2016-05-28 18:52:03,509 [batch-job-SampleBatch-work-manager.02] com.mulesoft.module.batch.engine.DefaultBatchEngine: Finished
execution for instance 'dc959d50-24fc-11e6-b3de-30d420524153' of job 'SampleBatch'. Total Records processed: 4. Successful records: 3. Failed
Records: 1
• INFO 2016-05-28 18:52:03,511 [batch-job-SampleBatch-work-manager.02] com.mulesoft.module.batch.engine.DefaultBatchEngine:
• *************************************************************************************************************************
***************************************
• * - - + Exception Type + - - * - - + Step + - - * - - + Count + - - *
• *************************************************************************************************************************
***************************************
• * com.mulesoft.module.batch.exception.BatchException * Batch_Step * 1 *
• *************************************************************************************************************************
***************************************
• INFO 2016-05-28 18:52:03,524 [batch-job-SampleBatch-work-manager.02] com.mulesoft.module.batch.DefaultBatchStep: Step Batch_Step1
finished processing all records for instance dc959d50-24fc-11e6-b3de-30d420524153 of job SampleBatch
13. • Flow of execution:
1. URL to trigger the service from browser
http://localhost:8089
2. a. Service will update the status to ‘Success’ for
the records whose num is other than 5
b. It will fail while trying to update the status as
‘successsuccesssuccesssuccesssuccesssuccess’ for
the num 5 because the length of the status excides
the size of the column. Hence, the failed map will
process in Batch_Step1 and will update the status
as ‘batch fail’ for the record whose num is 5