SlideShare a Scribd company logo
Testing Spark and scala
https://github.com/ganeshayadiyala/Scalatest-library-to-unit-test-spark/
● Ganesha Yadiyala
● Big data consultant at
datamantra.io
● Consult in spark and scala
● ganeshayadiyala@gmail.com
Agenda
● What is testing
● Different types of testing process
● Unit tests using scalatest
● Different styles in scalatest
● Using assertions
● Sharing fixtures
● Matchers
● Async Testing
● Testing of spark batch operation
● Unit testing streaming operation
What is testing
Software testing is a process of executing a program or application with the intent
of finding the software bugs.
It can also be stated as the process of validating and verifying that a software
application,
● Meets the business and technical requirements that guided it’s design and
development
● Works as expected
Few of the types of tests
● Unit tests
● Integration tests
● Functional tests
Unit tests
● Unit testing simply verifies that individual units of code (mostly functions) work
as expected
● Assumes everything else works
● Tests one specific condition or flow.
Advantages :
● Codes are more reusable. In order to make unit testing possible, codes need
to be modular. This means that codes are easier to reuse.
● Debugging is easy. When a test fails, only the latest changes need to be
debugged.
Integration tests
● Tests the interoperability of multiple subsystem
● Includes real components, databases etc
● Tests the connectivity of the components
● Hard to test all the cases (combination of tests are more)
● Hard to localize the errors ( may break different reasons)
● Much slower than unit tests
Functional tests
● Functional Testing is the type of testing done against the business
requirements of application
● Use real components and real data
Unit Test in scala
Scalatest
● We use scalatest for unit tests in scala
● For every class in src/main/scala write a test class in src/test/scala
● Consists of suite (collection of test cases)
● You define test classes by composing Suite style and mixin traits.
● You can test both scala and java code
● offers deep integration with tools such as JUnit, TestNG, Ant, Maven, sbt,
ScalaCheck, JMock, EasyMock, Mockito, ScalaMock, Selenium, Eclipse,
NetBeans, and IntelliJ.
Using the scalatest maven plugin
We have to disable maven surefire plugin and enable scalatest plugin
● Specify <skipTests>true</skipTests> in maven surefire plugin
● Add the scalatest-maven plugin and set the goals to test
Different styles in scalatest
● FunSuite
● FlatSpec
● FunSpec
● WordSpec
● FreeSpec
● PropSpec
● FeatureSpec
FunSuite
● In a FunSuite, tests are function values.
● You denote tests with test and provide the name of the test as a string
enclosed in parentheses, followed by the code of the test in curly braces
Ex : com.ganesh.scalatest.specs.FunSuitTest.scala
FlatSpec
● No nesting approach contrasts with the traits FunSpec and WordSpec.
● Uses behavior of clause
Ex : com.ganesh.scalatest.specs.FlatSpecTest.scala
FunSpec
● Tests are combined with text that specifies the behavior of the test.
● Uses describe clause
Ex : com.ganesh.scalatest.specs.FunSpecTest.scala
WordSpec
● your specification text is structured by placing words after strings
● Uses should and in clause
Ex : com.ganesh.scalatest.specs.WordSpecTest.scala
Using Assertions
ScalaTest makes three assertions available by default in any style trait
● assert - for general assertion.
● assertResult - to differentiate expected from actual values.
● assertThrows - to ensure a bit of code throws an expected exception.
Scalatest assertions are defined in trait Assertions. Assertions also provide some
other API’s.
Ex : com.ganesh.scalatest.features.AssertionsTest.scala
Ignoring the test
● Scalatest allows to ignore the test.
● We can ignore the test if we want it to change it implementation and run later
or if the test case is slow.
● We use ignore clause to ignore the test
● We use @Ignore annotation to ignore all the test in a suite.
Ex : com.ganesh.scalatest.features.IgnoreTest.scala
Sharing fixture
A test fixture is composed of the objects and other artifacts, which tests use to do
their work.
When multiple tests needs to work with the same fixture, we can share the fixture
between them.
It will reduce the duplication of code.
By calling get-fixture methods
If you need to create the same mutable fixture objects in multiple tests we can use
get-fixture method
● A get-fixture method returns a new instance of a needed fixture object each
time it is called
● Not appropriate to use if we need to cleanup those objects
Ex : com.ganesh.scalatest.fixtures.GetFixtureTest.scala
By Instantiating fixture-context objects
When different tests need different combinations of fixture objects, define the
fixture objects as instance variables of fixture-context objects.
● In this approach we initialize a fixture object inside trait/class.
● We create a new instance of the fixture trait in the test we need them.
● We can even mix in these fixture traits we created.
Ex : com.ganesh.scalatest.fixtures.FixtureContextTest.scala
By using withFixture
● Allows cleaning up of fixtures at the end of the tests
● If we have no object to pass to the test case, then we can use
withFixture(NoArgTest).
● If we have one or more objects to be passed to test case, then we need to
use withFixture(OneArgTest).
Ex : com.ganesh.scalatest.fixtures.WithFicture*.scala
By using BeforeAndAfter
● Methods which we used till now for sharing fixtures are performed during the
test.
● If exception occurs while creating this fixture then it’ll be reported as test
failure.
● If we use BeforeAndAfter setup happens before the test execution starts, and
cleanup happens once the test is completed
● So if any exception happens in the setup, it’ll abort the entire suit and no more
tests are attempted.
Ex : com.ganesh.scalatest.fixtures.BeforeAndAfterTest.scala
Matchers
ScalaTest provides a domain specific language (DSL) for expressing assertions in
tests using the word should.
Ex : com.ganesh.scalatest.features.MatchersTest.scala
Asynchronous testing
● Given a Future returned by the code you are testing, you need not block until
the Future completes before performing assertions against its value.
● We can instead map those assertions onto the Future and return the resulting
Future[Assertion] to ScalaTest.
● This result is executed asynchronously.
Ex : com.ganesh.scalatest.features.AsyncTest.scala
Testing private methods
● If the method is private in a class we can test it using scalatest.
● We can use PrivateMethodTester trait to achieve this.
● We can use invokePrivate operator to call the private method
Ex : com.ganesh.scalatest.features.PrivateMethodTest.scala
Mocking
Scalatest supports following mock libraries,
● ScalaMock
● EasyMock
● JMock
● Mockito
Ex : com.ganesh.scalatest.mock.MockTest.scala
Testing Spark
Complexities
● Needs spark context for all the tests
● Testing operations such as map, flatmap and reduce.
● Testing streaming application (Dstream operations).
● Making sure that there is only one context for each test case.
Setup
● Instead of creating contexts which are needed for each test suite, we create
the trait which extends BeforeAndAfter, and all our suites will extend this trait.
● In that trait we try to initialize all the contexts in before method
● All the contexts will be destroyed in after method
● Extend this trait in all the test suites
Ex : com.ganesh.scalatest.sparkbatch.EnvironmentInitializerSC.scala
Spark Streaming test
● The full control over clock is needed to manually manage batches, slides and
windows.
● Spark Streaming provides necessary abstraction over system clock,
ManualClock class.
● But its private class, we cannot access it in our testcases
● So we use a wrapper class to use the ManualClock instance in our test case.
Ex : com.ganesh.scalatest.sparkstreaming
Summary
● We can select any of the styles provided by the scalatest, it just differs in how
we write test but will have all the features.
● Make use of assertions and matchers provided by scalatest for better test
cases.
● While testing spark we need to test the logic, so keep your code modular so
that each logic can be tested individually.
● There is a external library called spark testing base which provides many
functions to assert on dataframe level and it has traits which provides you the
contexts needed for the test.
References
● http://www.scalatest.org/
● http://mkuthan.github.io/blog/2015/03/01/spark-unit-testing/
● https://www.slideshare.net/remeniuk/testing-in-scala-adform-research

More Related Content

What's hot

testng
testngtestng
TestNG Session presented in PB
TestNG Session presented in PBTestNG Session presented in PB
TestNG Session presented in PB
Abhishek Yadav
 
Mutation testing
Mutation testingMutation testing
Mutation testing
Łukasz Cieśluk
 
Sanity testing and smoke testing
Sanity testing and smoke testingSanity testing and smoke testing
Sanity testing and smoke testing
MUHAMMAD FARHAN ASLAM
 
Code Quality in Ruby and Java
Code Quality in Ruby and JavaCode Quality in Ruby and Java
Code Quality in Ruby and Java
Steve Hayes
 
Model-Based Testing: Why, What, How
Model-Based Testing: Why, What, HowModel-Based Testing: Why, What, How
Model-Based Testing: Why, What, How
Bob Binder
 
Unit Testing And Mocking
Unit Testing And MockingUnit Testing And Mocking
Unit Testing And Mocking
Joe Wilson
 
Unit Testing and Coverage for AngularJS
Unit Testing and Coverage for AngularJSUnit Testing and Coverage for AngularJS
Unit Testing and Coverage for AngularJS
Knoldus Inc.
 
Test cases
Test casesTest cases
Test cases
Chandra Maddigapu
 
Maven
MavenMaven
Getting started with karate dsl
Getting started with karate dslGetting started with karate dsl
Getting started with karate dsl
Knoldus Inc.
 
Type script - advanced usage and practices
Type script  - advanced usage and practicesType script  - advanced usage and practices
Type script - advanced usage and practices
Iwan van der Kleijn
 
Mutation Testing: Testing your tests
Mutation Testing: Testing your testsMutation Testing: Testing your tests
Mutation Testing: Testing your tests
Stephen Leigh
 
Test Case Design
Test Case DesignTest Case Design
Test Case Design
acatalin
 
Unit testing JavaScript: Jasmine & karma intro
Unit testing JavaScript: Jasmine & karma introUnit testing JavaScript: Jasmine & karma intro
Unit testing JavaScript: Jasmine & karma intro
Maurice De Beijer [MVP]
 
TESTING STRATEGY.ppt
TESTING STRATEGY.pptTESTING STRATEGY.ppt
TESTING STRATEGY.ppt
FawazHussain4
 
Angular Unit Testing
Angular Unit TestingAngular Unit Testing
Angular Unit Testing
Shailendra Chauhan
 
Unit testing best practices
Unit testing best practicesUnit testing best practices
Unit testing best practices
nickokiss
 
Sirius Web Advanced : Customize and Extend the Platform
Sirius Web Advanced : Customize and Extend the PlatformSirius Web Advanced : Customize and Extend the Platform
Sirius Web Advanced : Customize and Extend the Platform
Obeo
 
TestNG Session presented in Xebia XKE
TestNG Session presented in Xebia XKETestNG Session presented in Xebia XKE
TestNG Session presented in Xebia XKE
Abhishek Yadav
 

What's hot (20)

testng
testngtestng
testng
 
TestNG Session presented in PB
TestNG Session presented in PBTestNG Session presented in PB
TestNG Session presented in PB
 
Mutation testing
Mutation testingMutation testing
Mutation testing
 
Sanity testing and smoke testing
Sanity testing and smoke testingSanity testing and smoke testing
Sanity testing and smoke testing
 
Code Quality in Ruby and Java
Code Quality in Ruby and JavaCode Quality in Ruby and Java
Code Quality in Ruby and Java
 
Model-Based Testing: Why, What, How
Model-Based Testing: Why, What, HowModel-Based Testing: Why, What, How
Model-Based Testing: Why, What, How
 
Unit Testing And Mocking
Unit Testing And MockingUnit Testing And Mocking
Unit Testing And Mocking
 
Unit Testing and Coverage for AngularJS
Unit Testing and Coverage for AngularJSUnit Testing and Coverage for AngularJS
Unit Testing and Coverage for AngularJS
 
Test cases
Test casesTest cases
Test cases
 
Maven
MavenMaven
Maven
 
Getting started with karate dsl
Getting started with karate dslGetting started with karate dsl
Getting started with karate dsl
 
Type script - advanced usage and practices
Type script  - advanced usage and practicesType script  - advanced usage and practices
Type script - advanced usage and practices
 
Mutation Testing: Testing your tests
Mutation Testing: Testing your testsMutation Testing: Testing your tests
Mutation Testing: Testing your tests
 
Test Case Design
Test Case DesignTest Case Design
Test Case Design
 
Unit testing JavaScript: Jasmine & karma intro
Unit testing JavaScript: Jasmine & karma introUnit testing JavaScript: Jasmine & karma intro
Unit testing JavaScript: Jasmine & karma intro
 
TESTING STRATEGY.ppt
TESTING STRATEGY.pptTESTING STRATEGY.ppt
TESTING STRATEGY.ppt
 
Angular Unit Testing
Angular Unit TestingAngular Unit Testing
Angular Unit Testing
 
Unit testing best practices
Unit testing best practicesUnit testing best practices
Unit testing best practices
 
Sirius Web Advanced : Customize and Extend the Platform
Sirius Web Advanced : Customize and Extend the PlatformSirius Web Advanced : Customize and Extend the Platform
Sirius Web Advanced : Customize and Extend the Platform
 
TestNG Session presented in Xebia XKE
TestNG Session presented in Xebia XKETestNG Session presented in Xebia XKE
TestNG Session presented in Xebia XKE
 

Similar to Testing Spark and Scala

Scala test
Scala testScala test
Scala test
Meetu Maltiar
 
Scala test
Scala testScala test
Angular Unit testing.pptx
Angular Unit testing.pptxAngular Unit testing.pptx
Angular Unit testing.pptx
RiyaBangera
 
JAVASCRIPT Test Driven Development & Jasmine
JAVASCRIPT Test Driven Development & JasmineJAVASCRIPT Test Driven Development & Jasmine
JAVASCRIPT Test Driven Development & Jasmine
Anup Singh
 
Cypress Testing.pptx
Cypress Testing.pptxCypress Testing.pptx
Cypress Testing.pptx
JasmeenShrestha
 
JUnit- A Unit Testing Framework
JUnit- A Unit Testing FrameworkJUnit- A Unit Testing Framework
JUnit- A Unit Testing Framework
Onkar Deshpande
 
S313352 optimizing java device testing with automatic feature discovering
S313352 optimizing java device testing with automatic feature discoveringS313352 optimizing java device testing with automatic feature discovering
S313352 optimizing java device testing with automatic feature discovering
romanovfedor
 
Java Unit Test - JUnit
Java Unit Test - JUnitJava Unit Test - JUnit
Java Unit Test - JUnit
Aktuğ Urun
 
Unit testing
Unit testingUnit testing
Unit testing
Pooya Sagharchiha
 
[FullStack NYC 2019] Effective Unit Tests for JavaScript
[FullStack NYC 2019] Effective Unit Tests for JavaScript[FullStack NYC 2019] Effective Unit Tests for JavaScript
[FullStack NYC 2019] Effective Unit Tests for JavaScript
Hazem Saleh
 
Unit testing in xcode 8 with swift
Unit testing in xcode 8 with swiftUnit testing in xcode 8 with swift
Unit testing in xcode 8 with swift
allanh0526
 
Intro to junit
Intro to junitIntro to junit
Intro to junit
Rakesh Srivastava
 
Kirill Rozin - Practical Wars for Automatization
Kirill Rozin - Practical Wars for AutomatizationKirill Rozin - Practical Wars for Automatization
Kirill Rozin - Practical Wars for Automatization
Sergey Arkhipov
 
Automation for developers
Automation for developersAutomation for developers
Automation for developers
Dharshana Kasun Warusavitharana
 
Annotations
AnnotationsAnnotations
Annotations
Knoldus Inc.
 
Unit testing
Unit testingUnit testing
Unit testing
Panos Pnevmatikatos
 
Unit Testing in Angular
Unit Testing in AngularUnit Testing in Angular
Unit Testing in Angular
Knoldus Inc.
 
Wso2 test automation framework internal training
Wso2 test automation framework internal trainingWso2 test automation framework internal training
Wso2 test automation framework internal training
Dharshana Kasun Warusavitharana
 
Quick tour to front end unit testing using jasmine
Quick tour to front end unit testing using jasmineQuick tour to front end unit testing using jasmine
Quick tour to front end unit testing using jasmine
Gil Fink
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Development
Anand Kumar Rajana
 

Similar to Testing Spark and Scala (20)

Scala test
Scala testScala test
Scala test
 
Scala test
Scala testScala test
Scala test
 
Angular Unit testing.pptx
Angular Unit testing.pptxAngular Unit testing.pptx
Angular Unit testing.pptx
 
JAVASCRIPT Test Driven Development & Jasmine
JAVASCRIPT Test Driven Development & JasmineJAVASCRIPT Test Driven Development & Jasmine
JAVASCRIPT Test Driven Development & Jasmine
 
Cypress Testing.pptx
Cypress Testing.pptxCypress Testing.pptx
Cypress Testing.pptx
 
JUnit- A Unit Testing Framework
JUnit- A Unit Testing FrameworkJUnit- A Unit Testing Framework
JUnit- A Unit Testing Framework
 
S313352 optimizing java device testing with automatic feature discovering
S313352 optimizing java device testing with automatic feature discoveringS313352 optimizing java device testing with automatic feature discovering
S313352 optimizing java device testing with automatic feature discovering
 
Java Unit Test - JUnit
Java Unit Test - JUnitJava Unit Test - JUnit
Java Unit Test - JUnit
 
Unit testing
Unit testingUnit testing
Unit testing
 
[FullStack NYC 2019] Effective Unit Tests for JavaScript
[FullStack NYC 2019] Effective Unit Tests for JavaScript[FullStack NYC 2019] Effective Unit Tests for JavaScript
[FullStack NYC 2019] Effective Unit Tests for JavaScript
 
Unit testing in xcode 8 with swift
Unit testing in xcode 8 with swiftUnit testing in xcode 8 with swift
Unit testing in xcode 8 with swift
 
Intro to junit
Intro to junitIntro to junit
Intro to junit
 
Kirill Rozin - Practical Wars for Automatization
Kirill Rozin - Practical Wars for AutomatizationKirill Rozin - Practical Wars for Automatization
Kirill Rozin - Practical Wars for Automatization
 
Automation for developers
Automation for developersAutomation for developers
Automation for developers
 
Annotations
AnnotationsAnnotations
Annotations
 
Unit testing
Unit testingUnit testing
Unit testing
 
Unit Testing in Angular
Unit Testing in AngularUnit Testing in Angular
Unit Testing in Angular
 
Wso2 test automation framework internal training
Wso2 test automation framework internal trainingWso2 test automation framework internal training
Wso2 test automation framework internal training
 
Quick tour to front end unit testing using jasmine
Quick tour to front end unit testing using jasmineQuick tour to front end unit testing using jasmine
Quick tour to front end unit testing using jasmine
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Development
 

More from datamantra

Multi Source Data Analysis using Spark and Tellius
Multi Source Data Analysis using Spark and TelliusMulti Source Data Analysis using Spark and Tellius
Multi Source Data Analysis using Spark and Tellius
datamantra
 
State management in Structured Streaming
State management in Structured StreamingState management in Structured Streaming
State management in Structured Streaming
datamantra
 
Spark on Kubernetes
Spark on KubernetesSpark on Kubernetes
Spark on Kubernetes
datamantra
 
Understanding transactional writes in datasource v2
Understanding transactional writes in  datasource v2Understanding transactional writes in  datasource v2
Understanding transactional writes in datasource v2
datamantra
 
Introduction to Datasource V2 API
Introduction to Datasource V2 APIIntroduction to Datasource V2 API
Introduction to Datasource V2 API
datamantra
 
Exploratory Data Analysis in Spark
Exploratory Data Analysis in SparkExploratory Data Analysis in Spark
Exploratory Data Analysis in Spark
datamantra
 
Core Services behind Spark Job Execution
Core Services behind Spark Job ExecutionCore Services behind Spark Job Execution
Core Services behind Spark Job Execution
datamantra
 
Optimizing S3 Write-heavy Spark workloads
Optimizing S3 Write-heavy Spark workloadsOptimizing S3 Write-heavy Spark workloads
Optimizing S3 Write-heavy Spark workloads
datamantra
 
Structured Streaming with Kafka
Structured Streaming with KafkaStructured Streaming with Kafka
Structured Streaming with Kafka
datamantra
 
Understanding time in structured streaming
Understanding time in structured streamingUnderstanding time in structured streaming
Understanding time in structured streaming
datamantra
 
Spark stack for Model life-cycle management
Spark stack for Model life-cycle managementSpark stack for Model life-cycle management
Spark stack for Model life-cycle management
datamantra
 
Productionalizing Spark ML
Productionalizing Spark MLProductionalizing Spark ML
Productionalizing Spark ML
datamantra
 
Introduction to Structured streaming
Introduction to Structured streamingIntroduction to Structured streaming
Introduction to Structured streaming
datamantra
 
Building real time Data Pipeline using Spark Streaming
Building real time Data Pipeline using Spark StreamingBuilding real time Data Pipeline using Spark Streaming
Building real time Data Pipeline using Spark Streaming
datamantra
 
Understanding Implicits in Scala
Understanding Implicits in ScalaUnderstanding Implicits in Scala
Understanding Implicits in Scala
datamantra
 
Migrating to Spark 2.0 - Part 2
Migrating to Spark 2.0 - Part 2Migrating to Spark 2.0 - Part 2
Migrating to Spark 2.0 - Part 2
datamantra
 
Migrating to spark 2.0
Migrating to spark 2.0Migrating to spark 2.0
Migrating to spark 2.0
datamantra
 
Scalable Spark deployment using Kubernetes
Scalable Spark deployment using KubernetesScalable Spark deployment using Kubernetes
Scalable Spark deployment using Kubernetes
datamantra
 
Introduction to concurrent programming with akka actors
Introduction to concurrent programming with akka actorsIntroduction to concurrent programming with akka actors
Introduction to concurrent programming with akka actors
datamantra
 
Functional programming in Scala
Functional programming in ScalaFunctional programming in Scala
Functional programming in Scala
datamantra
 

More from datamantra (20)

Multi Source Data Analysis using Spark and Tellius
Multi Source Data Analysis using Spark and TelliusMulti Source Data Analysis using Spark and Tellius
Multi Source Data Analysis using Spark and Tellius
 
State management in Structured Streaming
State management in Structured StreamingState management in Structured Streaming
State management in Structured Streaming
 
Spark on Kubernetes
Spark on KubernetesSpark on Kubernetes
Spark on Kubernetes
 
Understanding transactional writes in datasource v2
Understanding transactional writes in  datasource v2Understanding transactional writes in  datasource v2
Understanding transactional writes in datasource v2
 
Introduction to Datasource V2 API
Introduction to Datasource V2 APIIntroduction to Datasource V2 API
Introduction to Datasource V2 API
 
Exploratory Data Analysis in Spark
Exploratory Data Analysis in SparkExploratory Data Analysis in Spark
Exploratory Data Analysis in Spark
 
Core Services behind Spark Job Execution
Core Services behind Spark Job ExecutionCore Services behind Spark Job Execution
Core Services behind Spark Job Execution
 
Optimizing S3 Write-heavy Spark workloads
Optimizing S3 Write-heavy Spark workloadsOptimizing S3 Write-heavy Spark workloads
Optimizing S3 Write-heavy Spark workloads
 
Structured Streaming with Kafka
Structured Streaming with KafkaStructured Streaming with Kafka
Structured Streaming with Kafka
 
Understanding time in structured streaming
Understanding time in structured streamingUnderstanding time in structured streaming
Understanding time in structured streaming
 
Spark stack for Model life-cycle management
Spark stack for Model life-cycle managementSpark stack for Model life-cycle management
Spark stack for Model life-cycle management
 
Productionalizing Spark ML
Productionalizing Spark MLProductionalizing Spark ML
Productionalizing Spark ML
 
Introduction to Structured streaming
Introduction to Structured streamingIntroduction to Structured streaming
Introduction to Structured streaming
 
Building real time Data Pipeline using Spark Streaming
Building real time Data Pipeline using Spark StreamingBuilding real time Data Pipeline using Spark Streaming
Building real time Data Pipeline using Spark Streaming
 
Understanding Implicits in Scala
Understanding Implicits in ScalaUnderstanding Implicits in Scala
Understanding Implicits in Scala
 
Migrating to Spark 2.0 - Part 2
Migrating to Spark 2.0 - Part 2Migrating to Spark 2.0 - Part 2
Migrating to Spark 2.0 - Part 2
 
Migrating to spark 2.0
Migrating to spark 2.0Migrating to spark 2.0
Migrating to spark 2.0
 
Scalable Spark deployment using Kubernetes
Scalable Spark deployment using KubernetesScalable Spark deployment using Kubernetes
Scalable Spark deployment using Kubernetes
 
Introduction to concurrent programming with akka actors
Introduction to concurrent programming with akka actorsIntroduction to concurrent programming with akka actors
Introduction to concurrent programming with akka actors
 
Functional programming in Scala
Functional programming in ScalaFunctional programming in Scala
Functional programming in Scala
 

Recently uploaded

Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
janvikumar4133
 
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy DsouzaOpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata
 
Biometric Question Bank 2021 - 1 Soln-1.pdf
Biometric Question Bank 2021 - 1 Soln-1.pdfBiometric Question Bank 2021 - 1 Soln-1.pdf
Biometric Question Bank 2021 - 1 Soln-1.pdf
Joel Ngushwai
 
Research proposal seminar ,Research Methodology
Research proposal seminar ,Research MethodologyResearch proposal seminar ,Research Methodology
Research proposal seminar ,Research Methodology
doctorzlife786
 
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
saadkhan1485265
 
Willis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdfWillis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdf
LINAT
 
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
45unexpected
 
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
norina2645
 
History and Application of LLM Leveraging Big Data
History and Application of LLM Leveraging Big DataHistory and Application of LLM Leveraging Big Data
History and Application of LLM Leveraging Big Data
Jongwook Woo
 
transgenders community data in india by govt
transgenders community data in india by govttransgenders community data in india by govt
transgenders community data in india by govt
palanisamyiiiier
 
Harendra Singh, AI Strategy and Consulting Portfolio
Harendra Singh, AI Strategy and Consulting PortfolioHarendra Singh, AI Strategy and Consulting Portfolio
Harendra Singh, AI Strategy and Consulting Portfolio
harendmgr
 
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
birajmohan012
 
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
kinni singh$A17
 
potential development of the A* search algorithm specifically
potential development of the A* search algorithm specificallypotential development of the A* search algorithm specifically
potential development of the A* search algorithm specifically
huseindihon
 
potential usefulness of multi-agent maze-solving in general
potential usefulness of multi-agent maze-solving in generalpotential usefulness of multi-agent maze-solving in general
potential usefulness of multi-agent maze-solving in general
huseindihon
 
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
vrvipin164
 
DU degree offer diploma Transcript
DU degree offer diploma TranscriptDU degree offer diploma Transcript
DU degree offer diploma Transcript
uapta
 
Oracle Database Desupported Features on 23ai (Part A)
Oracle Database Desupported Features on 23ai (Part A)Oracle Database Desupported Features on 23ai (Part A)
Oracle Database Desupported Features on 23ai (Part A)
Alireza Kamrani
 
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
satpalsheravatmumbai
 
Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...
Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...
Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...
sharonblush
 

Recently uploaded (20)

Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
 
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy DsouzaOpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
 
Biometric Question Bank 2021 - 1 Soln-1.pdf
Biometric Question Bank 2021 - 1 Soln-1.pdfBiometric Question Bank 2021 - 1 Soln-1.pdf
Biometric Question Bank 2021 - 1 Soln-1.pdf
 
Research proposal seminar ,Research Methodology
Research proposal seminar ,Research MethodologyResearch proposal seminar ,Research Methodology
Research proposal seminar ,Research Methodology
 
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
 
Willis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdfWillis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdf
 
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
Female Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service A...
 
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
 
History and Application of LLM Leveraging Big Data
History and Application of LLM Leveraging Big DataHistory and Application of LLM Leveraging Big Data
History and Application of LLM Leveraging Big Data
 
transgenders community data in india by govt
transgenders community data in india by govttransgenders community data in india by govt
transgenders community data in india by govt
 
Harendra Singh, AI Strategy and Consulting Portfolio
Harendra Singh, AI Strategy and Consulting PortfolioHarendra Singh, AI Strategy and Consulting Portfolio
Harendra Singh, AI Strategy and Consulting Portfolio
 
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
Beautiful Girls Call Pune 000XX00000 Provide Best And Top Girl Service And No...
 
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
Noida Girls Call Noida 9873940964 Unlimited Short Providing Girls Service Ava...
 
potential development of the A* search algorithm specifically
potential development of the A* search algorithm specificallypotential development of the A* search algorithm specifically
potential development of the A* search algorithm specifically
 
potential usefulness of multi-agent maze-solving in general
potential usefulness of multi-agent maze-solving in generalpotential usefulness of multi-agent maze-solving in general
potential usefulness of multi-agent maze-solving in general
 
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
 
DU degree offer diploma Transcript
DU degree offer diploma TranscriptDU degree offer diploma Transcript
DU degree offer diploma Transcript
 
Oracle Database Desupported Features on 23ai (Part A)
Oracle Database Desupported Features on 23ai (Part A)Oracle Database Desupported Features on 23ai (Part A)
Oracle Database Desupported Features on 23ai (Part A)
 
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
VIP Kanpur Girls Call Kanpur 0X0000000X Doorstep High-Profile Girl Service Ca...
 
Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...
Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...
Best Girls Call Navi Mumbai 9930245274 Provide Best And Top Girl Service And ...
 

Testing Spark and Scala

  • 1. Testing Spark and scala https://github.com/ganeshayadiyala/Scalatest-library-to-unit-test-spark/
  • 2. ● Ganesha Yadiyala ● Big data consultant at datamantra.io ● Consult in spark and scala ● ganeshayadiyala@gmail.com
  • 3. Agenda ● What is testing ● Different types of testing process ● Unit tests using scalatest ● Different styles in scalatest ● Using assertions ● Sharing fixtures ● Matchers ● Async Testing ● Testing of spark batch operation ● Unit testing streaming operation
  • 4. What is testing Software testing is a process of executing a program or application with the intent of finding the software bugs. It can also be stated as the process of validating and verifying that a software application, ● Meets the business and technical requirements that guided it’s design and development ● Works as expected
  • 5. Few of the types of tests ● Unit tests ● Integration tests ● Functional tests
  • 6. Unit tests ● Unit testing simply verifies that individual units of code (mostly functions) work as expected ● Assumes everything else works ● Tests one specific condition or flow. Advantages : ● Codes are more reusable. In order to make unit testing possible, codes need to be modular. This means that codes are easier to reuse. ● Debugging is easy. When a test fails, only the latest changes need to be debugged.
  • 7. Integration tests ● Tests the interoperability of multiple subsystem ● Includes real components, databases etc ● Tests the connectivity of the components ● Hard to test all the cases (combination of tests are more) ● Hard to localize the errors ( may break different reasons) ● Much slower than unit tests
  • 8. Functional tests ● Functional Testing is the type of testing done against the business requirements of application ● Use real components and real data
  • 9. Unit Test in scala
  • 10. Scalatest ● We use scalatest for unit tests in scala ● For every class in src/main/scala write a test class in src/test/scala ● Consists of suite (collection of test cases) ● You define test classes by composing Suite style and mixin traits. ● You can test both scala and java code ● offers deep integration with tools such as JUnit, TestNG, Ant, Maven, sbt, ScalaCheck, JMock, EasyMock, Mockito, ScalaMock, Selenium, Eclipse, NetBeans, and IntelliJ.
  • 11. Using the scalatest maven plugin We have to disable maven surefire plugin and enable scalatest plugin ● Specify <skipTests>true</skipTests> in maven surefire plugin ● Add the scalatest-maven plugin and set the goals to test
  • 12. Different styles in scalatest ● FunSuite ● FlatSpec ● FunSpec ● WordSpec ● FreeSpec ● PropSpec ● FeatureSpec
  • 13. FunSuite ● In a FunSuite, tests are function values. ● You denote tests with test and provide the name of the test as a string enclosed in parentheses, followed by the code of the test in curly braces Ex : com.ganesh.scalatest.specs.FunSuitTest.scala
  • 14. FlatSpec ● No nesting approach contrasts with the traits FunSpec and WordSpec. ● Uses behavior of clause Ex : com.ganesh.scalatest.specs.FlatSpecTest.scala
  • 15. FunSpec ● Tests are combined with text that specifies the behavior of the test. ● Uses describe clause Ex : com.ganesh.scalatest.specs.FunSpecTest.scala
  • 16. WordSpec ● your specification text is structured by placing words after strings ● Uses should and in clause Ex : com.ganesh.scalatest.specs.WordSpecTest.scala
  • 17. Using Assertions ScalaTest makes three assertions available by default in any style trait ● assert - for general assertion. ● assertResult - to differentiate expected from actual values. ● assertThrows - to ensure a bit of code throws an expected exception. Scalatest assertions are defined in trait Assertions. Assertions also provide some other API’s. Ex : com.ganesh.scalatest.features.AssertionsTest.scala
  • 18. Ignoring the test ● Scalatest allows to ignore the test. ● We can ignore the test if we want it to change it implementation and run later or if the test case is slow. ● We use ignore clause to ignore the test ● We use @Ignore annotation to ignore all the test in a suite. Ex : com.ganesh.scalatest.features.IgnoreTest.scala
  • 19. Sharing fixture A test fixture is composed of the objects and other artifacts, which tests use to do their work. When multiple tests needs to work with the same fixture, we can share the fixture between them. It will reduce the duplication of code.
  • 20. By calling get-fixture methods If you need to create the same mutable fixture objects in multiple tests we can use get-fixture method ● A get-fixture method returns a new instance of a needed fixture object each time it is called ● Not appropriate to use if we need to cleanup those objects Ex : com.ganesh.scalatest.fixtures.GetFixtureTest.scala
  • 21. By Instantiating fixture-context objects When different tests need different combinations of fixture objects, define the fixture objects as instance variables of fixture-context objects. ● In this approach we initialize a fixture object inside trait/class. ● We create a new instance of the fixture trait in the test we need them. ● We can even mix in these fixture traits we created. Ex : com.ganesh.scalatest.fixtures.FixtureContextTest.scala
  • 22. By using withFixture ● Allows cleaning up of fixtures at the end of the tests ● If we have no object to pass to the test case, then we can use withFixture(NoArgTest). ● If we have one or more objects to be passed to test case, then we need to use withFixture(OneArgTest). Ex : com.ganesh.scalatest.fixtures.WithFicture*.scala
  • 23. By using BeforeAndAfter ● Methods which we used till now for sharing fixtures are performed during the test. ● If exception occurs while creating this fixture then it’ll be reported as test failure. ● If we use BeforeAndAfter setup happens before the test execution starts, and cleanup happens once the test is completed ● So if any exception happens in the setup, it’ll abort the entire suit and no more tests are attempted. Ex : com.ganesh.scalatest.fixtures.BeforeAndAfterTest.scala
  • 24. Matchers ScalaTest provides a domain specific language (DSL) for expressing assertions in tests using the word should. Ex : com.ganesh.scalatest.features.MatchersTest.scala
  • 25. Asynchronous testing ● Given a Future returned by the code you are testing, you need not block until the Future completes before performing assertions against its value. ● We can instead map those assertions onto the Future and return the resulting Future[Assertion] to ScalaTest. ● This result is executed asynchronously. Ex : com.ganesh.scalatest.features.AsyncTest.scala
  • 26. Testing private methods ● If the method is private in a class we can test it using scalatest. ● We can use PrivateMethodTester trait to achieve this. ● We can use invokePrivate operator to call the private method Ex : com.ganesh.scalatest.features.PrivateMethodTest.scala
  • 27. Mocking Scalatest supports following mock libraries, ● ScalaMock ● EasyMock ● JMock ● Mockito Ex : com.ganesh.scalatest.mock.MockTest.scala
  • 29. Complexities ● Needs spark context for all the tests ● Testing operations such as map, flatmap and reduce. ● Testing streaming application (Dstream operations). ● Making sure that there is only one context for each test case.
  • 30. Setup ● Instead of creating contexts which are needed for each test suite, we create the trait which extends BeforeAndAfter, and all our suites will extend this trait. ● In that trait we try to initialize all the contexts in before method ● All the contexts will be destroyed in after method ● Extend this trait in all the test suites Ex : com.ganesh.scalatest.sparkbatch.EnvironmentInitializerSC.scala
  • 31. Spark Streaming test ● The full control over clock is needed to manually manage batches, slides and windows. ● Spark Streaming provides necessary abstraction over system clock, ManualClock class. ● But its private class, we cannot access it in our testcases ● So we use a wrapper class to use the ManualClock instance in our test case. Ex : com.ganesh.scalatest.sparkstreaming
  • 32. Summary ● We can select any of the styles provided by the scalatest, it just differs in how we write test but will have all the features. ● Make use of assertions and matchers provided by scalatest for better test cases. ● While testing spark we need to test the logic, so keep your code modular so that each logic can be tested individually. ● There is a external library called spark testing base which provides many functions to assert on dataframe level and it has traits which provides you the contexts needed for the test.