Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Jenkins World Tour 2015
London, UK 23-24 June
Footer
#jenkinsconf
Jenkins-based CI for heterogeneous
Hardware/Software projects
Oleg Nenashev
Jenkins Developer @ CloudB...
Footer
#jenkinsconf
> whoami
•  https://ru.linkedin.com/in/onenashev
•  Background: ~10 years of…
–  R&D: HW and Embedded ...
Footer
#jenkinsconf
> whoami --list jenkins
•  >6 years of Hudson/Jenkins experience
•  3 years of open-source contributio...
Footer
#jenkinsconf
Outline
•  CI in HW/Embedded world. Why Jenkins?
•  Integrating Jenkins with the common stuff
–  EDA t...
Footer
#jenkinsconf
Jenkins users from HW world
•  The most of HW companies have Jenkins instances
•  Examples from a 5min...
Footer
#jenkinsconf
Related talks / articles
•  Robert Martin, “Multi-Stage-CI with Jenkins in an Embedded
World” @ JUC201...
Footer
#jenkinsconf
INTRODUCTION
CI in HW/Embedded world
8
Footer
#jenkinsconf
What is a hardware product?
. . .
Not so obvious…
•  IP Libraries –
reusable blocks
•  Prototypes
•  D...
Footer
#jenkinsconf
HW Development Flow
10
Synthesis
Place & Route
RTL Design
Specification
= Compilation
= Linking
Design...
Footer
#jenkinsconfHW CI Example
11
Get sources
SCMs
Binary repositories
Build artifacts
Setup environment
“Build” = S&P&R...
Footer
#jenkinsconf
Embedded SW build process
12
Other
software
Embedded
Software
•  Different tools, but the approaches a...
Footer
#jenkinsconf
And what about an HW IP Product?
Hardware
part
External IP
Firmware /
Embedded
SW
External
SW
Developm...
Footer
#jenkinsconf
Welcome to CI hell!
•  Almost everything needs to be automated
•  Multiple technologies and tools
•  U...
Footer
#jenkinsconf
Why Jenkins?
•  It’s a framework => you can do everything
•  It’s open-source:
–  You can customize Je...
Footer
#jenkinsconf
AREA SPECIFICS
•  EDA Software
•  Computing grids
•  FPGAs
16
Footer
#jenkinsconf
EDA Software
•  Almost no OSS plugins
–  Exception - Cadence vManager Plugin
•  Specifics:
–  Dozens o...
Footer
#jenkinsconf
EDA Software. Possible solution
•  Store tools on shared folders
–  On Windows services kohsuke/winsw/...
Footer
#jenkinsconf
Custom Tools. Global Configuration
19
Can be stored remotely =>
automatic tool deployment
Node and glo...
Footer
#jenkinsconf
Custom Tools. Job Configuration
20
Allow
versioning
Automatic
installation
Footer
#jenkinsconf
Custom Tools. Usage
21
In build steps:
In build launch dialog:
Footer
#jenkinsconf
EDA Software. Publishers
•  Jenkins has publishers for many SW dev. tools
–  Unit test results (xUnit ...
Footer
#jenkinsconf
EDA Software. Licenses
•  FLEX License Manager is everywhere in EDA
•  Main problem: shared usage of l...
Footer
#jenkinsconf
Hardware testing
•  Simulators – specialized EDA tools
•  Hardware – FPGAs, ASIC prototypes
•  Combine...
Footer
#jenkinsconf
Cloudless skies
•  Simulation tests require…
•  Public/private clouds are not cost-effective
•  HW Com...
Footer
#jenkinsconf
Computing grids
Examples:
•  Sun Grid Engine (SGE, UGE)
•  Platform Load Sharing Facility (LSF)
•  Net...
Footer
#jenkinsconf
Computing Grids Usage
27
Manage tasks from
CLI (script steps)
“Transparent” –
launch Jenkins nodes
on ...
Footer
#jenkinsconf
Simulations are slow, so…
28
•  S - EZ1CUSB @ http://www.easyfpga.com
•  M - http://www.xilinx.com/uni...
Footer
#jenkinsconf
29
And XXL…
•  http://blogs.synopsys.com/
breakingthethreelaws/2014/04/
the-secret-ninja-fu-for-higher...
Footer
#jenkinsconf
Mapping CI to existing HW
30
Nodes @ Site A Nodes @ Site CNodes @ Site B
“We borrowed FPGAs
here, some...
Footer
#jenkinsconf
How it works
31
Power
When it hangs
“Deploy bit-files”
Blasting cables
Debugging cables,
firmware uplo...
Footer
#jenkinsconf
How do I implement jobs?
•  Throttling of builds
•  Shared functionality is decoupled to external proj...
Footer
#jenkinsconf
Throttling of FPGA jobs
•  Throttle Concurrent Builds Plugin
–  Jobs can be throttled by a category
– ...
Footer
#jenkinsconf
Throttling of FPGA jobs
Job configuration:
34
Footer
#jenkinsconf
FPGAs monitoring
•  Impossible to use an external system – shared resources
•  Solution in Jenkins:
1....
Footer
#jenkinsconf
ENTERPRISE FEATURES
Getting added value from CloudBees Jenkins Platform
36
CJE – CloudBees Jenkins Ent...
Footer
#jenkinsconf
Common features All common features are
applicable, out of the scope
See CloudBees Docs
Footer
#jenkinsconf
Templates!
•  OSS Plugins have a limited functionality:
–  Template Project, Inheritance Plugin, Job G...
Footer
#jenkinsconf
Easy failover implementation
•  Workflow allows to transparently re-run
unreliable stages on other nod...
Footer
#jenkinsconf
Shared slaves for rare hardware
•  Use-case: multiple Jenkins instances in a single org
•  Instances m...
Footer
#jenkinsconf
Durable tasks
•  Endurance tests may take weeks
•  Jenkins slaves may be disconnected, masters can
be ...
Footer
#jenkinsconf
Summary
•  Jenkins is not only about software
•  Jenkins can be used to run CI in
hardware projects
• ...
Footer
#jenkinsconf
My HW-related wish-list
•  Sun Grid Engine Cloud plugin
•  FLEX LM plugin
•  Flexible Windows Services...
Footer
#jenkinsconf
Q&A
Upcoming SlideShare
Loading in …5
×

of

JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 1 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 2

YouTube videos are no longer supported on SlideShare

View original on YouTube

JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 4 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 5 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 6 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 7 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 8 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 9 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 10 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 11 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 12 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 13 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 14 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 15 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 16 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 17 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 18 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 19 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 20 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 21 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 22 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 23 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 24 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 25 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 26 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 27 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 28 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 29 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 30 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 31 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 32 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 33 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 34 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 35 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 36 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 37 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 38 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 39 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 40 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 41 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 42 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 43 JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 44

YouTube videos are no longer supported on SlideShare

View original on YouTube

JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects Slide 46
Upcoming SlideShare
Power Up Your People: Emerging Tech and the Frictionless Employee Experience (webinar slides)
Next
Download to read offline and view in fullscreen.

2 Likes

Share

Download to read offline

JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects

Download to read offline

By Oleg Nenashev, CloudBees, Inc.

This talk will address Jenkins-based continuous integration (CI) in the area of embedded systems, which include both hardware and software components. An overview of common automation cases, challenges and their solutions based on Jenkins CI services will be presented. The specifics of Jenkins usage in the hardware area (available plugins and workarounds, environment and desired high availability features) will also be discussed. The session will cover several automation examples and case studies.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardware/Software Projects

  1. 1. Jenkins World Tour 2015 London, UK 23-24 June
  2. 2. Footer #jenkinsconf Jenkins-based CI for heterogeneous Hardware/Software projects Oleg Nenashev Jenkins Developer @ CloudBees 2
  3. 3. Footer #jenkinsconf > whoami •  https://ru.linkedin.com/in/onenashev •  Background: ~10 years of… –  R&D: HW and Embedded systems –  CI in HW, Embedded and EDA areas –  Department-wide CI frameworks •  Current: –  Jenkins CI contributor –  Jenkins Developer at CloudBees –  PhD student, EDA Lab @ SPBPU 3
  4. 4. Footer #jenkinsconf > whoami --list jenkins •  >6 years of Hudson/Jenkins experience •  3 years of open-source contribution to Jenkins •  Maintenance of large-scale installations •  I maintain about 10 open-source plugins •  Contributions to other Jenkins stuff and INFRA •  Ownership •  Job Restrictions •  Role Strategy (legacy) •  Custom Tools (legacy) •  Extra Tool Installers •  Cygwin Process Killer Tool belts Ownership-based security
  5. 5. Footer #jenkinsconf Outline •  CI in HW/Embedded world. Why Jenkins? •  Integrating Jenkins with the common stuff –  EDA tools, FPGA farms, Computing Grids •  Several examples and case studies •  Getting added value from CloudBees Jenkins Platform 5
  6. 6. Footer #jenkinsconf Jenkins users from HW world •  The most of HW companies have Jenkins instances •  Examples from a 5min search in public materials 6
  7. 7. Footer #jenkinsconf Related talks / articles •  Robert Martin, “Multi-Stage-CI with Jenkins in an Embedded World” @ JUC2014/Berlin •  Martin d’Anjou, “FPGA Continuous Integration with Jenkins” @SNUG2013/Canada •  Alan Fitch, “Continuous Integration for FPGA Design and Verification” @ Verification Futures, 2015 •  JL Gray, Gordon McGregor, “A 30 Minute Project Makeover Using Continuous Integration” @ DVCon, 2012 •  … •  Sarah Woodall, “The Famous Cows of Cambridge: A Non- Standard Use Case for Jenkins” @JUC2015/London
  8. 8. Footer #jenkinsconf INTRODUCTION CI in HW/Embedded world 8
  9. 9. Footer #jenkinsconf What is a hardware product? . . . Not so obvious… •  IP Libraries – reusable blocks •  Prototypes •  Development kits •  …
  10. 10. Footer #jenkinsconf HW Development Flow 10 Synthesis Place & Route RTL Design Specification = Compilation = Linking Design = Coding VHDL Verilog / SystemVerilog SystemC… Verification Packaging
  11. 11. Footer #jenkinsconfHW CI Example 11 Get sources SCMs Binary repositories Build artifacts Setup environment “Build” = S&P&R “Test” “Analyze” Publish reports Deploy artifacts Static analysis Formal verification Timing analysis Specialized tools Unit tests Integration tests EDA Tools Electronic Design Automation Test results Coverages System resources utilization Much parallel tasks in flows
  12. 12. Footer #jenkinsconf Embedded SW build process 12 Other software Embedded Software •  Different tools, but the approaches are similar •  Verification is a real challenge
  13. 13. Footer #jenkinsconf And what about an HW IP Product? Hardware part External IP Firmware / Embedded SW External SW Development tools / EDAs Demo SW Demo hardware Developer kits Yes, we need to automate everything “Main product”
  14. 14. Footer #jenkinsconf Welcome to CI hell! •  Almost everything needs to be automated •  Multiple technologies and tools •  Unstable hardware & networks •  Legacy stuff is everywhere •  HW engineers are not SW experts •  … •  Limited resources •  Limited OSS usage and contribution options
  15. 15. Footer #jenkinsconf Why Jenkins? •  It’s a framework => you can do everything •  It’s open-source: –  You can customize Jenkins core and plugins –  Plenty of available materials and examples •  “Stable” LTS versions •  Commercial support for self-hosted installations
  16. 16. Footer #jenkinsconf AREA SPECIFICS •  EDA Software •  Computing grids •  FPGAs 16
  17. 17. Footer #jenkinsconf EDA Software •  Almost no OSS plugins –  Exception - Cadence vManager Plugin •  Specifics: –  Dozens of tools in a common design flow –  Different platform requirements –  Multiple tool versions are required –  Big size of tools – up to 100 Gigabytes –  Limited licenses being shared with externals 17
  18. 18. Footer #jenkinsconf EDA Software. Possible solution •  Store tools on shared folders –  On Windows services kohsuke/winsw/pull/71 may be used –  Extra Tool Installers Plugin provide a stub installer •  Access tools: Labels + Custom Tools Plugin •  Idea of the plugin: –  Configurable installers –  Setup of a custom environment (PATH, env. vars) –  Built-in versioning based on Extended Choice Parameter 18
  19. 19. Footer #jenkinsconf Custom Tools. Global Configuration 19 Can be stored remotely => automatic tool deployment Node and global env. vars are supported
  20. 20. Footer #jenkinsconf Custom Tools. Job Configuration 20 Allow versioning Automatic installation
  21. 21. Footer #jenkinsconf Custom Tools. Usage 21 In build steps: In build launch dialog:
  22. 22. Footer #jenkinsconf EDA Software. Publishers •  Jenkins has publishers for many SW dev. tools –  Unit test results (xUnit et all.) –  Performance test results –  Coverages •  No solutions for hardware, but you can use converters •  Example for VCS coverages: 1.  Get an HTML or raw XML report 2.  Use XSLT to convert it to Cobertura format 3.  Publish data using a standard plugin 22
  23. 23. Footer #jenkinsconf EDA Software. Licenses •  FLEX License Manager is everywhere in EDA •  Main problem: shared usage of licenses –  No sporadic prioritization in FLEX => long waiting –  Dedicated licenses are not cost-effective –  Limited availability of tools => custom accounts •  Minimal solution – Throttling, custom “wait for a license” scripts •  Best solution - FLEX LM plugin, no OSS ones 23
  24. 24. Footer #jenkinsconf Hardware testing •  Simulators – specialized EDA tools •  Hardware – FPGAs, ASIC prototypes •  Combined solutions (HW Accelerators, etc.) 24
  25. 25. Footer #jenkinsconf Cloudless skies •  Simulation tests require… •  Public/private clouds are not cost-effective •  HW Companies mostly use computing grids 25 Thousands CPU hours Terabytes of TMP files and reports All RAM you have
  26. 26. Footer #jenkinsconf Computing grids Examples: •  Sun Grid Engine (SGE, UGE) •  Platform Load Sharing Facility (LSF) •  NetBatch @ Intel •  … 26 •  Datacenter-wide grids •  Sharing across departments “We run hello-world in 2 hours”
  27. 27. Footer #jenkinsconf Computing Grids Usage 27 Manage tasks from CLI (script steps) “Transparent” – launch Jenkins nodes on the grid •  Use-case: Legacy scripts •  Difficult management (job abort, status polling, etc.) •  Manual launch •  “Start by script on Master” •  LSF Cloud Plugin •  No FOSS plugins for other grids L “Launch task” build & workflow steps •  No FOSS plugins !
  28. 28. Footer #jenkinsconf Simulations are slow, so… 28 •  S - EZ1CUSB @ http://www.easyfpga.com •  M - http://www.xilinx.com/univ/xupv5-lx110t.htm •  L - HAPS-64, www.synopsys.com S - x00$ M – x000$ L –x0000$ FPGA accelerators & prototyping boards
  29. 29. Footer #jenkinsconf 29 And XXL… •  http://blogs.synopsys.com/ breakingthethreelaws/2014/04/ the-secret-ninja-fu-for-higher- performance-prototype- operation/ HAPS S96 Guess the price… Could CI have its own stuff? Your CPU prototype is here
  30. 30. Footer #jenkinsconf Mapping CI to existing HW 30 Nodes @ Site A Nodes @ Site CNodes @ Site B “We borrowed FPGAs here, somebody takes them randomly” “This stuff has Windows drivers only” “No infra team members here” “We run nodes on dev. laptops” Frankenboards “We built it from the garbage we had” Infrastructure after the “budget optimization” L L L L L
  31. 31. Footer #jenkinsconf How it works 31 Power When it hangs “Deploy bit-files” Blasting cables Debugging cables, firmware upload We don’t change them, huh? Jenkins node drivers tools peripherals Attached hardware runtime
  32. 32. Footer #jenkinsconf How do I implement jobs? •  Throttling of builds •  Shared functionality is decoupled to external project (Parameterized Trigger or Template Project) –  FPGA blasting –  Firmware Upload –  Run of a test case (binary upload) using a debugger provided by the toolchain (gdb, etc.) –  etc. •  Failover and restart on infra issues => Naginator 32
  33. 33. Footer #jenkinsconf Throttling of FPGA jobs •  Throttle Concurrent Builds Plugin –  Jobs can be throttled by a category –  Performance issues (esp. till 1.609.1), can be partially solved by Label Throttle plugin (CloudBees Jenkins Enterprise) –  Job developers must follow rules, no enforcement Global configuration: 33
  34. 34. Footer #jenkinsconf Throttling of FPGA jobs Job configuration: 34
  35. 35. Footer #jenkinsconf FPGAs monitoring •  Impossible to use an external system – shared resources •  Solution in Jenkins: 1.  Use periodic Multi-configuration Matrix Job for a monitoring 2.  Use Elastic Axis to generate label axes by an expression 3.  Run a spot-check on FPGA with blasting and a firmware test 4.  If it fails, disable the node using System Groovy Script 5.  Send notifications to INFRA admins 35
  36. 36. Footer #jenkinsconf ENTERPRISE FEATURES Getting added value from CloudBees Jenkins Platform 36 CJE – CloudBees Jenkins Enterprise CJOC – CloudBees Jenkins Operations Center
  37. 37. Footer #jenkinsconf Common features All common features are applicable, out of the scope See CloudBees Docs
  38. 38. Footer #jenkinsconf Templates! •  OSS Plugins have a limited functionality: –  Template Project, Inheritance Plugin, Job Generator, etc. •  Parameterized trigger causes UI overheads –  Multi-page configuration in UI: “Trigger with several parameters => wait => copy artifacts => analyze” •  Templates allow to save much time on the implementations •  Workflow + Scriptler Plugin are tasty as well 38
  39. 39. Footer #jenkinsconf Easy failover implementation •  Workflow allows to transparently re-run unreliable stages on other nodes. for (def board : boards) { try { // blasting … // run test … // test finished break; } catch (FPGAError error) { // try another board continue; } } System failure Node #1 Node #2. . . . . .
  40. 40. Footer #jenkinsconf Shared slaves for rare hardware •  Use-case: multiple Jenkins instances in a single org •  Instances may require a specific resource •  No cross-Jenkins throttling features •  Out-of-the-box throttling in CJOC shared slaves 40
  41. 41. Footer #jenkinsconf Durable tasks •  Endurance tests may take weeks •  Jenkins slaves may be disconnected, masters can be restarted for maintenance purposes •  No stable solutions in Jenkins OSS if you don’t use computing grids •  Durable tasks allow to overcome the issue in many cases –  Shell scripts are running independently –  Jenkins polls for a completion asynchronously –  Serious limitation: cross-site installations 41
  42. 42. Footer #jenkinsconf Summary •  Jenkins is not only about software •  Jenkins can be used to run CI in hardware projects •  You can use Jenkins even without domain-specific plugins •  CJE/CJOC features greatly help to decrease the maintenance cost 42
  43. 43. Footer #jenkinsconf My HW-related wish-list •  Sun Grid Engine Cloud plugin •  FLEX LM plugin •  Flexible Windows Services •  Plugins for common EDA tools and report formats –  Build steps are OK => just use CLI utilities –  Publishing requires much efforts –  Logs parsing would be a killer feature 43
  44. 44. Footer #jenkinsconf Q&A
  • anirupmukherjee

    Nov. 10, 2019
  • zahernourredine

    Dec. 12, 2018

By Oleg Nenashev, CloudBees, Inc. This talk will address Jenkins-based continuous integration (CI) in the area of embedded systems, which include both hardware and software components. An overview of common automation cases, challenges and their solutions based on Jenkins CI services will be presented. The specifics of Jenkins usage in the hardware area (available plugins and workarounds, environment and desired high availability features) will also be discussed. The session will cover several automation examples and case studies.

Views

Total views

1,585

On Slideshare

0

From embeds

0

Number of embeds

7

Actions

Downloads

27

Shares

0

Comments

0

Likes

2

×