Your SlideShare is downloading. ×
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Groovy DevOps in the Cloud for Devoxx UK 2014
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Groovy DevOps in the Cloud for Devoxx UK 2014

359

Published on

This talk focuses on a set of tools to automate the provisioning of virtual machines on Amazon EC2 using Groovy programming language and libraries. …

This talk focuses on a set of tools to automate the provisioning of virtual machines on Amazon EC2 using Groovy programming language and libraries.
We will explore how to leverage those to create an infrastructure for building, configuring and testing the provisioning of boxes in the cloud – elegant and groovy.

Published in: Software, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
359
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 01
  • 2. About me 02
  • 3. Andrey Adamovich Bio: Developer, coach, speaker, author Company: Aestas/IT (http://aestasit.com) E-mail: andrey@aestasit.com Linkedin: http://www.linkedin.com/in/andreyadamovich Twitter: @aestasit • • • • • 03
  • 4. What's this presentation about? Our take on: DevOps Intrastructure Provisioning Continuous Integration Continuous Delivery • • • • 04
  • 5. Technologies Groovy - http://groovy.codehaus.org Gradle - http://gradle.org Jenkins - http://jenkins-ci.org Puppet - http://puppetlabs.com AWS - http://aws.amazon.com • • • • • 05
  • 6. Developers + Operations = ?06
  • 7. Silos 07
  • 8. Conflicts 08
  • 9. Risk 09
  • 10. Agile 10
  • 11. What is DevOps? 11
  • 12. C.A.M.S. Culture : People over processes and tools. Software is made by and for people. Automation : Automation is essential for DevOps to gain quick feedback. Measurement : DevOps finds a specific path to measurement. Quality and shared (or at least aligned) incentives are critical. Sharing : Creates a culture where people share ideas, processes, and tools. • • • • 12
  • 13. It's not about tools! 13
  • 14. It's about culture and process!14
  • 15. But without tools... 15
  • 16. ...it's definitely harder!16
  • 17. DevOps imply automation! 17
  • 18. DevOps imply structure! 18
  • 19. Infrastructure as code 19
  • 20. Infrastructure as code Automate the provisioning and maintenance of servers: Build from source control Utilize existing tools Ensure testability • • • 20
  • 21. Configuration propagation 21
  • 22. Configuration propagation 22
  • 23. Changes Imagine uploading *.class files and repackaging JAR directly on production servers when you have an urgent code change. 23
  • 24. Deployment is automatic! 24
  • 25. And, so, should be... 25
  • 26. infrastructure configuration changes!26
  • 27. No manual changes! 27
  • 28. Building an automation toolkit Automation is key We are JVM hackers Fragmented ecosystem • • • 28
  • 29. Initial toolset Gradle Groovy Ant Python/WLST Shell scripts • • • • • 29
  • 30. Required tooling Infrastructure connectivity Infrastructure provisioning Infrastructure virtualization Infrastructure testing • • • • 30
  • 31. First Blood 31
  • 32. Ant + Gradle ant.taskdef( name: 'scp', classname: 'o.a.t.a.t.o.ssh.Scp', classpath: configurations.secureShell.asPath) ant.taskdef( name: 'sshexec', classname: 'o.a.t.a.t.o.ssh.SSHExec', classpath: configurations.secureShell.asPath) 01. 02. 03. 04. 05.06. 07. 08. 09. 32
  • 33. Simple call ant.sshexec( host: host, username: user, password: password, command: command, trust: 'true', failonerror: failOnError) 01. 02. 03. 04. 05. 06. 07. 33
  • 34. Next step: wrapper function def ssh(String command, Properties props, boolean failOnError = false, String suCommandQuoteChar = "'", String outputProperty = null) { ... } 01. 02. 03. 04. 05. 06. 07. 34
  • 35. Next step: wrapper function def scp(String file, String remoteDir, Properties props) { ... } 01. 02. 03. 04. 05. 35
  • 36. Task example I task installFonts << { forAllServers { props -> ssh('yes | yum install *font*', props) } } 01. 02. 03. 04. 05. 36
  • 37. Task example II task uninstallNginx << { forAllServers { props -> ssh('/etc/init.d/nginx stop', props) ssh('yes | yum remove nginx', props, true) ssh('rm -rf /etc/yum.repos.d/nginx.repo', props) ssh('rm -rf /var/log/nginx', props) ssh('rm -rf /etc/nginx /var/nginx', props) } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 37
  • 38. Drawbacks New connection each time Excplicit repeating parameters Complex scripts are hard to maintain Tasks are not idempotent • • • • 38
  • 39. Sshoogr 39
  • 40. Sshoogr features Groovy-based SSH DSL for: Remote command execution File uploading/downloading Tunneling • • • 40
  • 41. Why Groovy? Groovy is perfect choice for scripting Gradle build scripts are Groovy Very mature, concise syntax Extremely easy to produce DSL We wrote a book about it! • • • • • 41
  • 42. Shameless plug 42
  • 43. Sshoogr usage (import) @Grab( group='com.aestasit.infrastructure.sshoogr', module='sshoogr', version='0.9.16') import static com.aestasit.ssh.DefaultSsh.* 01. 02. 03. 04. 05. 43
  • 44. Sshoogr usage (defaults) defaultUser = 'root' defaultKeyFile = new File('secret.pem') execOptions { verbose = true showCommand = true } 01. 02. 03. 04. 05. 06. 44
  • 45. Sshoogr usage (connection) remoteSession { url = 'user2:654321@localhost:2222' exec 'rm -rf /tmp/*' exec 'touch /var/lock/my.pid' remoteFile('/var/my.conf').text = "enabled=true" } 01. 02. 03. 04. 05. 06. 45
  • 46. Sshoogr usage (multi-line content) remoteFile('/etc/yum.repos.d/puppet.repo').text = ''' [puppet] name=Puppet Labs Packages baseurl=http://yum.puppetlabs.com/el/ enabled=0 gpgcheck=0 ''' 01. 02. 03. 04. 05. 06. 07. 46
  • 47. Sshoogr usage (file copying) remoteSession { scp { from { localDir "$buildDir/application" } into { remoteDir '/var/bea/domain/application' } } } 01. 02. 03. 04. 05. 06. 47
  • 48. Sshoogr usage (command result) def result = exec(command: '/usr/bin/mycmd', failOnError: false, showOutput: false) if (result.exitStatus == 1) { result.output.eachLine { line -> if (line.contains('WARNING')) { throw new RuntimeException("Warning!!!") } } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 48
  • 49. Sshoogr usage (shortcuts) if (ok('/usr/bin/mycmd')) { ... } if (fail('/usr/bin/othercmd')) { ... } 01. 02. 03. 04. 05. 06. 49
  • 50. Sshoogr usage (tunnels) tunnel('1.2.3.4', 8080) { int localPort -> def url = "http://localhost:${localPort}/flushCache" def result = new URL(url).text if (result == 'OK') { println "Cache is flushed!" } else { throw new RuntimeException(result) } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 50
  • 51. Sshoogr usage (prefix/suffix) prefix('sudo ') { exec 'rm -rf /var/log/abc.log' exec 'service abc restart' } suffix(' >> output.log') { exec 'yum -y install nginx' exec 'yum -y install mc' exec 'yum -y install links' } 01. 02. 03. 04. 05. 06. 07. 08. 09. 51
  • 52. Still problems Complex scripts are still not easy to maintain Scripts are usually not idempotent • • 52
  • 53. Puppet 53
  • 54. Why Puppet? More mature than competition Large community Readable DSL Good acceptance from DEVs and OPs No need to learn Ruby ;) • • • • • 54
  • 55. Puppet example 55
  • 56. Puppet provisioning 56
  • 57. Puppet provisioning 57
  • 58. Puppet provisioning 58
  • 59. Puppet provisioning 59
  • 60. Puppet state management 60
  • 61. Puppet state management 61
  • 62. Puppet state management 62
  • 63. Puppet modules 63
  • 64. Puppet modules 64
  • 65. Puppet modules 65
  • 66. Sshoogr + Gradle + Puppet66
  • 67. Upload modules task uploadModules << { remoteSession { exec 'rm -rf /tmp/repo.zip' scp { from { localFile "${buildDir}/repo.zip" } into { remoteDir "/root" } } ... 01. 02. 03. 04. 05. 06. 07. 08. 67
  • 68. Upload modules ... exec 'rm -rf /etc/puppet/modules' exec 'unzip /tmp/repo.zip -d /etc/puppet/modules' } } 01. 02. 03. 04. 05. 68
  • 69. Apply manifests task puppetApply(dependsOn: uploadModules) << { remoteSession { scp { from { localFile "${buildDir}/setup.pp" } into { remoteDir "/tmp" } } exec 'puppet apply /tmp/setup.pp' } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 69
  • 70. What we solved? Separated infrastructure state description and operations tasks Scripts became more maintainable and idempotent • • 70
  • 71. In the meanwhile... We started developing complex/generic Puppet modules Modules need proper testing ...on different platforms • • • 71
  • 72. Do you test, right? How to test this stuff? How to reuse a JUnit approach to testing? We wanted things to be SIMPLE! • • • 72
  • 73. PUnit 73
  • 74. PUnit Simple testing tool for verifying remote server state Uses Sshoogr and JUnit Reuse reporting features of JUnit As simple as ... • • • • 74
  • 75. PUnit example (derby) class DerbyInstallTest extends BasePuppetIntegrationTest { @Before void installDerby() { apply("include derby") } ... } 01. 02. 03. 04. 05. 06. 07. 08. 75
  • 76. PUnit example (derby) @Test void ensureDerbyRunning() { command('service derby status > derbystatus.log') assertTrue fileText("/root/derbystatus.log") .contains('Derby') assertTrue fileText("/root/derbystatus.log") .contains('is running.') } 01. 02. 03. 04. 05. 06. 07. 08. 76
  • 77. PUnit example (derby) @Test void ensureCanConnect() { Thread.sleep(10000) uploadScript() command('/opt/derby/db-derby-10.9.1.0-bin/bin/ij ' + 'testDataScript.sql > derbytest.log') ... 01. 02. 03. 04. 05. 06. 07. 77
  • 78. PUnit example (derby) ... // Check if the log of the insert // operation contains the word ERROR. assertFalse( "The script should return at least one error", fileText("/root/derbytest.log") .contains('ERROR') ) ... 01. 02. 03. 04. 05. 06. 07. 08. 09. 78
  • 79. PUnit example (derby) ... // Check on data that was inserted into a table. assertTrue( "The log should contain a SELECT result", fileText("/root/derbytest.log") .contains('Grand Ave.') ) } 01. 02. 03. 04. 05. 06. 07. 08. 79
  • 80. PUnit example (jenkins) session { tunnel ('127.0.0.1', 8080) { int localPort -> def driver = new HtmlUnitDriver(false) driver.manage() .timeouts() .pageLoadTimeout(300, TimeUnit.SECONDS) .implicitlyWait(30, TimeUnit.SECONDS) driver.get("http://127.0.0.1:${localPort}/login") ... 01. 02. 03. 04. 05. 06. 07. 08. 09. 10. 80
  • 81. PUnit example (jenkins) ... def input = driver.findElement(By.name('j_username')) input.sendKeys('john') input = driver.findElement(By.name('j_password')) input.sendKeys('123456') input.submit() ... 01. 02. 03. 04. 05. 06. 07. 81
  • 82. PUnit example (jenkins) ... def wait = new WebDriverWait(driver, 30) wait.until ExpectedConditions. presenceOfElementLocated (By.linkText('John Doe')) ... } } 01. 02. 03. 04. 05. 06. 07. 82
  • 83. PUnit example (svn) session { tunnel ('127.0.0.1', 80) { int localPort -> // Initilize repository connection data. DAVRepositoryFactory.setup() def url = SVNURL.create('http', null, '127.0.0.1', localPort, 'repos/cafebabe', true) def repository = SVNRepositoryFactory.create(url) println "Verifying SVN repository at ${url}" ... 01. 02. 03. 04. 05. 06. 07. 08. 09. 83
  • 84. PUnit example (svn) ... // Setup credentials. def authManager = SVNWCUtil. createDefaultAuthenticationManager('joe', '123456') repository.setAuthenticationManager(authManager) // Verify repository is at revision 0. assertEquals 0, repository.getLatestRevision() ... 01. 02. 03. 04. 05. 06. 07. 08. 09. 84
  • 85. PUnit example (svn) ... // Commit first revision. ISVNEditor editor = repository. getCommitEditor("Initial commit.", null) editor.with { openRoot(-1) addFile('dummy.txt', null, -1) applyTextDelta('dummy.txt', null) def deltaGenerator = new SVNDeltaGenerator() 01. 02. 03. 04. 05. 06. 07. 08. 09. 85
  • 86. PUnit example (svn) ... def checksum = deltaGenerator.sendDelta('dummy.txt', new ByteArrayInputStream("data".getBytes()), editor, true) closeFile('dummy.txt', checksum) def commitInfo = closeEdit() println commitInfo } ... 01. 02. 03. 04. 05. 06. 07. 08. 09. 86
  • 87. PUnit example (svn) ... // Verify repository is at revision 1 now. assertEquals 1, repository.getLatestRevision() } } 01. 02. 03. 04. 05. 87
  • 88. Continuous integration 88
  • 89. Why Jenkins? De-facto standard Stable There is a plugin for that! • • • 89
  • 90. Jenkins build 90
  • 91. Next problem? 91
  • 92. Scalability How do we test on different OS? How do we run parallel tests on multiple architectures? How do we avoid selling our houses? • • • 92
  • 93. Amazon Web Services 93
  • 94. Elastic Compute Cloud Mature Great API Virtual hardware variety OS variety • • • • 94
  • 95. Gramazon 95
  • 96. Gramazon Groovy-based API for interacting with EC2 Integration with Gradle • • 96
  • 97. Gramazon example I task startInstance(type: StartInstance) { keyName 'cloud-do' securityGroup 'cloud-do' instanceName 'gramazon/cloud-do' stateFileName 'cloud-do.json' ami 'ami-6f07e418' instanceType 't1.micro' waitForStart true } 01. 02. 03. 04. 05. 06. 07. 08. 09. 97
  • 98. Gramazon example II task terminateInstance(type: TerminateInstance) { stateFileName 'cloud-do.json' } 01. 02. 03. 98
  • 99. The flow Start instance(s) Upload manifests Run tests Generate report Terminate instance(s) 1. 2. 3. 4. 5. 99
  • 100. Next issue? 100
  • 101. Imgr 101
  • 102. Imgr A tool for building images Inspired by Packer • • 102
  • 103. Supports Shell Puppet • • 103
  • 104. Configuration example 104
  • 105. Summary 105
  • 106. Images, manifests, tasks 106
  • 107. The big picture 107
  • 108. Aetomation 108
  • 109. Conclusions Reuse your existing Java knowledge ...to build a bridge between DEVs and OPs Reuse development best practices for OPs Don't be afraid to try new technologies Automate! • • • • • 109
  • 110. Next steps? Create more documentation and examples Add more DSL convience methods Extend integration with Gradle Add Windows connectivity/scripting support Define richer model for EC2 and potentially other clouds Extend support for other provisioning tools • • • • • • 110
  • 111. Reading material 111
  • 112. The Phoenix Project 112
  • 113. Continuous Delivery 113
  • 114. Release It 114
  • 115. Programming Amazon EC2 115
  • 116. Gradle in Action 116
  • 117. Groovy 2 Cookbook 117
  • 118. Technologies to follow Vagrant - http://www.vagrantup.com/ Docker - https://www.docker.io/ Packer - http://www.packer.io/ Qemu - http://wiki.qemu.org/ jclouds - http://jclouds.apache.org/ Cloudbees - http://www.cloudbees.com/ • • • • • • 118
  • 119. One more thing... 119
  • 120. It's all Open Source! 120
  • 121. Source code Sshoogr: https://github.com/aestasit/sshoogr Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle PUnit: https://github.com/aestasit/puppet-unit Gramazon: https://github.com/aestasit/gramazon Imgr: https://github.com/aestasit/imgr • • • • • 121
  • 122. Seeking contributors! 122
  • 123. Questions? 123
  • 124. Thank you! 124

×