Chef Cookbook Testing and Continuous Integration

  • 12,012 views
Uploaded on

Delivered at Chef-Boston meetup on June 13, 2013.

Delivered at Chef-Boston meetup on June 13, 2013.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
12,012
On Slideshare
0
From Embeds
0
Number of Embeds
13

Actions

Shares
Downloads
118
Comments
5
Likes
22

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Chef Cookbook Testing andContinuous IntegrationJulian C. DunnSenior Consultant, Opscode, Inc.<jdunn@opscode.com>Friday, June 14, 13
  • 2. Why Write Cookbook Tests?• Catch development errors• Catch regressions• Code to requirements (TDD, BDD)• Smaller batch sizes == lower blast radius• Continuous delivery of cookbooks!• Build confidence that your infrastructure isworking correctlyFriday, June 14, 13
  • 3. Software DevelopmentLifecycle forSysadminsFriday, June 14, 13
  • 4. Devops Is A Two-Way Street• It’s great whendevelopers careabout• uptime!• scaling!• deployment!• Put them on call!etc. etc. etc.Friday, June 14, 13
  • 5. Devops Is A Two-Way Street• Systems Administratorsalso have as much or more tolearn from developers as well!Friday, June 14, 13
  • 6. SourceCodeCompilerArtifactTestTypical Development Workflow• Typical Software Development Workflow:• Write Source• Compile Source• Deploy Artifacts• Write Tests• Run TestsFriday, June 14, 13
  • 7. SourceCodeCompilerArtifactTestTypical Development Workflow• Source Code• The recipe for a computer program• Edited directly• Managed via Source Control SoftwareFriday, June 14, 13
  • 8. SourceCodeCompilerArtifactTestTypical Development Workflow• Compiler• Takes source code and converts it intoexecutable programsFriday, June 14, 13
  • 9. SourceCodeCompilerArtifactTestTypical Development Workflow• Artifact• Artifacts are executable programscreated by compilers.• Compiled artifacts cannot be editeddirectly. Source code must be changedand re-compiled to produce a new buildartifact.Friday, June 14, 13
  • 10. SourceCodeCompilerArtifactTestTypical Development Workflow• Testing• Write tests to verify that code works asintended• Run tests at different stages of the codelifecycle to ensure correctnessFriday, June 14, 13
  • 11. SourceCodeCompilerArtifactTestTypical Development Workflow• When developing software, most timeisn’t actually spent coding• 10-second changes to source code cantake minutes to vet• Compiling code• Deploying code• Writing & Running testsFriday, June 14, 13
  • 12. Friday, June 14, 13
  • 13. SourceCodeCompilerArtifactTestTypical Development WorkflowFun!Boring!• Things that are fun:• Designing programs!• Writing source code!• Things that are boring:• Compiling code• Deploying artifacts• Running testsFriday, June 14, 13
  • 14. Look familiar to any Chefs?Friday, June 14, 13
  • 15. Worksta(on:+Knife+cookbook+create+Worksta(on:+Edit+cookbook+Worksta(on:+Knife+cookbook+upload+Provision+target+Bootstrap+target+Worksta(on:+Edit+target+run+list+ssh+target+Target:+Run+chef>client+Traditional Cookbook Development: First Chef RunFriday, June 14, 13
  • 16. Worksta(on:+edit+cookbook+Worksta(on:+knife+cookbook+upload+Target:+Run+chef9client+Traditional Cookbook Development: Subsequent RunsFriday, June 14, 13
  • 17. SourceCodeCompilerAr/factTestBoooooooring• Too much time doing“paperwork”• vi recipes/something.rb• knife cookbook upload• sudo pkill -USR1 chef-client• #%$#%$ somethingbroke, let me do that allagain• Not enough time doing funstuff!• Writing recipesLess Fun!More Boring!XFriday, June 14, 13
  • 18. SourceCodeCompilerAr/factTestBoooooooring• Too much time doing“paperwork”• vi recipes/something.rb• knife cookbook upload• sudo pkill -USR1 chef-client• #%$#%$ somethingbroke, let me do that allagain• Not enough time doing funstuff!• Writing recipesLess Fun!More Boring!XTHISSUCKS!Friday, June 14, 13
  • 19. Chefs on a PlaneWhat if...Friday, June 14, 13
  • 20. Chefs on a PlaneWhat if...Worksta(on:+Knife+cookbook+create+Worksta(on:+Edit+cookbook+Worksta(on:+Knife+cookbook+upload+Provision+target+Bootstrap+target+Worksta(on:+Edit+target+run+list+ssh+target+Target:+Run+chef>client+Target:(Run(chef/client(Worksta6on:(edit(cookbook(Worksta6on:(knife(cookbook(upload(...we couldautomate allof this...Friday, June 14, 13
  • 21. Chefs on a PlaneWhat if...Worksta(on:+Knife+cookbook+create+Worksta(on:+Edit+cookbook+Worksta(on:+Knife+cookbook+upload+Provision+target+Bootstrap+target+Worksta(on:+Edit+target+run+list+ssh+target+Target:+Run+chef>client+Target:(Run(chef/client(Worksta6on:(edit(cookbook(Worksta6on:(knife(cookbook(upload(...we couldautomate allof this......to run entirely on this...Friday, June 14, 13
  • 22. Chefs on a PlaneWhat if......even whileaboard this?Worksta(on:+Knife+cookbook+create+Worksta(on:+Edit+cookbook+Worksta(on:+Knife+cookbook+upload+Provision+target+Bootstrap+target+Worksta(on:+Edit+target+run+list+ssh+target+Target:+Run+chef>client+Target:(Run(chef/client(Worksta6on:(edit(cookbook(Worksta6on:(knife(cookbook(upload(...we couldautomate allof this......to run entirely on this...Friday, June 14, 13
  • 23. SourceCodeCompilerAr/factTestDeveloping for Chef: Rapid Iteration• Less time waiting aroundfor cookbook deploys andChef runs• More frequent testing• Better code• Business needs metmore quicklyMore Fun!Less Boring!XFriday, June 14, 13
  • 24. Code Can (Help)Friday, June 14, 13
  • 25. The ToolchainFriday, June 14, 13
  • 26. The Toolchain• You’re becoming a developer!• Reasonably powerful computer• Good editorFriday, June 14, 13
  • 27. Have a Good Computer• Virtualization is used to runacceptance tests• Running on a real virtualized“node”• Lots of memory (4GB+, 8GBrecommended)• Fast disk (SSD)• Good processor (Intel i5+)• Modern OSXFriday, June 14, 13
  • 28. Tools• Chef (obviously)• Virtualization provider• Virtualization automation(Vagrant)• Cookbook dependencymanager (Berkshelf,Librarian)• Unit and acceptance testingframeworksFriday, June 14, 13
  • 29. Software TestingTerminology and ChefFriday, June 14, 13
  • 30. Different Types of Testing• There are formalSoftware Engineeringdefinitions for testingFriday, June 14, 13
  • 31. Different Types of Testing• There are formalSoftware Engineeringdefinitions for testing• Unit testFriday, June 14, 13
  • 32. Different Types of Testing• There are formalSoftware Engineeringdefinitions for testing• Unit test• Integration testFriday, June 14, 13
  • 33. Different Types of Testing• There are formalSoftware Engineeringdefinitions for testing• Unit test• Integration test• Acceptance testFriday, June 14, 13
  • 34. Different Types of Testing• There are formalSoftware Engineeringdefinitions for testing• Unit test• Integration test• Acceptance test• An easy explanation forSAs is ...Friday, June 14, 13
  • 35. • Unit Test: Signal InputWhat and When To TestFlickr user: Rain RabbitFriday, June 14, 13
  • 36. • Unit Test: Signal Input• Did I send Chef the correctcommand?What and When To TestFlickr user: Rain RabbitFriday, June 14, 13
  • 37. • Unit Test: Signal Input• Did I send Chef the correctcommand?• Signal ProcessingWhat and When To TestFlickr user: Rain RabbitFriday, June 14, 13
  • 38. • Unit Test: Signal Input• Did I send Chef the correctcommand?• Signal Processing• Did Chef interpret my commandcorrectly?What and When To TestFlickr user: Rain RabbitFriday, June 14, 13
  • 39. • Unit Test: Signal Input• Did I send Chef the correctcommand?• Signal Processing• Did Chef interpret my commandcorrectly?• Acceptance Test: Signal OutputWhat and When To TestFlickr user: Rain RabbitFriday, June 14, 13
  • 40. • Unit Test: Signal Input• Did I send Chef the correctcommand?• Signal Processing• Did Chef interpret my commandcorrectly?• Acceptance Test: Signal Output• Did my expressed intent, executedby Chef, achieve the desiredresult?What and When To TestFlickr user: Rain RabbitFriday, June 14, 13
  • 41. Chef Testing ToolsFriday, June 14, 13
  • 42. Chef Testing Tools• Signal Input• ChefSpecFriday, June 14, 13
  • 43. Chef Testing Tools• Signal Input• ChefSpec• Signal out• Chef Minitests & Minitest HandlerFriday, June 14, 13
  • 44. Chef Testing Tools• Signal Input• ChefSpec• Signal out• Chef Minitests & Minitest Handler• Signal processing• Who cares; that’s what Opscode’s own tests forChef are for!Friday, June 14, 13
  • 45. General Philosophy for Chef TestingFriday, June 14, 13
  • 46. General Philosophy for Chef Testing• Unit tests• ChefSpec assertion for every resource of interest• Each recipe should get its own test fileFriday, June 14, 13
  • 47. General Philosophy for Chef Testing• Unit tests• ChefSpec assertion for every resource of interest• Each recipe should get its own test file• Acceptance tests• Anything that you couldn’t test in a unit testFriday, June 14, 13
  • 48. General Philosophy for Chef Testing• Unit tests• ChefSpec assertion for every resource of interest• Each recipe should get its own test file• Acceptance tests• Anything that you couldn’t test in a unit test• Don’t repeat yourself!Friday, June 14, 13
  • 49. Unit Testing: ChefSpecFriday, June 14, 13
  • 50. ChefSpec Overview• Built on top of RSpec• Converge a Chef run in memory• Overrides all providers to take no action• Make assertions about the in-memory Chef run• Mock Ohai data (automatic attributes) with FauxhaiFriday, June 14, 13
  • 51. Example Recipe Part Oneinclude_recipe “java”user node[sauceproxy][server][user] do  comment "SauceLabs Proxy User"  system true  action :createenddirectory node[sauceproxy][server][install_dir] do  owner node[sauceproxy][server][user]  mode 00755  action :createend# Cant assume we have unzippackage "unzip" do  action :installendexecute "unzip-saucelabs-proxy" do  cwd node[sauceproxy][server][install_dir]  command "unzip -o #{Chef::Config[:file_cache_path]}/#{node[sauceproxy][server][zipfile]}"  action :nothing  notifies :restart, "service[sauceproxy]"endhttps://github.com/juliandunn/sauceproxy/blob/master/recipes/server.rbFriday, June 14, 13
  • 52. Example Recipe Part Tworemote_file "#{Chef::Config[:file_cache_path]}/#{node[sauceproxy][server][zipfile]}" dosource "#{node[sauceproxy][server][download_url]}/#{node[sauceproxy][server][zipfile]}"  action :create_if_missing  notifies :run, "execute[unzip-saucelabs-proxy]", :immediatelyendtemplate "/etc/init.d/sauceproxy" do  source "sauceproxy.init.erb"  mode 00755  owner "root"  group "root"  action :createendtemplate "/etc/sysconfig/sauceproxy" do  source "sauceproxy.sysconfig.erb"  mode 00644  owner "root"  group "root"  action :createendservice "sauceproxy" do  supports :restart => true  action [:enable, :start]endFriday, June 14, 13
  • 53. Example ChefSpec Test: Setupdescribe sauceproxy::server dolet (:chef_run) dorunner = ChefSpec::ChefRunner.new(platform: centos,version: 6.3)runner.node.set[sauceproxy][server][user] = fakerunner.node.set[sauceproxy][server][install_dir] =/tmp/fakerunner.node.set[sauceproxy][server][version] = 3.14159runner.converge(sauceproxy::server)endFriday, June 14, 13
  • 54. Example ChefSpec Test: Expectationsit should create a sauceproxy directory with the rightownership doexpect(chef_run).to create_directory(/tmp/fake)expect(chef_run.directory(/tmp/fake)).tobe_owned_by(fake)endit should start a service called sauceproxy doexpect(chef_run).to start_service(sauceproxy)expect(chef_run).toset_service_to_start_on_boot(sauceproxy)endendFriday, June 14, 13
  • 55. ChefSpec In ActionFriday, June 14, 13
  • 56. ChefSpec: Expectations• Many expectations (assertions) possible• https://github.com/acrmp/chefspec#making-assertions• Don’t (just) restate static resources• Mock out data• Static resource assertions: good for regressions• Whoops, I deleted the service resourceFriday, June 14, 13
  • 57. Acceptance Testing:MiniTestFriday, June 14, 13
  • 58. Chef Client RunFriday, June 14, 13
  • 59. Chef Client RunFriday, June 14, 13
  • 60. Acceptance Tests as Report Handler• Acceptance testing as a Chef Report Handler• Many tools (serverspec, bats, minitest)• I’ll demo Chef-MiniTest-Handler; most widely usedFriday, June 14, 13
  • 61. MiniTest Handler Cookbook• http://community.opscode.com/cookbooks/minitest-handler• Installs Minitest Gems• Installs Chef Minitest Gems• Installs the Chef-Minitest-Handler NotificationHandler for Chef-Client• Places test files from cookbooks on the target nodeas part of a Chef-client runFriday, June 14, 13
  • 62. MiniTest Files• Each recipe gets an individual test file.• recipes/server.rb• files/default/test/server_test.rb• Tests for each recipe are automatically loaded by thehandlerFriday, June 14, 13
  • 63. Example MiniTestrequire minitest/specdescribe_recipe sauceproxy::server doit runs as a daemon doservice(sauceproxy).must_be_runningendendhttp://tinyurl.com/minitest-examplesfor many more example testsFriday, June 14, 13
  • 64. Make this go on awith virtualization!Friday, June 14, 13
  • 65. Using Vagrant & Berkshelf to Iterate• Vagrant is virtualization middleware• Driven from Vagrantfile• Defines VM images, customization parameters,provisioners (Chef in our case)• Berkshelf is a cookbook dependency manager• Get dependent cookbooks from community API• Feed them to Vagrant Chef provisionerFriday, June 14, 13
  • 66. Example VagrantfileVagrant.configure("2") do |config|config.vm.hostname = "sauceproxy-berkshelf"config.vm.box = "opscode-centos-6.4"config.vm.box_url = "https://opscode-vm.s3.amazonaws.com/vagrant/opscode_centos-6.4_chef-11.4.4.box"config.ssh.max_tries = 40config.ssh.timeout = 120config.vm.provision :chef_solo do |chef|chef.json = {}chef.run_list = ["recipe[minitest-handler]","recipe[sauceproxy::server]"]endendFriday, June 14, 13
  • 67. % vagrant upBringing Up Vagrantborkbork:~/Dropbox/devel/github/juliandunn/sauceproxy (master)$ vagrant upBringing machine default up with virtualbox provider...[default] Importing base box centos-6.4...[default] Matching MAC address for NAT networking...[default] Setting the name of the VM...[default] Clearing any previously set forwarded ports...[Berkshelf] Updating Vagrants berkshelf: /Users/juliandunn/.berkshelf/vagrant/berkshelf-20130611-20810-rt7k01[Berkshelf] Using sauceproxy (0.1.8) at path: /Users/juliandunn/Dropbox/devel/github/juliandunn/sauceproxy[Berkshelf] Using minitest-handler (0.2.1)[Berkshelf] Using java (1.11.4)[Berkshelf] Using windows (1.8.10)[Berkshelf] Using chef_handler (1.1.4)[default] Creating shared folders metadata...[default] Clearing any previously set network interfaces...[default] Preparing network interfaces based on configuration...[default] Forwarding ports...[default] -- 22 => 2222 (adapter 1)[default] Booting VM...[default] Waiting for VM to boot. This can take a few minutes.Friday, June 14, 13
  • 68. % vagrant upBringing Up Vagrantborkbork:~/Dropbox/devel/github/juliandunn/sauceproxy (master)$ vagrant upBringing machine default up with virtualbox provider...[default] Importing base box centos-6.4...[default] Matching MAC address for NAT networking...[default] Setting the name of the VM...[default] Clearing any previously set forwarded ports...[Berkshelf] Updating Vagrants berkshelf: /Users/juliandunn/.berkshelf/vagrant/berkshelf-20130611-20810-rt7k01[Berkshelf] Using sauceproxy (0.1.8) at path: /Users/juliandunn/Dropbox/devel/github/juliandunn/sauceproxy[Berkshelf] Using minitest-handler (0.2.1)[Berkshelf] Using java (1.11.4)[Berkshelf] Using windows (1.8.10)[Berkshelf] Using chef_handler (1.1.4)[default] Creating shared folders metadata...[default] Clearing any previously set network interfaces...[default] Preparing network interfaces based on configuration...[default] Forwarding ports...[default] -- 22 => 2222 (adapter 1)[default] Booting VM...[default] Waiting for VM to boot. This can take a few minutes.Friday, June 14, 13
  • 69. Vagrant Run Continued[default] Waiting for VM to boot. This can take a few minutes.[default] VM booted and ready for use!GuestAdditions 4.2.12 running --- OK.[default] Setting hostname...[default] Configuring and enabling network interfaces...[default] Mounting shared folders...[default] -- /vagrant[default] -- /tmp/vagrant-chef-1/chef-solo-1/cookbooks[default] Running provisioner: chef_solo...Generating chef JSON and uploading...Running chef-solo...[2013-06-12T02:52:26+00:00] INFO: *** Chef 11.4.4 ***[2013-06-12T02:52:26+00:00] INFO: Setting the run_list to ["recipe[minitest-handler]", "recipe[sauceproxy::server]"] from JSON[2013-06-12T02:52:26+00:00] INFO: Run List is [recipe[minitest-handler],recipe[sauceproxy::server]][2013-06-12T02:52:26+00:00] INFO: Run List expands to [minitest-handler,sauceproxy::server][2013-06-12T02:52:26+00:00] INFO: Starting Chef Run for sauceproxy-berkshelfFriday, June 14, 13
  • 70. Vagrant Run Continued[2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds[2013-06-12T02:55:22+00:00] INFO: Running report handlersRun options: -v --seed 31883# Running tests:recipe::java::openjdk#test_0001_installs the correct version of the jdk =0.11 s = .recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =0.04 s = .recipe::sauceproxy::server#test_0001_runs as a daemon =0.09 s = .Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.3 tests, 3 assertions, 0 failures, 0 errors, 0 skips[2013-06-12T02:55:22+00:00] INFO: Report handlers completeFriday, June 14, 13
  • 71. Vagrant Run Continued[2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds[2013-06-12T02:55:22+00:00] INFO: Running report handlersRun options: -v --seed 31883# Running tests:recipe::java::openjdk#test_0001_installs the correct version of the jdk =0.11 s = .recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =0.04 s = .recipe::sauceproxy::server#test_0001_runs as a daemon =0.09 s = .Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.3 tests, 3 assertions, 0 failures, 0 errors, 0 skips[2013-06-12T02:55:22+00:00] INFO: Report handlers completeFriday, June 14, 13
  • 72. Minitests from Java cookbook ran too![2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds[2013-06-12T02:55:22+00:00] INFO: Running report handlersRun options: -v --seed 31883# Running tests:recipe::java::openjdk#test_0001_installs the correct version of the jdk =0.11 s = .recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =0.04 s = .recipe::sauceproxy::server#test_0001_runs as a daemon =0.09 s = .Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.3 tests, 3 assertions, 0 failures, 0 errors, 0 skips[2013-06-12T02:55:22+00:00] INFO: Report handlers completeFriday, June 14, 13
  • 73. Minitests from Java cookbook ran too![2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds[2013-06-12T02:55:22+00:00] INFO: Running report handlersRun options: -v --seed 31883# Running tests:recipe::java::openjdk#test_0001_installs the correct version of the jdk =0.11 s = .recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =0.04 s = .recipe::sauceproxy::server#test_0001_runs as a daemon =0.09 s = .Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.3 tests, 3 assertions, 0 failures, 0 errors, 0 skips[2013-06-12T02:55:22+00:00] INFO: Report handlers completeFriday, June 14, 13
  • 74. New Test/Dev CycleWrite Unit TestsWorked?Run ChefSpecNoWrite RecipeCodeWrite AcceptanceTestsYesWorked?Run Vagrant + Minitest HandlerNovagrant destroyYes Commit/Tag Code,etc.Friday, June 14, 13
  • 75. CI Pipeline IntegrationFriday, June 14, 13
  • 76. Continuous Integration Pipeline• Run unit tests in your pipeline• Run acceptance tests in yourpipeline• Drive cookbook uploads as theoutput!Friday, June 14, 13
  • 77. Travis-CI Example for SauceProxylanguage: rubygemfile:  - gemfiles/travis.gemfilervm:  - "1.9.2"  - "1.9.3"script: bundle exec rake test:syntax test:lint test:specnotifications:  email:    - jdunn@opscode.comFriday, June 14, 13
  • 78. Rakefile Example• Too long to post here• Rake tasks for Foodcriticand ChefSpec• https://github.com/juliandunn/sauceproxy/blob/master/RakefileFriday, June 14, 13
  • 79. Travis-CI Example DashboardFriday, June 14, 13
  • 80. ^#$%(#$ I broke the buildFriday, June 14, 13
  • 81. Test Kitchen (alpha)• Run acceptance test suite onmultiple OSes• Different fixtures for each, if desired• Different drivers (vagrant, ec2, lxc,etc.)• Hook up to Jenkins or other CIsystem if desiredFriday, June 14, 13
  • 82. Test Kitchen Config File---driver_plugin: vagrantplatforms:- name: centos-6.4driver_config:box: opscode-centos-6.4box_url: https://opscode-vm.s3.amazonaws.com/vagrant/opscode_centos-6.4_provisionerless.boxrequire_chef_omnibus: true- name: centos-5.9driver_config:box: opscode-centos-5.9box_url: https://opscode-vm.s3.amazonaws.com/vagrant/opscode_centos-5.9_provisionerless.boxrequire_chef_omnibus: truesuites:- name: defaultrun_list: ["recipe[minitest-handler]", "recipe[sauceproxy::server]"]attributes: {}Friday, June 14, 13
  • 83. Test Kitchen DemoFriday, June 14, 13
  • 84. ResourcesFriday, June 14, 13
  • 85. Resources• ChefSpec: https://github.com/acrmp/chefspec• Chef Minitest Handler: https://github.com/calavera/minitest-chef-handler• Berkshelf: http://berkshelf.com/• Vagrant: http://vagrantup.com/• Test Kitchen: https://github.com/opscode/test-kitchen• Bento: https://github.com/opscode/bento• SauceProxy Sample Cookbook: https://github.com/juliandunn/sauceproxyFriday, June 14, 13
  • 86. More on Testing• Chef-NYC: The Hows and Whysof Cookbook Testing• Seth Vargo (Author of ChefSpec,Fauxhai, many others)• http://www.meetup.com/Chef-NYC/events/122219772/• June 25th in ManhattanFriday, June 14, 13
  • 87. Chef Fundamentals Training• Boston, July 11-12(CompuWorks, 263Summer St.)• eventbrite.com/event/6652057483• Use code BOSTON-MEETUP to save 25%!Friday, June 14, 13
  • 88. Questions and PrizesAlso, we’re hiring! Automate all the things for fun and profit!http://www.opscode.com/blog/careers/(and/or see me after)Friday, June 14, 13