Incident
  reporting

S. Freitag, F.
  Feldhaus
                     Incident reporting
Before you
report            GridKa Summer School 2010
Incident
Scenarios

Incident
handling         Stefan Freitag, Florian Feldhaus

                      Robotics Research Institute
                            TU Dortmund


                      September 10, 2010
Contents

   Incident
  reporting

S. Freitag, F.
  Feldhaus

Before you
report            1 Before you report
Incident
Scenarios

Incident
handling
                  2 Incident Scenarios



                  3 Incident handling
Do you know....?

   Incident
  reporting
                 Security Incident Response Policy1
S. Freitag, F.
  Feldhaus
                        objective: ensure that all incidents are investigated as fully
Before you
report                  as possible and that sites promptly report intrusions.
Incident
Scenarios
                        As a grid participant, you agree to
Incident
                             report suspected security incidents that have impact or
handling                     relationship to grid resources, services, or identities
                             respond to and investigate incident reports regarding
                             resources, services, or identities for which you are
                             responsible
                             perform appropriate investigations and forensics and share
                             the results with the incident coordinator
                             follow the incident response procedure

                 Next question: what is the incident response procedure?
                   1
                       https://edms.cern.ch/document/428035/7
EGEE incident response procedure2

   Incident
  reporting

S. Freitag, F.
  Feldhaus

                 Audience
Before you
report
                 grid site security officers and site administrators
Incident
Scenarios

Incident         Definition of security incident
handling
                 The act of violating an explicit or implied security policy

                 Definition of actions for the case of a security incident
                 More on this in a few minutes . . .




                   2
                       https://edms.cern.ch/document/867454
Security incident - scenario A (2009)

   Incident
  reporting

S. Freitag, F.        Some grid sites allow gsissh-based access to VoBoxes (e.g.
  Feldhaus
                      for VO software managers)
Before you
report                On a VoBox Grid users are mapped to local accounts
Incident
Scenarios
                 Initial step for an attacker
Incident
handling
                      gain access to user credentials (certificate or proxy)


                 What happens next ?

                      Connect to VoBox using stolen credentials
                      Running e.g. a kernel exploit to gain root privileges
Security incident - scenario A (2009)

   Incident
  reporting

S. Freitag, F.
  Feldhaus

                   # s h −x w u n d e r b a r e m p o r i u m . s h
Before you
report             [...]
Incident
                    [+] got r i n g 0 !
Scenarios           [+] d e t e c t e d 2.6 s t y l e 4k s t a c k s
Incident            [ + ] D i s a b l e d s e c u r i t y o f : n o t h i n g , what an
handling
                          i n s e c u r e machine !
                    [ + ] Got r o o t !

                   sh −3.00# i d
                   u i d =0( r o o t ) g i d =0( r o o t ) g r o u p s =64004( hepcg )
                          c o n t e x t=u s e r u : s y s t e m r : i n i t r c t
Security incident - scenario B (2010)

   Incident
  reporting
                                 Department A     The Grid
S. Freitag, F.
  Feldhaus

Before you
report

Incident
Scenarios

Incident
handling
Security incident - scenario B (2010)

   Incident
  reporting
                                     Department A   The Grid
S. Freitag, F.
  Feldhaus
                             CERTIFICATE
                             X.509
Before you
report
                             CERTIFICATE
Incident                     X.509

Scenarios

Incident
handling
Security incident - scenario B (2010)

   Incident
  reporting

S. Freitag, F.
                                                        Department A   The Grid
  Feldhaus

                                                CERTIFICATE
Before you                                      X.509

report

Incident
Scenarios                         CERTIFICATE
                                  X.509

Incident
handling




                         Alien
                       attacker
Security incident - scenario B (2010)

   Incident
  reporting

S. Freitag, F.
  Feldhaus                                               The Grid

Before you
report

Incident
Scenarios
                                  stolen
Incident                         CERTIFICATE
                                 X.509
handling




                    Alien
                  attacker
Incident handling

   Incident
  reporting

S. Freitag, F.
  Feldhaus       For the next slides please keep in mind:
Before you
report

Incident         The red block describes actions required by the EGEE Incident
Scenarios
                 Response Procedure document
Incident
handling



                 The blue block contains information about actions carried out
                 during a security incident at the Grid resource in Dortmund

                 Down here you will find additional information, e.g. max.
                 response times
Incident handling

   Incident
  reporting
                 First action
S. Freitag, F.
  Feldhaus       Inform immediately your local security team and your ROC
Before you       Security Contact
report

Incident
Scenarios        Action
Incident
handling              Sent E-Mail to Ursula Epting
                      Read Incident response procedure
                      Informed 2nd site security officer and local security team




                                   max. 4 hours           or
Incident handling

   Incident
  reporting

S. Freitag, F.
  Feldhaus

Before you       Response procedure
report

Incident         In case no support is shortly available [...] try to contain the
Scenarios
                 incident. For instance by unplugging the network cable
Incident
handling         connected to the host. Do NOT reboot or power off the host.

                 Action

                     Disconnected affected workernodes from network
Incident handling

   Incident
  reporting      Response procedure
S. Freitag, F.
  Feldhaus       Assist your local security team and your ROC Security Contact
                 to confirm and investigate the incident. Announce the incident
Before you
report           to all the sites.
Incident
Scenarios
                 Actions
Incident
handling
                     Send a heads-up e-mail (template: next slide)
                     Arranged meeting with local security team
                     Network guys were asked to check logs




                              max. 4 hours (Announcement)
Heads-up E-mail

   Incident
  reporting

S. Freitag, F.
  Feldhaus

Before you
report
                 ** PLEASE DO NOT REDISTRIBUTE ** EGEE-<DATE> (ex: EGEE-20090531)
Incident         ** This message is sent to the EGEE CSIRTs and must NOT be publicly archived **
Scenarios        Dear CSIRTs,
                 It seems a security incident has been detected at <your site>.
Incident         Summary of the information available so far:
handling
                 Ex: A malicious SSH connection was detected from XXXXX. The extent of the
                 incident is unclear for now, and more information will be published in the coming
                 hours as forensics are progressing at our site. However, all sites should check for
                 successful SSH connection from XXXXX as a precautionary measure.
Incident handling

   Incident
  reporting      Response procedure
S. Freitag, F.
  Feldhaus           Report a downtime for the affected hosts on the GOCDB
Before you
report
                     → Send an EGEE broadcast announcing the downtime for
Incident
                     the affected hosts Use ”Security operations in progress” as
Scenarios            the reason with no additional detail both for the broadcast
Incident
handling
                     and the GOCDB.

                 Actions

                     Created downtime for possibly affected hosts udo-ce01/
                     udo-dcache01



                                      max. 1 day after discovery
Incident handling

   Incident
  reporting

S. Freitag, F.   Response procedure
  Feldhaus
                 Perform appropriate forensics and take necessary actions to
Before you
report           prevent further damage
Incident
Scenarios
                     Identify and kill suspicious process(es) as appropriate, but
Incident
                     aim at preserving the information they could have
handling
                     generated
                     If it is suspected that some grid credentials have been
                     abused or compromised, you MUST ensure the relevant
                     accounts become suspended
                     If it is suspected that some grid credentials have been
                     abused, you MUST ensure that the relevant VO
                     manager(s) have been informed.
Incident handling

   Incident
  reporting

S. Freitag, F.
  Feldhaus

Before you
                 Response procedure
report

Incident
                 Perform appropriate forensics and take necessary actions to
Scenarios        prevent further damage
Incident
handling             If it is suspected that some grid credentials have been
                     compromised, you MUST ensure that the relevant
                     certification authority gets informed.
                     If needed, seek for help from your local security team or
                     from your ROC Security Contact
Incident handling

   Incident
  reporting

S. Freitag, F.
  Feldhaus

Before you
report           Action
Incident
Scenarios            Banned affected users on our compute elements by adding
Incident             their DN to the blacklist in
handling
                     /opt/glite/etc/lcas/ban users.db
                     E-Mail to VO manager regarding compromised user
                     Contacted the certification authority
Incident handling

   Incident
  reporting

S. Freitag, F.   Response procedure
  Feldhaus
                 As part of the security incident resolution process, sites are
Before you
report           expected to report the following information:
Incident
Scenarios             affected hosts and hosts used as entry point to the site
Incident              remote IP address(es) of the attacker
handling
                      evidence of the compromise, including timestamps
                      what was lost, details of the attack
                      list of other sites possibly affected (if available)
                      possible vulnerabilities exploited by the attacker (if
                      available)
                      actions taken to resolve the incident
Incident handling

   Incident
  reporting

S. Freitag, F.
  Feldhaus

Before you
                 Response procedure
report

Incident
Scenarios
                     Tracked down the UI that was used by the attacker for job
Incident
                     submission (checking logs of batchsystem, Compute
handling             Element, . . . )
                     Analyzed netflow to/fro affected workernode
                     Analyzed executables deployed by the attacker
                     Updated incident report regularly
Incident handling

   Incident
  reporting

S. Freitag, F.
  Feldhaus       Response procedure
Before you       Coordinate with your local security team and your ROC
report
                 Security Contact to send an incident closure report including
Incident
Scenarios        lessons learnt and measures taken to prevent future incidents.
Incident
handling
                 Actions
                 Preparation and submission of final report



                                       max. 1 months
Incident handling

   Incident
  reporting

S. Freitag, F.
  Feldhaus
                 Response procedure
Before you
report
                 Restore the service, and if needed, send an EGEE broadcast,
Incident
Scenarios        update the GOCDB, service documentation and procedures to
Incident         prevent recurrence as necessary
handling


                 Actions

                     Re-installation of affected workernode
                     Safety tuning
Incident
  reporting

S. Freitag, F.
  Feldhaus

Before you
report

Incident
Scenarios

Incident
handling
                 Thanks for your attention!

Talk at the Security Workshop, GridKA Summerschool 2010

  • 1.
    Incident reporting S.Freitag, F. Feldhaus Incident reporting Before you report GridKa Summer School 2010 Incident Scenarios Incident handling Stefan Freitag, Florian Feldhaus Robotics Research Institute TU Dortmund September 10, 2010
  • 2.
    Contents Incident reporting S. Freitag, F. Feldhaus Before you report 1 Before you report Incident Scenarios Incident handling 2 Incident Scenarios 3 Incident handling
  • 3.
    Do you know....? Incident reporting Security Incident Response Policy1 S. Freitag, F. Feldhaus objective: ensure that all incidents are investigated as fully Before you report as possible and that sites promptly report intrusions. Incident Scenarios As a grid participant, you agree to Incident report suspected security incidents that have impact or handling relationship to grid resources, services, or identities respond to and investigate incident reports regarding resources, services, or identities for which you are responsible perform appropriate investigations and forensics and share the results with the incident coordinator follow the incident response procedure Next question: what is the incident response procedure? 1 https://edms.cern.ch/document/428035/7
  • 4.
    EGEE incident responseprocedure2 Incident reporting S. Freitag, F. Feldhaus Audience Before you report grid site security officers and site administrators Incident Scenarios Incident Definition of security incident handling The act of violating an explicit or implied security policy Definition of actions for the case of a security incident More on this in a few minutes . . . 2 https://edms.cern.ch/document/867454
  • 5.
    Security incident -scenario A (2009) Incident reporting S. Freitag, F. Some grid sites allow gsissh-based access to VoBoxes (e.g. Feldhaus for VO software managers) Before you report On a VoBox Grid users are mapped to local accounts Incident Scenarios Initial step for an attacker Incident handling gain access to user credentials (certificate or proxy) What happens next ? Connect to VoBox using stolen credentials Running e.g. a kernel exploit to gain root privileges
  • 6.
    Security incident -scenario A (2009) Incident reporting S. Freitag, F. Feldhaus # s h −x w u n d e r b a r e m p o r i u m . s h Before you report [...] Incident [+] got r i n g 0 ! Scenarios [+] d e t e c t e d 2.6 s t y l e 4k s t a c k s Incident [ + ] D i s a b l e d s e c u r i t y o f : n o t h i n g , what an handling i n s e c u r e machine ! [ + ] Got r o o t ! sh −3.00# i d u i d =0( r o o t ) g i d =0( r o o t ) g r o u p s =64004( hepcg ) c o n t e x t=u s e r u : s y s t e m r : i n i t r c t
  • 7.
    Security incident -scenario B (2010) Incident reporting Department A The Grid S. Freitag, F. Feldhaus Before you report Incident Scenarios Incident handling
  • 8.
    Security incident -scenario B (2010) Incident reporting Department A The Grid S. Freitag, F. Feldhaus CERTIFICATE X.509 Before you report CERTIFICATE Incident X.509 Scenarios Incident handling
  • 9.
    Security incident -scenario B (2010) Incident reporting S. Freitag, F. Department A The Grid Feldhaus CERTIFICATE Before you X.509 report Incident Scenarios CERTIFICATE X.509 Incident handling Alien attacker
  • 10.
    Security incident -scenario B (2010) Incident reporting S. Freitag, F. Feldhaus The Grid Before you report Incident Scenarios stolen Incident CERTIFICATE X.509 handling Alien attacker
  • 11.
    Incident handling Incident reporting S. Freitag, F. Feldhaus For the next slides please keep in mind: Before you report Incident The red block describes actions required by the EGEE Incident Scenarios Response Procedure document Incident handling The blue block contains information about actions carried out during a security incident at the Grid resource in Dortmund Down here you will find additional information, e.g. max. response times
  • 12.
    Incident handling Incident reporting First action S. Freitag, F. Feldhaus Inform immediately your local security team and your ROC Before you Security Contact report Incident Scenarios Action Incident handling Sent E-Mail to Ursula Epting Read Incident response procedure Informed 2nd site security officer and local security team max. 4 hours or
  • 13.
    Incident handling Incident reporting S. Freitag, F. Feldhaus Before you Response procedure report Incident In case no support is shortly available [...] try to contain the Scenarios incident. For instance by unplugging the network cable Incident handling connected to the host. Do NOT reboot or power off the host. Action Disconnected affected workernodes from network
  • 14.
    Incident handling Incident reporting Response procedure S. Freitag, F. Feldhaus Assist your local security team and your ROC Security Contact to confirm and investigate the incident. Announce the incident Before you report to all the sites. Incident Scenarios Actions Incident handling Send a heads-up e-mail (template: next slide) Arranged meeting with local security team Network guys were asked to check logs max. 4 hours (Announcement)
  • 15.
    Heads-up E-mail Incident reporting S. Freitag, F. Feldhaus Before you report ** PLEASE DO NOT REDISTRIBUTE ** EGEE-<DATE> (ex: EGEE-20090531) Incident ** This message is sent to the EGEE CSIRTs and must NOT be publicly archived ** Scenarios Dear CSIRTs, It seems a security incident has been detected at <your site>. Incident Summary of the information available so far: handling Ex: A malicious SSH connection was detected from XXXXX. The extent of the incident is unclear for now, and more information will be published in the coming hours as forensics are progressing at our site. However, all sites should check for successful SSH connection from XXXXX as a precautionary measure.
  • 16.
    Incident handling Incident reporting Response procedure S. Freitag, F. Feldhaus Report a downtime for the affected hosts on the GOCDB Before you report → Send an EGEE broadcast announcing the downtime for Incident the affected hosts Use ”Security operations in progress” as Scenarios the reason with no additional detail both for the broadcast Incident handling and the GOCDB. Actions Created downtime for possibly affected hosts udo-ce01/ udo-dcache01 max. 1 day after discovery
  • 17.
    Incident handling Incident reporting S. Freitag, F. Response procedure Feldhaus Perform appropriate forensics and take necessary actions to Before you report prevent further damage Incident Scenarios Identify and kill suspicious process(es) as appropriate, but Incident aim at preserving the information they could have handling generated If it is suspected that some grid credentials have been abused or compromised, you MUST ensure the relevant accounts become suspended If it is suspected that some grid credentials have been abused, you MUST ensure that the relevant VO manager(s) have been informed.
  • 18.
    Incident handling Incident reporting S. Freitag, F. Feldhaus Before you Response procedure report Incident Perform appropriate forensics and take necessary actions to Scenarios prevent further damage Incident handling If it is suspected that some grid credentials have been compromised, you MUST ensure that the relevant certification authority gets informed. If needed, seek for help from your local security team or from your ROC Security Contact
  • 19.
    Incident handling Incident reporting S. Freitag, F. Feldhaus Before you report Action Incident Scenarios Banned affected users on our compute elements by adding Incident their DN to the blacklist in handling /opt/glite/etc/lcas/ban users.db E-Mail to VO manager regarding compromised user Contacted the certification authority
  • 20.
    Incident handling Incident reporting S. Freitag, F. Response procedure Feldhaus As part of the security incident resolution process, sites are Before you report expected to report the following information: Incident Scenarios affected hosts and hosts used as entry point to the site Incident remote IP address(es) of the attacker handling evidence of the compromise, including timestamps what was lost, details of the attack list of other sites possibly affected (if available) possible vulnerabilities exploited by the attacker (if available) actions taken to resolve the incident
  • 21.
    Incident handling Incident reporting S. Freitag, F. Feldhaus Before you Response procedure report Incident Scenarios Tracked down the UI that was used by the attacker for job Incident submission (checking logs of batchsystem, Compute handling Element, . . . ) Analyzed netflow to/fro affected workernode Analyzed executables deployed by the attacker Updated incident report regularly
  • 22.
    Incident handling Incident reporting S. Freitag, F. Feldhaus Response procedure Before you Coordinate with your local security team and your ROC report Security Contact to send an incident closure report including Incident Scenarios lessons learnt and measures taken to prevent future incidents. Incident handling Actions Preparation and submission of final report max. 1 months
  • 23.
    Incident handling Incident reporting S. Freitag, F. Feldhaus Response procedure Before you report Restore the service, and if needed, send an EGEE broadcast, Incident Scenarios update the GOCDB, service documentation and procedures to Incident prevent recurrence as necessary handling Actions Re-installation of affected workernode Safety tuning
  • 24.
    Incident reporting S.Freitag, F. Feldhaus Before you report Incident Scenarios Incident handling Thanks for your attention!