Exadata cell update

Exadata
Cell Storage update
X2-2 quarter rack
image version from 11.2.3.3.0 to 11.2.3.3.1
Infrastructure & Governance
Date : 2015. 1.30
작성자 : 김 종 인

■ Cell node update
- patchmgr 란? ( = DB node 의 dbnodeupdate.sh)
exadata 의 cell node 의 new release 를 적용하기 위한 모든 수동작업을 대체하고 작업전check 및 적
용과정을 자동으로 수행해주는 기능을 한다.
cell node image version / Infiniband Firmware 를 최신으로 Update 한다.
- patchmgr 버전 (별도로 다운받을 경우 11.2.3.x 와 12.1.x 버전용 따로 존재하므로 확인후 다운받아
야 한다. 문서 888828.1 항상확인)
- 2015년 1월 29일 기준 최신버전
11.2.3.3.1 - Patch 19166601 - Storage server and InfiniBand switch software
12.1.2.1.0 - Patch 17885582 - Storage server and InfiniBand switch software
Patch 19681939 - Exadata 12.1.1.1.1 patchmgr plug-ins
해당 패치압축을 풀면 patchmgr 가 내장되어 있음.
- 작업후 Linux Kernel version Updata 됨
Image 11.2.3.3.0 ☞ 2.6.39-400.126.1 (RHEL 5.9)
Image 11.2.3.3.1 ☞ 2.6.39-400.128.17 (RHEL 5.10)
Image 12.1.2.1.0 ☞ 2.6.39-400.243.1 (RHEL 6.6)

작업순서
1. cell 의 ip 정보등 확인 작업
[EXA1]root@exa1:/root# dcli -g cell_group -l root /opt/oracle.cellos/ipconf -verify
gtceladm01: Verifying of Exadata configuration file /opt/oracle.cellos/cell.conf
gtceladm01: Done. Configuration file /opt/oracle.cellos/cell.conf passed all verification checks
2. 해당 Patch 다운로드 후 압축해제
-[EXA1]root@exa1:/root# unzip p19166601_112331_Linux-x86-64.zip
-Archive: p19166601_112331_Linux-x86-64.zip
- creating: patch_11.2.3.3.1.140708/
- inflating: patch_11.2.3.3.1.140708/sundcs_36p_repository_2.1.3_4.pkg
- inflating: patch_11.2.3.3.1.140708/upgradeIBSwitch.sh
- creating: patch_11.2.3.3.1.140708/linux.db.rpms/
- inflating: patch_11.2.3.3.1.140708/linux.db.rpms/perl-XML-Parser-2.34-6.1.2.2.1.x86_64.rpm
- creating: patch_11.2.3.3.1.140708/plugins/
- inflating: patch_11.2.3.3.1.140708/plugins/010-check_17854520.sh
....................

작업순서
3. [EXA1]root@exa1:/root# crsctl stop crs
4. [EXA1]root@exa1:/root# dcli -g /root/cell_group -l root 'service celld stop‘
5. [EXA1]root@exa1:/root# dcli -g /root/cell_group -l root 'service celld status‘
6. [EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -
reset_force (patchmgr -h 시 -reset_force 옵션은 없음)
2015-01-20 13:36:41 +0900 :DONE: reset_force
7. [EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -cleanup
2015-01-20 13:37:54 +0900 :Working: DO: Cleanup ...
2015-01-20 13:37:55 +0900 :SUCCESS: DONE: Cleanup
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -
patch_check_prereq -ignore_alerts
(-ignore_alerts 옵션없이 진행할 경우 Cell alert 등으로 인해 Fail 됨)

작업순서
8. [EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -patch -
ignore_alerts
(마찬가지로 -ignore_alerts 옵션없이 진행할 경우 Cell alert 등으로 인해 Fail 됨)
pre-reboot 절차에 45분 소요된다고 표시가 됨. (실제론 10분정도 소요)
2015-01-20 13:58:26 +0900 Wait for patch pre-reboot procedures
||||| Minutes left 045///// Minutes left 045----- Minutes left 045 Minutes left 045||||| Minutes
left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045 Minutes left 045|||||
Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045 Minutes
left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left
045 Minutes left 045||||| Minutes left 045||||| Minu
2015-01-29 16:15:20 +0900 2 of 5 :SUCCESS: DONE: Waiting to finish pre-reboot patch actions.
2015-01-29 16:15:20 +0900 :Working: DO: Execute plugin check for Patching ...
2015-01-29 16:15:21 +0900 :SUCCESS: DONE: Execute plugin check for Patching.
2015-01-29 16:15:21 +0900 3 of 5 :Working: DO: Finalize patch on cells. Cells will reboot. Up to 5
minutes ...
2015-01-29 16:15:43 +0900 3 of 5 :SUCCESS: DONE: Finalize patch on cells.

작업순서
pre-reboot 절차가 끝나면 120분 소요메세지와 함께 작업진행됨 (시간은 예측치이기 때문에 해당경우
의 실제 작업은 78분 남기고 종료되었음)
2015-01-29 16:15:43 +0900 4 of 5 :Working: DO: Wait for cells to reboot and come online. Up to 120 minutes ...
2015-01-29 16:16:43 +0900 Wait for patch finalization and reboot
||||| 2015-01-29 16:16:45 +0900 Minutes left 120///// 2015-01-29 16:16:45 +0900 Minutes left 120----- 2015-01-29 16:16:45 +0900 Minutes
left 120 2015-01-29 16:16:45 +0900 Minutes left 120||||| 2015-01-29 16:16:45 +0900 Minutes left 120||||| 2015-01-29 16:16:45
+0900 Minutes left 120///// 2015-01-29 16:16:45 +0900 Minutes left 120----- 2015-01-29 16:16:45 +0900 Minutes left 120 2015-
01-29 16:16:45 +0900 Minutes left 120||||| 2015-01-29 16:16:45 +0900 Minutes ft
……………………………………………………………………………………..
078///// 2015-01-29 17:12:14 +0900 Minutes left 078----- 2015-01-29 17:12:14 +0900 Minutes left 078 2015-01-29 17:12:14
+0900 Minutes left 078||||| 2015-01-29 17:12:14 +0900 Minutes left 078||||| 2015-01-29 17:12:14 +0900 Minutes left 078///// 2015-01-29
17:12:14 +0900 Minutes left 078----- 2015-01-29 17:12:14 +0900 Minutes left 078 2015-01-29 17:12:14 +0900 Minutes left 078|||||
2015-01-29 17:12:14 +0900 Minutes left 078
2015-01-29 17:13:20 +0900 4 of 5 :SUCCESS: DONE: Wait for cells to reboot and come online.
2015-01-29 17:13:20 +0900 5 of 5 :Working: DO: Check the state of patch on cells. Up to 5 minutes ...
2015-01-29 17:14:02 +0900 5 of 5 :SUCCESS: DONE: Check the state of patch on cells.
2015-01-29 17:14:02 +0900 :Working: DO: Execute plugin check for Post Patch ...
2015-01-29 17:14:03 +0900 :SUCCESS: DONE: Execute plugin check for Post Patch.
9. Cell Rebooting 후 상태 확인 (service celld status)
10. [EXA1]root@exa1:/root# crsctl start crs

작업순서
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# dcli -g /root/cell_group -l root imageinfo
gtceladm01:
gtceladm01: Kernel version: 2.6.39-400.128.17.el5uek #1 SMP Tue May 27 13:20:24 PDT 2014 x86_64
gtceladm01: Cell version: OSS_11.2.3.3.1_LINUX.X64_140708
gtceladm01: Cell rpm version: cell-11.2.3.3.1_LINUX.X64_140708-1
gtceladm01:
gtceladm01: Active image version: 11.2.3.3.1.140708
gtceladm01: Active image activated: 2015-01-20 15:00:37 +0900
gtceladm01: Active image status: success
gtceladm01: Active system partition on device: /dev/md6
gtceladm01: Active software partition on device: /dev/md8
gtceladm01:
gtceladm01: In partition rollback: Impossible
gtceladm01:
gtceladm01: Cell boot usb partition: /dev/sdm1
gtceladm01: Cell boot usb version: 11.2.3.3.1.140708
gtceladm01:
gtceladm01: Inactive image version: 11.2.3.3.0.131014.1
gtceladm01: Inactive image activated: 2015-01-06 17:03:56 +0900
gtceladm01: Inactive image status: success
gtceladm01: Inactive system partition on device: /dev/md5
gtceladm01: Inactive software partition on device: /dev/md7
gtceladm01:
gtceladm01: Boot area has rollback archive for the version: 11.2.3.3.0.131014.1
gtceladm01: Rollback to the inactive partitions: Possible

작업순서
[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# dcli -g /root/cell_group -l root imagehistory
gtceladm01: Version : 11.2.3.3.0.131014.1
gtceladm01: Image activation date : 2015-01-06 17:03:56 +0900
gtceladm01: Imaging mode : fresh
gtceladm01: Imaging status : success
gtceladm01:
gtceladm01: Version : 11.2.3.3.1.140708
gtceladm01: Imaging mode : out of partition upgrade
gtceladm01:
gtceladm02:
gtceladm02:
gtceladm03:

작업 유의사항
- 작업로그는 아래의 로그를 tail -f 로 열어서 확인 (vi 등 wrte 작업 절대금지)
tail -f /u01/patch_12.1.2.1.0.141206.1/patchmgr.stdout
각 노드별로 성공적으로 진행되었음을 확인 할수 있음
gtceladm03: _EXIT_PASS_Cell gtceladm03 172.16.40.103 2015-01-20 14:44:08 +0900: Found: 11.2.3.3.1.140708 state success Expected:
11.2.3.3.1.140708 state success.
2015-01-20 15:02:04 +0900 ++++++++++++++++++ Logs so far end ++++++++++
2015-01-20 15:02:04 +0900 5 of 5 :SUCCESS: DONE: Check the state of patch on cells.
2015-01-20 15:02:04 +0900 :Working: DO: Execute plugin check for Post Patch ...
2015-01-20 15:02:04 +0900 ++++++++++++++++++ Logs so far begin ++++++++++
2015-01-20 15:02:05 +0900 ++++++++++++++++++ Logs so far end ++++++++++
2015-01-20 15:02:05 +0900 :SUCCESS: DONE: Execute plugin check for Post Patch.
================PatchMgr run ended 2015-01-20 15:02:05 +0900 ===========
patch 시작시 표시되는 Warning 메시지 반드시 준수
********************************************************************************
NOTE Cells will reboot during the patch or rollback process.
NOTE For non-rolling patch or rollback, ensure all ASM instances using
NOTE the cells are shut down for the duration of the patch or rollback.
NOTE For rolling patch or rollback, ensure all ASM instances using
NOTE the cells are up for the duration of the patch or rollback.
WARNING Do not start more than one instance of patchmgr.
WARNING Do not interrupt the patchmgr session.
WARNING Do not alter state of ASM instances during patch or rollback.
WARNING Do not resize the screen. It may disturb the screen layout.
WARNING Do not reboot cells or alter cell services during patch or rollback.
WARNING Do not open log files in editor in write mode or try to alter them.
NOTE All time estimates are approximate.
NOTE You may interrupt this patchmgr run in next 60 seconds with CONTROL-c.
********************************************************************************

작업 유의사항
- patchmgr -h
[EXA1]root@exa1:/u01/patch_12.1.2.1.0.141206.1# ./patchmgr -h
Usage:
./patchmgr -cells cell_host_file
[-patch_check_prereq | -rollback_check_prereq [-rolling] [-ignore_alerts]]
[-patch | -rollback [-rolling] [-ignore_alerts]]
[-cleanup]
./patchmgr -ibswitches [ibswitch_list_file]
<-upgrade | -downgrade> [-ibswitch_precheck] [-force]]
./patchmgr -h
OPTIONS
-h
Displays this screen
Cell patching and rollback
The following options are supported for cell patching and(or) rollback:
-cells cell_list_file
Specifies the name of the cell list file. The file contains one
cell hostname or ip per line. The cell patching will fail if the
list file is not specified.
-cleanup
Cleanup mode. Cleans up all patch files and temporary content on all
cells. Before cleaning up, collects logs and information for problem
diagnostics and analysis. Cleaning up patch files can be done manually
if patch fails by removing directory /root/_cellupd_dpullec_ on each cell.
-ignore_alerts
Ignore any active hardware alerts on the Exadata cell and proceed
with the patching.
-patch
Applys the patch, including firmware updates, wherever possible
(BIOS,
Disk Controller and if possible disk drives) to all cells in the cell
list file.
-patch_check_prereq
Runs prerequisite check on all the cells to determine if the patch
can be applied to the cells.
-rollback
Rolls back the patch.
-rollback_check_prereq
Runs prerequisite check on all the cells to determin if the cells
can be rolled back for this specific patch.
-rolling
Applies the patch or executes the rollback in rolling fashion, one
cell at a time.
Environment variable EXA_PATCH_ACTIVATE_TIMEOUT_SECONDS
controls the timeout value waiting for the grid disks to be
activated.
The default is set to 36000 (10 hours).

작업 유의사항
- patchmgr -h
-smtp_from "addr"
The address patchmgr notification sent from.
sendmail daemon must be started to send email notification.
-smtp_to "addr1 addr2 addr3 ..."
The address(es) patchmgr notification sent to.
sendmail daemon must be started to send email notification.
-smtp_set_envelope_sender
Use the same from address in Return-Path: mail header.
Infiniband switch upgrade and downgrade
The following options are supported for infiniband switch upgrade and(or)
downgrade:
-ibswitches [ibswitch_list_file]
Specifies the name of the InfiniBand switch list file. The file has one
switch hostname or ip per line. If no filename is provided, then it
runs the command on all InfiniBand switches discovered from this host
by running ibswitches command.
-downgrade
Downgrade the InfiniBand switches in the list file to 2.1.3-4
-force
Specifies to proceed with the upgrade or downgrade even on non-critical
failures.
-ibswitch_precheck
Runs the pre-update validation checks on the InfiniBand switches in the
list file.
-upgrade
Upgrade the InfiniBand switches in the list file to 2.1.5-1

Exadata cell update

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (19)

Similar to Exadata cell update

Similar to Exadata cell update (20)

Recently uploaded

Recently uploaded (20)

Exadata cell update