10 Sep 12 Hot replacement of an HBA/NIC card (rx8640). No downtime.
Take note of the fcmsutil output below. One of the fiber cards has failed here.
We became aware of the problem due to a resmon message and errors generated on the SAN side.
[shmuel01]/root # olrad -q
Driver(s)
Capable
Slot Path Bus Max Spd Pwr Occu Susp OLAR OLD Max Mode
Num Spd Mode
0-0-0-1 0/0/8/1 140 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-2 0/0/10/1 169 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-3 0/0/12/1 198 266 133 On Yes No Yes N/A PCI-X PCI-X
0-0-0-4 0/0/14/1 227 266 133 On Yes No Yes N/A PCI-X PCI-X
0-0-0-5 0/0/6/1 112 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-6 0/0/4/1 84 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-7 0/0/2/1 56 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-8 0/0/1/1 28 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-1 1/0/8/1 396 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-2 1/0/10/1 425 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-3 1/0/12/1 454 266 133 On Yes No Yes N/A PCI-X PCI-X
0-0-1-4 1/0/14/1 483 266 133 On Yes No Yes N/A PCI-X PCI-X
0-0-1-5 1/0/6/1 368 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-6 1/0/4/1 340 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-7 1/0/2/1 312 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-8 1/0/1/1 284 133 133 Off No N/A N/A N/A PCI-X PCI-X
[sap1pbw0]/root # ioscan -fnCfc
Class I H/W Path Driver S/W State H/W Type Description
===========================================================================
fc 0 0/0/12/1/0/4/0 fcd CLAIMED INTERFACE HP AD193-60001 PCI/PCI-X Fibre Channel 1-port 4Gb FC/1-port 1000B-T Combo Adapter (FC Port 1)
/dev/fcd0
fc 1 0/0/14/1/0/4/0 fcd CLAIMED INTERFACE HP AD193-60001 PCI/PCI-X Fibre Channel 1-port 4Gb FC/1-port 1000B-T Combo Adapter (FC Port 1)
/dev/fcd1
fc 2 1/0/12/1/0/4/0 fcd CLAIMED INTERFACE HP AD193-60001 PCI/PCI-X Fibre Channel 1-port 4Gb FC/1-port 1000B-T Combo Adapter (FC Port 1)
/dev/fcd2
fc 3 1/0/14/1/0/4/0 fcd CLAIMED INTERFACE HP AD193-60001 PCI/PCI-X Fibre Channel 1-port 4Gb FC/1-port 1000B-T Combo Adapter (FC Port 1)
/dev/fcd3
[shmuel01]/root # fcmsutil /dev/fcd0
Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012de
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x500110a0003db94b
N_Port Port World Wide Name = 0x500110a0003db94a
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/0/12/1/0/4/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17
[shmuel01]/root # fcmsutil /dev/fcd1
Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012de
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0xdc000b
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x500110a0003db94f
N_Port Port World Wide Name = 0x500110a0003db94e
Switch Port World Wide Name = 0x2043000dec8812c0
Switch Node World Wide Name = 0x2046000dec2adb01
Driver state = ONLINE
Hardware Path is = 0/0/14/1/0/4/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17
[shmuel01]/root # fcmsutil /dev/fcd2
Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012de
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0x9d000b
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x500110a0003db93b
N_Port Port World Wide Name = 0x500110a0003db93a
Switch Port World Wide Name = 0x2043000dec881f00
Switch Node World Wide Name = 0x2050000dec2adcc1
Driver state = ONLINE
Hardware Path is = 1/0/12/1/0/4/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17
[shmuel01]/root # fcmsutil /dev/fcd3
Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012de
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0x9d000a
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x500110a0003db8e1
N_Port Port World Wide Name = 0x500110a0003db8e0
Switch Port World Wide Name = 0x2003000dec881f00
Switch Node World Wide Name = 0x2050000dec2adcc1
Driver state = ONLINE
Hardware Path is = 1/0/14/1/0/4/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17
/usr/bin/olrad -r slot_ID
/usr/bin/olrad -r 0-0-0-3
[shmuel01]/root # olrad -q
Driver(s)
Capable
Slot Path Bus Max Spd Pwr Occu Susp OLAR OLD Max Mode
Num Spd Mode
0-0-0-1 0/0/8/1 140 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-2 0/0/10/1 169 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-3 0/0/12/1 198 266 266 Off Yes Yes Yes N/A PCI-X PCI-X
0-0-0-4 0/0/14/1 227 266 266 On Yes No Yes N/A PCI-X PCI-X
0-0-0-5 0/0/6/1 112 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-6 0/0/4/1 84 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-7 0/0/2/1 56 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-8 0/0/1/1 28 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-1 1/0/8/1 396 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-2 1/0/10/1 425 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-3 1/0/12/1 454 266 133 On Yes No Yes N/A PCI-X PCI-X
0-0-1-4 1/0/14/1 483 266 133 On Yes No Yes N/A PCI-X PCI-X
0-0-1-5 1/0/6/1 368 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-6 1/0/4/1 340 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-7 1/0/2/1 312 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-8 1/0/1/1 284 133 133 Off No N/A N/A N/A PCI-X PCI-X
Cards to be replaced are flashing yellow.
Replace card now.
/usr/bin/olrad -A 0-0-0-3
/usr/bin/olrad -R 0-0-0-3
Tags: fiber card replacement, fiber channel adaptor, hba replacement, hot swap hba card, rx8640, systems administration
11 Jul 12 Online JFS filesystem expansion without umount
This requires add in software from Veritas/Symmantec that can be expensive on HP-UX.
Lets say we want to extend /var which is lvol8.
It was originally 600 LE and we extended it as follows.
lvextend -l 800 /dev/vg00/lvol8 /dev/disk/disk25_p2
The 800 refers to LE logical extents. I prefer to use this rather than MB because it corresponds to the values displayed by most lvm utilities. -L <value in megabytes> can be used. I assume you can do the math here.
[system099]/root # lvdisplay /dev/vg00/lvol8
— Logical volumes —
LV Name /dev/vg00/lvol8
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 12800
Current LE 800
Allocated PE 1600
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation non-strict
IO Timeout (Seconds) default
It was originally 600 LE and we extended it as follows.
lvextend -l 800 /dev/vg00/lvol8 /dev/disk/disk25_p2
fsadm -F vxfs -b 12800M /var
This command will use online JFS to extend the filesystem. The alternative is extendfs. But that requires a umount. Or perhaps xvumount -o force. Your choice, I don’t recommend it, but did test it the other day in the sandbox.
Now if you want to be cute and use all your space, try this.
SIZE=$(lvdisplay /dev/vg00/lvol8 | awk ‘/LV Size/ {print $NF}’);
ESIZE=”${SIZE}M”
fsadm -F vxfs -b $ESIZE /var
You can get more cute and do it with a single command line but I’m not going to do that. I have not broken any systems this week and am trying to avoid that.
Tags: file system expansion, Online JFS
01 Jul 10 Quick and dirty awk trick to see all Fiber Channel status
I needed to update documentation and I needed more information that my prior syslayout.sh script provided on fiber.
# Improvement by JRF on the ITRC forums.
ioscan -kfnC fc | awk ‘/fcd/ {getline;fcd=$NF;print fcd,$2}’ | while read -r fdev
do
fcmsutil ${fdev} | awk ‘/Hardware / {print $5};/World / { print $7}’
done
Ouput looks like this:
0/0/12/1/0/4/0
0x5001438002a24979
0x5001438002a24978
0x204f000dec81b540
0x200f000dec81b541
0/0/12/1/0/4/1
0x5001438002a2497b
0x5001438002a2497a
0x2093000dec81b480
0x2010000dec81b481
2/0/12/1/0/4/0
0x5001438002a24d35
0x5001438002a24d34
0x2050000dec81b480
0x2010000dec81b481
2/0/12/1/0/4/1
0x5001438002a24d37
0x5001438002a24d36
0x2093000dec81b540
0x200f000dec81b541
Second, new improved version (Girsah Chadash)
ioscan -kfnC fc | awk ‘/fcd/ {getline;fcd=$NF;print fcd,$2}’ | while read -r fde
v
do
fcmsutil ${fdev} | awk ‘/Hardware / {print $5};/World / { print $7}’ | awk’ {printf “%s %s %s %s %s”,$1, $2, $3, $4, $5;}’
fcmsutil ${fdev} | awk ‘/Hardware Path/ {PATH=$5};
/N_Port Node/ {NNODE=$7};
/N_Port Port/ {NPORT=$7};
/Switch Port/ {SPORT=$7};
/Switch Node/ {SNODE=$7};
END{print PATH, NNODE, NPORT, SPORT, SNODE}’
done
Next innovation would be to combine the awk statements in the while loop and pull off the output with a single awk command. I may do this, but I might need help from an awk guru.
The ioscan output is multi line, so the real innovation is using the getline function of awk to get the second line of data and ignore the first. The filter /fcd gets rid of the column format information.
Yes it could be done with grep, but it is more AWKFUL this way.
Updated document to provide some indentation. Trying to make the code easier to read. That may force me to add formatted scripts to download, which I will get to as time permits.
SEP
Tags: Capacity Planning, high capacity volume group, superdome
27 Oct 09 HP-UX Hardware: How to tell your fiber channel adapter is KIA
Important information in bold. Must run as root. Serial numbers changed to protect the innocent.
No electrons were harmed in the creation of this message.
root # fcmsutil /dev/fcd1
Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012c9
PCI Mode = PCI-X 133 MHz
ISP Code version = 3.3.18
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x50060b0000c1bd8b
N_Port Port World Wide Name = 0x50060b0000c1bd8a
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 3/0/12/1/0/4/1
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.06 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Jun 23 2006,14:05:23
Driver state = AWAITING_LINK_UP (Sure sign of a dead card)
root # fcmsutil /dev/fcd1 vpd
V I T A L P R O D U C T D A T A
——— ————- ——-
Product Description : “PCI-X 2-port 2Gb FC/2-port 1000B-T HBA ”
Part number : “AB465-60001”
Engineering Date Code : “A-4620”
Part Serial number : “PQR06501L0”
Misc. Information : “PW=25W;PCI 66MHZ;PCI-X 133MHZ”
Mfd. Date : “4651”
Check Sum : 0xf9
EFI version : “001.47”
ROM Firmware version : “003.003.154”
Asset Tag : “NA”
