Data looks like this
DFW1HCMDB999
dfw1hcmdb998
:%s/.*/\L&/g
Suddenly the data is lower case.
We want storage to check performance on three possibly problematic LUNS.
Need to get the 4 character LUN ID’s on three disks:
disk82 disk83 and disk123
/usr/bin/inq -nodots -sym_wwn | egrep “disk82|disk83|disk123″| awk ‘{print $3}’ |awk ‘{ print substr( $0, length($0) – 3, length($0) ) }’
Output:
5422
5423
5826
HP-UX 11.31 September 2011 OE.
A good day is an awkful day.
So you lose a boot disk on a vxvm booted system. You hot replace the disk, but now you need to make sure the software mirror is good.
Lets say for example its rootdisk02
If the disk stayed alive and did not power down you can run vxbrk_rootmirror and cleanly replace the disk.
What do do if the disk powered down?
/opt/VRTS/bin/vxdiskunsetup -C rootdisk02
vxrecover -g rootdg -sb
You more than likely end up with a disk named like this that is totally useless:
rootdg.26724.31359
vxdg destroy rootdg.26724.31359 (on rootdisk02)
Remove the incomplete mirror copy
/etc/vx/bin/vxbrk_rootmir -v rootdisk02
And mirror it back ,
/opt/VRTS/bin/vxrootmir -v -g rootdg rootdisk02
Tags: hpux 11.23, hpux 11.23 pa-risc, vxvm, vxvm boot disk failure, vxvm boot mirror
This is an improvement to fixing the problems if you do blow things up. Click here to see.
Here is the thing. VXvM is messed up on HP-UX. The mirror break command is broken on 11.23 and 11.31.
That being said depending on how you use it, you can have a mess to clean up or not.
Scenario:
…
Note the disks are supposedly failing. Easy fix, though I can’t say how long this will last.
Now we look at them.
Now they are fixed.
Now to the heart of the matter. Lets say you want to break c2t0d0 out of the mirror and say make a drd image. The man page and HP support says you can use this form.
If you use that form on many HP-UX systems the mirror break will fail and you will have a mess to clean up. If you want to prove your skills go ahead and use that form and click the link above to find the fix.
If you would rather look smart and say cruise the Internet, do this form.
You get the following UGLY results.
However the only thing that actually goes wrong is removing the disk rootdisk02 from the roodg.
Easily fixed with a single command.
vxdg -g rootdg rmdisk rootdisk02
vxdisk list shows:
A healthy ready for DRD cloning rootdg
Tags: how to break up a vxvm mirror without blowing up your rootdg, mirror break, vxvm, vxvm mirror break
Take note of the fcmsutil output below. One of the fiber cards has failed here.
We became aware of the problem due to a resmon message and errors generated on the SAN side.
[shmuel01]/root # olrad -q
Driver(s)
Capable
Slot Path Bus Max Spd Pwr Occu Susp OLAR OLD Max Mode
Num Spd Mode
0-0-0-1 0/0/8/1 140 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-2 0/0/10/1 169 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-3 0/0/12/1 198 266 133 On Yes No Yes N/A PCI-X PCI-X
0-0-0-4 0/0/14/1 227 266 133 On Yes No Yes N/A PCI-X PCI-X
0-0-0-5 0/0/6/1 112 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-6 0/0/4/1 84 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-7 0/0/2/1 56 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-8 0/0/1/1 28 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-1 1/0/8/1 396 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-2 1/0/10/1 425 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-3 1/0/12/1 454 266 133 On Yes No Yes N/A PCI-X PCI-X
0-0-1-4 1/0/14/1 483 266 133 On Yes No Yes N/A PCI-X PCI-X
0-0-1-5 1/0/6/1 368 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-6 1/0/4/1 340 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-7 1/0/2/1 312 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-8 1/0/1/1 284 133 133 Off No N/A N/A N/A PCI-X PCI-X
[sap1pbw0]/root # ioscan -fnCfc
Class I H/W Path Driver S/W State H/W Type Description
===========================================================================
fc 0 0/0/12/1/0/4/0 fcd CLAIMED INTERFACE HP AD193-60001 PCI/PCI-X Fibre Channel 1-port 4Gb FC/1-port 1000B-T Combo Adapter (FC Port 1)
/dev/fcd0
fc 1 0/0/14/1/0/4/0 fcd CLAIMED INTERFACE HP AD193-60001 PCI/PCI-X Fibre Channel 1-port 4Gb FC/1-port 1000B-T Combo Adapter (FC Port 1)
/dev/fcd1
fc 2 1/0/12/1/0/4/0 fcd CLAIMED INTERFACE HP AD193-60001 PCI/PCI-X Fibre Channel 1-port 4Gb FC/1-port 1000B-T Combo Adapter (FC Port 1)
/dev/fcd2
fc 3 1/0/14/1/0/4/0 fcd CLAIMED INTERFACE HP AD193-60001 PCI/PCI-X Fibre Channel 1-port 4Gb FC/1-port 1000B-T Combo Adapter (FC Port 1)
/dev/fcd3
[shmuel01]/root # fcmsutil /dev/fcd0
Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012de
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Previous Topology = UNINITIALIZED
Link Speed = UNKNOWN
Local N_Port_id is = None
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x500110a0003db94b
N_Port Port World Wide Name = 0x500110a0003db94a
Switch Port World Wide Name = 0x0000000000000000
Switch Node World Wide Name = 0x0000000000000000
Driver state = AWAITING_LINK_UP
Hardware Path is = 0/0/12/1/0/4/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17
[shmuel01]/root # fcmsutil /dev/fcd1
Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012de
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0xdc000b
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x500110a0003db94f
N_Port Port World Wide Name = 0x500110a0003db94e
Switch Port World Wide Name = 0x2043000dec8812c0
Switch Node World Wide Name = 0x2046000dec2adb01
Driver state = ONLINE
Hardware Path is = 0/0/14/1/0/4/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17
[shmuel01]/root # fcmsutil /dev/fcd2
Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012de
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0x9d000b
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x500110a0003db93b
N_Port Port World Wide Name = 0x500110a0003db93a
Switch Port World Wide Name = 0x2043000dec881f00
Switch Node World Wide Name = 0x2050000dec2adcc1
Driver state = ONLINE
Hardware Path is = 1/0/12/1/0/4/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17
[shmuel01]/root # fcmsutil /dev/fcd3
Vendor ID is = 0x001077
Device ID is = 0x002422
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012de
PCI Mode = PCI-X 133 MHz
ISP Code version = 4.0.90
ISP Chip version = 3
Topology = PTTOPT_FABRIC
Link Speed = 4Gb
Local N_Port_id is = 0x9d000a
Previous N_Port_id is = None
N_Port Node World Wide Name = 0x500110a0003db8e1
N_Port Port World Wide Name = 0x500110a0003db8e0
Switch Port World Wide Name = 0x2003000dec881f00
Switch Node World Wide Name = 0x2050000dec2adcc1
Driver state = ONLINE
Hardware Path is = 1/0/14/1/0/4/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx & 24xx Driver B.11.23.08 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Mar 27 2007,15:29:17
/usr/bin/olrad -r slot_ID
/usr/bin/olrad -r 0-0-0-3
[shmuel01]/root # olrad -q
Driver(s)
Capable
Slot Path Bus Max Spd Pwr Occu Susp OLAR OLD Max Mode
Num Spd Mode
0-0-0-1 0/0/8/1 140 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-2 0/0/10/1 169 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-3 0/0/12/1 198 266 266 Off Yes Yes Yes N/A PCI-X PCI-X
0-0-0-4 0/0/14/1 227 266 266 On Yes No Yes N/A PCI-X PCI-X
0-0-0-5 0/0/6/1 112 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-6 0/0/4/1 84 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-7 0/0/2/1 56 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-0-8 0/0/1/1 28 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-1 1/0/8/1 396 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-2 1/0/10/1 425 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-3 1/0/12/1 454 266 133 On Yes No Yes N/A PCI-X PCI-X
0-0-1-4 1/0/14/1 483 266 133 On Yes No Yes N/A PCI-X PCI-X
0-0-1-5 1/0/6/1 368 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-6 1/0/4/1 340 266 266 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-7 1/0/2/1 312 133 133 Off No N/A N/A N/A PCI-X PCI-X
0-0-1-8 1/0/1/1 284 133 133 Off No N/A N/A N/A PCI-X PCI-X
Cards to be replaced are flashing yellow.
Replace card now.
/usr/bin/olrad -A 0-0-0-3
/usr/bin/olrad -R 0-0-0-3
Tags: fiber card replacement, fiber channel adaptor, hba replacement, hot swap hba card, rx8640, systems administration
This requires add in software from Veritas/Symmantec that can be expensive on HP-UX.
Lets say we want to extend /var which is lvol8.
It was originally 600 LE and we extended it as follows.
lvextend -l 800 /dev/vg00/lvol8 /dev/disk/disk25_p2
The 800 refers to LE logical extents. I prefer to use this rather than MB because it corresponds to the values displayed by most lvm utilities. -L <value in megabytes> can be used. I assume you can do the math here.
[system099]/root # lvdisplay /dev/vg00/lvol8
— Logical volumes —
LV Name /dev/vg00/lvol8
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 12800
Current LE 800
Allocated PE 1600
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation non-strict
IO Timeout (Seconds) default
It was originally 600 LE and we extended it as follows.
lvextend -l 800 /dev/vg00/lvol8 /dev/disk/disk25_p2
fsadm -F vxfs -b 12800M /var
This command will use online JFS to extend the filesystem. The alternative is extendfs. But that requires a umount. Or perhaps xvumount -o force. Your choice, I don’t recommend it, but did test it the other day in the sandbox.
Now if you want to be cute and use all your space, try this.
SIZE=$(lvdisplay /dev/vg00/lvol8 | awk ‘/LV Size/ {print $NF}’);
ESIZE=”${SIZE}M”
fsadm -F vxfs -b $ESIZE /var
You can get more cute and do it with a single command line but I’m not going to do that. I have not broken any systems this week and am trying to avoid that.
Tags: file system expansion, Online JFS
Scenario:
vxvm boot system shut down decommissioned. It was left off for over a year. Hardware is needed. System is brought up.
Now you have rootdisk2 part of the boot disk group and to do anything useful you need to clean it up.
Get data:
vxprint -ht -g rootdg
I thought to write this only after I was done. So I don’t have a before state picture.
vxdg -g rootdg rmdisk rootdisk02
Getting rid of the sub disks (BE CAREFUL)
sd rootdisk02-11 lpvol-02 rootdisk02 68419584 524288 0 – RMOV
sd rootdisk02-12 – rootdisk02 68943872 3145728 – – RMOV
sd rootdisk02-13 – rootdisk02 63438848 1048576 – – RMOV
pl crashvol-01 – DISABLED – 0 CONCAT – RW
pl homevol-02 – DISABLED – 0 CONCAT – RW
pl optvol-02 – DISABLED – 0 CONCAT – RW
pl standvol-02 – DISABLED – 0 CONCAT – RW
pl swapvol-02 – DISABLED – 0 CONCAT – RW
pl swvol-02 – DISABLED – 0 CONCAT – RW
pl tmpvol-02 – DISABLED – 0 CONCAT – RW
pl usrvol-02 – DISABLED – 0 CONCAT – RW
pl varvol-02 – DISABLED – 0 CONCAT – RW
v crashvol – ENABLED ACTIVE 15728640 SELECT – fsgen
pl crashvol-02 crashvol ENABLED ACTIVE 15728640 CONCAT – RW
sd rootdisk01-13 crashvol-02 rootdisk01 118226944 15728640 0 c0t8d0s2 ENA
v homevol – ENABLED ACTIVE 1048576 SELECT – fsgen
pl homevol-01 homevol ENABLED ACTIVE 1048576 CONCAT – RW
sd rootdisk01-04 homevol-01 rootdisk01 35127296 524288 0 c0t8d0s2 ENA
sd rootdisk01-11 homevol-01 rootdisk01 66322432 524288 524288 c0t8d0s2 ENA
v lpvol – ENABLED ACTIVE 524288 SELECT – fsgen
pl lpvol-01 lpvol ENABLED ACTIVE 524288 CONCAT – RW
sd rootdisk01-12 lpvol-01 rootdisk01 71565312 524288 0 c0t8d0s2 ENA
pl lpvol-02 lpvol DISABLED REMOVED 524288 CONCAT – WO
v optvol – ENABLED ACTIVE 8388608 SELECT – fsgen
pl optvol-01 optvol ENABLED ACTIVE 8388608 CONCAT – RW
sd rootdisk01-05 optvol-01 rootdisk01 35651584 8388608 0 c0t8d0s2 ENA
v rootvol – ENABLED ACTIVE 1048576 SELECT – root
pl rootvol-01 rootvol ENABLED ACTIVE 1048576 CONCAT – RW
sd rootdisk01-03 rootvol-01 rootdisk01 34078720 1048576 0 c0t8d0s2 ENA
pl rootvol-02 rootvol DISABLED RECOVER 0 CONCAT – RW
v standvol – ENABLED ACTIVE 524288 SELECT – fsgen
pl standvol-01 standvol ENABLED ACTIVE 524288 CONCAT – RW
sd rootdisk01-01 standvol-01 rootdisk01 0 524288 0 c0t8d0s2 ENA
v swapvol – ENABLED ACTIVE 33554432 SELECT – swap
pl swapvol-01 swapvol ENABLED ACTIVE 33554432 CONCAT – RW
sd rootdisk01-02 swapvol-01 rootdisk01 524288 33554432 0 c0t8d0s2 ENA
v swapvol2 – ENABLED ACTIVE 46137344 SELECT – fsgen
pl swapvol2-01 swapvol2 ENABLED ACTIVE 46137344 CONCAT – RW
sd rootdisk01-06 swapvol2-01 rootdisk01 72089600 46137344 0 c0t8d0s2 ENA
v swvol – ENABLED ACTIVE 3145728 SELECT – fsgen
pl swvol-01 swvol ENABLED ACTIVE 3145728 CONCAT – RW
sd rootdisk01-10 swvol-01 rootdisk01 63176704 3145728 0 c0t8d0s2 ENA
v tmpvol – ENABLED ACTIVE 2097152 SELECT – fsgen
pl tmpvol-01 tmpvol ENABLED ACTIVE 2097152 CONCAT – RW
sd rootdisk01-07 tmpvol-01 rootdisk01 44302336 2097152 0 c0t8d0s2 ENA
v usrvol – ENABLED ACTIVE 8388608 SELECT – fsgen
pl usrvol-01 usrvol ENABLED ACTIVE 8388608 CONCAT – RW
sd rootdisk01-08 usrvol-01 rootdisk01 46399488 8388608 0 c0t8d0s2 ENA
v varvol – ENABLED ACTIVE 9437184 SELECT – fsgen
pl varvol-01 varvol ENABLED ACTIVE 9437184 CONCAT – RW
sd rootdisk01-09 varvol-01 rootdisk01 54788096 8388608 0 c0t8d0s2 ENA
sd rootdisk01-14 varvol-01 rootdisk01 66846720 1048576 8388608 c0t8d0s2 ENA
Time to clean up the mess we made. attach and disassociate properly.
532 vxplex -o force -g rootdg att swvol swvol-02
533 vxplex -o force -o rm -g rootdg dis swvol-02
575 vxprint -ht -g rootdg | grep RMOV | awk ‘{print $2}’ | while read -r sd
do
vxedit -g rootdg rm $sd
done
vxedit -g rootdg rm rootdisk02
I’ve just been through another frustrating battle with swinstall and wanted a complete what to check list in the event that it won’t install software:
Tags: /etc/hosts, hpux, swinstall
Lets say we have a file system mounted as /stuck
umount /stuck
Mount Point busy.
fuser -cu /stuck
# shows no open processes on the filesystem. umount won’t work.
umount has no documented -f (force) paramter
vxumount is the tool of choice here.
vxumount -o force /stuck
/stuck is no longer stuck
vxmount is not always in the PATH. It is something worth searching for and is available on most HP-UX systems.
Only works on vxfs file systems, not HFS or NFS.
Ever try and do system reporting based on bdf?
Annoying as all heck that sometimes the output is two line and sometimes one? I was forced to solve that problem today.
if [ “$OS” = “HP-UX” ]
then
dcmd=bdf
fi
arraypointer=0
exec $dcmd | egrep -v “%used|/dev/deviceFileSystem” | awk ‘{lvn=$1;v=$2;if (v==””) {getline;cap=$1;ucap=$2;acap=$3;puse=$4;mp=$5;printf “%s %s %s %s %s %s\n”, lvn,cap,ucap,acap,puse,mp} else {printf “%s %s %s %s %s %s\n”, $1,$2,$3,$4,$5,$6} }’ | while read -r p1 p2 p3 p4 p5 p6
do
#### calculations
done
Typical bdf output:
/dev/vg00/lvol9 4096000 3140019 896285 78% /var/adm/crash
/dev/vg_stgb1/lvol1
1572765696 1382813225 178080447 89% /steven05stgb
More when the script is done
Tags: awk, awk with if else logic, bdf, hpux, systems administration