I'm copying and pasting from Stout's upgrade plan here - please forgive any places I don't fix the copy and paste.

This group of the Hub, I've started calling the '2nd Hub' in order to shorten 'Whitewater, River Falls, Parkside, and Colleges'. If I say '2nd hub' or 'hub2', it's referring to the new shared instance that these four are using.

Think of this as my high level plan for the 2nd hub's upgrade. It's going to be slightly different from the other upgrades in some details, mainly because they're getting split into their own zone, rather than a shared one, or staying in the same zone they're using now.

Here are the high level steps for preparation

1. Create new enterprise storage for the second hub's zone (completed August 10th, 2008)
2. Create new hvoydb2 zone
   1. Build zone to look like hvoydb (completed August 10th, 2008)
      
   2. Copy enterprise storage mountpoints from previous day's backups (completed August 10th, 2008)
   3. Modify firewall rules for garrison to allow 1521 traffic in to new zone's temporary IP address (completed August 10th, 2008)
   4. Add users and groups, shared memory, /etc/system, /etc/services, /etc/inetd lines for the 2nd hub members, /etc/init.d scripts from hvoydb (completed August 10th, 2008)
      
   5. Modify Oracle listener configuration (completed August 10th, 2008)
      1. Saved modified copy to home directory (completed August 10th, 2008)
         
   6. Attempt to start up Oracle and troubleshoot as needed (completed August 10th, 2008)
      1. Had to set the lofs filesystems to mount up with seteuid and setuid bits allowed (completed August 10th, 2008)
         
   7. Verify not connecting to current instance of Oracle (completed August 10th, 2008)
      
   8. Modify Voyager configuration (list files here as doing it)
      1. On web server - modified /m1/voyager/xxxdb/etc/webvoyage/voyager.ini (need read-only instance up first at 2am)
      2. On db server - verified everything referred to localhost - no 'voyapp' or 'voydb' entries (grep -i voyapp */*...) (completed August 10th, 2008)
         
   9. Attempt to startup Voyager and troubleshoot as needed (completed August 10th, 2008)
      
   10. Verify not connecting to current instance of Oracle (completed August 10th, 2008)
3. Create new hublib2 zone
   1. Clone hublib zone (completed August 8th, 2008)
   2. Use production IP addresses for the four members of this hub instance (completed August 8th, 2008)
   3. Modify the apache configuration to supply these four members apache instances and not the rest (completed August 8th, 2008)
   4. Start apache and troubleshoot as needed (completed August 8th, 2008)
   5. Verify hublib2 webvoyage is working pointing to production zone (hvoydb.wisconsin.edu) (completed August 8th, 2008)
   6. Modify Voyager and apache configuration to use the hvoydb2 zone instead
   7. Start apache
   8. Verify Stout's webvoyage working pointed to new stovoydb zone
      1. Had to modify my local hosts file to point xxxlib.wisconsin.edu at it's temporary address for testing
         

Prep work in progress for building new zones.
-bw 05:50pm, August 10th, 2008

New zone builds are complete. The hublib2 zone is up and running as the current live production webserver for these four, so it can't be reconfigured until after 2am when the upgrade starts and the readonly instance is up. The hvoydb2 zone has been tested with colleges using the ascii based opac at the command line and appears to work. All should be set for upgrade to start at 2am, with some IP address shuffling.

-bw 10:26pm, August 10th, 2008

Read only instances are back up now. Starting work on the upgrade zones and verifying they're working.
-bw 3:47am, August 11th, 2008

Have both the read only instances and the upgrade instances up and running. Going to do some work on the upgrade instance before kicking off the upgrade.
-bw 4:23am, August 11th, 2008

Cleanup work done on the upgrade instances. Starting up the upgrade.
-bw 5:14am, August 11th, 2008

Regens are running for all four second hub instances in the upgrade zone - kicked off between 6:00am and 6:10am.
-bw 6:11am, August 11th, 2008

Everything's set, webvoyage looks good and the regens all finished and are pushed into place. I've contacted all four locations and am waiting to hear back if the client testing goes alright, then will try to pick a time to switch over to live (if all is well).
-bw 1:20pm, August 11th, 2008

Here are the high level steps for the actual upgrade - starting at 2am, Monday July 21st

1. After the nightly hub backup is run normally, set the second hub schools to read-only mode - the current install will be the read-only OPAC used for the second hub's upgrade (this will be different from other upgrades in this cycle) (completed August 11th, 2008)
2. Read only mode notes
3. Comment out the second hub's cron jobs on hvoydb read only zone - add them to hvoydb2 zone commented out (will uncomment after upgrade is complete) (completed August 11th, 2008)
4. Save any changed configuration files on /oracle Copy the database mounts (/oracle and oracle data partitions) over from the backup just made via rsync (completed August 11th, 2008)
5. Put modified files in place for listener, reconfigure oracle, and put the voyager.ini file in place on the new web zone
   1. /oracle/app/oracle/product/9.2.0/network/admin/tnsnames.ora (db) (completed August 11th, 2008)
      
   2. /oracle/app/oracle/product/9.2.0/network/admin/listener.ora (db) (completed August 11th, 2008)
      
   3. /m1/voyager/xxxdb/etc/webvoyage/voyager.ini (web) (completed August 11th, 2008)
      
6. Start everything up in new zones and troubleshoot as needed (completed August 11th, 2008)
   
7. Verify with new hublib2 zone that the new hvoydb2 zone is working (completed August 11th, 2008)
   1. Will have to rotate through all four instances with only 1 IP address available
   2. Actually - reconfigured the apache configs for the upgrade ot use NameVirtualHost directives to split out to the various web servers, working well with hacked hosts file on workstation. (completed August 11th, 2008)
8. Verify upgrade instance isn't talking to read only instance (completed August 11th, 2008)
9. In the new hvoydb2 zone kick off the voy654 patch (follow their directions) (completed August 11th, 2008)
10. In the new hvoydb2 zone run the run_as_root.ksh script provided (completed August 11th, 2008)
11. In the new hvoydb2 zone fix anything the run_as_root.ksh script breaks (completed August 11th, 2008)
12. 1. Had to manually install their service methods into global zone's /lib/svc/method directory and fix ownership/permission
    2. Had to manually clear listener service - svcadm clear ELGolisten
13. In the new hvoydb2 zone run index regen (their directions) (completed August 11th, 2008)
14. In the new hublib2 zone kick off the voy654 patch (completed August 11th, 2008)
15. Had to run it 3 times before it was successful. (completed August 11th, 2008)
16. In the new hublib2 zone run the run_as_root.ksh script (completed August 11th, 2008)
17. In the new hublib2 zone fix anything the run_as_root.ksh script breaks (completed August 11th, 2008)
1. enabled the web service
   
18. In the z3950 zone kick off the voy654 patch
19. In the z3950 zone run the run_as_root.ksh script if necessary
20. In the z3950 zone fix anything the run_as_root.ksh scripts breaks
21. In the new hvoydb2 zone drop all tablespaces that aren't hvoydb2 (completed August 11th, 2008)
    1. Drop tablespaces notes
22. Contact the four second hub members (completed August 11th, 2008)
    1. Have them point staff clients at hvoydb2 address for the new hvoydb2 zone and test(completed August 11th, 2008)
    2. Switch live IP address to the new hvoydb2 zone (completed August 11th, 2008)
    3. Have them walk me through testing the new Webvoyage zone (completed August 11th, 2008)
    4. Modify Oracle listener and tnsnames configuration files above and restart if needed(completed August 11th, 2008)
    5. In the z3950 zone update the keyword files from the hvoydb2 zone (completed)
    6. In the z3950 zone update the oracle config for the second hub members(completed August 11th, 2008)
    7. Had to modify /oracle/app/oracle/product/9.2.0/network/admin/tnsnames.ora (add VGERHUB2 entry)
    8. Had to modify /var/opt/oracle/oratab (add VGERHUB2 entry)(completed August 11th, 2008)
    9. Had to modify /m1/voyager/xxxdb/ini/voyager.env (first VGER entry in login string changed to VGERHUB2)(completed August 11th, 2008)

1. Switch live IP address over to new hublib2 zone(completed August 11th, 2008)
2. Verify everything is working, all cron jobs running, and backups configured (completed August 11th, 2008)
   
3. Verify backups run that night as planned, troubleshoot as necessary