I'm copying in from Stout's - please don't mind any copy/paste inaccuracies I haven't fixed yet.

Think of this as my high level plan for Stout's upgrade. It's going to be slightly different from the other upgrades in some details, mainly because they're getting split into their own zone, rather than a shared one, or staying in the same zone they're using now.

Here are the high level steps for preparation

1. Create new enterprise storage for Milwaukee's read-only zone (completed August 2nd, 2008)
2. Copy enterprise storage mountpoints from previous day's backups (completed August 2nd, 2008)
   
3. Create new milvoydb zone on Widener for readonly (in progress August 2nd, 2008)
   
1. Configure zone from milvoydb zone on ambrosia (completed August 2nd, 2008)
   
2. Take down zone Tuesday at 2:10am on ambrosia (completed August 5th, 2008)
   
3. Detach milvoydb zone on ambrosia (completed August 5th, 2008)
   
4. Copy over milvoydb zoneroot on ambrosia to widener (completed August 5th, 2008)
   
5. Enable firewall on widener and copy over rules from ambrosia (completed August 5th, 2008)
   
6. Modify firewall rules on ambrosia to allow 1521 traffic in to new zone's temporary IP address (completed August 5th, 2008)
   
7. Modify milvoydb zone on ambrosia for upgrade and temp IP address (completed August 5th, 2008)
   
1. Modify Oracle listener configuration (completed August 5th, 2008)
   
2. Saved modified copy to home directory (made backup copy in oracle directory)
   
3. Attempt to restart up Oracle and troubleshoot as needed (completed August 5th, 2008)
4. Had to set the lofs filesystems to mount up with seteuid and setuid bits allowed
   
5. Verify not connecting to current instance of Oracle (put in firewall rule - (completed August 5th, 2008))
6. Modify Voyager configuration (list files here as doing it)
7. On db server - verified everything referred to localhost - no 'voyapp' or 'voydb' entries (grep -i voyapp */*...) (completed August 5th, 2008)
   
8. Attempt to startup Voyager and troubleshoot as needed
   
9. Verify not connecting to current instance of Oracle again
4. Create new millib zone on widener for readonly
   
1. Configure millib zone on widener from millib on alexandria (completed August 2nd, 2008)
2. Take millib zone down on alexandria at 2:10am on day of upgrade (completed August 5th, 2008)
   
3. Detach millib zone on alexandria (completed August 5th, 2008)
   
4. Copy zoneroot from alexandria to widener (completed August 5th, 2008)
   
5. Attach zone on widener and start it up (completed August 5th, 2008)
   
6. Test millib zone on widener with milvoydb zone on widener (completed August 5th)
7. On millib on alexandria for upgrade -
   
1. Configure millib zone on alexandria with temp IP address (completed August 5th)
   
2. Modify apache configuration on millib on alexandria for new IP address (completed August 5th
1. /m1/uwmapache/conf/httpd.conf
   
2. /m1/uwmapache/conf/extra/httpd-ssl.conf
3. On web server - modify /m1/voyager/milwaukeedb/etc/webvoyage/voyager.ini and /m1/voyager/rescatdb/etc/webvoyage/voyager.ini
   
8. Start apache and troubleshoot as needed (completed August 5th, 2008)
   
9. Verify Milwaukee's webvoyage working (completed August 5th, 2008)
   

Prep work completed - I have the milvoydb/milvoyapp and millib zones up and running on widener. Keyword searches through webvoyage seem to be working and sorting properly (it took about 5-10 minutes for the keyword service to finish initial indexing). Only catches I ran into were zone configuration and filesystem link issues (/opt/zones was symlinked to /zoneroot, which milvoydb didn't like, /usr/local/bin/coraenv, oraenv, dbhome links didn't exist on widener, they do now, I had previously configured the zones to use the temp IP addresses).

Now I'm going to work on bringing up the milvoydb/milvoyapp on ambrosia again - with temporary IP address.

That's completed - There's something still trying to connect to the read-only database running on widener - but it's firewalled off now from allowing it to connect and still seems to be working, so I don't believe it matters.

Going to get the upgrade stuff into place now and get it kicked off. - 4:28am, 8/5/2008

As seen below - the regen is off and running. It's chewing up 1 CPU core on ambrosia at this point pretty steadily, so it's working along alright I think. - 6:29am, 8/5/2008

Now at the point where Webvoyage seems to be working with the newly upraded zones - have to hack millib.wisconsin.edu to the temp address in the hosts file to test it out. Ready for client testing first thing in the morning.

Point client at hubtmp.doit.wisc.edu (same port numbers as old) and test out the clients.
-bw 11:38pm, August 5th, 2008

Here are the high level steps for the actual upgrade - starting at 2am, Monday July 21st

1. After the nightly hub backup is run normally, do the detach, attach, rsync notes above for all storage for milvoydb and millib putting them onto widener (completed August 5th, 2008)
   
2. Bring up milvoydb and millib zones on widener (completed August 5th, 2008)
   
3. Set the milvoydb zone instance to read only (completed August 5th, 2008)
   
1. Read only mode notes
2. Comment out cron jobs in milvoydb and millib on widener (completed August 5th, 2008)
4. Setup milvoydb and millib zones on ambrosia and alexandria with temp IP addresses (completed August5th, 2008)
5. Modify files as needed on upgrade zones - milvoydb on ambrosia and millib on alexandria (see prep steps)
6. Boot zones on milvoydb and milib on ambrosia/alexandria (completed August 5th, 2008)
   
7. Start everything up in upgrade zones and troubleshoot as needed (completed August 5th, 2008)
   
8. In the milvoydb zone on ambrosia kick off the voy654 patch (milwaukeedb, rescatdb) - (completed August 5th, 2008)
9. In the milvoydb zone on ambrosia run the run_as_root.ksh script provided (completed August 5th, 2008)
10. In the new milvoydb zone fix anything the run_as_root.ksh script breaks (completed August 5th, 2008)
1. Had to manually install their service methods into global zone's /lib/svc/method directory and fix ownership/permission (completed August 5th, 2008)
2. Had to manually clear listener service - svcadm clearELGolisten (completed August 5th, 2008)
11. In the upgrade milvoydb zone run index regen (their directions) (completed 11:27am, August 5th 2008)
1. It's sitting at the 'Performing Indexgen' step. Guessing it will be there quite a while.
2. Going to query Exlibris if it's safe to kick of RescatDB index regen at the same time or not.
   
12. In the upgrade millib zone kick off the voy654 patch (completed 11pm, August 5th, 2008)
1. Had to run it 3 times before it was successful. - true again
   
13. In the upgrade millib zone run the run_as_root.ksh script (completed 11pm, August 5th, 2008)
    
14. In the upgrade millib zone fix anything the run_as_root.ksh script breaks (completed August 5th, 2008)
1. Did not enable the web service because we're using a separate apache for SSL
   
15. In the z3950 zone kick off the voy654 patch (completed August 6th, 2008)
16. In the z3950 zone run the run_as_root.ksh script if necessary(completed August 6th, 2008)
17. In the z3950 zone fix anything the run_as_root.ksh scripts breaks(completed August 6th, 2008)
18. Contact UW Milwaukee (ready for this at 11:36pm, August 5th, 2008)
    
1. Have them point staff clients at hubtmp address for the upgraded milvoydb zone and test(completed August 6th, 2008)
2. Switch live IP address to the upgraded milvoydb zone(completed August 6th, 2008)
3. Switch live IP address over to new upgraded millib zone (completed August 6th, 2008)
4. Modify Oracle listener and tnsnames configuration files above and restart if needed(completed August 6th, 2008)
   
5. Have them walk me through testing the new Webvoyage zone(completed August 6th, 2008)
6. In the z3950 zone update the keyword files from the milvoydb zone(completed August 6th, 2008)
19. Verify everything is working, all cron jobs running, and backups configured (everything but backups completed)
    
20. Verify backups run that night as planned, troubleshoot as necessary (in progress as of August 7th, 2008)