This upgrade will start at approximately 2:00am on Wednesday, August 15th. It should be completed before end of day on Friday, August 17th.

Read-only mode outline - getting started 2:35am, 8/15

• (before that night) modify firewall rules as needed between production and test to allow test to work as production - completed 7/28
• Confirm backup completed successfully - completed 2:50am, 8/15
• Take down production and disable cron jobs - completed 2:55am, 8/15
• Re-do backup manually if needed - completed 2:50am, 8/15
• Copy backup volumes to test instances with latest production data - completed 3:30am, 8/15
• Copy production app server keyword directories to test DB server (production has split app server, test does not) - completed 3:30am, 8/15
• Bring up test instances with production IP addresses, and with Oracle tablespace(s) set to read-only - completed 4am, 8/15
  • hvoyapp2 IP address on hvoydb2upg zone
  • Save a copy of /m1/voyager/xxxdb/tomcat/vxws/conf/server.xml for each instance on hvoyapp2 and hvoydb2 (for the RemoteAddr list)
• Verify and test/fix that test instances are working, and looking like production - completed 4am, 8/15
• Migrate app server zone hvoyapp to copley from riggs, and to ZFS pools from UFS mounts - completed 4:15pm, 8/15
• Bring up production instances with test IP adresses - completed 4:15pm, 8/15

Upgrade

bw - 4:15am, 8/15 - taking a break here, will work on bringing up production zones with alternate IP addresses and starting the upgrade later this morning.
bw - 11:45pm, 8/15, breaking off again here for the night, Oracle upgrade is done, regens are running, will pick up after they're done in the morning

• put VIK on the production DB, app and web zones, and use it to download all installation files - in progress 4:30pm, 8/15
• copy the DB server's xxxdb/ini/voyager.env to web server and update as needed - completed 4:30pm, 8/15
• Run the VIK steps on DB server to menu 5 step 16 (Oracle catalog upgrade) - completed 8pm, 8/15
• unmount keyword directory mounts on the DB server - completed 10pm, 8/15
• Run through up to Menu 5 on web server and app server - completed 6:30pm, 8/15
• Finish menu 5 on DB server, and go to index gen on app server - Menu 5 on DB server completed 10:30pm, 8/15
• After index regen is 'setup' by script, manually login to app server, move and symlink from /m1/voyager/utility to /m1/incoming as needed to provide disk space - completed 11:30pm, 8/15
• Modify the REGEN.xxxdb files as needed to enable all indexes to be regenerated instead of the short list - completed 11:30pm, 8/15
• Run menu 6 and start the regen on the app server - started 11:32pm, 8/15 - letting run, will check on them in the morning - completed 3:30am, 8/15 4 hour run time
• On the DB and web server, run through menu 6, skipping steps 9 and 14-17 (steps 3 and 6 will try to access Oracle) - in progress 8:30am, 8/15
• Once regens are done, bring DB server through menu 7 step 2 - completed 4pm, 8/16
• Bring web server and app server through menu 7 step 2 - completed 4pm, 8/16
• Re-mount keyword directory mounts on the DB server here - completed 4pm, 8/16
• Delete any keyword server SMF services created on the DB server - completed 4pm, 8/16
• Do through start Oracle steps on DB server, then start Voyager steps on app server and then the DB server, then start everything else, on web server last - completed 4pm, 8/16
• Modify the /m1/voyager/xxxdb/tomcat/conf/server.xml file to allow all appropriate IP ranges to access VXWS. - completed 5:10pm, 8/16
• Install any needed hotfixes. - completed 5pm, 8/16
• Run post-hotfixes regen - completed 9pm, 8/16 (about 4 hours again)

Test

• At this point the upgraded production zones will be named 'hublib2upg.wisconsin.edu' and 'hvoydb2upg.wisconsin.edu' with the same port range, so the OPAC should be tested as needed, and so should at least one set of clients.

Switch Back production - 1 pm, 8/17

• Reconfigure zone files so that the prod IP addresses will come up on prod after a reboot - completed 12:50pm, 8/17
• Reconfigure Oracle listener config and hosts file for production to come up correctly - completed 12:55pm, 8/17
• Reconfigure firewall if needed - completed 12:55pm, 8/17
• Shut down test/read-only instances - completed 1pm, 8/17
• Reboot the production zones to come up with production IPs - completed 1:05pm, 8/17
• Verify/fix anything needed - completed 1:10pm, 8/17
• Setup newBooks and ShelfLister - completed 2pm, 8/17