This upgrade will start at approximately 2:00am on Friday, August 10th. It should be completed before end of day on Monday, August 13th.

bw - 2:20am, 8/10 - getting started
Read-only mode outline

• (before that night) modify firewall rules as needed between production and test to allow test to work as production - completed 7/28
• Confirm backup completed successfully - completed 2:25am, 8/10
• Comment out voyager user and backup cron jobs in global zone - completed 2:25am, 8/10
• Take down production - completed 2:25am, 8/10
• Re-do backup manually if needed - completed 2:30am, 8/10
• Copy backup volumes to test instances with latest production data - in progress 2:35am, 8/10
• Copy production app server keyword directories to test DB server (production has split app server, test does not) - started 2:55am, completed 4:20am, 8/10
• Update /etc/inet/hosts for hvoyapp entry so it points at hvoydb IP address (then can skip next step) - completed 4:45am, 8/10
• Update /m1/voyager/xxxdb/ini/voyager.ini to point mfhd and data keyword servers at localhost - completed 4:25am, 8/10
• Bring up test instances with production IP addresses, and with Oracle tablespace(s) set to read-only - completed 5:06am, 8/10
  • Make sure endeavor group still exists for old definitions of Global_bin service
  • Make sure /m1/voyager/xxxdb mount point isn't pointing to old data/volume
  • Add lines to ipf.conf to block traffic into read-only oracle listener from test IP addresses
• Verify and test/fix that test instances are working, and looking like production - completed 5:06am, 8/10
• Destroy and copy backup database data files to production instance (after re-verifying backup) - to take advantage of proper ZFS block size - completed 3:10am, 8/10
• Modify production zones as needed to come up on test IP addresses - completed 8:10am, 8/10
  • each zones /etc/inet/hosts entries for all IP addresses in the zone set (5 for web, 5 for db)
  • /m1/shared/apache2/conf/ActivatedVirtualHosts/*.conf modified
• Verify that production zones are working using the test IP addresses - completed 8:30am, 8/10
  • Modify production firewall rules needing to be updated for the "*upg" IP addresses to work on the production servers
  bw - 5:10am, 8/10 - taking a break here, will get the prod zones up and running with test IP addresses and get the upgrade started later this morning. bw - 7:30am, 8/10 - back from break, working on production zones

Upgrade
bw - 8:30am, 8/10 - short break here. Then will start with the VIK upgrade setup and steps
bw - lunch break ~noon-1pm, 8/10

• (before that night) put VIK on the production DB, app and web zones, and use it to download all installation files - completed 10:54am, 8/10
• (before that night) copy the DB server's xxxdb/ini/voyager.env to web server and update as needed - completed 10:54am, 8/10
• Run the VIK steps on DB server to menu 5 step 16 (Oracle catalog upgrade) - that will take a while, so while it's running do - completed 3pm, 8/10
• unmount keyword directory mounts on the DB server - completed 1:45pm, 8/10
• Run through up to Menu 5 on web server and app server - completed 3pm, 8/10
• Finish menu 5 on DB server, and go to index gen on app server - completed 3pm, 8/10
• After index regen is 'setup' by script, manually login to app server, move and symlink from /m1/voyager/utility to /m1/incoming as needed to provide disk space - completed 4pm, 8/10
• Modify the REGEN.xxxdb files as needed to enable all indexes to be regenerated instead of the short list - completed 4pm, 8/10
• Run menu 6 and start the regen on the app server - in completed 6:30pm, 8/10, regens started at 4pm, finished 6:30pm 8/10
  • VIK runs them in series, not in parallel, just left them to run overnight.
• On the DB and web server, run through menu 6, skipping steps 9 and 14-17 (steps 3 and 6 will try to access Oracle) - completed 7:45am, 8/11
• Once regens are done, bring DB server through menu 7 step 2 - completed 7:45am, 8/11
• Bring web server and app server through menu 7 step 2 - completed 7:45am, 8/11
• Re-mount keyword directory mounts on the DB server here - completed 7:45am, 8/11
• Delete any keyword server SMF services created on the DB server - completed 8am, 8/11
• Do through start Oracle steps on DB server, then start Voyager steps on app server and then the DB server, then start everything else, on web server last 0 - completed 8:15am, 8/11
• Modify the /m1/voyager/xxxdb/tomcat/conf/server.xml file to allow all appropriate IP ranges to access VXWS. - in progress 8:15am, 8/11
• Delete Pjobd and Pgdc servces from app server (?) - completed 9:15am, 8/11
• Install any needed hotfixes. - completed 9:30am, 8/11
• Put the new services under TCP wrappers on DB and app servers - completed 8:30am, 8/11
• Migrate app server zone hvoyapp to copley from riggs, and to ZFS pools from UFS mounts - started 9:30am, completed 2pm, 8/11
  • Firewall rules, zone detach, zpool export/import, zone attach, migrate NFS shares to copley, update hvoydb /etc/vfstab to copley

Test

• At this point the upgraded production zones will be named 'hublibupg.wisconsin.edu' and 'hvoydbupg.wisconsin.edu' with the same port range, so the OPAC should be tested as needed, and so should at least one set of clients.
• Need for additional hotfix identified - coordinating when to do hotfix and regen versus the switchover to live.

Switch Back production

• Reconfigure zone files so that the prod IP addresses will come up on prod after a reboot - completed 11:50am, 8/13
• Reconfigure Oracle listener config and hosts file for production to come up correctly - completed 11:50am, 8/13
• Reconfigure firewall if needed - completed 11:55am, 8/13
• Shut down test/read-only instances - completed 12:05pm, 8/13
• Reboot the production zones to come up with production IPs - completed 12:10pm, 8/13
• Verify/fix anything needed - in progress/completed 12:10pm, 8/13
• uncomment voyager and backup cron jobs - completed 12:30pm, 8/13
• Bring up test zone as 7.2.3 as needed - not doing at this time
• Install newBooks on new instance (not tab, just files and script) and test script - completed 12:30pm, 8/13
• Reconfigure and/or test Shelflister on new instance - completed 12:45pm, 8/13
• Final hotfix - fix staff client search failures, etc. - scheduling for 6pm, 8/13
• Copy hotfix to server before
• Take down Voyager
• Run hotfix script
• Bring up Voyager
• Test things are functional
• Run regens