Introduction
We recently upgraded one of our customers from APEX 5.1.2 to 18.1. In this post, I will make the case why you should consider upgrading APEX, describe the planning and preparation that goes into an APEX upgrade, describe the upgrade itself and finally take a look at some of the issues encountered during this particular upgrade. While many of my points are applicable to any APEX upgrade, some are specific to upgrading from APEX 5.1 to 18.1.
Why Upgrade APEX?
Before I start, let's cover some of the reasons you may want to upgrade APEX:
- New Functionality - These days, technology and business are changing constantly. Upgrading to the latest release of APEX provides you with the latest features so you are ready to meet these demands.
- Staying Current - The longer you leave it between upgrades the more complex the upgrade will be.
- Security - If security issues are reported they are typically resolved in newer versions of APEX.
Background
The customer in question uses APEX extensively to extend Oracle e-Business Suite. For non-production, they have a single ORDS server (running under Tomcat) which services the development and three other non-production EBS databases. There is a separate Tomcat/ORDS server which is dedicated to production.
The customer has a total of 20 APEX Applications, more than half of which are tightly integrated with EBS. The average daily page views are around 37K with up to 200 different users logging into an APEX application every day. Many of these applications are business critical, so bugs and long outages were out of the question. Given these constraints, we obviously couldn't just download the latest version and give it a whirl. We needed a plan and some careful preparation.
APEX Architecture
The customer has a total of 20 APEX Applications, more than half of which are tightly integrated with EBS. The average daily page views are around 37K with up to 200 different users logging into an APEX application every day. Many of these applications are business critical, so bugs and long outages were out of the question. Given these constraints, we obviously couldn't just download the latest version and give it a whirl. We needed a plan and some careful preparation.
APEX Architecture
Planning
This may sound obvious but having a well-documented (and well-communicated) plan is critical to the success of any upgrade. You can safely assume that if it is not in the plan (and you did not communicate the plan) then no one knows it is going to happen.
The plan should clearly lay out the following:
The plan should clearly lay out the following:
- Who is accountable for what and exactly when should they be doing it?
- What communications should happen and when should they be sent?
- What should happen if it all goes horribly wrong?
- When instances need to be cloned.
- When outages will occur and how long they will be for.
- How long activities should take. (Note: These timings should be updated as you go through the process, so you have an accurate timing for the production cutover).
Preparation
Prepare your APEX Apps
There are a number of things you can do before the upgrade. Probably the most important of which is to bring all of your existing Apps as up to date as possible. The goal here is to do as much of the work as you possibly can before you even start the upgrade. For this particular upgrade, this meant the following:
Prepare your Test Scripts
Make sure all of your regression test scripts are documented before the upgrade. Also make sure that you have documented how to log into each application before the upgrade. You do not want to waste time trying to figure out how to login to application X, once the upgrade is underway.
Stage the Install Files
Stage the APEX install zip on the database server prior to the upgrade. You may also want to create a new table space (for the new APEX schema) ahead of time. You should also stage the images folder from the APEX install ZIP on the ORDS server. These steps will save you minutes during the upgrade.
There are a number of things you can do before the upgrade. Probably the most important of which is to bring all of your existing Apps as up to date as possible. The goal here is to do as much of the work as you possibly can before you even start the upgrade. For this particular upgrade, this meant the following:
- Make sure all Apps are using the Universal Theme
- Make sure all Apps are using the latest subscription to the Universal Theme
- Make sure all of your Apps are running with 'Include Legacy JavaScript' and 'Include jQuery Migrate' switched off
- Make sure all plugins are on the latest version
- Remove any un-used plugins
- Figure out if you can replace any plugins with native functionality introduced in the new version
- Review the release notes for APEX 18.1 to see if there was any code we could remediate ahead of the upgrade
Prepare your Test Scripts
Make sure all of your regression test scripts are documented before the upgrade. Also make sure that you have documented how to log into each application before the upgrade. You do not want to waste time trying to figure out how to login to application X, once the upgrade is underway.
Stage the Install Files
Stage the APEX install zip on the database server prior to the upgrade. You may also want to create a new table space (for the new APEX schema) ahead of time. You should also stage the images folder from the APEX install ZIP on the ORDS server. These steps will save you minutes during the upgrade.
The Upgrade
Non-Production
The upgrade process itself will differ for every implementation. One of our core challenges was to upgrade an environment, isolate it and perform necessary testing and remediation without impacting the other non-production environments. This is only really a challenge because our customer has one Tomcat Server, serving four databases. For 364 days a year, this is an optimal configuration. During an upgrade, however, it is pretty inconvenient. The problem being, the moment you upgrade one database to 18.1 and then update the images folder on the ORDS server, the other three environments will stop working (see APEX architecture diagram above). To get around this, we took the sandbox instance out of the ORDS url mapping file, stood up a standalone version of ORDS and pointed it to the upgraded sandbox database. This gave us time to iterate through testing and remediating issues until we were comfortable with proceeding on to upgrade the other 3 non-production instances.
The upgrade process itself will differ for every implementation. One of our core challenges was to upgrade an environment, isolate it and perform necessary testing and remediation without impacting the other non-production environments. This is only really a challenge because our customer has one Tomcat Server, serving four databases. For 364 days a year, this is an optimal configuration. During an upgrade, however, it is pretty inconvenient. The problem being, the moment you upgrade one database to 18.1 and then update the images folder on the ORDS server, the other three environments will stop working (see APEX architecture diagram above). To get around this, we took the sandbox instance out of the ORDS url mapping file, stood up a standalone version of ORDS and pointed it to the upgraded sandbox database. This gave us time to iterate through testing and remediating issues until we were comfortable with proceeding on to upgrade the other 3 non-production instances.
Upgrading an APEX Application
Once you have run the APEX upgrade scripts, there are a few steps you need to run for each of your APEX Apps. This is in addition to any remediation that comes out of regression testing.
Refresh the Theme
Assuming you are using the Universal theme, you should refresh the theme subscription to the master theme. This will ensure you application is using the latest templates and template options from APEX 18.1.
Once you have run the APEX upgrade scripts, there are a few steps you need to run for each of your APEX Apps. This is in addition to any remediation that comes out of regression testing.
Refresh the Theme
Assuming you are using the Universal theme, you should refresh the theme subscription to the master theme. This will ensure you application is using the latest templates and template options from APEX 18.1.
Legacy JavaScript and jQuery in APEX 18.1
During the upgrade, Oracle turns on legacy JavaScript support for every application by default. We chose to take the extra time to turn all of these options off prior to our testing and remediation phase. We believe this puts our customer in a much better place for future upgrades and the inevitable 18.1.1 (or 18.2) bug fix release.
Testing
Once all of the non-production instances were upgraded we handed over to the business to run through their test scripts. It is very important (but rarely easy) to involve the business in testing and infuse in them a sense of ownership and accountability in the entire process. In fact, our biggest issue was discovered during business testing (described in the Issues section below). In terms of hours, our testing was pretty evenly split between the initial testing and remediation cycle and business testing.
Handling Production Issues During the Non-Production Upgrade
Of course life goes on during an upgrade, bugs crop up and need to be dealt with. This is where communication of the process to everyone (including the business) really pays off. If users know you are upgrading and potentially improving their lives they are more likely to be ok with holding off on a bug fix until after the upgrade.
Sometimes a bug really needs to be fixed right now. You will need to accommodate this while you are upgrading and testing the non-production environments. This can be achieved by isolating a non-production instance (without upgrading it) and give it its own ORDS server so it can serve as an environment to make and test break fix issues during this period. Don't forget to merge any bug fixes into your upgraded code!
Production Cutover
The goal is to keep the production outage window as small as possible whilst keeping risk to a minimum. To achieve this, you need to automate as much as you can. The upgrades of your test instances should be scripted, and you should be actively updating your scripts and run book throughout the process. All remediated code should be packaged into an automated deployment which can be quickly deployed immediately after the upgrade. In our case, we were able to complete the APEX 18.1 upgrade in less than 10 minutes (including updating the images on the ORDS server). We then ran an automated deployment of all of the upgraded application which again took under 10 minutes. Finally, we then smoke tested for 10 minutes before releasing the system back to the users for a total elapsed time of 30 minutes.
Top Tips:
Once all of the non-production instances were upgraded we handed over to the business to run through their test scripts. It is very important (but rarely easy) to involve the business in testing and infuse in them a sense of ownership and accountability in the entire process. In fact, our biggest issue was discovered during business testing (described in the Issues section below). In terms of hours, our testing was pretty evenly split between the initial testing and remediation cycle and business testing.
Handling Production Issues During the Non-Production Upgrade
Of course life goes on during an upgrade, bugs crop up and need to be dealt with. This is where communication of the process to everyone (including the business) really pays off. If users know you are upgrading and potentially improving their lives they are more likely to be ok with holding off on a bug fix until after the upgrade.
Sometimes a bug really needs to be fixed right now. You will need to accommodate this while you are upgrading and testing the non-production environments. This can be achieved by isolating a non-production instance (without upgrading it) and give it its own ORDS server so it can serve as an environment to make and test break fix issues during this period. Don't forget to merge any bug fixes into your upgraded code!
Production Cutover
The goal is to keep the production outage window as small as possible whilst keeping risk to a minimum. To achieve this, you need to automate as much as you can. The upgrades of your test instances should be scripted, and you should be actively updating your scripts and run book throughout the process. All remediated code should be packaged into an automated deployment which can be quickly deployed immediately after the upgrade. In our case, we were able to complete the APEX 18.1 upgrade in less than 10 minutes (including updating the images on the ORDS server). We then ran an automated deployment of all of the upgraded application which again took under 10 minutes. Finally, we then smoke tested for 10 minutes before releasing the system back to the users for a total elapsed time of 30 minutes.
Top Tips:
- Clone production to your non-production instances just before the upgrade. This will ensure you are working on the latest code (and data).
- Engage the business in the entire upgrade process (especially testing).
- Don't be afraid to sell the features of the latest version of APEX. Get people excited about what they will get from the new version rather than focusing on the inconvenience of the upgrade itself.
- Read the release notes carefully. They provide useful pointers to what code needs to be remediated and where you need to focus your testing.
- Automate as much as you possibly can and stage the install files prior to the install.
Issues
As great as the APEX team is at ensuring your apps survive an upgrade, for systems with any real complexity, there are always some issues. if you read my previous blog on Building APEX Applications Responsibly, you will not be surprised where most of these issues came from.
Plugins
The customer was using six different plugins. We encountered one fatal issue where a plugin was using de-supported jQuery. Luckily, we were able to replace this with native functionality.
JavaScript / jQuery
We ran into three occurrences of de-supported jQuery being used which we had to re-code.
Native APEX Issues
We encountered the following issues in native 18.1 code:
Plugins
The customer was using six different plugins. We encountered one fatal issue where a plugin was using de-supported jQuery. Luckily, we were able to replace this with native functionality.
JavaScript / jQuery
We ran into three occurrences of de-supported jQuery being used which we had to re-code.
Native APEX Issues
We encountered the following issues in native 18.1 code:
- The field type 'Text Field with autocomplete' (when used with a dependent list of values) does not work in an Interactive Grid in APEX 18.1. Oracle development is aware, and we have our fingers crossed for a fix in APEX 18.2.
- We encountered an issue with an Interactive Grid in a modal window freezing in Internet Explorer. We were able to make this page a non-modal page and work around this issue.
Conclusion
For a complex architecture such as this, we felt the upgrade went smoothly. It was certainly less painful than the upgrade from APEX 4.2 to APEX 5.0 and even a little easier than the APEX 5.0 to 5.1 upgrade. Hats of to the APEX development team for continuing to prioritize the ease of upgrade of APEX.
While this may not have been the most scintillating blog post you read this week, hopefully it inspired you to consider upgrading to APEX 18.1. If you do upgrade, then please take the time to plan it thoroughly and involve the business in the process from start to finish.
While this may not have been the most scintillating blog post you read this week, hopefully it inspired you to consider upgrading to APEX 18.1. If you do upgrade, then please take the time to plan it thoroughly and involve the business in the process from start to finish.
Author
Jon Dixon, Co-Founder of JMJ Cloud
RSS Feed