Tuning GlassFish for Oracle APEX & ORDS in Production

Introduction

Installing GlassFish and deploying Oracle ORDS is well documented and fairly routine these days. It's really just a couple of downloads some unzipping, enter a few commands and you are pretty much there. That get's you a functional install but does not get you a production ready setup able to stand up to the rigors of a high volume and highly available production system.










​How far short of production ready is it? It may surprise you to know that the default installs will leave you with the following shortcomings:

• When ORDS starts up, only 1 JDBC connection will be available incurring an overhead to start up other connections as users access ORDS
• A maximum of 10 users can be accessing APEX at the same time
• GlassFish will be running in development mode (which incurs an overhead in checking to see if code has changed)
• None of your static files will be cached
• If you have more than 1 CPU, you will not be taking advantage of it
• GlassFish will only assign .5GB ram to Oracle ORDS, even if you have more available
• Logging will be set to Info, adding a significant burden to GlassFish
• ORDS will not log to the GlassFish log files
• You are in danger of timing out on large file uploads

BTW, the list is pretty much the same for Tomcat. In this post, I'll point you in the right direction to get Oracle ORDS and GlassFish ready for production.

---

Setting the Scene

Of course every use case is different. In my example I am going to assume an actively used system with 250 users and an anticipated peak load of 50 users running in parallel. Remember, because of the stateless nature of APEX and ORDS, your database connection is returned to the pool as soon as your page renders (or your web-service returns a response). That means in our example we are assuming that 50 users could be clicking on a button at exactly the same time requiring all 50 connections (that is a lot). We are assuming a normal load of 20 concurrently active users. I am going to assume we have a dedicated web server running ORDS and GlassFish with 4GB RAM and 1 Quadcore CPU.

Oracle REST Data Services (ORDS)

You may have noticed the warning message when starting ORDS with default settings. There are four settings in the default.xml fle that need to be adjusted from the out of the box settings:

• jdbc.InitialLimit (the number of DB connections that will be created when the server is started) & jdbc.MinLimit (minimum number of DB connections that will be kept alive at idle, or low water mark)
  • The default for these settings are 3 and 1 respectively
  • There is an overhead in ORDS establishing a new connection. In view of this we would want to set these values to our anticipated normal load.
  • In our example this would be 20.
  • I can't think of any reason these two values should be different.
• jdbc.MaxLimit (maximum number of DB connections that can be active)
  • The default value is 10
  • This is the high water mark. As connections close through inactivity the number of open connections will idle down to jdbc.MinLimit
  • This should be set to the maximum anticipated concurrent load which in our case is 50
• jdbc.MaxStatementsLimit (the maximum number of SQL statements to cache for each connection)
  • The default value is 10.
  • If we have available memory, we should up this to 20.
  • Note: If all our connections are busy this would be 50 connections * 20 cached SQL statements * (let's say average SQL statement size is 250 bytes) = 244K

---

GlassFish

There is a lot more to do on the GlassFish side. For each of the configurations, I will mention the human readable option name from the GlassFish Admin Console. You can also set most of these options directly in the domain.xml configuration file if you prefer.

​

• Turn Off GlassFish Development Settings
  • Turn off Auto-Deploy  & Dynamic Re-Loading. Both of these features are great for development, but can affect performance.
• JVM Settings:
  • –Xms (Initial Memory Allocated to JVM at Startup) & –Xmx (Max Memory Assigned to JVM). As we have 4GB, let’s assign half to the JVM. We should also start the JVM with all 2GB allocated so there is not work for the JVM to do in allocating memory as it goes.
  • -server (use the server VM as opposed to the Client VM). The Server VM has been specially tuned to maximize peak operating speed.

                      -Xmx2048m
                      -Xms2048m
                      -server

• Acceptor Threads:
  • Acceptor threads are used to accept new connections to the server and to schedule existing connections when a new request comes in. Set this value equal to the number of CPU cores in your server.
  • In our case we have 1 Quadcore CPU so we should set this to 4.
• Thread Pool:
  • We will set both the min and max to 100. Specifying the same values for Min and Max will allow GlassFish to use a slightly more optimized thread pool.
• Timeouts (set to 10 minutes):
  • http-listener-1 > Connection Upload Timeout: 600000 (Milliseconds)
  • http-listener-1 > Request Timeout: 600 (Seconds)
• HTTP Connections:
  • http-listener-1 > Max Connections = 200
  • The number of connections specified by Max Connections is divided equally among the threads.
• Enable Caching for http-listener-1:
  • Status: Enabled
  • Max Age: 21600 (seconds) - 6 Hours
  • Max Cache Size: 52428800 (bytes) - 50 MB
  • Max File Count: 3000
• Disable http-listener-2
  • We typically only need one listener so we disable "http-listener-2"
• ​Logging :
  • Add the ORDS logger 'oracle.dbtools'
    • oracle.dbtools.level=WARNING
  • Set all other loggers to 'SEVERE'. This will reduce the workload on the web server as INFO is the default value which generates a lot of messages.

---

Screen Shots

Conclusion

The above settings are a guideline only for a fictitious work load. As with all things performance, you will still need to go through the exercise of adding load to your system and finding the right settings for your hardware, users, data etc. The good news is, you now know which levers to pull!