The APEX_DATA_PARSER PL/SQL package was introduced in APEX 19.1. It comes from a long line of useful APIs produced by the APEX development team. APEX_DATA_PARSER solves a number of challenges with loading file based data that I have faced throughout my Oracle career of 25 years. In this post I will explain what problems it solves and how you can use it to make importing data from files much easier.

Almost all projects whether they are custom APEX applications or full-blown ERP implementations require that you import data into a database table from a file. As Oracle developers our peak productivity zone begins when we can handle data using SQL. The quicker we can start handling data using SQL, the quicker development can be completed.
 
APEX_DATA_PARSER provides APIs to help us to:

• Analyze a file to understand its format and structure.
• Identify the field names and datatypes of fields contained in the file.
• Use a SELECT statement to pull data from a file without having to develop complex parsing logic.

 
APEX_DATA_PARSER supports the following file formats:

• CSV
• JSON
• Excel (XLSX)
• XML

Previously, we have had to employ one of the following methods to get external file data into a database table to start handling the data using SQL:

• Use SQL*Loader to load data into a staging table and then kick off a PL/SQL process to perform transformations.
• Use UTL_FILE within PL/SQL to read data from files on the file system.
• Use ‘External Table’ functionality within PL/SQL to read data from files as if they were a table.
• Build a custom program using a language like Java to read data from the file, parse it and insert it into a table using JDBC.

 
All of these methods come with challenges, which include:

• None of them support parsing and loading JSON, Excel or XML files.
• None of these methods easily handle the structure of your file changing.
• All of these methods require that you copy the file to a specific server that the database is able to access.
• SQL*Loader and External Tables require that you manually build a definition file to define the structure of the file you are loading.
• UTL_FILE and External Tables require that you configure database directories to define the location of the external file on the file system.
• UTL_FILE requires that you manually build the parsing logic to split records into fields etc.

APEX_DATA_PARSER handles all of the challenges I described above (and more). It contains a series of APIs that allow you to discover the structure of a file and then parse the data in the file directly from a SQL statement.

The File
In order or for APEX_DATA_PARSER to act on your file, you first have to get it into a BLOB. You might now be thinking if I have to get the file into a BLOB then this really isn’t solving the problem of having to move the file to a location where the database is aware of it. Enter APEX_WEB_SERVICE. You could use APEX_WEB_SERVICE to first get the file BLOB from somewhere like AWS S3 and then consume the response using APEX_DATA_PARSER e.g.

​Discovery
One of the most interesting features for me is the ability to analyze a file and generate a JSON document containing everything you need to know about the file. 

APEX_DATA_PARSER.DISCOVER returns a JSON document with details of the file encoding, delimiters, and the names and data types for each of the fields in the file. This information is extremely useful. For example, you can use it to write code to handle changes to the numbers of columns in files that are uploaded. Today your user uploads a file with 10 columns and tomorrow they add a column. Being able to inspect the format of the file allows you to dynamically handle that 11th column. This is something you could never do with other Oracle loader tools.
 
To make things even easier, there is a companion API to APEX_DATA_PARSER.DISCOVER called APEX_DATA_PARSER.GET_COLUMNS.

​APEX_DATA_PARSER.GET_COLUMNS produces a PL/SQL array of the columns from the JSON profile. 

​Parsing
Now that we know what our file looks like we would like to parse and process the data within the file using SQL. This can be done using APEX_DATA_PARSER.PARSE.

​APEX_DATA_PARSER.PARSE returns a PL/SQL array with 301 columns, the line_number from the file and up to 300 columns of data from the file.

​Although the column names are generic (col001, col002 etc.) we know exactly what is in each column because we have the profile generated earlier.
 
At every turn, APEX_DATA_PARSER is making it easier for you to deal with information about the file and the file itself using SQL.

​There a few considerations that are worth noting.

Many More Parameters
There are many more parameters to the APIs I have described above. I encourage you to read the documentation to learn about these.
 
It Supports a Flat File Structure

• Although it supports JSON and XML file types, APEX_DATA_PARSER can only handle data in a flat structure.
• You cannot, therefore, parse out data from nested objects in JSON or XML.
• During discovery of JSON files, APEX_DATA_PARSER will try and determine the main array and parse the top level of that array only.

 
Basic DATE Type Identification

• It seems to rely on clues in the formatting of date fields to identify the format string.
• For example:
  • DD/MM/YYYY is not identified as a DATE while DD.MM.YYYY is.
  • MM/DD/YYYY is identified as a DATE.
• It seems to me if it tested a number of different date format strings against records in the file, it could improve its success rate in identifying DATE fields.

 
Performance
APEX_DATA_PARSER relies on underlying code to handle the actual data parsing. For example, if you are on 11g of the database it uses APEX_JSON to parse JSON. If you are on 18c of the database it will use the native JSON parser. The native JSON parser in 18c is orders of magnitude faster than APEX_JSON. In view of this the parsing speed can vary dramatically depending on the version of the DB you are on.
 
This is actually very clever. By encapsulating file parsing logic within APEX_DATA_PARSER you will automatically gain performance improvements when you upgrade from 11g to 18c.

​We have already used APEX_DATA_PARSER for loading payroll data from external payroll providers into Oracle EBS. These payroll files have a varying number of columns depending on the different pay elements in the payroll file (e.g. gross pay, sick pay, holiday, vacation, etc.). Being able to analyze the profile produced by APEX_DATA_PARSER at run time allowed us to pick up columns added to the file dynamically. As long as these columns matched Oracle payment elements, we could then load them right into Oracle.
 
Prior to APEX_DATA_PARSER this would have been nearly impossible. At the very least APEX_DATA_PARSER cut the development time in half.

For on-premise and cloud-based ERP projects, a significant portion of the technical budget is spent converting data (e.g. items, customers, suppliers, orders, invoices etc.) from a legacy system to a new Oracle ERP. APEX_DATA_PARSER can significantly reduce the cost and complexity of these conversions by removing the complexity of getting you to the stage where you can work with the file data in SQL.
 
There are many more examples where APEX_DATA_PARSER can save you time and allow you to provide real business benefit. For example, you can save users from manual data entry by allowing them to upload data from an Excel spreadsheet. APEX_DATA_PARSER even handles fetching data from a specific tab of a spreadsheet.
 
Here is the link to the APEX 19.1 documentation for APEX_DATA_PARSER

​In this post I will demonstrate a useful feature of APEX that does not get a lot of attention. ‘User Preferences’. It’s another one of those APEX features that saves you time. This allows you to focus more time on building important business logic that adds value to your business.

APEX User Preferences allow you to store name value pairs by user and APEX application. These name value pairs are persisted using an internal APEX table. You can set, get and delete user preferences declaratively or via APEX provided APIs.
 
You can imagine these name value pairs are stored as follows:

There are many reasons why you may want to store values related to a particular user:

• Default Time zone
• Default language
• Notification preference
• Default home screen
• Default unit of measure
• URL to profile picture
• etc.

​For the remainder of this post, I will demonstrate how to get and set preferences using a simple example. In this example, we have a user preference which determines if the user sees a Detailed or Summarized dashboard view when the user logs in.
 
Here is a screen shot of the a sample preferences page which I created where the user can set their default dashboard view.

​There is no need to formally create a user preference before setting a value for it. When you set a user preference for the first time, APEX creates a record in an internal table to store the preference and its value. This value is stored until you either change or delete the preference.
 
There are two ways to set a preference value.
 
1. Declaratively in a Page Process
One of the great things about user preferences is that you can set a user preference declaratively (no code) using a page process. The below page process is set to run when the user clicks the ‘Apply’ button (from our example above). When it runs, the user preference ‘DASHBOARD_VIEW’ is set to the current value of the page item ‘P3_DASHBOARD_VIEW’. 

​The key here is the type ‘User Preferences’ (under the ‘Identification’ section).
 
The type under the ‘Settings’ section has three options, two of which (in bold) can be used to set a user preference. Their names are pretty self-explanatory.

• Reset Preferences (remove all preferences for current user)
• Set to Item Value
• Set to Item Value if Not Null

More about the first option later in the post.
 
2. Programmatically Using an API
APEX also provides and API which you can use to set User Preferences. In the below code snippet, we are setting the user preference ‘DASHBOARD_VIEW’ to the value ‘DETAILED’ for the currently logged in user.

​There are also two ways to get a preference value.
 
1. Declaratively Using Page Item Source
You can declare the source for a page item to be a User Preference (again no code). This means that when the page is rendered, the page item automatically picks up the value for the preference.

​In our example, we need to initialize the item ‘P3_DASHBOARD_VIEW’ with the current value for the user preference ‘DASHBOARD_VIEW’.
 
2. Programmatically Using an API
APEX also provides and API which you can use to get the value for a user preference. In the below code snippet, we are getting the current value for user preference ‘DASHBOARD_VIEW’ for the currently logged in user.

​It may not be completely necessary to purge user preferences but it may be useful when you are de-activating users who no longer work for your company. There are three options for purging user preferences.
 
1. Declaratively Using a Page Process
With this option you can purge all current user preference entries for a particular user and APEX Application. When the below page process runs, it will purge all user preferences for the current user and APEX application. 

​In the page process above, we again have Identification > Type set to ‘User Preferences’. This time, we have the Settings > Type set to ‘Reset Preferences (remove all preferences for current user)’.
 
2. Via Workspace Administration
As a workspace administrator, you can view and purge user preferences for the currently logged in user or a specified user. You can access these options using the following navigation from Workspace Administration: Manage Service > Session State > Manage Preferences

​The screen shot below (from the ‘Purge Preferences by User’ option) shows preferences for user ‘JDIXON’. Clicking ‘Purge User Preferences’ will purge all preferences for this user.

​Finally, there is an API you can call to purge a single user preference for a given user in the current application. The following code will delete the preference ‘DASHBOARD_VIEW’ for the currently logged in user in the current application.

Without User Preferences, you would have to build a table and PL/SQL package to manage this kind of data in your APEX applications. The fact that this feature is baked into APEX saves you time and lets you focus on value-add features of your Application.
 
Here is the link to the APEX 19.1 documentation on user preferences:

​Full disclosure, I have absolutely no experience in trying to convince my boss to try APEX or ORDS. I do, however, have experience in basking in the glow of clients who have reaped the benefits of implementing mission critical systems using these two no-cost options of the Oracle database.
 
TL;DR; If you are a boss, try out APEX and ORDS today. If you already have an Oracle database, it costs you nothing extra and the upsides for you and your business are tremendous.

​If you are a developer looking to convince your boss to adopt APEX and ORDS, then this post is for you. My goal is to arm you with the material you need to make a compelling case for adopting APEX and ORDS. I have broken the post into several major talking points. At the start of each talking point, I have a headline statement (for bosses with short attention spans). I then go on to explain my reasoning in a little more detail (for bosses that need some evidence).

​Data Centric - APEX, because of its architecture, is very data centric. Both the code and the data exist in the same database enabling zero data access latency. This architecture did no come about by accident; there are good reasons why your code should be right next to your data. I have been using APEX for a while now, but I love this new term ‘zero data latency’. Of course, this has been true since the days of HTMLDB. But in this world of super complex, mega frameworks that are so abstracted from the data they need a map to find it, it is very relevant today.
 
Stateless – Much of the scalability of APEX comes from its stateless nature. The API calls to the Oracle APEX engine use the standard Oracle database connection pool. This means that once an API call is processed and the response is sent back to the browser, the connection is returned to the database connection pool and can be used by any other request. The database sessions are only active when performing a request, otherwise, the they are inactive and do not consume any database resources. This allows APEX to handle tens of thousands of users simultaneously.

APEX is very stable, easy to upgrade and new features are coming thick and fast.
 
You may think stability and innovation don’t belong in the same sentence. When you are talking about APEX, however, they really do. I have never undertaken a large JavaScript framework project. I can imagine, however, that as a boss it is a little un-nerving heading into a multi-year project with half a dozen technologies, many of which may not exist by the time the project is over. APEX has been around for 15+ years and there is no sign of it going anywhere. You may think, “So what, Windows has been around for longer and it is still awful.”. APEX is different. The pace of new (and useful) features over the past 2-3 years (especially in the areas of UI, Single Sign On and integration capabilities) has been fantastic. BTW, there is no sign of that pace letting up. With multiple releases a year you get the option of up taking these improvements at regular intervals.
 
Hold on a minute, that doesn’t sound very stable. What I haven’t mentioned is that the APEX development team is world famous for providing upgrades that do not break your existing Apps. One of the ways they achieve this is by allowing apps to run in previous versions with ‘Compatibility Mode’. This means that in most cases, you can do the upgrade and then switch on new functionality in individual apps, as and when you need it. There are stories every month of APEX users who have upgraded from APEX 3 (released in 2007) to the current version without anything breaking. Can you imagine doing that with any other development framework?

Your users will love using APEX Applications!
 
If you had experience of APEX from version 4 (or before), boy are you in for a treat. Applications built with APEX look as good as any other framework and are quite frankly a joy to use. We are long past the dystopian days of automaton workers using systems they are forced to use. When people use Enterprise systems these days they expect more. While you may feel like dark mode is a fad, Apple is about to release it across all of the Operating Systems. Believe me, your Enterprise users will be using dark mode soon enough. Wouldn’t it be great if your development platform could make dark mode happen at the flick of a switch? APEX has your back. Dark mode is already available in APEX Builder and will be coming soon as an end user theme style. It goes much further than that too. APEX’s Universal Theme delivers UI components that would otherwise consume hundreds of hours of custom development. For a peak at what is available, check out the Universal Theme demo App https://apex.oracle.com/ut
 
Finally, with accessibility concerns finally being taken seriously, it is good to know that the APEX team has baked accessibility checks right into the product. They even have accessibility tested themes available. Check out this video for more details on Accessibility in APEX.

​APEX and ORDS provide you everything you need to host and consume REST based web services to meet all of your integration needs.
 
Oracle REST Data Services (ORDS) has two roles in an APEX world. One is to service APEX pages to end users and the other is to host REST based web services. The ability to host REST based web services based on SQL and PL/SQL is extremely powerful. It allows you to provide external systems secure access to the data in your Oracle database. This can be useful for integrations and even for hosting web services to be consumed by native mobile applications. See our post on ORDS for details.
 
APEX comes with a number of features to facilitate consumption of any REST based web services. For example, Web Source Modules provide a wizard-based approach to configuring components that allow you to create reports and forms directly on web services. This extends the reach of APEX outside the database and prevents un-necessary syncing of data to and from other systems. 

With APEX Social Sign-In you can use just about any OAuth identity provider to authenticate your users.

Authentication can be a complex proposition. If you store usernames and passwords yourself you really need to know what you are doing. The last thing you want is for your user account data to be compromised. It is generally far safer to outsource this task to a trustworthy identity provider such as Okta or Auth0. You could also outsource this task to platforms your users already use such as Office 365, Google and even Facebook. 

​By providing the ability to create custom authentication schemes, APEX has always provided flexibility in authenticating users. Custom authentication schemes allow you to create code that could use just about anything to authenticate a user. Since APEX Social Sign-In was introduced in APEX 18.1, APEX authentication has reached a whole new level. You can now setup declarative authorization schemes for OAuth-capable identity providers and APEX handles calling the necessary web services to authenticate your user. Taking Okta for example. Okta can now be your source of truth for handling all authentication in your company. All of your other systems (Email, ERP and your APEX Apps) can then use Okta to authenticate your users. See our post on APEX Social Sign-In for more details.

​APEX provides many useful features and utilities out of the box, that would otherwise require numerous third-party libraries.
 
To name a few:

• Out of the Box Authorization
• Multi-Language and globalization features
• Sending Emails
• Email Templates
• Extensible plugin framework with hundreds of available plugins
• Building and parsing JSON
• Excel and CSV parsing
• Zipping and Unzipping files
• JWT Handling
• String manipulation utilities
• and many more…

 
The benefit of having these utilities bundled with APEX are that fixes and enhancements are handled by Oracle and released with each new version of APEX. In contrast, having these types of utilities spread out amongst a number of providers means you get fixes when they feel like. It also means that as those providers drop by the wayside you are left having to merge new libraries into your applications.

​APEX applications can be hosted in the cloud just as easily as on-premise.
 
The APEX architecture lends itself well to running in the cloud and there are numerous options for hosting APEX in the cloud.
 
PaaS - There are turn key PaaS solutions (e.g. Exadata Express from Oracle) that host the entire APEX and ORDS architecture at a very reasonable price. All you need is a credit card and a browser to get started. See our post on using APEX to extend Oracle ERP Cloud.
 
IaaS - You can also install APEX and ORDS yourself on infrastructure from just about any cloud provider. This allows you complete flexibility to size the environment as you wish, but you will need to perform much of the maintenance (backups, upgrades etc.) yourself.

​APEX is building on its high productivity heritage to establish itself as a legitimate contender in the Low Code space.
 
APEX has always been known for allowing developers to build useful, scalable applications in a very short period of time. Over the past year and a half, more and more features have been added to establish APEX as a top contender in the low code pace. These features include wizard-based development of web service integrations and pre-built components that can be added to existing applications or used to build brand new applications. APEX is already at the stage where you can build fully functional applications starting from the upload of an Excel spreadsheet. These kinds of features are allowing APEX to break out of the IT department and into the business. It is here where (handled the right way), citizen developers can become productive with APEX with just a small amount of oversight from IT. 

​Many of our customers run Oracle e-Business Suite and have a number of PL/SQL developers on staff. How to leverage this talent and still provide your users with applications they will love using? An experienced PL/SQL developer can be up and running on APEX in a matter of days. Five years of Oracle database, SQL and PL/SQL experience gives you about 70% of the skills to be a high performing APEX developer. Sprinkle in a little JavaScript, CSS and web services experience and you are 100% of the way there.

​APEX and ORDS are no cost options of the Oracle Database.  If you own an Oracle database license you own APEX and ORDS.
 
I purposely left this item to last because in my opinion, APEX and ORDS stand on their own against any enterprise-ready development platform. Having said that, if you own an Oracle database license you really should be using APEX and ORDS because it isn’t going to cost you a penny extra. In addition to the no extra cost for the software, you will also need very little additional hardware.  A VM with 4GB of RAM and 15GB disk is more than enough to host ORDS, which is the only other component you need.

It’s very hard to write a pithy conclusion to summarize everything I have said in this post. There are also, no doubt, a number of important APEX benefits that I have not included. If you read this post and you are a developer keen to bring APEX and ORDS into your organization, I encourage you to use some of this material to make a case. If you are a boss and you are looking to adopt a development platform that will deliver real value to your business, then I urge you to visit the following links to learn more about what APEX and ORDS can offer your organization:

• APEX Shortcuts
• APEX Office Hour Videos
• APEX Learning