How to configure hugepages for Oracle EBS

Hugepages is a Linux memory management feature that allows the operating system to use larger memory pages (2 MB by default) instead of the standard 4 KB pages. Configuring hugepages for Oracle databases significantly improves performance by reducing the overhead of memory management and TLB (Translation Lookaside Buffer) misses. Many customers are missing this critical configuration, which can lead to suboptimal database performance.

This guide walks through the complete steps to configure hugepages on a Red Hat-based Linux system for Oracle EBS.

Note that hugepages is a Linux-specific feature, so steps 1 to 10 below apply only to EBS database tiers running on Linux (Oracle Linux, Red Hat Enterprise Linux or SUSE). The other Unix platforms certified for the Oracle EBS database tier – Oracle Solaris, IBM AIX on Power and HP-UX Itanium – have their own large page mechanisms, most of which need no manual configuration at all. See 11 Large pages on other operating systems at the end of this article for a platform-by-platform summary.

1 Stop EBS applications

Before making any changes, stop all running EBS applications to free up memory resources.

2 Calculate the number of required hugepages

Divide the SGA size in bytes by 2048*1024 (2048 kB is the default hugepage size).

Example for an 80 GB (81920 MB) SGA:

80*1024*1024*1024/(2048*1024) = 80 * 512 + 5 = 40965 hugepages

As a quick rule of thumb: multiply the number of Gigabytes by 512, then add 5 extra hugepages as a buffer.

3 Update /etc/sysctl.conf

Update /etc/sysctl.conf with the recommended setting. Open the file with vi and set the number of hugepages:

vi /etc/sysctl.conf
vm.nr_hugepages=40965

Then take the parameters into effect by running:

sysctl -p

4 Update the limits configuration file

Add the following entries into a config file under /etc/security/limits.d/. This is usually a config file created by the oracle preinstall RPM and looks similar to: oracle-database-preinstall-19c.conf

If there is no such file, update the default Linux file /etc/security/limits.conf:

oracle hard memlock unlimited
oracle soft memlock unlimited

The server does not need to be restarted after applying these changes, but Linux users need to re-login to see the difference. You may use the following command to check the current memlock limit for the current session:

ulimit -l

5 Check/change the parameter to force the instance to use large pages

Connect to the database as sysdba and verify/set the use_large_pages parameter:

sqlplus '/as sysdba'
show parameter use_large_pages
alter system set use_large_pages=ONLY scope=spfile;

6 Set the SGA parameters

Change sga_target and sga_max_size to the required values. Note that sga_max_size should be set only on the CDB level, while sga_target should be set to the same value on both the CDB and PDB levels.

sqlplus '/as sysdba'
alter system set sga_max_size=10G scope=spfile;
alter system set sga_target=10G scope=spfile;
alter system set sga_target=10G container=all scope=spfile;

Result cache should be modified only on the PDB level:

sqlplus '/as sysdba'
alter session set container=XXXX;
alter system set RESULT_CACHE_MAX_SIZE=600M;

7 Restart the database in open mode

shu immediate
startup

8 Check available hugepages

After the database has started, verify that hugepages are being used:

[oracle@apps ~]$ grep Huge /proc/meminfo
HugePages_Total:    6150
HugePages_Free:       13
HugePages_Rsvd:        9
HugePages_Surp:        0
Hugepagesize:       2048 kB

9 Check the Oracle database alert log

Check the Oracle database alert log for confirmation that hugepages are configured. You should see output similar to the following:

2018-07-19 06:52:34.199000 +02:00
Starting ORACLE instance (normal) (OS id: 26730)
Dump of system resources acquired for SHARED GLOBAL AREA (SGA):
Per process system memlock (soft) limit = 14G
Expected per process system memlock (soft) limit to lock
SHARED GLOBAL AREA (SGA) into memory: 12G
Available system pagesizes:
  4K, 2048K
Supported system pagesize(s):
PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s)
  2048K             6150            6146             6146       NONE

10 Check the memory usage

Verify the memory state after database startup with hugepages configured. Hugepages are reported in the used section as they are allocated at system startup:

[root@r12_local ~]# free -mh
              total        used        free      shared  buffers   cached
Mem:           5.8G        3.1G        2.8G       340K    25M      935M
-/+ buffers/cache:          2.1G        3.7G
Swap:          1.7G          0B        1.7G

11 Large pages on other operating systems

Hugepages as configured above is a Linux concept: the vm.nr_hugepages kernel parameter, /proc/meminfo and the use_large_pages initialization parameter do not exist on other platforms. Oracle documents that USE_LARGE_PAGES is applicable only on the Linux operating system.

That does not mean large pages are irrelevant elsewhere – every Unix platform certified for the EBS database tier uses them – but on Solaris and HP-UX the operating system and the database negotiate the page sizes automatically, so there is nothing for the DBA to configure. The table below summarises the position for each operating system.

Oracle Solaris – nothing to configure

Solaris is the simplest case. Oracle Database automatically uses Optimized Shared Memory where it is available, and otherwise selects ISM or DISM depending on whether SGA_MAX_SIZE is larger than the SGA components actually in use. All of these mechanisms allocate the SGA in large, non-pageable pages. Oracle states that the oradism utility is automatically set up during installation and that it is not required to perform any configuration tasks to use dynamic SGA.

Important: do not set LOCK_SGA=TRUE on Solaris. Oracle Database 12c and later will fail to start if you do.

IBM AIX – 64 KB automatic, 16 MB optional

AIX supports 4 KB, 64 KB, 16 MB and 16 GB pages. Since POWER5, the 64 KB page size is used automatically for the SGA without any configuration and without lock_sga, which is already a large improvement over 4 KB pages and is sufficient for the majority of EBS installations.

16 MB large pages are the closest AIX equivalent to Linux hugepages and are worth considering for very large SGAs. They must be reserved up front by the system administrator – the number of regions is the SGA size divided by 16 MB:

# /usr/sbin/vmo -r -o lgpg_regions=10250 -o lgpg_size=16777216
# /usr/sbin/vmo -p -o v_pinshm=1
# chuser capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGATE oracle

The database must then be told to pin the SGA, which is what causes it to use the reserved 16 MB pages:

SQL> alter system set lock_sga=TRUE scope=spfile;

Two caveats. First, 16 MB pages are always pinned and are permanently removed from the pool of normal memory, even if the database never uses them, so over-allocating them starves the rest of the server – including the EBS application tier if it runs on the same host. Second, use_large_pages has no effect on AIX; lock_sga is the equivalent switch. For Oracle RAC, the root user needs the same capabilities when running srvctl.

HP-UX Itanium – automatic, and usually best left alone

By default Oracle Database uses the largest virtual memory page size available on HP-UX, expressed as the value L in the page size hint of the executable. HP-UX then allocates memory in progressively larger pages – 1 MB, 4 MB, 16 MB and so on, up to 1 GB – as the allocation grows. No sysctl-style tuning is needed.

The chatr command exists mainly to lower this setting in the rare case where a server runs a very large number of processes and reports Out of memory errors as a result of the aggressive page size escalation:

# /usr/bin/chatr +pd <newsize> $ORACLE_HOME/bin/oracle

For a dedicated EBS database server, the default is normally the right choice.

Summary

If your EBS database tier runs on Linux, the hugepages configuration described in steps 1 to 10 is essential, and it is one of the most common omissions we see on customer systems. If it runs on Oracle Solaris or HP-UX, large pages are already in use and there is nothing to configure. On IBM AIX you get 64 KB pages for free and can opt into 16 MB pages for very large SGAs. In every case it is worth checking the database alert log after startup, which reports the page sizes actually used for the SGA on all platforms.