Recently one of our customers experienced issues with adop phase=fs_clone. It just hanged for a day without any message and then failed. I spent some time debugging as there were no errors in the adop log files and adopscanlog did not help.
Then I found that adop also created logs in the following directory:
$INST_TOP/logs/appl/rgf/TXK |
In the directory CLONINGCLIENT-XXXXXX I found that unpack.sh script just hanged with no error messages. I checked the domain directory on the patch filesystem for any messages and found the following log file:
$FMW_HOME/user_projects/domains/EBS_domain/hs_err_pid377803.log |
It contained the following error stack:
# # There is insufficient memory for the Java Runtime Environment to continue. # Cannot create GC thread. Out of system resources. # Possible reasons: # The system is out of physical RAM or swap space # In 32 bit mode, the process size limit was hit # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Use 64 bit Java on a 64 bit OS # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (gcTaskThread.cpp:46), pid=377803, tid=0xf6f5bb40 # # JRE version: (7.0_401-b07) (build ) # Java VM: Java HotSpot(TM) Server VM (24.401-b07 mixed mode linux-x86 ) # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # --------------- T H R E A D --------------- Current thread (0xf6e09800): JavaThread "Unknown thread" [_thread_in_vm, id=377807, stack(0xf6f0c000,0xf6f5c000)] Stack: [0xf6f0c000,0xf6f5c000], sp=0xf6f5acf0, free space=315k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x80f8f0] VMError::report_and_die()+0x1c0 V [libjvm.so+0x34c882] report_vm_out_of_memory(char const*, int, unsigned int, char const*)+0x72 V [libjvm.so+0x40c153] GCTaskThread::GCTaskThread(GCTaskManager*, unsigned int, unsigned int)+0xe3 V [libjvm.so+0x40b7a7] GCTaskManager::initialize()+0x277 V [libjvm.so+0x6c8883] ParallelScavengeHeap::initialize()+0x563 V [libjvm.so+0x7e0b64] Universe::initialize_heap()+0xa4 V [libjvm.so+0x7e1cb8] universe_init()+0x58 V [libjvm.so+0x463205] init_globals()+0x55 V [libjvm.so+0x7c8ae0] Threads::create_vm(JavaVMInitArgs*, bool*)+0x200 V [libjvm.so+0x4d7e5c] JNI_CreateJavaVM+0x5c C [libjli.so+0x2738] JavaMain+0x98 C [libpthread.so.0+0x6ed9] start_thread+0xf9 |
It was misleading as I thought that there is a problem with available memory on the application server. But there was a plenty of RAM on the server. Some articles pointed to the Linux resource limits, but increasing them also did not help. So after searching for similar issues I found the following workaround which helped:
export CONFIG_JVM_ARGS="-Xms2048m -Xmx4096m -XX:MaxPermSize=2048m" |
And the root cause of the issue was too many OHS and Weblogic logfiles which occupied to much space. It is adviced to schedule a logs cleanup script on the Oracle EBS application server.
There are several MOS Docs that describe similar issues:
Adop Fs_clone Run For Days No Error Just Keep Running (Doc ID 2671167.1)
ADOP Fails with OutOfMemoryError: Java Heap Space Error (Doc ID 1679761.1)
12.2 EBS adop phase=fs_clone fails during wlsT2PApply on unpack.sh with error “Java Heap Space”/”java.lang.OutOfMemoryError: GC overhead limit exceeded” (Doc ID 2429369.1)
Also there are several blog posts on the same topic, but the symptoms differ. So I decided to publish this article which might help someone with the investigation.