Oracle jdbc driver was unable to connect the database server

Today we faced a problem that a scheduled job in our production server was unable to connect to Oracle database server. The error reported was,

java.sql.SQLRecoverableException: No more data to read from socket
        at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1157)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
        at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:204)
        at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1041)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1329)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
        at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3665)
        at oracle.jdbc.driver.OracleCallableStatement.executeUpdate(OracleCallableStatement.java:4739)
        at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1352)
        at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
        at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)

We found some details in the oracle forums that the cause for this could be due to the driver version incompatibility with the Oracle database version. But on analyzing the logs, we found that the job was failing only for the past two days. After checking this with the Oracle database team, we were informed that there was no change or upgrade done to the oracle database, but they have changed the infrastructure of the servers, which means the IP address of the database server was changed. But this should not affect us since we have configured the server url with the host name and not the IP address.

On further analysis, we found that this was due to the dns cache which was enabled (networkaddress.cache.ttl=-1) in the jvm (where the production tomcat was running) in  ~\jre\lib\security\java.security

'networkaddress.cache.ttl' is a security property which works for the below settings:

# any negative value: caching forever   (default)
# any positive value: the number of seconds to cache an address for

# zero: do not cache   

 

It is also advised that in the comments that setting this to anything other than the default value can have serious security implications. Do not set it unless you are sure you are not exposed to DNS spoofing attack.

Back to the problem - Since the dns caching was enabled, the old IP address was cached which was the cause for this problem. As a fix to this problem, we restarted the tomcat servers to flush the jvm dns cache.

Basic Git commands

1. Create a Git repository and push it to remote

Open the Git bash (Unix bash prompt) for the respective folder and execute the following commands

git init
git add *
git commit -m "initial commit message"
git remote add origin https://github.com/xxxxx/xxxx.git
git push -u origin master 

2. Clone repository

 git clone https://github.com/xxxxx/xxxx.git

3. Create new branch

git branch branch-name

git checkout branch-name

or

git checkout -b branch-name

4. Delete a local branch

git branch -d branch-name

Delete a Remote branch 

git push origin --delete branch-name

Tips to configure Git in Windows

We were facing the problem with 'EGit- Synchronize workspace' which showed all the files as changed, even if we haven't changed the content of those files. This happened only when we clone/ pull the repository using Git bash (Unix bash prompt). As we all know the reason for this problem - Windows uses both a carriage-return character and a linefeed character for newlines in its files, whereas Linux/ Mac use only the linefeed character.

Git Hub provides some configurations to override these cross-platform issues.
Do the following before cloning the remote repository, I have the folder structure as C:\Git\Project\, where I would clone the repository inside this folder.

1. Open Git bash at C:\Git\Project\
2. Enter, git config --system core.autocrlf true and click enter
3. Enter, git config --system core.whitespace cr-at-eol and click enter
4. Then clone the repository as usual, git clone 'https://github.com/xxxxx/xxxx.git' 
5. Then open the local repository in your Eclipse and import projects. 
6. Right click on your projects > Team > Share to Git
7. Right click on your projects > Team > Synchronize workspace
8. In your Team Synchronize perspective, check the list of modified files. No files will be shown.

To know more about the commands that we used in step 2 and 3, refer http://git-scm.com/book/en/Customizing-Git-Git-Configuration

Memory leak and performance analysis

Tools:

JVM Profiler - We have a plugin for it which could be installed in Eclipse.

MAT - Memory Analyzer Tool for analyzing the heap dump.

---

JVM monitor plugin for Eclipse

http://www.jvmmonitor.org/doc/index.html

http://help.eclipse.org/helios/index.jsp?topic=%2Forg.eclipse.tptp.platform.doc.user%2Fconcepts%2Fceproftl.xhtml

Note:
CW-Tomcat is running with java 32 bit and Eclipse also should be 32 bit.

---

Steps:

1. Installed Eclipse helios 32 bit for windows
2. Installed Liferay Plugin, EGit plugins for Eclipse
   1. Through Install New Software -  http://releases.liferay.com/tools/ide/eclipse/helios/stable/
   2. Through Install New Software -  http://download.eclipse.org/egit/updates-2.1
3. Installed JVM Monitor plugin for Eclipse
   1. Through Eclipse Marketplace - Search JVM monitor and Click Install
   2. Through Preferences > Java > Monitor > Tools, Set JDK Root directory as C:\....\Softwares\jdk1.6.0_21
4. Launch eclipse helios 32 bit from command prompt making Eclipse to use the same jdk that is set for JVM monitor, C:\...\Softwares\eclipse-jee-helios-SR2-win32\eclipse>eclipse -vm "C:\Mahesh\Softwares\jdk1.6.0_21\bin\javaw"

---

Monitoring JVM on remote host

To monitor JVM on remote host:

1. Add the following configurations to JAVA_OPTS in ..../WAR/tomcat_folder/bin/setenv.sh

       -Dcom.sun.management.jmxremote.port=9876
       -Dcom.sun.management.jmxremote.ssl=false
       -Dcom.sun.management.jmxremote.authenticate=false

2. Press the button Add JVM Connection on Remote Host on JVM Explorer to open New JVM Connection dialog. 
   1. Enter the ip address of the remote server
   2. Enter the port: 9876

Note: Sometimes we may not be able to connect to remote server even after setting the above three JVM arguments. In that case, add the below argument as well.        

              -Djava.rmi.server.hostname= mention the host-name here

---

How to capture heap dump of JVM running in Remote host

Since 'JVM monitor' does not support viewing the Memory details when monitoring remote host, we may need to capture the heap dump for analyzing the heap memory usage. Below are the steps to capture the heap dump (Ref: http://www.jvmmonitor.org/doc/, under section "How can I enable the BCI mode of CPU profiler on remote host?")

1. In the Eclipse JVM monitoring view, select the server to monitor. In the Properties screen, you will find Timeline, Threads, Memory, CPU, MBeans and Overview. Select 'Memory' tab.
2. Click on the 'heap dump' symbol in the top-right corner to capture the heap dump.
3. A dialog box will be shown with the path where the dump file should be saved in the remote server (eg. /home/liferay\32323432323.hprof). And it may not allow you to select the 'Transfer ....' checkbox or to click on OK button to initiate the dump. An info could be displayed saying that ' agent is not loaded'.
4. Now follow the instructions given in  http://www.jvmmonitor.org/doc/, under section "How can I enable the BCI mode of CPU profiler on remote host?"
5. Copy the jvmmonitor-agent.jar from ~..\eclipse-jee-helios-SR2-win32\plugins\
   org.jvmmonitor.core_3.8.1.201302030008\lib\jvmmonitor-agent.jar and drop it
   into '/home/liferay' directory of Staging server (remote server) using WinSCP tool.
6. Add the following configurations to JAVA_OPTS in  .../WAR/tomcat_folder/bin/setenv.sh

                     -javaagent:/home/liferay/jvmmonitor-agent.jar     

     7. Done. Now we are ready to dump the file. The heap dump .hprof file will be saved at '/home/liferay/'

---

How to use JVM Profiler:

http://www.jvmmonitor.org/doc/index.html#Monitoring_threads

---

Suggestions from different sites:

It is definitely advisable to have on the -XX:+HeapDumpOnOutOfMemoryError setting.   What is dumped is a binary .hprof file.   The setting for the file location shuold be:

-XX:HeapDumpPath=/niku/logs/heapdump.hprof

Ref: https://communities.ca.com/web/ca-clarity-global-user-community/message-board/-/message_boards/view_message/2290442

Setup for this in setenv.sh
-XX:HeapDumpOnOutOfMemoryError –XX:HeapDumpPath=d:\heapDumps

---

 How to open or analyze the heap dump .hprof files?

Some of the tools suggested in net are,

1. MAT - Eclipse Memory Analyzer Tool - (Free) - http://www.eclipse.org/mat/downloads.php
2. Visual VM - https://visualvm.dev.java.net/
3. HAT -  Java Heap Analysis Tool - http://docs.oracle.com/javase/6/docs/technotes/tools/share/jhat.html
4. YourKit Java Profiler

 I used MAT to analyze the heap dump (.hprof file).

MAT for Windows 32 bit was unable to open large dump files. (I tried with a .hprof file of  ~2.2 GB). Also the max heap size of MAT-32 Bit was by default 1GB, which cannot be tweaked. As per suggestion from an Eclipse forum - installed MAT for Windows 64 bit which did the job and allowed to tweak the max heap size. Have configured the vm argument -Xmx to 3GB in MemoryAnalyzer.ini file. Then the heap dump loaded in MAT without errors.