Oracle jdbc driver was unable to connect the database server

Today we faced a problem that a scheduled job in our production server was unable to connect to Oracle database server. The error reported was,

java.sql.SQLRecoverableException: No more data to read from socket
        at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1157)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
        at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:204)
        at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1041)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1329)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
        at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3665)
        at oracle.jdbc.driver.OracleCallableStatement.executeUpdate(OracleCallableStatement.java:4739)
        at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1352)
        at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
        at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)

We found some details in the oracle forums that the cause for this could be due to the driver version incompatibility with the Oracle database version. But on analyzing the logs, we found that the job was failing only for the past two days. After checking this with the Oracle database team, we were informed that there was no change or upgrade done to the oracle database, but they have changed the infrastructure of the servers, which means the IP address of the database server was changed. But this should not affect us since we have configured the server url with the host name and not the IP address.

On further analysis, we found that this was due to the dns cache which was enabled (networkaddress.cache.ttl=-1) in the jvm (where the production tomcat was running) in  ~\jre\lib\security\java.security

'networkaddress.cache.ttl' is a security property which works for the below settings:

# any negative value: caching forever   (default)
# any positive value: the number of seconds to cache an address for

# zero: do not cache   

 

It is also advised that in the comments that setting this to anything other than the default value can have serious security implications. Do not set it unless you are sure you are not exposed to DNS spoofing attack.

Back to the problem - Since the dns caching was enabled, the old IP address was cached which was the cause for this problem. As a fix to this problem, we restarted the tomcat servers to flush the jvm dns cache.