Tuesday, September 23, 2014

Authentication using Apache Shiro

Introduction

Authentication is the process of identity verification. A user should provide some information as an identity that the system understands so as to allow the user to login the system. This document covers one of Shiro's authentication mechanism that can be used for a java RESTful application.

Shiro

Apache Shiro is a java security framework that offers solution for authentication, authorization and session management for an application. Since  RESTful application are stateless, we will not deal with sessions. For every request URL, the user is authenticated. Four basic terms that Shiro uses are: 
(1) Subject: refers to the user who is using the system. It can be human-being, another application, cron job etc.
(2) Principals: refers to attributes that identify the subject, like username first name, last name, ssn
(3) Credentials: refers to secret data like password, certificates etc.
(4) Realms: this is Shiro's security specific DAO which talks to the application's back end datasource or another DAO.

Shiro configuration file (shiro.ini)

During the application deployment, Shiro loads its configuration file (shiro.ini). They call it as the initialization file where the Realm, Credential matcher, authentication filter and few more settings that are to be used in the application are configured.

shiro.ini
 [main]  
 authc.loginUrl = /web/login.jsp  
 customRealm = com.app.security.shiro.realm.CustomRealm  
 credentialsMatcher = com.app.security.shiro.checker.CustomCredentialsMatcher  
 credentialsMatcher.hashAlgorithmName = SHA-512  
 credentialsMatcher.hashIterations = 1024  
 # base64 encoding, not hex since it requires less space than hex  
 credentialsMatcher.storedCredentialsHexEncoded = false  
 customRealm.credentialsMatcher = $credentialsMatcher  
 # customRealm.dataSourceName should be the same that is used in "persistence.xml"  
 customRealm.dataSourceName = java:jboss/datasources/appDS  
 securityManager.realms = $customRealm  
 [urls]  
 # authc means the default FormAuthenticationFilter used for FORM level login  
 /web/login.jsp = authc  
 # REST URL authentication. authcBasic means the default BasicHttpAuthenticationFilter  
 # "rest": HttpMethodPermissionFilter, to be used in future for authorization  
 /rest/** = authcBasic, rest  

Shiro Filter

Shiro provides a request filter called ShiroFilter which needs to be added in our web.xml such that all the request to the application pass through this filter before being processed by the server. The Shiro filter internally forwards the request to the Authentication filter that we have configured in shiro.ini file. For our application, we used 'BasicHttpAuthenticationFilter' (which is marked as 'authcBasic' in shiro.ini). This filter internally uses the Realm DAO and the CredentialsMatcher to validate the credentials for authentication. If the user is successfully validated, then the request is forwarded to the application's REST layer to process further. 

web.xml
 <?xml version="1.0" encoding="UTF-8"?>  
 <web-app xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">  
   <listener>  
     <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class>  
   </listener>  
   <filter>  
     <filter-name>ShiroFilter</filter-name>  
     <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class>  
   </filter>  
   <filter-mapping>  
     <filter-name>ShiroFilter</filter-name>  
     <url-pattern>/*</url-pattern>  
   </filter-mapping>  
   <session-config>  
    <session-timeout>30</session-timeout>  
   </session-config>  
 </web-app>  

Realm and CredentialMatcher

The principal (username) and credential (password) are retrieved from the "Authorization" URL header attribute. They are base64 encoded and are given in the following format, "Basic YWRtaW46YWRtaW4=" where 'YWRtaW46YWRtaW4=' is the base 64 encode of username:password.
The ShiroFilter internally constructs the AuthenticationToken using the username & password. The BasicAuthenticationFilter internally validates this token with the user authentication information in the application database. This is done using CustomRealm and CustomCredentialsMatcher.
CustomRealm is a custom Realm which has its own implementation to fetch the authentication information of the respective user from the application database. We use just plain JDBC to fetch the user record matching the username in the Authentication token. Plain JDBC has been used due to a Shiro limitation which is given below. Finally the AuthenticationToken and the user info fetched from database are passed to the CustomCredentialsMatcher for validation.
Shiro limitation: Apache Shiro is unaware of CDI injection in Realm. The injected UserDAO was not instantiated during CustomRealm object instantiation. There are a couple of tickets in Shiro's queue for this. Refer: https://issues.apache.org/jira/browse/SHIRO-337, https://issues.apache.org/jira/browse/SHIRO-422. Hence as a workaround, currently CustomRealm extends JdbcRealm and we fetch the user object matching username through direct JDBC queries. In future if Shiro resolves the above tickets, we can remove the plain JDBC stuff and enable the injected UserDAO code. 

CustomRealm.java
 public class CustomRealm extends JdbcRealm { //JdbcRealm extends AuthorizingRealm and AuthenticatingRealm   
   private static final Logger log = Logger.getLogger(CustomRealm.class);  
   protected static final String DEFAULT_AUTHENTICATION_QUERY = "SELECT user_name, password, password_salt "  
       + "FROM user WHERE user_name = ?";   
   // @Inject protected UserDAO userDAO;  
   public CustomRealm() {  
     super();  
     setName("CustomRealm");  
     CustomCredentialsMatcher credentialMatcher = new CustomCredentialsMatcher();  
     setCredentialsMatcher(credentialMatcher);  
     log.info("CustomRealm instantiated!");      
   }  
   protected String dataSourceName; // java:jboss/datasources/appDS - configured in shiro.ini  
   public String getDataSourceName() {   
     return dataSourceName; // java:jboss/datasources/appDS  
   }   
   public void setDataSourceName(String dataSourceName) {   
     this.dataSourceName = dataSourceName;   
     this.dataSource = getDataSourceFromJNDI(dataSourceName); // setting datasource to JdbcRealm.datasource  
   }   
   private DataSource getDataSourceFromJNDI(String jndiDataSourceName) {   
     try {   
       InitialContext ic = new InitialContext();   
       return (DataSource) ic.lookup(jndiDataSourceName);   
     } catch (NamingException e) {   
       log.error("JNDI error while retrieving " + jndiDataSourceName, e);   
       throw new AuthorizationException(e);   
     }   
   }   
   @Override  
   protected AuthenticationInfo doGetAuthenticationInfo(AuthenticationToken token)  
       throws AuthenticationException {  
     log.info("Do Shiro doGetAuthenticationInfo...");  
     //identify account to log to  
     String username = String.valueOf(token.getPrincipal());  
     checkNotNull(username, "Null usernames are not allowed by this realm.");  
     log.info("---------------------------------------------------------------------");  
     log.info("Principal: " + String.valueOf(token.getPrincipal())  
         + " - Credentials: " + String.valueOf(token.getCredentials()));     
     log.info("---------------------------------------------------------------------");  
     String hashedPassword = null;  
     ByteSource credentialsSalt = null;  
     if("admin".equalsIgnoreCase(username)) {  
       hashedPassword = "admin";  
     } else {  
       IUser user = null;  
       // user = userDAO.findByUserName(username);    
       Connection conn = null;  
       try {  
         conn = dataSource.getConnection();  
         user = getUserByUserName(conn, username);  
       } catch (SQLException e) {  
         throw new AuthenticationException("Unable to fetch user by username ["   
             + username + "].", e);  
       } finally {  
         JdbcUtils.closeConnection(conn);  
       }  
       checkNotNull(user, "No account found for user [" + username + "]");  
       hashedPassword = user.getPassword();  
       credentialsSalt = new SimpleByteSource(Base64.decode(user.getPasswordSalt()));  
     }  
     // SimpleAuthenticationInfo implements SaltedAuthenticationInfo  
     return new SimpleAuthenticationInfo(token.getPrincipal(), hashedPassword, credentialsSalt, getName());  
   }  
   @Override  
   protected AuthorizationInfo doGetAuthorizationInfo(PrincipalCollection arg0) {  
     // TODO Auto-generated method stub  
     return null;  
   }  
   private IUser getUserByUserName(Connection conn, String username)   
       throws SQLException {   
     PreparedStatement ps = null;   
     ResultSet rs = null;   
     IUser user = null;  
     try {   
       ps = conn.prepareStatement(DEFAULT_AUTHENTICATION_QUERY);   
       ps.setString(1, username);   
       rs = ps.executeQuery();   
       // Loop over results - although we are only expecting one result,   
       // since usernames should be unique   
       boolean foundResult = false;   
       while (rs.next()) {   
         // Check to ensure only one row is processed   
         if (foundResult) {   
           throw new AuthenticationException(   
               "More than one user row found for user ["   
                   + username + "]. Usernames must be unique."); // TODO work on this warning  
         }   
         user = new User();  
         user.setUserName(rs.getString("user_name"));  
         user.setPassword(rs.getString("password"));  
         user.setPasswordSalt(rs.getString("password_salt"));  
         foundResult = true;   
       }   
     } finally {   
       JdbcUtils.closeResultSet(rs);   
       JdbcUtils.closeStatement(ps);   
     }   
     return user;   
   }   
   private void checkNotNull(Object reference, String message) {  
     if (reference == null) {  
       throw new AuthenticationException(message);  
     }  
   }  
 }  

CustomCredentialsMatcher.java
 public class CustomCredentialsMatcher extends HashedCredentialsMatcher {  
      
   private static final Logger log = Logger.getLogger(CustomCredentialsMatcher.class);  
      private static final SHIRO_CREDENTIAL_HASH_INTERATION = 1024;  
      
   public CustomCredentialsMatcher() {  
     super();  
     log.info("CustomCredentialsMatcher instantiated!");  
     setHashAlgorithmName(Sha512Hash.ALGORITHM_NAME);  
     setHashIterations(SHIRO_CREDENTIAL_HASH_INTERATION);  
     setStoredCredentialsHexEncoded(false); // Base64-encode, not hex-encode  
   }  

   @Override  
   public boolean doCredentialsMatch(AuthenticationToken token, AuthenticationInfo info) {  
        
     log.info("Shiro doCredentialsMatch...");  
        
     log.info("AuthenticationToken.credentials: " + new String((char[]) token.getCredentials()));  
     log.info("AuthenticationToken.principals: " + token.getPrincipal());  
     log.info("AuthenticationInfo.credentials: " + info.getCredentials());  
     log.info("AuthenticationInfoprincipals: " + info.getPrincipals());  
     if("admin".equalsIgnoreCase(String.valueOf(token.getPrincipal()))) {  
       return true;  
     } else {  
       // verifies by comparing the credential in token and info  
       return super.doCredentialsMatch(token, info);  
     }   
   }  
 }  

Schema design: User (user)
  • user_name : username to login
  • password  : password to login
  • passwordSalt : a secure random number used as a salt while encrypting/decrypting the password.
It is insecure to store the password as plain string in the database. Hence we encrypt the password using Shiro's Sha512Hash algorithm, also a password salt is used for this to make it more secure. Note that the hashing algorithm (Sha512Hash) used for this password crypt is configured in shiro.ini.
UserService.java
 public class UserService {  
     @Inject UserDAO userDAO;  
       
     public void createUser(UserBean userBean) {  
         String id = // generate a uuid  
         userBean.setId(id);           
         if(userBean.getPassword != null) {  
             // We'll use a Random Number Generator to generate salts. This is much more secure  
             // than using a username as a salt or not having a salt at all.  
             RandomNumberGenerator rng = new SecureRandomNumberGenerator();  
             Object salt = rng.nextBytes();  
             //hash the plain-text password with the random salt and multiple  
             //iterations and then Base64-encode the value (requires less space than Hex)  
             String final SHIRO_CREDENTIAL_HASH_INTERATION = 1024;  
             String hashedPasswordBase64 = new Sha512Hash(this.password, salt,  
                     SHIRO_CREDENTIAL_HASH_INTERATION).toBase64();  
             userBean.setPassword(hashedPasswordBase64);  
             userBean.setPasswordSalt(salt.toString());              
         } else {  
             throw new ApplicationRuntimeException("password is null");  
         }  
         userDAO.createUser(userBean);  
     }  
 }      

Saturday, May 24, 2014

How to implement Jackson filters to filter the REST JSON response

Background

We are building an application that supports REST calls to our API. We use JBoss AS 7.1.1 and its default framework 'RestEasy' and Jackson v1.9 JSON processor. Since it is a REST application, we may need to provide provision for the clients to filter the JSON response. As other RESTFul service application does, we planned to have a query parameter called "fields" to do this. For instance, the URL that the client has to submit will be,https:///rest/product/123?fields=id,name

The above GET request URL should return the JSON object of type "product" matching id=123. But note that the response JSON object should contain only the "id" and "name" fields in it as follows: {"id": "123", "name":"xyz"}

Implementation

To do this, we may need to fetch the value of query parameter "fields" from HttpRequest. This could be achieved using javax.ws.rs annotation @QueryParam as given below.


@GET
@Produces (MediaType. APPLICATION_JSON) 
@Path( "/{id:" + ID_PATTERN + "}" ) 
public String getProduct(@PathParam ("id" ) String id, 
        @QueryParam ("fields" ) String fields) { 
    String productJson = null ; 
    final ProductModel model = productService.read(id);         
    try { 
        productJson = filterJsonFields(model, ProductModel.class.getName(), fields); 
    } catch (IOException e) { 
        throw new ApplicationRuntimeException( 
                e.getCause() + ": " + e.getLocalizedMessage()) 
                .setWebStatus(Status. INTERNAL_SERVER_ERROR ); 
    } 
  return productJson ; 
}


After fetching the whole Product object from the database, we could filter the fields of the object while serializing the Java object to JSON object, which will be returned as the response. This filtering mechanism can be done by many ways using Jackson processor. But I know two ways of doing this.
  1. Using @JsonFilter
  2. Overiding the JacksonAnnotationIntrospector

1. Using @JsonFilter

This is the simple way to filter the response object. For this, we need to declare the @JsonFilter annotation on top of our model bean classes. 

@JsonFilter(com.test.ProductModel)  
public class ProductModel implements IModel {  
    private String id ;  
    private String name ;  
    ....  
    ....  
    ...getter and setter methods...
}


Filtering mechanism

 public static String filterJsonFields(IModel model,  
  String jsonFilter, String fieldsQueryParam) throws IOException { 
          
    String jsonResult = null ;  
    SimpleBeanPropertyFilter filter = SimpleBeanPropertyFilter.  
                     serializeAllExcept(Collections.<String> emptySet());  
    Set<String> filterProps = new HashSet<String>();      
    if (fieldsQueryParam != null) {  
        StringTokenizer st = new StringTokenizer(fieldsQueryParam, "," );  
        while (st.hasMoreTokens()) {  
            filterProps.add(st.nextToken());  
        }  
        filter = SimpleBeanPropertyFilter.filterOutAllExcept(filterProps);  
    }         
    ObjectMapper mapper = new ObjectMapper();  
    FilterProvider fProvider = new SimpleFilterProvider().addFilter(jsonFilter, filter)  
                        // setFailOnUnknownId: Ignore filtering the reference member fields  
                        // Refer: https://jira.codehaus.org/browse/JACKSON-650  
                        .setFailOnUnknownId( false );  
    try {  
        jsonResult = mapper.writer(fProvider).writeValueAsString(model);  
    } catch (IOException e) {  
        throw e;  
    }  
    return jsonResult;  
}   
   

The "jsonFilter" method parameter is the class name of the model bean that is passed from the caller method (getProduct()).

Caveat: But this didn't work well with JBoss's RestEasy.


2. Overriding the JacksonAnnotationIntrospector

In this method, we need to extend the JacksonAnnotationIntrospector and override the method "findFilterId". During serialization, Jackson calls the JacksonAnnotationIntrospector.findFilterId() to identify the filter of the model bean. In our case, we are using the name of the model bean class as its filter name. 

import org.codehaus.jackson.map.introspect.AnnotatedClass;  
import org.codehaus.jackson.map.introspect.JacksonAnnotationIntrospector;  
   
public class MyFilteringIntrospector extends JacksonAnnotationIntrospector {  
       
    @Override  
    public Object findFilterId(AnnotatedClass ac) {  
        Object id = super .findFilterId(ac); 
        if (id == null ) {  
            // the name of the class will be used as its filter name  
            id = ac.getName();  
        }  
        return id;  
    }  
} 

We may need to tell Jackson to use our custom AnnotationIntrospector and not JacksonAnnotationIntrospector. For this, 

AnnotationIntrospector myFilterIntrospector = new MyFilteringIntrospector();
mapper.setAnnotationIntrospector(myFilterIntrospector );

Hence the filterJsonFields() method is updated with the above lines of code.


Filtering mechanism

 public static String filterJsonFields(IModel model,  
           String jsonFilter, String fieldsQueryParam) throws IOException {  
   
    String jsonResult = null ;  
    SimpleBeanPropertyFilter filter = SimpleBeanPropertyFilter.  
                    serializeAllExcept(Collections.<String> emptySet());  
    Set<String> filterProps = new HashSet<String>();      
    if (fieldsQueryParam != null) {  
        StringTokenizer st = new StringTokenizer(fieldsQueryParam, ",");  
        while (st.hasMoreTokens()) {  
            filterProps.add(st.nextToken());  
        }  
        filter = SimpleBeanPropertyFilter.filterOutAllExcept(filterProps);  
    }       
    ObjectMapper mapper = new ObjectMapper();  
    AnnotationIntrospector myFilterIntrospector = new MyFilteringIntrospector();  
    mapper.setAnnotationIntrospector(myFilterIntrospector);  
    FilterProvider fProvider = new SimpleFilterProvider().addFilter(jsonFilter, filter)  
                  // setFailOnUnknownId: Ignore filtering the reference member fields  
                  // Refer: https://jira.codehaus.org/browse/JACKSON-650  
                  .setFailOnUnknownId( false );  
    try {  
        jsonResult = mapper.writer(fProvider).writeValueAsString(model);  
    } catch (IOException e) {  
        throw e;  
    }  
    return jsonResult;  
}   

Friday, May 23, 2014

How to enable RestEasy in JBoss AS 7.1 to use Jackson processor

Problem 

The environment used is JBoss AS 7.1 which owns RestEasy (a framework to build RESTful webservices). JBoss by default uses RestEasy framework. I added the Jackson annotation @JsonSerialize (include=JsonSerialize.Inclusion.NON_NULL) on top of my REST model classes to filter the NULL fields from the JSON response during serialization. But, RestEasy ignored this Jackson annotation.

Solution

With reference to some forums in internet, I came to know the following. To enable Jackson processor for RestEasy, the below maven dependency and configuration need to be done. If not, RestEasy may use Jettison processor.

1. Maven Jackson dependency
<groupId>org.jboss.resteasy</groupId>  
      <artifactId>resteasy-jaxrs</artifactId>  
      <version>${version.jboss.resteasy}</version>  
      <scope>provided</scope>  
 </dependency>  
   
 <dependency>  
      <groupId>org.jboss.resteasy</groupId>  
      <artifactId>resteasy-jackson-provider</artifactId>  
      <version>${version.jboss.resteasy}</version>  
      <scope>provided</scope>  
 </dependency>  
2. Create a file under named "jboss-deployment-structure.xml" under the WEB-INF folder and include the jackson modules in the dependencies.
<? xml version= "1.0" encoding = "UTF-8"?>  
< jboss-deployment-structure>  
   <deployment >  
     < dependencies>  
       <!-- added the below jackson modules to enable JBoss RestEasy to  
       use Jackson -->  
       < module name= "org.codehaus.jackson.jackson-core-asl" />  
       < module name= "org.codehaus.jackson.jackson-mapper-asl" />  
     </ dependencies>  
   </deployment >  
 </ jboss-deployment-structure>  

Monday, September 23, 2013

Liferay: Create a quartz job scheduler

In two ways a job can be scheduled in Liferay.
  1. Configure  in liferay-portlet.xml
  2. Implement it in code.
1. Configure  in liferay-portlet.xml
  1. Override the property 'scheduler.enabled = true' in portal-ext.properties.
  2. Create a new dummy portlet in liferay-portlet.xml as follows,
     <portlet>  
         <portlet-name>job</portlet-name>  
         <scheduler-entry>  
             <scheduler-event-listener-class>com.madrone.portal.job.PortalJob</scheduler-event-listener-class>  
             <trigger>  
                 <simple>  
                     <simple-trigger-value>5</simple-trigger-value>  
                     <time-unit>minute</time-unit>  
                 </simple>  
                 <!-- cron>  
                     <property-key>0 0 0 * * ?</property-key>  
                 </cron-->  
             </trigger>  
         </scheduler-entry>  
     </portlet>  
    
  3. Add this portlet in portlet.xml
     <portlet>  
          <portlet-name> job</portlet-name>  
          <display-name>Job</display-name>  
          <portlet-class>  
              com.madrone.portal.job.PortalJob  
          </portlet-class>  
          <supports>  
               <mime-type>text/html</mime-type>  
               <portlet-mode>view</portlet-mode>  
          </supports>  
          <security-role-ref>  
               <role-name>administrator</role-name>  
          </security-role-ref>  
          <security-role-ref>  
               <role-name>guest</role-name>  
          </security-role-ref>  
          <security-role-ref>  
               <role-name>power-user</role-name>  
          </security-role-ref>  
          <security-role-ref>  
               <role-name>user</role-name>  
          </security-role-ref>  
     </portlet>  
    
  4. Create a job listener
     public class PortalJob implements MessageListener {  
         private static final Log LOG = LogFactory.getLog(PortalJob.class);  
        
         @Override  
       public void receive(Message arg0) {  
         LOG.info("PortalJob: Starts" );  
         // your code   
         LOG.info("PortalJob: Stops" );         
       }  
     }  
    
2. Implement it in code. 
  1. Override the property 'scheduler.enabled = true' in portal-ext.properties.
  2. Create an action class which is run during server startup
     public class PortalStartupAction extends SimpleAction {  
       private static final Log LOG = LogFactory.getLog(PortalStartupAction .class );  
       
       @Override  
       public void run(String[] arg0) throws ActionException {  
         LOG.info("Preparing to Schedule the job: PortalJob" );  
         Thread thread = Thread. currentThread();  
         try {  
                 SchedulerEngineUtil. schedule(PortalJob.SCHEDULER_ENTRY, thread.getContextClassLoader());  
         } catch (SchedulerException e) {  
           throw new ActionException(e);  
         }     
         LOG.info("PortalJob scheduled successfully." );  
       }  
     }  
    
  3. Configure the above action class in portal-ext.properties
     #  
     # Application startup event that runs once for every web site instance of  
     # the portal that initializes.  
     #  
     application.startup.events=com.madrone.portal.util.PortalStartupAction  
    
  4. Create a job
     public class PortalJob implements MessageListener {  
         private static final Log LOG = LogFactory.getLog(PortalJob.class);  
       
         @SuppressWarnings("serial" )  
         public static final SchedulerEntry SCHEDULER_ENTRY = new SchedulerEntryImpl() {  
              {  
              setDescription( "Job that get triggered every 5 minutes");  
                
              setTriggerType(TriggerType. SIMPLE);  
              setTimeUnit(TimeUnit. MINUTE);  
              setTriggerValue( "5");  
                
              //setTriggerType(TriggerType.CRON);  
              //setTriggerValue("0 0 0 * * ?"); // Fire at 12:00 AM every midnight  
              // Reference: http://www.quartz-scheduler.org/documentation/quartz-1.x/tutorials/crontrigger  
              setEventListenerClass(PortalJob.class.getName());  
              }  
         };  
       
         @Override  
         public void receive(Message arg0) {  
             LOG.info("CRPortalERPJob: Starts" );  
                 // your code  
           LOG.info("CRPortalERPJob: Stops" );         
       }  
     }  
    

Wednesday, September 4, 2013

Observer pattern

Java has two utility classes to serve Observer pattern - Observer and Observable. 
  • The aim of this pattern is to notify the objects of Observer class, when a change is done to the objects of Observable class.
  • Observer class should implement Observer interface and override the method 'update'.
  • Observable class should extend Observable class. The key method of this class is 'setChanged()'. This marks the Observable object being changed, only then the notifyObservers() will notify the observer objects that a change was done to the observable object. The notifyObservors() makes call to the Observor's update() method.
  • How do the observable object knows its observers? The observers should be added to the observable objects using the methods addObserver() of Observable class. An observer can be added or removed from the observable object.
 
1:  import java.util.ArrayList;  
2:  import java.util.List;  
3:  import java.util.Observable;  
4:  import java.util.Observer;  
5:    
6:  public class ObserverPattern {  
7:        
8:      public static void main(String []args) {          
9:          Stock s = Stock.getInstance();  
10:          s.setName("Madrone");  
11:          s.setNumShares(10);  
12:            
13:          // Observers  
14:          List<Buyer> buyerList = new ArrayList<Buyer>();  
15:          buyerList.add(new Buyer("A"));  
16:          buyerList.add(new Buyer("B"));  
17:            
18:          for(Buyer b : buyerList) {  
19:              s.addObserver(b);  
20:          }  
21:            
22:          for(int i=0; i<10; i++) {  
23:              s.setNumShares(s.getNumShares() + 1); // Observed object is modified   
24:              s.notifyObservers();  
25:          }      
26:            
27:          System.out.println("Shares available: " + s.getNumShares());  
28:          for(Buyer b : buyerList) {  
29:              System.out.println(b.getName() + " : " + b.getNumShares());  
30:          }  
31:      }  
32:  }  
33:    
34:  class Buyer implements Observer {  
35:        
36:      private String name;  
37:      private int numShares;  
38:        
39:      Buyer(String name) {  
40:          this.name = name;  
41:      }  
42:    
43:      @Override  
44:      public void update(Observable observable, Object ob) {          
45:          Stock s = (Stock) observable;  
46:          if(s.getNumShares() > 0) {  
47:              buyAShare();  
48:              s.setNumShares(s.getNumShares() - 1);  
49:          }      
50:      }  
51:        
52:      private void buyAShare() {  
53:          this.setNumShares(this.getNumShares() + 1);  
54:      }  
55:        
56:      public String getName() {  
57:          return name;  
58:      }  
59:      public void setName(String name) {  
60:          this.name = name;  
61:      }  
62:      public int getNumShares() {  
63:          return numShares;  
64:      }  
65:      public void setNumShares(int numShares) {  
66:          this.numShares = numShares;  
67:      }  
68:  }  
69:    
70:  // Singleton class  
71:  class Stock extends Observable {  
72:        
73:      private String name;  
74:      private int numShares;  
75:        
76:      public static final Stock INSTANCE = new Stock();  
77:      private Stock() {}  
78:            
79:      public static Stock getInstance() {  
80:          return INSTANCE;      
81:      }  
82:        
83:      public String getName() {  
84:          return name;  
85:      }  
86:      public void setName(String name) {  
87:          this.name = name;  
88:      }  
89:      public int getNumShares() {  
90:          return numShares;  
91:      }  
92:      public void setNumShares(int numShares) {  
93:          this.numShares = numShares;  
94:          setChanged();  
95:      }  
96:  }  

Wednesday, July 10, 2013

Oracle jdbc driver was unable to connect the database server

Today we faced a problem that a scheduled job in our production server was unable to connect to Oracle database server. The error reported was,

java.sql.SQLRecoverableException: No more data to read from socket
        at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1157)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531)
        at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:204)
        at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1041)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1329)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
        at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3665)
        at oracle.jdbc.driver.OracleCallableStatement.executeUpdate(OracleCallableStatement.java:4739)
        at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeUpdate(OraclePreparedStatementWrapper.java:1352)
        at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
        at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)

We found some details in the oracle forums that the cause for this could be due to the driver version incompatibility with the Oracle database version. But on analyzing the logs, we found that the job was failing only for the past two days. After checking this with the Oracle database team, we were informed that there was no change or upgrade done to the oracle database, but they have changed the infrastructure of the servers, which means the IP address of the database server was changed. But this should not affect us since we have configured the server url with the host name and not the IP address.

On further analysis, we found that this was due to the dns cache which was enabled (networkaddress.cache.ttl=-1) in the jvm (where the production tomcat was running) in  ~\jre\lib\security\java.security

'networkaddress.cache.ttl' is a security property which works for the below settings:

# any negative value: caching forever   (default)
# any positive value: the number of seconds to cache an address for
# zero: do not cache   
 
It is also advised that in the comments that setting this to anything other than the default value can have serious security implications. Do not set it unless you are sure you are not exposed to DNS spoofing attack.

Back to the problem - Since the dns caching was enabled, the old IP address was cached which was the cause for this problem. As a fix to this problem, we restarted the tomcat servers to flush the jvm dns cache.

Friday, July 5, 2013

Basic Git commands

1. Create a Git repository and push it to remote

Open the Git bash (Unix bash prompt) for the respective folder and execute the following commands

git init
git add *
git commit -m "initial commit message"
git remote add origin https://github.com/xxxxx/xxxx.git
git push -u origin master
 

2. Clone repository

 git clone https://github.com/xxxxx/xxxx.git

3. Create new branch

git branch branch-name
git checkout branch-name

or

git checkout -b branch-name

4. Delete a local branch

git branch -d branch-name

Delete a Remote branch 

git push origin --delete branch-name

Tips to configure Git in Windows

We were facing the problem with 'EGit- Synchronize workspace' which showed all the files as changed, even if we haven't changed the content of those files. This happened only when we clone/ pull the repository using Git bash (Unix bash prompt). As we all know the reason for this problem - Windows uses both a carriage-return character and a linefeed character for newlines in its files, whereas Linux/ Mac use only the linefeed character.
Git Hub provides some configurations to override these cross-platform issues.
Do the following before cloning the remote repository, I have the folder structure as C:\Git\Project\, where I would clone the repository inside this folder.
  1. Open Git bash at C:\Git\Project\
  2. Enter, git config --system core.autocrlf true and click enter
  3. Enter, git config --system core.whitespace cr-at-eol and click enter
  4. Then clone the repository as usual, git clone 'https://github.com/xxxxx/xxxx.git' 
  5. Then open the local repository in your Eclipse and import projects. 
  6. Right click on your projects > Team > Share to Git
  7. Right click on your projects > Team > Synchronize workspace
  8. In your Team Synchronize perspective, check the list of modified files. No files will be shown.

To know more about the commands that we used in step 2 and 3, refer http://git-scm.com/book/en/Customizing-Git-Git-Configuration

Thursday, July 4, 2013

Memory leak and performance analysis

Tools:

JVM Profiler - We have a plugin for it which could be installed in Eclipse.
MAT - Memory Analyzer Tool for analyzing the heap dump.



JVM monitor plugin for Eclipse


http://www.jvmmonitor.org/doc/index.html

http://help.eclipse.org/helios/index.jsp?topic=%2Forg.eclipse.tptp.platform.doc.user%2Fconcepts%2Fceproftl.xhtml

Note:
CW-Tomcat is running with java 32 bit and Eclipse also should be 32 bit.


Steps:
  1. Installed Eclipse helios 32 bit for windows
  2. Installed Liferay Plugin, EGit plugins for Eclipse
    1. Through Install New Software -  http://releases.liferay.com/tools/ide/eclipse/helios/stable/
    2. Through Install New Software -  http://download.eclipse.org/egit/updates-2.1
  3. Installed JVM Monitor plugin for Eclipse
    1. Through Eclipse Marketplace - Search JVM monitor and Click Install
    2. Through Preferences > Java > Monitor > Tools, Set JDK Root directory as C:\....\Softwares\jdk1.6.0_21
  4. Launch eclipse helios 32 bit from command prompt making Eclipse to use the same jdk that is set for JVM monitor, C:\...\Softwares\eclipse-jee-helios-SR2-win32\eclipse>eclipse -vm "C:\Mahesh\Softwares\jdk1.6.0_21\bin\javaw"

Monitoring JVM on remote host

To monitor JVM on remote host:

  1. Add the following configurations to JAVA_OPTS in ..../WAR/tomcat_folder/bin/setenv.sh
        -Dcom.sun.management.jmxremote.port=9876
        -Dcom.sun.management.jmxremote.ssl=false
        -Dcom.sun.management.jmxremote.authenticate=false
  2. Press the button Add JVM Connection on Remote Host on JVM Explorer to open New JVM Connection dialog. 
    1. Enter the ip address of the remote server
    2. Enter the port: 9876
Note: Sometimes we may not be able to connect to remote server even after setting the above three JVM arguments. In that case, add the below argument as well.        
              -Djava.rmi.server.hostname= mention the host-name here


How to capture heap dump of JVM running in Remote host

Since 'JVM monitor' does not support viewing the Memory details when monitoring remote host, we may need to capture the heap dump for analyzing the heap memory usage. Below are the steps to capture the heap dump (Ref: http://www.jvmmonitor.org/doc/, under section "How can I enable the BCI mode of CPU profiler on remote host?")
  1. In the Eclipse JVM monitoring view, select the server to monitor. In the Properties screen, you will find Timeline, Threads, Memory, CPU, MBeans and Overview. Select 'Memory' tab.
  2. Click on the 'heap dump' symbol in the top-right corner to capture the heap dump.
  3. A dialog box will be shown with the path where the dump file should be saved in the remote server (eg. /home/liferay\32323432323.hprof). And it may not allow you to select the 'Transfer ....' checkbox or to click on OK button to initiate the dump. An info could be displayed saying that ' agent is not loaded'.
  4. Now follow the instructions given in  http://www.jvmmonitor.org/doc/, under section "How can I enable the BCI mode of CPU profiler on remote host?"
  5. Copy the jvmmonitor-agent.jar from ~..\eclipse-jee-helios-SR2-win32\plugins\
    org.jvmmonitor.core_3.8.1.201302030008\lib\jvmmonitor-agent.jar
    and drop it
    into '/home/liferay' directory of Staging server (remote server) using WinSCP tool.
  6. Add the following configurations to JAVA_OPTS in  .../WAR/tomcat_folder/bin/setenv.sh
                     -javaagent:/home/liferay/jvmmonitor-agent.jar     

     7. Done. Now we are ready to dump the file. The heap dump .hprof file will be saved at '/home/liferay/'

How to use JVM Profiler:




Suggestions from different sites:

It is definitely advisable to have on the -XX:+HeapDumpOnOutOfMemoryError setting.   What is dumped is a binary .hprof file.   The setting for the file location shuold be:
-XX:HeapDumpPath=/niku/logs/heapdump.hprof


Setup for this in setenv.sh
-XX:HeapDumpOnOutOfMemoryError –XX:HeapDumpPath=d:\heapDumps




 How to open or analyze the heap dump .hprof files?

Some of the tools suggested in net are,
  1. MAT - Eclipse Memory Analyzer Tool - (Free) - http://www.eclipse.org/mat/downloads.php
  2. Visual VM - https://visualvm.dev.java.net/
  3. HAT -  Java Heap Analysis Tool - http://docs.oracle.com/javase/6/docs/technotes/tools/share/jhat.html
  4. YourKit Java Profiler
 I used MAT to analyze the heap dump (.hprof file).


MAT for Windows 32 bit was unable to open large dump files. (I tried with a .hprof file of  ~2.2 GB). Also the max heap size of MAT-32 Bit was by default 1GB, which cannot be tweaked. As per suggestion from an Eclipse forum - installed MAT for Windows 64 bit which did the job and allowed to tweak the max heap size. Have configured the vm argument -Xmx to 3GB in MemoryAnalyzer.ini file. Then the heap dump loaded in MAT without errors.