Rock-Solid Liferay Plugin Deployments

This blog post describes the overall process recommended by Xtivia for deploying custom Liferay plugin applications to Liferay.  Note that while this document is primarily targeted at the deployment of custom Liferay plugins, many of the concepts do apply to non-Liferay web applications being deployed to the Tomcat application server.

Hot deployments in Liferay bundles

Liferay bundles normally have Apache Tomcat’s hot deployment feature enabled; while hot deployment is a convenient process that allows the deployed artifacts to be available immediately, there are a number of problems that arise when Tomcat hot deployment is enabled which will cause the system to become unstable over time. As a result, Xtivia highly recommends disabling hot deployment at the application server level. Hot deployment is not recommended to be enabled on any environment except for lightweight testing purposes on local developer systems.

Problem Definition

The basic issues presented by the use of hot deployment on Apache Tomcat include the following:

• Ongoing leaks in the JVM’s permanent generation memory space. Enabling hot deployments on a Tomcat application server instance will cause the amount of memory consumed by the permanent generation to increase over time, eventually resulting in an outage. This is caused by the way that Tomcat handles web application deployments; the only remediation step is to restart the application server instance.
• Tomcat has also been observed to have problems cleaning up the global classloader during deployments, which can lead to class loader errors within the JVM, again causing an outage.
• The JVM will at times run into conflicts with the operating system when attempting to update filesystem content during a deployment; this can result in corrupted application deployments, as in-memory content fails to overwrite files which are marked as locked on the filesystem. These issues are extremely difficult to detect or triage.
• The preprocessing that Liferay does to any plugin is recommended to be executed at deploy time in order to minimize build overhead and avoid potential mismatches between the the target Liferay installation and the custom Liferay plugin.

Approach

The overall approach taken for persistent multi-user Apache Tomcat installations is to disable hot deployment at the application server level and include an application server restart in the application deployment process. The specifics of implementation may vary, depending on the delivery toolchain in place, but the overall process is as follows:

1. Disable Apache Tomcat’s hot deployment processor via modifications to the Tomcat configuration files.
2. Modify Liferay’s deployment process to force it to deploy an atomic WAR file, rather than an exploded application tree.
3. For each deployment, remove the target Tomcat instance from circulation during the deployment process.
4. During a deployment execution, clear the directories that Tomcat uses to store copies of the applications at runtime.
5. Execute validation of the deployment process prior to placing the Tomcat instance back in circulation.

Technical Details

Apache Tomcat Configuration

To turn off auto deployment in Tomcat, a configuration change needs to be made to Tomcat’s ${CATALINA_BASE}/conf/server.xml file. An attribute named autoDeploy with a value of “false” needs to be added to the Host Entity nested within the Engine defined within the server.xml file. An example follows:

<Service name="Catalina">
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" />
    <Engine name="Catalina" defaultHost="localhost">
        <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="false">
        </Host>
    </Engine>
</Service>

Liferay Configuration

To prevent the Liferay deployment process from automatically expanding deployed plugins into the ${CATALINA_BASE}/webapps directory, the following configuration needs to be added to the Liferay portal-ext.properties file for the instance.

auto.deploy.unpack.war=false

The purpose of this change is to provide some differentiation between applications that have only been processed by Liferay and applications which have been fully deployed by an application server restart.

Standard Plugin Deployment Process

Before any deployments are performed on a Liferay instance, that instance should be taken out of circulation at the load balancer or web server level. This prevents traffic from reaching the instance on which deployments are being performed, eliminating the risk that end user requests will inadvertently be routed to the target server during the deployment process. This has the secondary benefit of providing a safe window after the deployment has been done during which deployment validation can occur. Details on how to remove an individual Tomcat instance from circulation depend heavily on the load balancer or web server in use, and are beyond the scope of this document.

Once the application server has stopped receiving inbound traffic from the load balancers/web servers, the following steps should be executed:

1. Move the target web application WAR file from the Tomcat ${CATALINA_BASE}/webapps directory to a temporary backup location. If any step of the deployment process fails, restore to the previous state by copying this backed up WAR file back into the ${CATALINA_BASE}/webapps directory.
2. Delete the directory containing the expanded version of the target application from the ${CATALINA_BASE}/webapps directory.
3. Copy the plugin WAR that you want to be deployed to the Liferay deploy directory, located in ${liferay.home}/deploy.
4. Verify that the operating system user that owns the Liferay process has full write permissions on the copied artifact.
5. Wait for the new version of the application WAR to be available in the ${CATALINA_BASE}/webapps directory. Typically this is denoted by Liferay in the catalina.out and liferay-*.log files with the following message:

Deployment will start in a few seconds

At this point the Liferay plugin deployment processing activity is complete.

1. Stop the Liferay Apache Tomcat application server process.
2. Delete the contents of the ${CATALINA_BASE}/temp and ${CATALINA_BASE}/work directories.
3. Start the Liferay Apache Tomcat application server process.
4. Perform functional validation testing for each of the deployed applications.
5. Remove the original target web application from the temporary backup location.

Once this process is complete, the Apache Tomcat server can be placed back into circulation at the load balancer or web server level.

Liferay EXT Plugin Deployment

An edge case for the Liferay plugin deployment process is present for Liferay EXT plugins. These plugins actually modify the installed instance of the Liferay application itself, and they often include JAR files which need to be included in the application server’s global classpath; to accommodate these additional requirements, we recommend using a process similar to the one used to apply and deploy Liferay application patches/hotfixes. A sample process for deploying EXT plugins is as follows:

On a clean bundle matching the version of Liferay that you intend to deploy, do the following:

1. Unzip the target bundle into a temporary location.
2. Apply all necessary hotfixes & patches for the target environment to the bundle’s Liferay instance.
3. Start the bundle using the Tomcat startup.sh or startup.bat script.
4. Deploy the EXT plugin to the temporary bundle by dropping it into the bundle’s deploy directory.
5. Wait for the EXT plugin to be deployed by the temporary Liferay instance.
6. Shut down and restart the temporary Liferay instance.
7. Shut down the temporary Liferay instance.
8. Bundle the contents of the ROOT application in the temporary Liferay Tomcat instance’s webapps directory into a ROOT.war file.
9. Copy the ROOT.war file along with all EXT-generated JAR files from the temporary Liferay Tomcat instance’s lib/ext directory into a central location for deployment.
10. Remove the temporary Liferay instance.

On each of the application servers targeted for deployment, execute the following:

1. As for standard plugins, first remove the target Tomcat instance from the load balancer or web server.
2. Stop the Liferay Apache Tomcat application server process.
3. Move the following files from the Tomcat ${CATALINA_BASE}/webapps directory to a temporary backup location. If any step of the deployment process fails, restore to the previous state by copying these backed up files back into the ${CATALINA_BASE}/webapps directory.
   1. ROOT.war
   2. The WAR file for the target application
4. Move any JAR files which will be added by this EXT plugin from the Tomcat ${CATALINA_BASE}/lib directory to a temporary backup location. If any step of the deployment process fails, restore to the previous state by copying these files back into the source directory.
5. Delete the ROOT directory containing the expanded version of the Liferay Portal application from the ${CATALINA_BASE}/webapps directory.
6. Delete the contents of the ${CATALINA_BASE}/temp and ${CATALINA_BASE}/work directories.
7. Deploy the Liferay Portal application ROOT.war file to the target application server’s {CATALINA_BASE}/webapps directory.
8. Deploy all JAR files gathered from the temporary Liferay bundle’s lib/ext directory to the target application server’s ${CATALINA_BASE}/lib directory.
9. Start the Liferay Apache Tomcat application server process.
10. Perform functional validation testing for each of the deployed applications.
11. Remove the original ROOT.war and JAR files from the temporary backup location.
12. Place the target server back into circulation at the load balancer or web server level.