java.lang.SecurityException When Shutdowning JBoss

We can run following command when starting JBoss. We assume there are already jboss system user in OS.

# Centos
su jboss -c '$JBOSS_HOME/bin/run.sh -b 0.0.0.0 > /dev/null 2> /dev/null &'
# Ubuntu
sudo -u jboss $JBOSS_HOME/bin/run.sh -b 0.0.0.0 > /dev/null 2> /dev/null &

Stopping jboss actually should be done by running this

# Centos
su jboss -c '$JBOSS_HOME/bin/shutdown.sh -S &'
# for ubuntu
sudo -u jboss '$JBOSS_HOME/bin/shutdown.sh -S &'

But in my system this will Exception in thread "main" java.lang.SecurityException: Failed to authenticate principal=null, securityDomain=jmx-console. This is happen because likely we have enabled the interceptor in jmx-invoker-service.xml file in deploy folder. Then for shuttingdown jboss we should provide some credential that we can see at login-config.xml file in the conf folder.

So the shutdown command should be like this
# Centos
su jboss -c '$JBOSS_HOME/bin/shutdown.sh -S -u admin -p admin &'
# for ubuntu
sudo -u jboss '$JBOSS_HOME/bin/shutdown.sh -S -u admin -p admin &'

Actually there still some exception when running this.

javax.management.InstanceNotFoundException: jboss.remoting:service=invoker,transport= bisocket,host=sudirman.nfsint.com,port=4457,JBM_clientMaxPoolSize=200,clientLeasePeriod=10000,clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper,dataType=jms,marshaller=org.jboss.jms.wireformat.JMSWireFormat,numberOfCallRetries=1,numberOfRetries=10,pingFrequency=214748364,pingWindowFactor=10,socket.check_connection=false,timeout=0,unmarshaller=org.jboss.jms.wireformat.JMSWireFormat is not registered.

and

2011-07-27 10:06:07,835 ERROR [org.jboss.remoting.transport.Connector] invalid Object Name
javax.management.InstanceNotFoundException: jboss.remoting:service=invoker,transport= socket,host=sudirman.nfsint.com,port=3873 is not registered.

But after running

ps ax | grep java

I cannot see any JBoss instance that still running. So I conclude that shutdowning has been done correcly. I will check the later issue when have time. Meanwhile those shutdowning command can be put into some script.


reference

JBoss AS 7 has Released

It is little bit surprising when I browse to theserverside this evening and found a news that announce JBoss 7 has been released. I am surprised because what I remember JBoss 6 was released not too long a go. When I see the release date, JBoss 7 is released only 6 months after the JBoss 6.

JBoss application server is very important for me. Since my first step as Java Enterprise developer JBoss is the application server I have used most often. The funny thing, even though the version now already at 7 number, I myself still use the JBoss 4 version for development. This is happen because most of the project that I handled still use that version.

These are some important features of JBoss 7 Application server. Fast start up (< 3 seconds), lighweight, modular core, hot incremental deployment, elegant administration, domain management, first class component. I still don't have many comments for those features because I have to dirt my hand first playing the application. I will write another post later.

Troubleshooting When Cron Job wont Executed

What to check when Cron job/script wont executed? One of the possible answer is maybe the PATH declaration in your script which is not correct. Here are the sample story.

Currently I have to do application profiling in aim to analyst the posible bottleneck in application. The tools that I use like I have mentioned before is JBoss Profiler. Fortunately, this tool can be integrated easily with existing JBoss application server. JBoss profiler itself consist two modules. The first one is several jar files and configurations that is dipped into application server. The second part is small client Java application that will snapshot and produce profile report for current situation of method/function graph call from our application source code.

We have necessity to run the client snapshot application for every particular range time. The mean of this approach is when application that we profile start to run slowly in some particular module and some particular time we can dig to profile report and analyze the possible source of problem. To achieve this requirement we need some job scheduler tools. Because most of server live in Unix environment, we are very lucky because we have well known job scheduler in most Unix box. The scheduler name is Cron.

Scheduling a job with Cron is only the matter of editing crontab definition. Crontab itself is a configuration file that consist a list of job/command that will be executed for every particular time. This is the content of my Cron file. See the last line of the Cron files.

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin
MAILTO=root
HOME=/

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
#* * * * * root topCat.sh
#* * * * * root iostatCat.sh
* * * * * root snapshot.sh # this will be run every minute

This is the snapshot.sh script content.

#! /bin/sh
#This script is intended to take profiling snapshot

export PATH=$PATH:/usr/local/jdk1.6.0_20/bin
cd /opt/jboss-profiler-2.0.0.Beta5
# run the profiler snapshot capturer
java -Xmx512m -Djboss-profiler-client.properties=jboss-profiler-client.properties -jar jboss-profiler-client.jar snapshot
# move the produced report to other folder, in this case is to apache html folder so we can see remotely from browser client 
mv 201107* /var/www/html/profiler

#exit 0

I have run this configuration in my local successfully. But when I clone all the setting into the server the process seems not work as expected. I became more curious because after catching the log of cron tail -f /var/log/cron (or /var/log/syslog on ubuntu) the script actually already executed. The other of my confusedness, the script itself is worked perfectly if we executed it directly on the shell.

Jul 13 11:02:01 ID31-ND102 crond[13350]: (root) CMD (snapshot.sh)
Jul 13 11:03:01 ID31-ND102 crond[13368]: (root) CMD (snapshot.sh)
Jul 13 11:04:01 ID31-ND102 crond[13397]: (root) CMD (snapshot.sh)
Jul 13 11:05:01 ID31-ND102 crond[13418]: (root) CMD (snapshot.sh)
Jul 13 11:06:01 ID31-ND102 crond[13447]: (root) CMD (snapshot.sh)
Jul 13 11:07:01 ID31-ND102 crond[13465]: (root) CMD (snapshot.sh)
Jul 13 11:08:01 ID31-ND102 crond[13493]: (root) CMD (snapshot.sh)
Jul 13 11:09:01 ID31-ND102 crond[13512]: (root) CMD (snapshot.sh)
Jul 13 11:10:01 ID31-ND102 crond[13544]: (root) CMD (snapshot.sh)
Jul 13 11:10:01 ID31-ND102 crond[13546]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jul 13 11:11:01 ID31-ND102 crond[13564]: (root) CMD (snapshot.sh)
Jul 13 11:12:01 ID31-ND102 crond[13593]: (root) CMD (snapshot.sh)
Jul 13 11:13:01 ID31-ND102 crond[13612]: (root) CMD (snapshot.sh)
Jul 13 11:14:01 ID31-ND102 crond[13644]: (root) CMD (snapshot.sh)

Then after some hours of debugging I realize one possible source of problem is in PATH declaration in crontab files.

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin

Because I run java command then I have to add the path declaration to be like this.

PATH=/usr/local/jdk1.6.0_20/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin

Then the scheduler can run the script perfectly.

Notes: This command sometime takes long time to run. Please refer to jboss profiler documentation for more information.

java -Xmx512m -Djboss-profiler-client.properties=jboss-profiler-client.properties -jar jboss-profiler-client.jar snapshot

Detail Guide to Use JBoss Profiler

Long time I haven't write new post. Here the guide about remote profiling.

According to wikipedia, software profiling is a process of optimizing source code by analyzing the application running by profiler tools. By using profiler we can know how many times a method or function is invoked and also how long is taken for such invocation.

In the field Java, there so many profiler tools. Most known IDE usually have built in profiler tools. But there is sometime a need to do remote profiling. E.g profiling Java EE application that run in server. If we use JBoss application server then we are blessed by existence of JBoss profiler.

Excerpt from its website JBoss Profiler is a log based profiler using JVMPI and JVMTI. It uses an agent written in C that captures events from the JVM and logs to disk. A web application running on JBoss or another machine can be used to analyze these logs through a web browser.

The process of installing the profiler is very easy. First of course we need to download the newest version of the software from this link . In the time this post written I was using version 2.0 Beta5. Extract the files to some directory. I extract that to /opt directory. So the files will be available at /opt/jboss-profiler-<version>.

After extracting do the following steps exactly.

1. Copy jboss-profiler.jar to jbossas/bin
2. Copy jboss-profiler.properties to jbossas/bin
3. Copy javassist.jar to jbossas/bin
4. Edit jboss-profiler.properties in jbossas/bin to include the classes to be profiled
5. Copy jboss-profiler-plugins.jar to jbossas/bin
6. Edit run.conf (Unix) or run.bat (Windows) in jbossas/bin to include JBoss Profiler in JAVA_OPTS add the following argument: -javaagent:jboss-profiler.jar -Djboss-profiler.properties=jboss-profiler.properties
7. Copy jboss-profiler.sar to jbossas/server/<conf>/deploy

Note: you don't need to do step number 3 if you use JBoss 5 or JBoss 6 application server.

After finished copying the needed files, open jboss-profiler.properties. The most important key is "includes". Add the java package you wish to profile. After it's done, we are ready to do profiling.

Start the server then run the following command from directory of our extracted jboss-profiler. In my case I extract to /opt/jboss-profiler-<version>. The command is

java -Xmx512m -Djboss-profiler-client.properties=jboss-profiler-client.properties -jar jboss-profiler-client.jar

After running the that there will be a list of availables commands as follow.

startProfiler : Start the profiler
stopProfiler : Stop the profiler
snapshot : Take a snapshot
getSnapshot : Get a snapshot
listSnapshots : List snapshots
clearSnapshots : Clear snapshots
gc : Trigger garbage collection
enable : Enable the profiler
disable : Disable the profiler
load : Load a snapshot
save : Save a snapshot
diff : Compare snapshots
add : Add classes (public|package|protected|private)
remove : Remove classes
list : List classes

The most important command is snapshot. Everytime we run this command the profiler client will produce profiler report in current directory. The report consist html files with detail of method invocation for every classes that we have included in configuration.

Some problem that might appear:
1. Forget copying jboss-profiler.sar to jbossas/server/<conf>/deploy will make jboss-proiler-client.jar won't run
2. Make sure to chmod every jar in /opt/jboss-profiler-<version> directory to be excecuted. Otherwise the profiler client also won't run.

Happy profiling and let's find the thing to optimize let's kill the memory leak.