Posts Tagged JBoss
One of the development app servers that we recently upgraded to Java 6 and JBoss 5.0.0.GA’s hard drive filled up. I emptied 4GB out of the Recycle Bin, which let a build complete Friday, but by this morning it was full again.
A diligent co-worker discovered a server\default\tmp\vfs-nested.tmp\ directory containing 67,444 hexadecimally-named jar files totalling 16GB in size.
Excessive nestedjarNNN.tmp files in the tmp directory. The VFS unwraps nested jars by extracting the nested jar into a tmp file in the java tmp directory. This can result in a large number of files that fill up the tmp directory. You can disable this behavior by setting -Djboss.vfs.forceNoCopy=true on command line used to start jboss. This will be enabled by default in a future release, JBAS-4389.
That was 5.0.0 Beta 4. In 5.0.0.CR1, JBAS-4389 made NoCopy the default behavior. Also in CR1 though, JBAS-5262 replaced the jboss.vfs.forceNoCopy setting with a jboss.vfs.forceCopy setting defaulting to false; and JBAS-5633 enabled jboss.vfs.forceCopy by default, if unspecified.
SO THAT… by the time you get to the JBoss 5.0.0.GA Release Notes, we see:
JBoss VFS provides a set of different switches to control it’s internal behavior. JBoss AS sets jboss.vfs.forceCopy=true by default.
I gather that to turn off the copying for JBoss5.0.0.GA, we would set
on the JBoss command line.
Update: It looks like when we set the forceCopy flag to false, the heap space may fill up in memory instead. We may need a better solution!
I’ve got the fix locally, just need to create some proper long running tests. But this will be definitely included in the up-coming 5.0.1.
The “deployment fails the second time” issue has been resolved, and the CI servers are back in business.
For those who may be interested, the problem turned out to be a Maven config issue in [the functional test project]’s pom file.
(Note: you may need your techno-gibberish decoder goggles for the next details: )
In making ojdbc available to Liquibase for when we deliver schema changes, the ojdbc jar file got included in the war file. The config has been this way for several weeks, but we didn’t notice it until we just in the last few days got to the point of contributing Hibernate mappings, which starts the Persistence service, which needs ojdbc to actually work. At that time, the ojdbc jar in the war file fought with the ojdbc jar in JBoss’s server/default/lib directory, and just like when you were fighting with your sister over that cookie, nobody won and we got the NoClassDefFoundError.
(Or something vaguely similar to that.)
Thanks to [those who helped] for their help with this!
Now back to our regular programming…
This post is part 5 in a 9-part series on Envers, Spring, and JTA. The next post is beforeCompletion, I need to figure this out, and the previous post is The versioning isn’t happening, part 4: Registering with the wrong transaction?
The time had come to ask my questions at the Spring Data Access forum. Following is a copy of that forum post (Whew, does it take work to try to ask a question well!) :
Sychronization hook not called when using JTA txmgr
We’re using Spring 2.5.5 on JBoss 4.2.2.GA, along with JBoss envers version 1.1.0.GA. (Envers uses Hibernate’s onPostInsert() hook to register a javax.transaction.Synchronization callback that’s called during transaction commit, to write audit records about records that were created during that transaction.)When we use Spring’s HibernateTransactionManager and write a record, the registered envers javax.transaction.Synchronization.beforeCompletion () event handler fires as expected, but when we use Spring’s jta-transaction-manager, the registered envers beforeCompletion() event handler does not fire. In this post, after sketching our configuration, I discuss what I see working the same and different between the HibernateTransactionManager and jta-transaction-manager scenarios, and close by pointing out where I think the problem might be in Spring.
We’ve configured the Hibernate sessionFactory bean to hook it up to envers’ VersionsEventListener (which implements org.hibernate.event.PostInsertEventListener):
<bean id="versionsEventListener" class="org.jboss.envers.event.VersionsEventListener" /> <bean id="sessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean"> ... <property name="eventListeners"> <map> <entry key="post-insert"> <ref local="versionsEventListener" /> </entry> ... </map> </property> ... </bean>
With this configuration, every time a new row is inserted into the main table,
- org.jboss.envers.event.VersionsEventListener.onPos tInsert(PostInsertEvent event) is called
- onPostInsert() passes event.getSession() to org.jboss.envers.synchronization.VersionsSyncManag er.get(EventSource)
- VersionsSyncManager.get() then calls getTransaction() (which returns a JDBCTransaction) on the EventSource
- Finally, VersionsSyncManager.get() calls registerSynchronization() on this transaction to register a javax.transaction.Synchronization object whose beforeCompletion() method should be called during the transaction commit.
It is the code in this beforeCompletion() method that does the writing of the versioning records.
HibernateTransactionManager vs. JtaTransactionManager
When I set up my .war project’s Spring beans to use the Hibernate transaction manager:
<bean id="transactionManager" class="org.springframework.orm.hibernate3.HibernateTransactionManager"> <property name="sessionFactory" ref="sessionFactory" /> </bean>
then the VersionsEventListener.onPostInsert() call happens as I have descibed, and later the registered beforeCompletion() hook is called and envers writes my audit records to the database. But when I remove the above bean and use the JTA transaction manager:
the VersionsEventListener.onPostInsert() call still happens and I do see registerSynchronization() being called; but this time the envers beforeCompletion() hook never gets called (and thus the audit records are not written to the database).
Registering with a JDBCTransaction, Committing a UserTransaction?
That’s the main symptom. I’ll now speculate on the cause… : )
When I’m using the HibernateTransactionManager, I notice that its doCommit() method makes this call:
Here, getTransaction() returns an org.hibernate.transaction.JDBCTransaction (the same one that envers called registerSynchronization() on earlier). As part of this commit(), the JDBCTransaction calls its notifyLocalSynchsBeforeTransactionCompletion() method, which calls the javax.transaction.Synchonization.beforeCompletion( ) registered by envers. So I see how the hook works on the HibernateTransactionManager side. On the JtaTransactionManager side though, org.springframework.transaction.jta.JtaTransaction Manager.doCommit() makes this call:
Here, getUserTransaction() returns the javax.transaction.UserTransaction, and then we call commit() on that. This commit() call does not result in the envers javax.transaction.Synchronization.beforeCompletion () being called. Something I notice here is that envers’ onPostInsert() method caused registerSynchronization() to be called on a JDBCTransaction object, not the UserTransaction object. This JDBCTransaction object appears to be unused for the purposes of committing.
Why I think it might be an issue in Spring
Envers’ VersionsEventListener.onPostInsert(PostInsertEvent event) method receives an org.hibernate.event.PostInsertEvent, and it is from this event that envers eventually calls (effectively):
Envers is registering its Synchronization object with the transaction given to it by the PostInsertEvent — I don’t immediately see how it could do any differently (though I’m far from an expert at these things)…Meanwhile, I don’t see where Spring is informing the UserTransaction about these registered synchronizations, and I don’t see Spring taking care of them itself either. It seems to my uneducated eyes that Spring should be doing one or the other of these, so that the registered callbacks aren’t just lost. Is there something I’m missing?
Thank you for your patience with this long post!
This post is part 4 in a 9-part series on Envers, Spring, and JTA. The next post is The versioning isn’t happening, part 4a: phone a friend, and the previous post is The versioning isn’t happening, part 3: it’s in UserTransaction?
While looking at the javadoc for javax.transaction.Synchronization earlier, I had noticed this:
Using the registerSynchronization method, the application server registers a Synchronization object for the transaction currently associated with the target Transaction object.
Now, the only registerSynchronization() method I see is javax.transaction.Transaction’s registerSynchronization() method, and the one place that registerSynchronization() is called from the envers code is in org.jboss.envers.synchronization.VersionsSyncManager.get(EventSource).
Here’s an interesting thing: when I set a breakpoint on VersionsSyncManager.get(EventSource) and start stepping through the code, the first thing it does (line 47) is do a session.getTransaction(). I expected that underneath, this would be our javax.transaction.UserTransaction (since we’re running in JTA mode), but instead when we call transaction.registerSynchronization() I see that we have a org.hibernate.transaction.JDBCTransaction.
So to step back a moment, what we have here is a Synchronization being registered with a JDBCTransaction, when the actual work will be done by a javax.transaction.UserTransaction. It seems plausible that when the UserTransaction goes to commit, it won’t run these registered synchronizations (since it seems it wouldn’tknow about them).
Why does EventSource.getTransaction() return a JDBCTransaction instead of the UserTransaction?
EventSource extends org.hibernate.Session, and the javadoc for org.hibernate.Session.getTransaction() has this interesting tidbit:
The class of the returned Transaction object is determined by the property hibernate.transaction_factory.
The closest to this that I see in the AnnotationConfig is:
So what does SpringTransactionFactory do?
Hmm: org.springframework.orm.hibernate3.SpringTransactionFactory.createTransaction(JDBCContext, Context) returns a new JDBCTransaction, even if we’re in JTA mode. That is the transaction that is returned (I think) by org.hibernate.Session.getTransaction(). Is that a bug? Or alternatively, is the problem that Spring’s JtaTransactionManager calls commit() on the UserTransaction directly, without taking into account the synchronizations registered on the companion JDBCTransaction object? It seems that one way or another, someone should be taking responsibility for getting those registered callbacks called!
(Next: Time to ask for help!)
This post is part 3 in a 9-part series on Envers, Spring, and JTA. The next post is The versioning isn’t happening, part 4: Registering with the wrong transaction?, and the previous post is The versioning isn’t happening, part 2: beforeCompletion.
We’re trying to figure out why the envers versioning records are not created when our project uses a JTA transaction manager — it creates the versioning records just fine when we’re using local Hibernate transactions.
In part 2, we discovered that while org.jboss.envers.synchronization.VersionsSync.beforeCompletion() is called when we’re running in Hibernate mode, it is not getting called when we’re in JTA mode, even though onPostInsert is being called just like in Hibernate mode.
To dig farther toward the root of this, let’s see where the call to beforeCompletion() comes from when we’re in Hibernate mode…
1. Hibernate mode
I deployed the hibernate-mode version again and sent a message with a method-entry breakpoint on the VersionsSync beforeCompletion() call. This time, I set breakpoints on the lines farther up the call stack — the last six entries in the chain of calls yielding the call to beforeCompletion(). My list of breakpoints now looks like this:
AbstractPlatformTransactionManager [line: 678] - commit(TransactionStatus) AbstractPlatformTransactionManager [line: 709] - processCommit(DefaultTransactionStatus) HibernateTransactionManager [line: 655] - doCommit(DefaultTransactionStatus) JDBCTransaction [line: 109] - commit() JDBCTransaction [line: 228] - notifyLocalSynchsBeforeTransactionCompletion() TransactionAspectSupport [line: 321] - commitTransactionAfterReturning(TransactionInfo) VersionsSync [entry] - beforeCompletion()
2. JTA mode
We do break at TransactionInterceptor(TransactionAspectSupport).commitTransactionAfterReturning(TransactionAspectSupport$TransactionInfo) line: 321 (but… ohhhh, I stepped too slowly and the transaction timed out, yielding a stack trace! Trying again…)
We’re at JtaTransactionManager(AbstractPlatformTransactionManager).commit(TransactionStatus) line: 678 and then JtaTransactionManager.doCommit(DefaultTransactionStatus)… so far this is just like Hibernate mode. org.springframework.transaction.jta.JtaTransactionManager.doCommit(DefaultTransactionStatus) mainly just does txObjext.getUserTransaction().commit() (line 1028) — this is still analogous to what we see on the Hibernate-mode side (there, it’s org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(DefaultTransactionStatus), which mainly just calls txObject.getSessionHolder().getTransaction().commit() (line 655).)
2.1. A closer look at the Hibernate-mode commit() call
Let’s look more closely at this call:
The getTransaction() call returns a org.hibernate.transaction.JDBCTransaction object, which we then call commit() on. And it is this commit() method that calls org.hibernate.transaction.JDBCTransaction.notifyLocalSynchsBeforeTransactionCompletion(), which calls the beforeCompletion() method.
2.2. The Difference on the JTA Side
The call mentioned in step 1.2.1. is what causes beforeCompletion() to be called (which creates the envers versioning records) when we’re in Hibernate mode. The analogous call when we’re in JTA mode is:
(Called by org.springframework.transaction.jta.JtaTransactionManager.doCommit(DefaultTransactionStatus) on line 1028.)
Now, getUserTransaction() returns the javax.transaction.UserTransaction from JBoss; and then we call commit() on that. Somewhere inside that commit() call, javax.transaction.Synchronization.beforeCompletion()* is not being called.
*VersionsSync implements javax.transaction.Synchronization; any beforeCompletion() call in an app server’s UserTransaction would reference the javax.transaction.Synchronization interface rather than org.jboss.envers.synchronization.VersionsSync, I’m sure!
In our next installment, let’s look into why UserTransaction is not calling beforeCompletion().
This post is part 2 in a 9-part series on Envers, Spring, and JTA. The next post is The versioning isn’t happening, part 3: it’s in UserTransaction?, and the previous post is The versioning isn’t happening, part 1: wanderings.
We’re trying to figure out why the envers versioning records are not created when our project uses a JTA transaction manager — it creates the versioning records just fine when we’re using local Hibernate transactions.
Our approach last time was to see what differences there might be in the Hibernate configuration. I ended up that post with
Perhaps next I should capture the values of the rest of AnnotationConfiguration’s 47 fields, and compare them all between jta mode and Hibernate mode.
-1. The Other 47 Fields
After I stopped writing last time, I did go on to capture the 47 AnnotationConfiguration fields’ values when running in Hibernate mode and when running in JTA mode, and… they were the same! (Keith suspected they would be.)
0. No Other Apparent Magic During JBoss Startup
Still running in Hibernate mode, I followed the control flow after the creation of the Hibernate SessionFactory bean, but I didn’t see anything interesting-looking. It’s hard to let the idea go that something’s different at startup time, but I think next I need to take a closer look at what happens during a save to the database.
1. A list of work to do later
Let’s see where the envers onPostInsert() hook gets called from, in hibernate mode (i.e., where things are working correctly). Here’s the call stack:
Daemon Thread [http-127.0.0.1-8080-1] (Suspended (entry into method onPostInsert in VersionsEventListener)) VersionsEventListener.onPostInsert(PostInsertEvent) line: 88 EntityInsertAction.postInsert() line: 131 EntityInsertAction.execute() line: 110 ActionQueue.execute(Executable) line: 279 ActionQueue.executeActions(List) line: 263 ActionQueue.executeActions() line: 167 DefaultFlushEventListener(AbstractFlushingEventListener).performExecutions(EventSource) line: 298 DefaultFlushEventListener.onFlush(FlushEvent) line: 27 SessionImpl.flush() line: 1000 GreetingDAO(GenericDAO<PO,PK>).create(PO) line: 66 GreetingDBSvcImpl.saveGreeting(GreetingPTO) line: 25 GreetingSvcImpl.saveGreeting(GreetingDTO) line: 21 HelloWorldSvcImpl.writeGreetingToDatabase(Greeting) line: 24 NativeMethodAccessorImpl.invoke0(Method, Object, Object) line: not available [native method] NativeMethodAccessorImpl.invoke(Object, Object) line: 39
1.1. It’s all the same in JTA mode
I set breakpoints at most of these places. Let’s try it in jta mode:
- We get to the session.flush() in GreetingDAO(GenericDAO<PO,PK>).create(PO) line: 66
- We get to DefaultFlushEventListener.onFlush(FlushEvent) line: 27
- We get to DefaultFlushEventListener(AbstractFlushingEventListener).performExecutions(EventSource) line: 298
- We get to ActionQueue.executeActions() line: 167 with 1 element in our insertions ArrayList to do, just like in hibernate mode
- We get to EntityInsertAction.execute() line: 110
- We get into EntityInsertAction.postInsert(), with a org.jboss.envers.event.VersionsEventListener and a org.hibernate.search.event.FullTextIndexEventListenerin the postInsertEventListener array
- We even get into VersionsEventListener.onPostInsert(PostInsertEvent)!
So why doesn’t the versioning happen?
It turns out that VersionsEventListener.onPostInsert() isn’t where the real work is done.
1.2. What VersionsEventListener.onPostInsert() is responsible for
VersionsEventListener.onPostInsert() adds a VersionsWorkUnit to a list, to be picked up and performed later. It doesn’t actually do the work.
org.jboss.envers.event.VersionsEventListener.onPostInsert(PostInsertEvent) does is to add the org.jboss.envers.synchronization.work.VersionsWorkUnit to the org.jboss.envers.synchronization.VersionsSync.workUnits linked list (VersionsSync is the class that implements javax.transaction.Synchronization).
3. Who picks up the work unit?
Who is responsible for picking up the work unit, then? I notice that the javax.transaction.Synchronization specifies two methods: beforeCompletion() and afterCompletion(int). Let’s see if anyone calls beforeCompletion() when we’re in Hibernate mode… setting breakpoints on these two methods and re-running my hibernate mode test, we break at beforeCompletion() with this call stack:
Daemon Thread [http-127.0.0.1-8080-1] (Suspended (entry into method beforeCompletion in VersionsSync)) VersionsSync.beforeCompletion() line: 125 JDBCTransaction.notifyLocalSynchsBeforeTransactionCompletion() line: 228 JDBCTransaction.commit() line: 109 HibernateTransactionManager.doCommit(DefaultTransactionStatus) line: 655 HibernateTransactionManager(AbstractPlatformTransactionManager).processCommit(DefaultTransactionStatus) line: 709 HibernateTransactionManager(AbstractPlatformTransactionManager).commit(TransactionStatus) line: 678 TransactionInterceptor(TransactionAspectSupport).commitTransactionAfterReturning(TransactionAspectSupport$TransactionInfo)line: 321 TransactionInterceptor.invoke(MethodInvocation) line: 116 ReflectiveMethodInvocation.proceed() line: 171 ExposeInvocationInterceptor.invoke(MethodInvocation) line: 89 ReflectiveMethodInvocation.proceed() line: 171 JdkDynamicAopProxy.invoke(Object, Method, Object) line: 204 $Proxy68.writeGreetingToDatabase(Greeting) line: not available NativeMethodAccessorImpl.invoke0(Method, Object, Object) line: not available [native method] NativeMethodAccessorImpl.invoke(Object, Object) line: 39 DelegatingMethodAccessorImpl.invoke(Object, Object) line: 25 Method.invoke(Object, Object...) line: 585 JAXWSMethodInvoker(AbstractInvoker).performInvocation(Exchange, Object, Method, Object) line: 136 JAXWSMethodInvoker(AbstractInvoker).invoke(Exchange, Object, Method, List<Object>) line: 82 JAXWSMethodInvoker.invoke(Exchange, Object, Method, List<Object>) line: 97 JAXWSMethodInvoker(AbstractInvoker).invoke(Exchange, Object) line: 68 ServiceInvokerInterceptor$1.run() line: 56 SynchronousExecutor.execute(Runnable) line: 37 ServiceInvokerInterceptor.handleMessage(Message) line: 92 PhaseInterceptorChain.doIntercept(Message) line: 221 ChainInitiationObserver.onMessage(Message) line: 78 ServletDestination.invoke(ServletContext, HttpServletRequest, HttpServletResponse) line: 92 ServletController.invokeDestination(HttpServletRequest, HttpServletResponse, ServletDestination) line: 279 ServletController.invoke(HttpServletRequest, HttpServletResponse) line: 161 CXFServlet(AbstractCXFServlet).invoke(HttpServletRequest, HttpServletResponse) line: 174 CXFServlet(AbstractCXFServlet).doPost(HttpServletRequest, HttpServletResponse) line: 152 CXFServlet(HttpServlet).service(HttpServletRequest, HttpServletResponse) line: 710 CXFServlet(HttpServlet).service(ServletRequest, ServletResponse) line: 803 ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 290 ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 206 ReplyHeaderFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 96 ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 235 ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 206 StandardWrapperValve.invoke(Request, Response) line: 230 StandardContextValve.invoke(Request, Response) line: 175 SecurityAssociationValve.invoke(Request, Response) line: 179 JaccContextValve.invoke(Request, Response) line: 84 StandardHostValve.invoke(Request, Response) line: 127 ErrorReportValve.invoke(Request, Response) line: 102 CachedConnectionValve.invoke(Request, Response) line: 157 StandardEngineValve.invoke(Request, Response) line: 109 CoyoteAdapter.service(Request, Response) line: 262 Http11Processor.process(Socket) line: 844 Http11Protocol$Http11ConnectionHandler.process(Socket) line: 583 JIoEndpoint$Worker.run() line: 446 Thread.run() line: 595
So VersionsSync.beforeCompletion() does indeed get called when we’re in Hibernate mode.
3.1. Sidenote: The Point Where the Records Appear in the Tables
After some more lines of code, we get to org.hibernate.transaction.JDBCTransaction.commitAndResetAutoCommit(), which does a jdbcContext.connection().commit(). Before this line is executed, neither the main record nor the versioning record appear in the table; after that line, the records do appear.
4. JTA mode: no beforeCompletion call made
Next we’ll re-run the test we did in step 3 with the jta version deployed and with all the same breakpoints, to see if org.jboss.envers.synchronization.VersionsSync.beforeCompletion() gets called…
Tune in next time for an exploration of why VersionsSync.beforeCompletion() isn’t getting called in JTA mode!
Have you ever wished you could debug your war project in Eclipse while it was deployed and running on JBoss? This is called remote debugging, and here’s how you set up to do it (on JBoss 4.2.2.GA running on Java 5, anyway):
- Make a backup copy of your current JBoss run.bat
- Open run.bat and find the lines that look like this:
rem JPDA options. Uncomment and modify as appropriate to enable remote debugging. rem set JAVA_OPTS=-Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=y %JAVA_OPTS%
- Uncomment the second line, and save.
- Start Jboss. (It won’t do much until you connect a debugger to it.)
- In Eclipse, open your project, and on the toolbar click on the debugger bug arrow, then Open Debug Dialog.
- Choose Remote Java Application, right-click, New… and update the port to the address in the line you uncommented (possibly 8787).
- Click Debug
- Set a breakpoint somewhere in your code or a library that has source attached
- Exercise your project (via a web service, for instance) and when it hits one of your breakpoints, it will break in Eclipse.
Note to self: Avoid remote debugging for my own code — I should be adding log messages instead!