Blowing up in test mode

This is the final post in a 9-part series on Envers, Spring, and JTA. The previous post is The final answer.


Setting the hibernate.transaction.manager_lookup_class and org.hibernate.transaction.JTATransactionFactory properties in the Hibernate config:

<prop key="hibernate.transaction.manager_lookup_class">org.hibernate.transaction.JBossTransactionManagerLookup</prop>
<prop key="hibernate.transaction.factory_class">org.hibernate.transaction.JTATransactionFactory</prop>

fixed my issue where the envers beforeCompletion synchronization wasn’t-getting-called-when-we’re-in-JTA-mode… but with this change, the integration test stopped working — it now couldn’t find the JTA transaction manager by JNDI lookup.

Ah, yes — we’d need a JTA transaction manager to actually be available at integration test time — looks like another job for Bitronix Transaction Manager.

1. How I Set Up BTM

So, I put a jndi.properties file in the root of the classpath under src/test/java/ …

java.naming.factory.initial = bitronix.tm.jndi.BitronixInitialContextFactory

…put a bitronix-default-config.properties file beside it…

bitronix.tm.serverId = hibernate-support-module-btm-server
bitronix.tm.journal.disk.logPart1Filename = target/btm1.log
bitronix.tm.journal.disk.logPart2Filename = target/btm2.log

…and added a bitronixTransactionManager bean to the beans file under src/test/resources…

<bean id="bitronixTransactionManager" factory-method="getTransactionManager" class="bitronix.tm.TransactionManagerServices" destroy-method="shutdown"/>

2. One More Piece

The transaction manager JNDI lookup still was failing though, with this error:

org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.TransactionException: Naming service lookup for UserTransaction returned null [UserTransaction]

Caused by: org.hibernate.TransactionException: Naming service lookup for UserTransaction returned null [UserTransaction]

The BTM Hibernate documentation pointed out that I can set the

hibernate.transaction.manager_lookup_class to Hibernate’s BTMTransactionManagerLookup class.  (Huh!)

<prop key="hibernate.transaction.manager_lookup_class">org.hibernate.transaction.BTMTransactionManagerLookup</prop>

When I do this, the integration test now works again.

3. Now the Deploy to JBoss Fails

I was pretty sure that deploying to JBoss with the transaction manager lookup setting pointing to a BTMTransactionManagerLookup wasn’t going to work, and sure enough, on deploy I get this:

org.hibernate.HibernateException: Could not obtain BTM transaction manager instance

Caused by: java.lang.ClassNotFoundException: bitronix.tm.TransactionManagerServices

Right, this error is because we aren’t including BTM in the production war project (and we don’t want to).

4. Test and Production Both Need To Work At Once

So now we can run our tests if we specify the BTMTransactionManagerLookup, and we can run in production if we specify the JBossTransactionManagerLookup.  We need to find a configuration that will work both in test and production modes.

Another nagging issue is that we don’t want to hardcode our production configuration to be specific to JBoss — ideally we’d like the application server to be auto-detected; if we need to we could make it manually configurable.  Hardcoded JBossTransactionManagerLookup won’t do.

5. Spring to the Rescue?

Thinking of auto-detecting the application server reminds me of what Spring’s JtaTransactionManager does — it tries various JNDI lookups for the JTA TransactionManager and UserTransaction until it finds ones that work, so you’re not hardcoding support for only one app server and there is no user configuration needed .  I wonder if Spring has a TransactionManagerLookup that does that sort of thing to avoid  — is Spring’s LocalTransactionManagerLookup what we need?

When I switch the transaction manager lookup line to that:

<prop key="hibernate.transaction.manager_lookup_class">org.springframework.orm.hibernate3.LocalTransactionManagerLookup</prop>

…I get this error:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘sessionFactory’ defined in class path resource [com/ontsys/fw/hibernatesupport/startup/hibernate-support-beans.spring.xml]: Invocation of init method failed; nested exception is org.hibernate.HibernateException: Could not instantiate TransactionManagerLookup ‘org.springframework.orm.hibernate3.LocalTransactionManagerLookup’

Come to think of it, the javadoc for LocalTransactionManagerLookup does say:

Implementation of Hibernate’s TransactionManagerLookup interface that returns a Spring-managed JTA TransactionManager, determined by LocalSessionFactoryBean’s “jtaTransactionManager” property.

(Emphasis mine.)  My Hibernate configuration configures an AnnotationSessionFactoryBean (which is a LocalSessionFactoryBean), and I didn’t set the jtaTransactionManager property there.

5.1. Setting the jtaTransactionManager Property

As a first attempt, I set

<property name="jtaTransactionManager" ref="transactionManager" />

where the transactionManager reference refers to my

<tx:jta-transaction-manager />

bean.  I get:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘sessionFactory’ defined in class path resource [com/ontsys/fw/hibernatesupport/startup/hibernate-support-beans.spring.xml]: Initialization of bean failed; nested exception is org.springframework.beans.TypeMismatchException: Failed to convert property value of type [org.springframework.transaction.jta.JtaTransactionManager] to required type [javax.transaction.TransactionManager] for property ‘jtaTransactionManager’; nested exception is java.lang.IllegalArgumentException: Cannot convert value of type [org.springframework.transaction.jta.JtaTransactionManager] to required type [javax.transaction.TransactionManager] for property ‘jtaTransactionManager’: no matching editors or conversion strategy found

So the Spring JtaTransactionManager isn’t a javax.transaction.TransactionManager.  What do you specify for the jtaTransactionManager property, then?

In a helpful post from 2004, Juergen Hoeller speaks to Spring’s support for, and the limits of, container-independent JTA configuration:

…Spring’s JtaTransactionManager itself works just nicely in a container-independent fashion – in typical usage scenarios where you could use a plain JTA javax.transaction.UserTransaction too…you just need container-specific configuration for more advanced usage, namely transaction suspension and Hibernate cache completion callbacks. The simple reason for this is that you need the JTA javax.transaction.TransactionManager to suspend/resume transactions and register transaction synchronizations; the UserTransaction interface does not support that functionality.

Unfortunately, J2EE does not specify how to access the JTA TransactionManager, just the common JNDI location of the JTA UserTransaction – so the lookup of the JTA TransactionManager has to be container-specific. This is the case with Hibernate (see TransactionManagerLookup), this is the case with Spring, this is the case with any such tool.

For Hibernate cache completion callbacks with JTA, you need to specify a corresponding TransactionManagerLookup in the Hibernate properties. As an alternative, Spring allows you to pass a JTA TransactionManager into LocalSessionFactoryBean’s “jtaTransactionManager” property, to reuse a centrally defined TransactionManager reference that can be shared with JtaTransactionManager’s “transactionManager” property.

(Emphasis mine.) Juergen goes on to mention that as another alternative, Hibernate’s cache-completion callbacks are taken care of if you use Spring’s own transaction synchronization mechanism — this avoids the need for a container-specific TransactionManagerLookup.  I don’t think Spring’s synchronization mechanism is sufficient for what we need here though, because it looks like it only supports the afterCompletion() callback, and we (through envers) are making use of the beforeCompletion() callback.

6. A Table of Our Apparent Options

Opt # Configuration Viable Option? Comments
1 Plain ol’ Spring JtaTransactionManager and Hibernate LocalSessionFactoryBean without additional configuration No Hibernate doesn’t know about Spring JTA transaction manager, synchronizations get registered wrong and are not called
2 Use Spring’s synchonization mechanism No Appears to only support afterCompletion() synchronization callback, and envers needs and uses the beforeCompletion() one
3 Have a separate javax.transaction.TransactionManager that the Hibernate LocalSessionFactoryBean and Spring’s JtaTransactionManager both reference Yes The javax.transaction.TransactionManager may have to be defined in a container-specific way
4 Specify hibernate.transaction.manager_lookup_class to Hibernate’s LocalSessionFactoryBean config Yes Which TransactionManagerLookup class is needed is container-specific, so we’d need to abstract the choice out to a configuration property somehow

I’m leaning toward going with option #4 without even trying option #3 since I can see how we would do #4 and I’m afraid that even with #3’s greater complexity it still wouldn’t get us container-independent JTA configuration.

Update (1/2/2009): Yep, option #4 is what we went with.   To abstract away the container-specificness, we maintain a list of org.hibernate.transaction.*TransactionManagerLookup classes and iterate through the list trying getTransactionManager() on each until we find one that doesn’t return null — then we populate the hibernate.transaction.factory_class configuration property with that class name.  The org.hibernate.transaction.BTMTransactionManagerLookup last in the list, and the org.hibernate.transaction.JBossTransactionManagerLookup is currently the only other lookup class in the list; but as we add support more app servers, the idea is we can just add more of the specific org.hibernate.transaction.*TransactionManagerLookup classes to the list.

Why we’d want at least version 1.3.1

Bitronix Transaction Manager has been helpful for our integration tests, which can then run outside the app server and have a JTA transaction manager and enough of a JNDI provider to get by.

We have been using version 1.3, since that’s the version that first had the JNDI server.  But we had to specify disableJmx=true in the Configuration object so that the java: prefix could work.

This limitation has been removed as of BTM 1.3.1.  I’m BTM-enabling another module — maybe now’s the time to update to the current version of BTM…

We’ll want to JNDI-lookup the queue, too…

In our system we have proved out looking up the JMS connection factory from JNDI, but up to this point we have declared the Queue bean using an <amq:queue> element in the application bean files.  This ties our application code to the ActiveMQ broker, which we don’t want.  We want to move toward looking up the connection factory and the queue from JNDI.  That way the application bean file only needs to know the JNDI name, and we can configure the queue in the connection factory datasource configuration file on the app server (e.g., activemq-jms-ds.xml), probably as an mbean element in there:


   <mbean code="org.jboss.resource.deployment.AdminObject" name="activemq.queue:name=TheMBeanNameYouSeeInTheJMXConsole">
      <attribute name="JNDIName">this/is/the/jndi/name/the/bean/files/will/use</attribute>
      <depends optional-attribute-name="RARName">jboss.jca:service=RARDeployment,name='activemq-rar-5.1.0.rar'</depends>
      <attribute name="Type">javax.jms.Queue</attribute>
      <attribute name="Properties">PhysicalName=theQueueName.YouWillSee.In.ActiveMQ</attribute>
   </mbean>

But Who Will Care for the Babies?

This will be fine for a production environment — but what about our integration tests?  Right now we use Bitronix Transaction Manager 1.3’s JNDI server facilities to provide JNDI lookup of the JDBC datasource for our integration tests.  I thought we’d also use BTM to look up the JMS connection factory… but BTM does not support JNDI-looking-up queues as well.  Ludovic says:

If you really need a complete JNDI server, you can use Tomcat’s which is quite easy to reuse. This is what I recommended using before BTM 1.3, there is an example here: http://docs.codehaus.org/display/BTM/Hibernate

That’s probably what we’ll need to look into when the time comes…

java: Dropping the prefix

(Usually it’s good to drop one’s pretensions, but since all we have handy is a prefix we’ll just drop that instead…)

We are (for the moment, anyway) going to need to change our JBoss JDBC datasources so that the datasource resides in the global JNDI namespace, to be looked up under jndi-name Blah rather than java:Blah.  The reason we need to make this change is that Bitronix doesn’t currently support the java: prefix.

I seem to remember seeing a tag that you could put in your datasource configuration file that would tell JBoss to make the datasource available from the global JNDI namespace instead of from the JVM-specific java: one.

Ah yes, the JBoss 4 Application Server Guide and the DTD both document the use-java-context element.  Set that to false and it goes in the global JNDI namespace.  Using the java: namespace would be preferred; The JBoss 4 Application Server Guide also says,

DataSource wrappers are not usable outside of the server VM, so they are normally bound under the java:/, which isn’t shared outside the local VM.

That’s what we’ll need to set in our production datasources so that Bitronix Transaction Manager can exactly replicate the JNDI functionality at integration test time.

HSQLDB Says ‘No’; Bitronix Chafes

The currently-released version of HSQLDB, 1.8.0, doesn’t support XA transactions (though I see work thereon in the trunk).  This doesn’t please Bitronix, which declares the situation an unsafe one in which to start up:

bitronix.tm.utils.InitializationException: recovery failed, cannot safely start the transaction manager
...
Caused by: bitronix.tm.resource.ResourceConfigurationException: property <className> for resource <ds1> in resources configuration file must be the name of a class implementing either javax.sql.XADataSource or javax.jms.XAConnectionFactory

(To be fair, it’s hardly Bitronix’s fault we supplied it with a class that didn’t implement a required interface!)

Hmm… will we:

Or maybe we should try the H2 database engine, which already supports XA transactions and boasts a javax.sql.XADataSource-implementing JdbcDataSource class to back that up?  I wonder how much of our testing infrastructure that would gum up, changing test-databases.

Update 8/21/2008 – There’s a way around this limitation — see comment #1!

On the classpath

I guess I’d rather be on the classpath than the warpath…

Anyway, Bitronix’s bitronix.tm.Configuration class currently only supports specifying the resource loader .properties file’s location relative to the JVM’s current directory (think “argument to the constructor of a File object”), but for unit testing we would like to be able to plop the .properties file in our classpath and not have to hardcode a path to it.  Basically we wish we could give BTM a classpath-relative properties path-n-filename.  For the moment, we’ll use this hack:

    @BeforeClass
    public static void beforeAllTests() {
        //...

        final String configurationFilePath = getAbsoluteFilePath("/bitronix-resource-loader.properties");

        TransactionManagerServices.getConfiguration()
            .setResourceConfigurationFilename(configurationFilePath);
        TransactionManagerServices.getTransactionManager();
    }

    private static String getAbsoluteFilePath(String classPathBasedFilePath) {
        URL path = ThisClass.class.getResource(classPathBasedFilePath);
        if (path == null) {
            throw new RuntimeException("Can't find resource " + classPathBasedFilePath);
        }
        return path.getFile();
    }

In our case, path.getFile() returns something along the lines of:

/C:/path/to/eclipse/workspace/and/project/target/test-classes/bitronix-resource-loader.properties

The leading slash is kind of funky, but it works.

Why we wanted a JNDI server for integration testing

Why is having a JNDI server for our integration testing environment so helpful?  Well, just for our minimal XA integration test example, there were (at one point) fourteen Spring beans:

If we had to use a non-JNDI strategy for integration testing outside the app server, seven of these beans would have to be swapped out for test-only doubles — but we want our configuration for integration testing to match as closely as possible our production configuration so that we’re testing the same thing we’re deploying!

A minor simplification: the JtaTransactionManager bean

In a previous episode, I replaced a JTA transaction manager bean resembling this:


    <tx:jta-transaction-manager />

with a bean resembling this:


    <bean id="transactionManager" class="org.springframework.transaction.jta.JtaTransactionManager">
<property name="transactionManager" ref="BitronixTransactionManager" />
<property name="userTransaction" ref="BitronixTransactionManager" />
    </bean>

That’s kind of ok, but having to change out that transaction manager bean makes the configuration more complicated.

As it turns out though, my testing shows that we can keep the


    <tx:jta-transaction-manager />

version and it finds the BTM UserTransaction and TransactionManager fine.

XA, JNDI and Bitronix, part 4: dancing on the happy path

In part 3, we tried our first JMS message send; but even after we wrapped the message send in a transaction using AOP, BTM still reported that “resource ‘activemq/QueueConnectionFactory’ cannot be used outside XA transaction scope.”  Today we’ll work to resolve that issue.

Both Sends and Receives Need Transactions

Ludovic’s helpful comment on yesterday’s post confirms that we need to make sure both sends and receives are in transactions.  As of yesterday, we were only wrapping the message sends, but not receives, in a transaction — at least that was what we were trying to do!  As it turns out, the transaction-wrappage of the sends was unsuccessful, while on the other hand we were accidentally successfully wrapping the receives!  Here’s how it happened:

Wrapping Sends: If It Ain’t A Bean…

As you may recall from yesterday, the pointcut I had defined for applying transaction advice to message sends was as follows:

        <aop:pointcut id="messageReceiveOperation" expression="execution(void com.example.dmn.JndiBitronixTest.transactedSendMessage(..))" />

But as Keith points out, since my JndiBitronixTest class is not defined as a bean in my Spring beans files, the transaction advice will not be applied to the transactedSendMessage method execution.  Oops!  I didn’t realize that Spring AOP advice applied only to beans!  Fixing that is easy enough: the transactedSendMessage method calls one layer down into the MessageSender class, which is defined as a bean.  We can just start the transaction there instead:

        <aop:pointcut id="messageSendOperation" expression="execution(void com.example.dmn.MessageSender.sendMessage(..))" />

Wrapping Receives: Forgotten Advice

I was looking in the main beans file to verify that MessageSender was defined as a bean when I saw a surprising thing:  I already had transactional advice on the message receives, including the following pointcut:

        <aop:pointcut id="messageReceiveOperation" expression="execution(void com.example.dmn.MessageProcessor.processIt(..))" />

This advice came along when I brought this code over from my original XA example manual test that runs only on an app server.  I consolidated this advice with the send advice into the spring-beans-bitronix.xml file to yield this advice:

    
        
            
        
    

    
        <aop:pointcut id="messageSendOperation" expression="execution(void com.example.dmn.MessageSender.sendMessage(..))" />
        
    

    
        <aop:pointcut id="messageReceiveOperation" expression="execution(void com.example.dmn.MessageProcessor.processIt(..))" />
        
    

And this time both pointcuts point to classes that are Spring beans.  :)

Happy Path Seems to Work

Now our integration test can send a JMS message and the listener picks it up from the queue and writes it to the database, without exceptions or stack traces.  Furthermore, when I bump the log level to DEBUG for Bitronix classes, in my log4j.properties file:

log4j.logger.bitronix.tm=DEBUG

Then I see encouraging looking XA-type messages like the following:

DEBUG bitronix.tm.BitronixTransaction  - committing, 2 enlisted resource(s)
DEBUG bitronix.tm.BitronixTransaction  - changing transaction status to PREPARING
DEBUG bitronix.tm.journal.TransactionLogAppender  - between 19089 and 19193, writing a Bitronix TransactionLogRecord with status=PREPARING, recordLength=96, headerLength=28, time=1219154728262, sequenceNumber=11, crc32=-1825120209, gtrid=737072696E672D62746D0000011BDB48D3FF00000000, uniqueNames=XAOracleDS,activemq/QueueConnectionFactory
DEBUG bitronix.tm.journal.TransactionLogAppender  - disk journal appender now at position 19193
DEBUG bitronix.tm.twopc.AbstractPhaseEngine  - executing phase on 2 resource(s) enlisted in 1 position(s) in natural position order

...

DEBUG bitronix.tm.BitronixTransaction  - changing transaction status to PREPARED
...

DEBUG bitronix.tm.twopc.Preparer  - successfully prepared 2 resource(s)
DEBUG bitronix.tm.BitronixTransaction  - 2 interested resource(s)
DEBUG bitronix.tm.BitronixTransaction  - changing transaction status to COMMITTING (forced)
...

DEBUG bitronix.tm.BitronixTransaction  - changing transaction status to COMMITTED

There’s Also a One-Phase Commit In There (and that’s good)

Before I found that two-phase commit sequence in the log, I found another sequence farther up


DEBUG bitronix.tm.BitronixTransaction  - committing, 1 enlisted resource(s)
DEBUG bitronix.tm.BitronixTransaction  - changing transaction status to PREPARING
DEBUG bitronix.tm.journal.TransactionLogAppender  - between 18721 and 18813, writing a Bitronix TransactionLogRecord with status=PREPARING, recordLength=84, headerLength=28, time=1219154728200, sequenceNumber=7, crc32=-667556454, gtrid=737072696E672D62746D0000011BDB48D4AA00000003, uniqueNames=activemq/QueueConnectionFactory
DEBUG bitronix.tm.journal.TransactionLogAppender  - disk journal appender now at position 18813
DEBUG bitronix.tm.twopc.Preparer  - 1 resource enlisted, no prepare needed (1PC)
DEBUG bitronix.tm.BitronixTransaction  - changing transaction status to PREPARED
DEBUG bitronix.tm.journal.TransactionLogAppender  - between 18813 and 18905, writing a Bitronix TransactionLogRecord with status=PREPARED, recordLength=84, headerLength=28, time=1219154728200, sequenceNumber=8, crc32=-2001767057, gtrid=737072696E672D62746D0000011BDB48D4AA00000003, uniqueNames=activemq/QueueConnectionFactory
DEBUG bitronix.tm.journal.TransactionLogAppender  - disk journal appender now at position 18905
DEBUG bitronix.tm.BitronixTransaction  - 1 interested resource(s)
DEBUG bitronix.tm.BitronixTransaction  - changing transaction status to COMMITTING
DEBUG bitronix.tm.journal.TransactionLogAppender  - between 18905 and 18997, writing a Bitronix TransactionLogRecord with status=COMMITTING, recordLength=84, headerLength=28, time=1219154728200, sequenceNumber=9, crc32=236131807, gtrid=737072696E672D62746D0000011BDB48D4AA00000003, uniqueNames=activemq/QueueConnectionFactory
...

DEBUG bitronix.tm.BitronixTransaction  - changing transaction status to COMMITTED

I worried about line 8 where it said that only one resource was enlisted and so only a one-phase commit (1PC) was needed — “What happened to the database resource?” I wondered.  But then I remembered that this must be the message send (I tend to forget that that’s being transacted too), and the send does not involve the database resource.

Next Steps

I’d still like to have my integration test query the database to verify that the record really made it in there.

XA, JNDI and Bitronix, part 3: first send

In part two, we got the rest of the Spring beans connected up, and now we’re ready to try sending messages.

First Send

After initializing the application context, we’ll add this:


        MessageSender messageSender = (MessageSender) startupService.getContext().getBean("messageSender");
        messageSender.sendMessage("hello there");

We get this error:

org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: error enlisting a MessageProducerWrapper of a DualSessionWrapper in state ACCESSIBLE of a JmsPooledConnection of pool activemq/QueueConnectionFactory in state ACCESSIBLE with underlying connection ActiveMQConnection {id=ID:blahblah-1409-1219089747392-2:1,clientId=ID:blahblah-1409-1219089747392-3:1,started=false} with 1 opened session(s)

Caused by: bitronix.tm.internal.BitronixSystemException: resource ‘activemq/QueueConnectionFactory’ cannot be used outside XA transaction scope. Set allowLocalTransactions to true if you want to allow this and you know your resource supports this.

Local Transactions Not Enough Help

We can set the allowLocalTransactions property on our bitronix.tm.resource.jms.PoolingConnectionFactory bean:

But we still get an error, though a new one:

org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Session’s XAResource has not been enlisted in a distributed transaction.

I’ll take out the allowLocalTransactions setting since it didn’t help.

The Stack Trace is Right

Come to think of it, that stack trace has a point — we’re not in a transaction.  My original tied-to-an-app-server XA example used AOP to start a transaction at the service layer entry point, but I removed all that when I stripped off the service layer for this integration test.  Let’s add in the AOP transactional stuff to the Bitronix spring beans file:

   
       
           
       

   

   
       
       
   

We still get:

org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: error enlisting a MessageProducerWrapper of a DualSessionWrapper in state ACCESSIBLE of a JmsPooledConnection of pool activemq/QueueConnectionFactory in state ACCESSIBLE with underlying connection ActiveMQConnection {id=ID:blahblah-1759-1219092563720-2:1,clientId=ID:blahblah-1759-1219092563720-3:1,started=false} with 1 opened session(s)

Caused by: bitronix.tm.internal.BitronixSystemException: resource ‘activemq/QueueConnectionFactory’ cannot be used outside XA transaction scope. Set allowLocalTransactions to true if you want to allow this and you know your resource supports this.

Why aren’t we in an XA transaction?

Debug Statements

Let’s put some debug statements in there to


    @Test
    public void TestBitronix() {
        System.err.println("***Before transactedSendMessage");
        transactedSendMessage("hello there");
    }

    private void transactedSendMessage(String text) {
        MessageSender messageSender = (MessageSender) startupSvc.getContext().getBean("messageSender");
        System.err.println("***Before sendMessage");
        messageSender.sendMessage(text);
        System.err.println("***After sendMessage");
    }

When we run this, a curious thing happens: in the console output we see:

***Before transactedSendMessage
***Before sendMessage

But we should have seen a transaction start between those, shouldn’t we?

And on that unanswered question we must close for today!

(We’ll pick up next time.)