Posts Tagged experience

Automation-assisted manual transformations

I had been unit testing an SQL generator and had a bunch of tests that gave various input to the generator and tested its output against expected SQL.   The SQL in my tests looked like this:

Now I was ready to feed the SQL to a database engine to verify that it was validly formed.  I would generally use grep for this type of task; but here my SQL statements were formatted multiline for easier reading, and grep operates in a line-by-line mode.  There were over 100 test cases, so it was worth figuring out an automated solution.  I also wanted to avoid writing a single-purpose text-processing utility if possible.

I ended up writing down the following steps for myself:

  1. Turn the tests into a single line of text for ease of working with tools
    On command line: cat Test*.cpp | tr -d "\n\r" > all-one-line.out
  2. Discard everything but the queries, inserting a newline after each
    In editor (SlickEdit for me), open all-one-line.out and Replace All (using Perl-style regexes):
    .*?("(?:\( )*SELECT[^;]+?;)
    with
    \1\n
  3. Clean up what the regex didn’t
    Delete the last line
  4. Get rid of quotes
    Replace \" with nothing
  5. Get rid of semicolons
    Replace ;$ with nothing
  6. Get rid of extra spaces
    Replace <space>+ with <space>
  7. Save the file in the editor
  8. Get rid of Oracle-specific tests
    grep --invert-match TO_DATE < all-one-line.out > all-one-line.sql
  9. Let cool for 5 minutes before serving
    Paste all-one-line.sql into MS SQL Server Management Studio and execute (with Results to Text)

This may look like a large number of steps, but I got to where I could run through them in about 30 seconds and test all 130  queries on the server.  Nice!

Future improvements

Once I had the ability to test my test output against the database server, I wanted to do that each time the tests’ expected results changed.  So where I had originally envisioned a single smoke test run, I ended up going through these automation-assisted manual steps ten or twenty times.  In retrospect, the single-purpose utility script would clearly have been the better approach after all.  I need to get more comfortable whipping up such scripts to lower the barrier to writing them when these occasions arise.

Twelve years in, I would think I would be at the top of my craft by now, but there are still things that seem pretty basic that I’m learning.  Hmmm…I wonder if life really does begin at 40?

, , , , ,

Leave a comment

I was using the wrong kind of SessionFactoryBean

I was trying to change over from using a Hibernate mappings file to annotations, but my test kept blowing up while instantiating the Spring beans.

Then I realized that in my Spring bean file, my sessionFactory bean was declared as an org.springframework.orm.hibernate3.LocalSessionFactoryBean instead of an org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean.  The annotatedClasses property I was trying to set didn’t exist in the LocalSessionFactoryBean!

I had been focusing on how to specify the list of annotated classes (see this example in the Spring Javadoc) and hadn’t paid attention to what kind of SessionFactory bean I was working with.

, ,

Leave a comment

A simplifying regex

Regular expressions can get pretty complex.  I’m in the process of trying to master them, but from time to time I worry that I should just give up on the concept — that perhaps there are generally easier ways of accomplishing the same thing, and regexes are a waste of time.

Then I see an example like today’s that renews my belief that  it’s worthwhile to master these regexes:

Background

A co-worker had a method that received a String of comma-separated values.  Inside the method, he did a split(), with a comma as the delimiter. A simplified version of the class function in question:


public class Splitter {
    public String[] split(String s) {
        return s.split(",");
    }
}

…and its test:


public class SplitterTest {
	@Test
	public void testSplitBasic() {
		List<String> splitString = new Splitter().splitBasic("bob, sam,harry");
		assertEquals(3, splitString.size());
		assertEquals("bob", splitString.get(0));
		assertEquals(" sam", splitString.get(1));
		assertEquals("harry", splitString.get(2));
	}
}

This test passes.  So far, so good.

The Problem

He wanted to enhance the split method to support escaping a comma within the string.  When the split method encountered an escaped comma (“\,”) , it should not consider the comma a delimiter (and it should eat the backslash).

So we would want this test to pass:


        List<String> splitString = new Splitter().splitSmart("bob\\, sam,harry");
        assertEquals(2, splitString.size());
        assertEquals("bob, sam", splitString.get(0));
        assertEquals("harry", splitString.get(1));

splitSmart(): a Non-Regex Implementation

He had a working implementation that looked something like this:


    public List<String> splitSmart(String s) {
        List<String> list = new ArrayList<String>();

        String concat = "";
        boolean concatenating = false;
        for (String x : s.split(",")) {
            if (x.endsWith("\\")) {
                concat += x.substring(0, x.length() - 1) + ",";
                concatenating = true;
            } else if (concatenating) {
                concat += x;
                list.add(concat);
                concatenating = false;
            } else {
                list.add(x);
                concatenating = false;
            }
        }
        return list;
    }

This makes the test pass, but my co-worker was not happy with it.  Too clunky.

A Simplification using a Regex

I had to think about it for a few minutes, but eventually it came to me that what we wanted as a delimiter was a comma not preceded by a backslash.  Looks like a great opportunity to use… negative lookbehind!

The regex way to say “a comma not preceded by a backslash” is:

(?X) is the regex way of saying “not preceded by X” (in this case, a backslash, which has to be escaped) and the comma is the thing to match.

Now we can simplify splitSmart() down to this:

    public List splitSmart(String s) {
        List list = new ArrayList();

        for (String x : s.split(“(?remove a backslash that is followed by a comma, using positive lookahead!

Conclusion

My co-worker was pleased to use the regex-totin’ split-n-replace version of the code.  We both agreed it looked cleaner and simpler, even with the somewhat odd-looking lookbehind syntax and the double-escaped backslashes.  For my part, I was happy to be able to apply my regex learning to help someone.  :)

,

2 Comments

A problem that didn’t show up until later

Earlier this fall, a problem showed up when the database guys first contributed Hibernate mappings.  It was unclear how the Hibernate mappings could be causing the error we were getting — and it turned out that they weren’t; rather, they were exposing a different error that had previously been there, undetected.

I find these types of things really annoying before they’re figured out, and fascinating afterward.  Here’s an email I sent out afterward:


From: Daniel Meyer
Sent: Wednesday, September 24, 2008 3:42 PM
Subject: RE: Build Failures: resolved

The “deployment fails the second time” issue has been resolved, and the CI servers are back in business.

For those who may be interested, the problem turned out to be a Maven config issue in [the functional test project]’s pom file.

(Note: you may need your techno-gibberish decoder goggles for the next details: )

In making ojdbc available to Liquibase for when we deliver schema changes, the ojdbc jar file got included in the war file.  The config has been this way for several weeks, but we didn’t notice it until we just in the last few days got to the point of contributing Hibernate mappings, which starts the Persistence service, which needs ojdbc to actually work.  At that time, the ojdbc jar in the war file fought with the ojdbc jar in JBoss’s server/default/lib directory, and just like when you were fighting with your sister over that cookie, nobody won and we got the NoClassDefFoundError.

(Or something vaguely similar to that.)

Thanks to [those who helped] for their help with this!

Now back to our regular programming…

-Daniel-

, , ,

Leave a comment

NullPointerException on my test’s first use of a mock

This had happened before, but it got me again the other day: I was using Unitils‘ mock annotations (great stuff!) to set up my mocks, but my test failed with a NullPointerException the first place I tried to set an expection on the mock.  The reason being… I forgot to @RunWith(UnitilsJUnit4TestClassRunner.class) .  Sadly, I don’t think I was much faster diagnosing the problem this time than I was last time.

Goal: Next time this symptom appears, instantly suspect that I’m using the wrong runner!  :)

, , , ,

Leave a comment

I did it AGAIN!

Arrgh!

I had my mocks set up to expect calls and return things, but the class under test wasn’t getting as far as it should have been, given those inputs.

I began to suspect many things, both plausible and im-:

  • “Maybe JUnit’s gone haywire!”
  • “The coverage tool must be malfunctioning.”
  • “Maybe Eclipse has gotten fouled up building my project due to that unimplemented method I’m calling far away in a different as-yet-unused part of the project”
  • Maybe a system property or environment variable is floating around, somehow interfering (though that shouldn’t matter since I’m mocking the reading of those…)
  • Maybe the mock framework isn’t working (though it seems to be doing fine for all the other test methods…)

Finally, after over an hour had passed, I noticed that in my test I was INSTANTIATING A NEW INSTANCE of my class under test… which of course then wasn’t being injected with my mocks.

I’ve written about this at least twice before…but apparently I haven’t learned my lesson yet!

Note to self: Remember not to instantiate these things manually.

,

Leave a comment

Getting everything right

There are sometimes a bunch of things that you have to get right, and getting any one of them wrong keeps you from being where you need to be.   (I guess it’s a lot worse for space shuttle engineers!)

Things I Had Wrong

Here are six issues, mostly from today, that all needed to be corrected for my project to work:

  1. I didn’t have a jndi.properties file with a java.naming.factory.initial setting pointing to the bitronix.tm.jndi.BitronixInitialContextFactory.  This was causing a NoInitialContextException.
  2. I didn’t realize that Bitronix Transaction Manager supports (at least) two types of config file: the transaction manager config file and the resource loader config file.  I was putting settings in the transaction manager config file (defaults to bitronix-default-config.properties) that belong in the release loader config file.  This caused those settings to be ignored by BTM (if my bitronix-default-config.properties file is being read at all, which I’m not sure of).
  3. The bitronix.tm.resource.configuration property by which you can specify a resource loader config file expects a path relative to the folder in which you started the virtual machine, not relative to the classpath, so my resource loader configuration was never being found (that’s also why I got a NullPointerException when I tried to set the resourceConfigurationFilename property on the btmConfig bean).
  4. The uniqueName I was using for the JMS ConnectionFactory was java:activemq/QueueConnectionFactory, but BTM’s JNDI implementation doesn’t support the colon-ized namespace (i.e., I needed to remove the “java:” from the JNDI-name).  [Update 8/26/2008: It turns out you can use the java:-prefixed with BTM if you do it right]
  5. I neglected to set the init-method and destroy-method on my PoolingConnectionFactory bean, so the my org.apache.activemq.ActiveMQXAConnectionFactory underlying my bitronix.tm.resource.jms.PoolingConnectionFactory never got initialized (causing the uniqueName to never be registered with the JNDI server).
  6. I misspelled the name of the underlying class for my connection factory bean — as  org.activemq.ActiveMQXAConnectionFactory, when it’s actually org.apache.activemq.ActiveMQXAConnectionFactory — so I got a “ResourceConfigurationException: cannot create JMS connection factory named QueueConnectionFactory” error that had as its cause farther down a “java.lang.ClassNotFoundException: org.activemq.ActiveMQXAConnectionFactory”

Honorable Mention

I had also specified the wrong implementation of the slf4j-api (slf4j-jdk14 instead of slf4j-log4j12) in my pom file, which meant I didn’t see all of BTM’s messages when I set the logging level to DEBUG in log4j.properties.  This didn’t directly keep my project from doing what it oughta, but it might have slowed the problem analysis.

Unnecessary Changes

These are things that at one time or another I suspected I needed to change, but that ended up not being part of the problem:

  • Adding a jndiEnvironment section to the JndiObjectFactoryBean definition for the queueConnectionFactory.  This would have resulted in a queueConnectionFactory bean definition like this:
    <bean id="queueConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
    <property name="jndiName" value="java:activemq/QueueConnectionFactory" />
    <property name="jndiEnvironment">
    <props>
    <prop key="java.naming.factory.initial">bitronix.tm.jndi.BitronixInitialContextFactory</prop>
        </props>
      </property>
    </bean>

    This was taken care of by having this setting in the jndi.properties file.

  • Adding a depends-on attribute to the BitronixTransactionManager bean and/or queueConnectionFactory JndiObjectFactoryBean explicitly telling Spring which other beans needed to be instantiated before these beans.  Seems to work fine without that.
  • Adding bitronix.tm.resource.bind = true to a bitronix-default-config.properties in the root of the classpath.  The JNDI lookup seems to be doing fine without explicitly setting this property.

Analysis

Two quick points:

  • Experience will be a great help here, as inexperience with JNDI and BTM were a big part of why I took all these wrong turns; and
  • Hallelujah that it’s Friday!  :)

Where We Are Now

I have JNDI lookup working on both a JTA transaction manager and a JMS connection factory, kicked off from a JUnit-based integration test.

Next Steps

I need to get my JDBC datasource hooked up similarly, on the way to getting my whole XA example working as a JUnit test — sending a message to the JMS queue, receiving it and writing to the database, demonstrating both the commit and a rollback situations.

, , ,

Leave a comment