About Dovetail

Motivation
Architecture
Feature Summary Tables
Libraries Used By Dovetail

Motivation

Dovetail is a Java and XML toolkit designed to address some serious problems which arise in building a reliable, testable website with server-side Java and a relational database.

The standard model for building such a site is to start with a database like MySQL, and a servlet engine like Tomcat. Then we use SQL scripts to create a database schema, and we write some JDBC code in JavaBeans to create, modify, and delete records from tables in the database. Finally we write a layer of JSP pages or servlets which execute queries and render the results along with HTML markup, invoke methods on the Beans, and dispatch forms and links to other servlets and pages.

What problems arise when working in this mode? We can group them into several categories.

Loose binding between SQL statements and the database schema

As we write queries, inserts, and update statements, we can debug the SQL code against the database schema. But what happens when some kind of schema change is necessary? More interesting still, what happens if we want to migrate the application to a new database? Several problems can occur:

Queries can easily break when the schema is modified, or when a new database requires different SQL syntax
When query results are accessed by a name-lookup scheme (e.g. ResultSet#getString(String)), changing a query can cause downstream code to unexpectedly fail when named values can't be found.
It is hard to report errors that result from constraint violation, such as duplicate values of a primary key or unique index, since there is no easily available description of what those constraints are, and the exceptions reported by the JDBC drivers are usually terse.
It is hard to do type-checking, since Java and JDBC have different type systems. Type conversion code can be written which uses the DatabaseMetaData, but any errors which occur will result in run-time exceptions rather than compilation errors.

Loose binding between HTML forms and database tables

Typically, the input elements in an HTML form are meant to be bound to columns in a database table. For example, a user registration form will typically collect the user's e-mail address, and write it into an emailAddress column in a DB table. However, there is no mechanism for validating the elements in a form against the columns of the table. The form can easily omit data that is necessary for a valid record, or add extra data that is not stored in the database.

Poor handling of database errors which occur during form processing

When validation of form data fails on the server, an error needs to be reported back to the user so that they can fix it by changing their input. If these errors are handled on a case-by-case basis, there are going to be a lot of missed cases and bugs that result from un-handled or mis-handled errors. Furthermore, if robust handling for an error is implemented in a servlet or page, it is not inherited by other servlets and pages unless the error-handling code is copied into them.

Site navigation is implicit and fragile

Navigation from one page to another, through links rendered in servlets and JSP pages, is a fragile process. Some errors that can easily arise include:

Broken links. When pages are moved or renamed, any links to those pages have to be hunted down and corrected.
Mismatched or missing query parameters. When a JSP page or servlet requires a set of query parameters, there is no way to guarantee that a link to that page or servlet is passing them as required.

No easy way to test pages, servlets, and server-side beans

The implementations of the pages, servlets and server-side JavaBeans are all tied to the Java servlet classes and interfaces. Thus, the only way to write tests for them is to submit form data and somehow validate the form processing. This process is typically done by hand, which is both error-prone and not scaleable. It is impossible to do thourough manual testing of a site each time a change is made to the site schema, code, or data flow. Graphical HTML testing tools are available but they are extremely fragile and painful to use. Furthermore, testing a site's JavaBeans by writing HTML tests is a fundamentally bad idea because the tests are not at the right level of granularity.

Ad-hoc use of database connections

Database connections should be a carefully managed and controlled resource in any web application. Without a robust connection pool and a standard and failsafe way for acquiring and freeing database connections, a web site will quickly go down under load.

Architecture

Dovetail addresses all of these problems through the use of several standards-based techniques.

Database schema document

The database schema for a Dovetail website is described in an XML document. This document describes the tables, columns, constraints, and indexes that make up the database. This XML document is then leveraged in several ways:

XSL stylesheets can generate the SQL scripts for creating and dropping the database tables. Different stylesheets can be written for each new relational database, so that it is easy to switch between databases.
Another XSL stylesheet generates a set of Java classes, one for each table. Each table class has methods to access the columns, constraints, and indexes, and their properties.

Site document

Dovetail also includes a DTD for describing the core components of the web site. The site consists of locations (directories), pages, parameters, forms, queries, and links. Dovetail uses this document in several ways:

Elements in the site can be bound to tables and columns in the database schema document. Dovetail then uses this binding information at compile-time to check that the web site and database are in synch (e.g. there are no queries or forms that reference non-existant tables or columns)
A Java class is generated for each page, form, and query in the site. These 'convenience' classes contain typed methods for accessing various aspects of the site. They can then be referenced from within the site's JSP pages and servlets, and the Java compiler will signal any errors between what the pages and servlets expect, and what is described in the site document.

Test document

Testing is the missing link in many web development frameworks. A core feature of Dovetail is the ability to write robust tests for a web site that can be run quickly and easily. The primary mechanism for implementing Dovetail site tests is by writing (you guessed it) XML test documents. Each XML test document starts by specifying a page and the query parameters to that page. The document then proceeds to exercise the page by executing and validating queries, and by submitting and processing forms.

Database library

Dovetail includes a set of classes which can be used to simpify and standardize database access. The goal is to make it easy to perform common tasks such as using form data to create or update a database record, and implementing server-side validation of form data.

The database library includes built-in integration to DbConnectionBroker.

Site library

Dovetail also includes a set of classes for accessing information about the web site (provided in the site document), and for implementing site behavior. The site library includes classes for accessing pages, query parameters, forms, form elements, queries, query results, links, and more. It also includes an abstraction of the Servlet API which allows a Dovetail site to be exercised without a Servlet runner. This is very useful for writing and running test cases in a simple, single-process environment, without having to generate any HTTP request generation or response parsing.

Test library

The Dovetail test library is used to interpret and run a test document.

Java code generation

Some of the problems described in the previous section can be caught by using a Java compiler to look for mismatched names and data types. Dovetail generates typed Java classes with named accessor methods for many different aspects of the web site. This Java code generation is done using XSL stylesheets. Stylesheets are better than monolithic Java classes for implementing code generation, beacuse they are standardized, declarative, extensible, and easy to understand (once you understand XSL).

Feature Summary Tables

Here is a table which summarizes how Dovetail addresses the site-building problems described above.

Problem How Addressed

Fragile queries which can be easily broken when the database schema changes Queries are declared in an XML document. Query results are type-checked against the database schema. Typed Java classes are generated for each query, so that changes to the database schema cause compile-time errors in Java code.

Changes to schema cause run-time errors in code which accesses ResultSets Dovetail generates a Java class for each query with typed and named accessor methods, so that changes to queries or the database schema cause compile-time errors rather than run-time exceptions.

Consistent detection and reporting of database errors caused by constraint violations (e.g. primary key, unique index, NOT NULL) is difficult Dovetail includes centralized handling of SQLExceptions which can convert them to specialized exception types such as DuplicateKeyException and MissingValueException. It also includes a standard form error reporting and rendering mechanism which makes it easy to report such errors back to the user in a human-readable way

HTML forms which are meant to create or update database records may be mismatched with the database, or may break when the database schema is changed Dovetail can check the declarations of form input elements against database columns. It also warns (through the error log) of any form input elements which are declared in the site XML document but not rendered in the page or servlet. And for each form it generates a Java class with a named accessor method for each input element. Dovetail also includes a standard way to inform the user which inputs are required and which are optional.

Pages can easily render broken links to other pages. The site XML document includes a name for each page, and a format for defining links between pages which uses the page names. Links to non-existant pages are caught at compile-time, and when JSP pages are renamed, the site links are automatically updated with a recompile.

Form behavior must be tested via cumbersome & costly HTML testing tools Dovetail includes an IRequestContext abstraction which insulates the pages and forms from the details of the HTTP Servlet implementation. This makes it possible to write page, query, and form tests which can be run in a single Java process, without a web server or browser. In addition, Dovetail includes an XML DTD for writing test cases which can then be run and verified by a simple Java program.

Performance of queries used by pages and servlets is hard to analyze Dovetail includes a simple mechanism to execute the queries defined in the site XML document and output timing information which can be used to find bottelnecks.

Database connections are mis-managed Any web application with any hope of resonable scalability will need to use a connection pool. However, using a connection pool introduces new problems because failure to free a connection will eventually cause the site to run out of connections and crash. Dovetail is integrated with DbConnectionBroker, and includes a standard mechanism for getting a database connection for each page request, and a standard mechanism for freeing them.

Links can be easily broken when pages are removed or renamed, or when query parameters are required for a link to work properly. Since all the pages, parameters, and links are described in the site XML document, Dovetail can check each link at compile-time to ensure that all links are valid, and that all required query parameters have been specified

Problem	How Addressed
Fragile queries which can be easily broken when the database schema changes	Queries are declared in an XML document. Query results are type-checked against the database schema. Typed Java classes are generated for each query, so that changes to the database schema cause compile-time errors in Java code.
Changes to schema cause run-time errors in code which accesses ResultSets	Dovetail generates a Java class for each query with typed and named accessor methods, so that changes to queries or the database schema cause compile-time errors rather than run-time exceptions.
Consistent detection and reporting of database errors caused by constraint violations (e.g. primary key, unique index, NOT NULL) is difficult	Dovetail includes centralized handling of SQLExceptions which can convert them to specialized exception types such as DuplicateKeyException and MissingValueException. It also includes a standard form error reporting and rendering mechanism which makes it easy to report such errors back to the user in a human-readable way
HTML forms which are meant to create or update database records may be mismatched with the database, or may break when the database schema is changed	Dovetail can check the declarations of form input elements against database columns. It also warns (through the error log) of any form input elements which are declared in the site XML document but not rendered in the page or servlet. And for each form it generates a Java class with a named accessor method for each input element. Dovetail also includes a standard way to inform the user which inputs are required and which are optional.
Pages can easily render broken links to other pages.	The site XML document includes a name for each page, and a format for defining links between pages which uses the page names. Links to non-existant pages are caught at compile-time, and when JSP pages are renamed, the site links are automatically updated with a recompile.
Form behavior must be tested via cumbersome & costly HTML testing tools	Dovetail includes an `IRequestContext` abstraction which insulates the pages and forms from the details of the HTTP Servlet implementation. This makes it possible to write page, query, and form tests which can be run in a single Java process, without a web server or browser. In addition, Dovetail includes an XML DTD for writing test cases which can then be run and verified by a simple Java program.
Performance of queries used by pages and servlets is hard to analyze	Dovetail includes a simple mechanism to execute the queries defined in the site XML document and output timing information which can be used to find bottelnecks.
Database connections are mis-managed	Any web application with any hope of resonable scalability will need to use a connection pool. However, using a connection pool introduces new problems because failure to free a connection will eventually cause the site to run out of connections and crash. Dovetail is integrated with DbConnectionBroker, and includes a standard mechanism for getting a database connection for each page request, and a standard mechanism for freeing them.
Links can be easily broken when pages are removed or renamed, or when query parameters are required for a link to work properly.	Since all the pages, parameters, and links are described in the site XML document, Dovetail can check each link at compile-time to ensure that all links are valid, and that all required query parameters have been specified

And here are some planned future enhancements to Dovetail:

Problem How Addressed

Website load tests can be difficult, cumbersome, and expensive to set up and run Dovetail can leverage its site testing functionality to provide a robust mechanism for load-testing and profiling.

Problem	How Addressed
Website load tests can be difficult, cumbersome, and expensive to set up and run	Dovetail can leverage its site testing functionality to provide a robust mechanism for load-testing and profiling.

Libraries Used By Dovetail

Dovetail uses a variety of 3rd party libraries. The versions listed below are the ones that I test; later versions may also work, earlier ones probably won't.

You'll need to have all of these libraries in your classpath (all the JAR and ZIP files in the lib folder) if you want Dovetail to work.

These libraries are distributed under a variety of licenses. You can read the licenses and see which libraries they apply to in this folder.

To run a web site, you will of course also need a web server and servlet runner. The examples included with this distribution have been tested with Jakarta-Tomcat 3.2.1. To run a real web site you'll want to front it with Apache.

And, you'll need a JDBC driver and a SQL database. While I have run Dovetail against Microsoft SQL Server 7 and PostgreSQL, it is currently fully functional and tested only with MySQL 3.23.24 and the MM MySQL JDBC driver 2.0.

Home