Maven Plugin-Testing Tools

This API is meant to provide a series of convenience tools for developers wishing to test Maven plugins. The primary focus of the library is on integration testing, since it is not well-supported with current tools and approaches.

Special Problems with Plugin Integration Testing

Maven plugins represent a somewhat special need in terms of integration testing. Like many component-oriented systems, Maven provides much of the runtime context essential for testing a plugin in its "natural environment". This means that, in order to properly test the integration of any particular plugin into the larger system, the developer should really execute Maven builds that test particular features of the plugin in question.

However, running test builds from within the plugin's own build presents some special problems. Since we're using Maven to build the Maven plugin, we have to deal with the fact that a previous version of the plugin could be active within the current build (not a great practice, but sometimes unavoidable). This means that Maven must either be smart enough to unload the existing plugin-version from the runtime context and load the newer version - and correspondingly, unload the test version and reload the older version for other, later builds in the same reactor. As an alternative, the integration-testing of the plugin could fork separate Maven builds, and track the output in order to verify results and determine whether the build succeeds. The latter approach, while much slower and somewhat dependent on local-system configuration, is currently the most robust way to avoid existing versions of the plugin under test.

In addition to plugin-version inconsistencies, integration-testing of Maven plugins must cope with the need to cleanup after themselves. Remember, Maven only loads plugins from its own local repository (possibly after resolving them from a remote repository). Because of this, the only practical way to use the current plugin within an integration-test build is to install it into some sort of local repository, then direct Maven to use that local repository for the build. To avoid side effects in other builds (remember, we're building Maven plugins here...and they're used to build other projects), it's critical to avoid using the main local repository used for normal Maven builds. Therefore, plugin integration tests require a custom local repository, to which the integration-testing builds can be directed. Further complicating this testing scenario is the fact that the plugin's parent POM or one of its ancestor POMs could be a snapshot which has been installed in the main local repository, but which is not reachable using the plugin-POM's <relativePath/> parent-element.+

+ All of this seems to indicate that the best approach for ensuring the presence of all required POMs and plugins is to use the normal local repository to construct a new local repository for testing purposes. Work on this is ongoing.

Finally, in order to support integration-testing of plugins, it's important to give the developer an API by which to bootstrap the testing environment (and ideally, detect when it's already been bootstrapped), and to also provide a robust API for executing builds and verifying results.

A Note on Test-Suite Maintenance

Since failure to grasp the use cases of any system is a virtual guarantee that it will not meet the needs of the user, it's important to understand how integration tests are really used.

First and foremost, integration tests are used as a one-way valve, to ensure that no feature of the application - in our case, a Maven plugin - regresses as the result of unrelated development. A good set of tests - of which the integration tests are, well, integral - can provide the developer with freedom to implement new features with the confidence that he's able to effectively build on the foundations of previous versions.

However, it's important to understand that regression-prevention is not the sole purpose of an integration test. A secondary but critical use case deals with how integration tests are added to the plugin. One has only to consider how bugs are reported on a flawed system to understand. Since users usually report bugs, bug reports originate outside of the system. In the case of a Maven plugin, this means users are most likely able to supply examples of the failing behavior in the form of failing builds. In order to make immediate and direct use of what the user gives us, it's critical that integration tests are designed to be augmented using build examples from the user, with as little rewriting as possible.

To that end, the approach for integration testing in this library will be the execution of sample builds that use the plugin in question, then verify the results, either in the logs, the resulting exit code, or the project files it produces.

Integration-Testing Requirements

To summarize, this library strives to satisfy the following requirements for integration testing of Maven plugins:

  • Trigger Maven builds using test projects, without incurring any sort of conflict with plugins already in use within the current (this plugin's) build.
  • Construct a local repository into which the test version of the plugin can be installed, allowing the main local repository to remain untouched by the tests. Integration tests can be directed to use this test-time local repository instead of the normal location.
  • Support tracking and analysis of results from Maven builds spawned during integration testing. As far as is reasonable, test builds should be configured in the same way as normal project builds, to make the most of test cases submitted by users along with the issues they file.

Utilities Included in this Library

For more information on the APIs described below, see the JavaDocs.

  • PluginTestTool - The main entry point for setting up the plugin-testing environment, this utility orchestrates the ProjectTool, RepositoryTool, and BuildTool in order to produce a local repository containing the plugin jar, plus any ancestor POM files that are reachable using the <relativePath/> element of the parent specification.

    In order to make test projects more resilient to changes in the plugin's version, this tool allows the developer to specify a testing version for the plugin, so any reference to the plugin can be unambiguous and avoid accidentally referring to a plugin version resolved from outside.

  • ProjectTool - This testing tool provides APIs for creating a testable plugin jar file, optionally skipping the unit tests for the plugin (after all, it's conceivable that the unit tests themselves may be using these tools).
  • RepositoryTool - The RepositoryTool provides methods for assembling artifact repositories to support testing. This tool is responsible for assembling the local repository that is used during integration testing - complete with the test version of the plugin itself, along with all of the ancestor POMs that are reachable via <relativePath/>.
  • BuildTool - This tool's purpose is fairly self-explanatory; it is used to execute Maven builds. It provides a fairly generic API for running builds, for the simple reason that a single test may need to run multiple builds. For normal use cases, it's a good idea to wrap the BuildTool API using a standard project-testing method that will standardize the options used to run a similar set of test builds.