Maven: multi-project and dashboard issues

16

Some Maven multiproject experiences will be described in this post. Special attention is paid to the dashboard plug-in.

Maven: multi-project and dashboard issues

Generic Maven project structure

Let us first reiterate the canonical structure of a Maven
(sub)project:

Canonical Maven Project Lay-out

This project structure will henceforth implicitly be assumed for
all subprojects constituting an overall superproject.

A canonical multiproject lay-out

Although most if not all plug-ins work flawlessly with the
multiproject structure as outlined below, it is not a universally
applicable lay-out. For example, this lay-out does’t integrate
well with the Eclipse or WSAD IDEs.

However, for a proper understanding, let us first shortly
(re)introduce the canonical multiproject structure, that is
(sometimes implicitly) assumed in most Maven and
Maven plug-in related documentation.

Canonical multiproject structure

The lay-out of a canonical multiproject project is as follows
(where we collapsed the subproject tree structure as was listed
above):

Canonical multi-project

Invoking

$ maven multiproject:site

a target directory is generated containing the maven generated
site.

The advantage is that set aside some customizations and/or
optimizations, a maven.xml is not needed, hence
your Maven (and Jelly) related knowledge can be restricted
to the bare minimum. Moreover, you project structure lay-out
will be Maven generic!

Below we will focus a bit more on the precise contents of
the project.xml and the project.properties,
in order to be able to point out the differences with our
more generalized case that will be discussed thereafter.

Contents project.xml

An example project.xml has been listed below.
We made use of the possibility to inherit (extend) from a top-level
project.xml file, located in one of the subprojects.
This is especially usefull when dealing with a project consisting
of a lot of subprojects and is almost always employed.

This project.xml is responsible for the main (overview)
project site. The dashboard and multiproject plug-ins are configured
here, the former generating a summary of project metrics
(which metrics are to be included can be configured, see
the project.properties below),
the latter analyzing the version convergency of the
dependencies of all subprojects on (external) artifacts,
such as JAR and ZIP files.

Of course, other plug-ins, such as the mutlichanges plug-in can
and may be configured here as well, but we will refrain from
doing so for the sake of clarity.

A template project.xml is shown below:

<?xml version="1.0" encoding="UTF-8"?>
<project>
  <extend>sub-project-m/top-level-project.xml</extend>
  <pomVersion>3</pomVersion>
  <id>site</id>
  <name>Overall Site</name>

  <reports>
    <report>maven-dashboard-plugin</report>
    <report>maven-multiproject-plugin</report>
  </reports>
</project>

Contents project.properties

Below we have only listed those properties
from the project.properties file that are relevant
to the multi-project plug-in (and reactor) and the
dashboard plug-in and associated overall report generation:

#
# Multiproject
#
maven.mutliproject.basedir=.
maven.multiproject.includes=sub-project-*/project.xml
maven.multiproject.excludes=sub-project-DontBuildMe
#
# Dashboard properties
#
# Tells the dashboard plugin not to run its own reactor.
# We’ll use the one run by the multiproject.
maven.dashboard.runreactor = false
#
# Tells the multiproject plugin to execute the dashboard:report-single
# goal on each project when you run it through multiproject:site.
maven.multiproject.site.goals=site, dashboard:report-single
#
maven.dashboard.aggregators = csfiles,cswarnings,cserrors,junittests,junitfailures,juniterrors,junitpassrate,pmdfiles,pmdviolations,simiantdl,jcoveragelipc,jcoverageloc

The multiproject related properties don’t need to be explained,
except maybe for the maven.multiproject.excludes: this
property is optional and can be used for those subprojects that must
(temporarily) be exempted from the build process.

Dashborad report generation

It turns out that the dashboard plug-in seamlessly integrates
with this canonical multiproject structure. An integrated
summary will be available under the project report section.
An example of such a summary is listed on the Dashboard plug-in page,

http://maven.apache.org/reference/plugins/dashboard/.

Of course, for those subprojects that have dashboard reporting
enabled in the report section of their POM (project.xml),
a dashboard report for the subproject itself is included in the
project reports section.

Multiproject with root subproject

The need for a more generalized lay-out has been motivated at the
beginning of this document. In practice, this "root"
subproject may be named differently, such as main project, site
project or deployment project.

Multiproject with root dir structure

The lay-out of a canonical multiproject project that contains
a root project on the same level as the subprojects is depicted
below:

Multi-project with root dir on same level as subprojects

The differences with the canonical case are immediately obvious:

  • The project.xml and
    project.properties have moved to the root
    subproject
  • A maven.xml is mandatory now
  • The root project is on the same level as the subprojects
  • Site pages reside in the site dir.

All these small differences imply many small intricate mutations
as compared to the canonical case. These changes will be
elaborated on below.

Contents maven.xml

Let us first focus on the maven.xml file.

<goal name="myProject:site"
      description="Build site for the master project and all subprojects">

  <echo message="+-----------------------------------------------+"/>
  <echo message="| First building sites of separate sub projects |"/>
  <echo message="+-----------------------------------------------+"/>

  <maven:reactor
             basedir="${basedir}/.."
             includes="${maven.multiproject.includes}"
             excludes="${maven.multiproject.excludes}"
             goals="dashboard:report-single, dashboard:report, xdoc"
             banner="Build site for subproject "
             postProcessing="true"
             ignoreFailures="true"/>

  <echo message="+----------------------------------+"/>
  <echo message="| Performing dashboard aggragation |"/>
  <echo message="+----------------------------------+"/>

  <attainGoal name="dashboard:aggregate"/>
  <attainGoal name="myProject:mainSite"/>
</goal>

<goal name="myProject:mainSite"
      description="Build main (root/overview) pages">

  <echo message="+-------------------------------------+"/>
  <echo message="| Building main (root/overview) pages |"/>
  <echo message="+-------------------------------------+"/>

  <maven:reactor
             basedir="${basedir}"
             includes="site/project.xml"
             goals="allSite"
             banner="Build site for myProject using "
             postProcessing="false"
             ignoreFailures="true"/>

  <echo message="+-----------------------------------+"/>
  <echo message="| Copying all to target destination |"/>
  <echo message="+-----------------------------------+"/>

  <copy todir="${basedir}/target">
      <fileset dir="${basedir}/site/target">
          <include name="**/*" />
      </fileset>
  </copy>
</goal>

The maven.xml is composed as follows:

  1. The myProject:site builds the whole
    site, i.e. the main site and of all subprojects.
  2. It starts off by invocation of the reactor (standard)
    for the site generation of all subprojects.
  3. Note the goals that are invoked for each separate subproject!
    This is essential to get the dashboard reports
    right.
  4. Thereafter, an aggregate dashboard report is generated.
  5. The overall site is generated. This will require a
    section by itself. Note how the generation of the
    main site is "handed over" to the
    allSite goal of the maven.xml
    in the site directory.
  6. All files are copied to the target
    directory in the root level project directory.
    Needless to say, any destination will do, as well as a
    deploy to web server, of course.

Contents project.properties

Below we have only listed those properties
from the project.properties file that are relevant
to the project lay-out (and dashboard report generation):

#
# Multiproject
#
maven.multiproject.basedir=..
maven.multiproject.includes=sub-project-*/project.xml
maven.multiproject.excludes=sub-project-root/project.xml
#
# Dashboard properties
#
# Tells the dashboard plugin not to run its own reactor.
# We’ll use the one run by the multiproject.
maven.dashboard.runreactor = false
#
maven.dashboard.aggregators = csfiles,cswarnings,cserrors,junittests,junitfailures,juniterrors,junitpassrate,pmdfiles,pmdviolations,simiantdl,jcoveragelipc,jcoverageloc,javancssncsstotal,javancssjavadocstotal

The properties and their values should be obvious from inspection.
Notice the dashboard relevant properties especially: not more, not
less is needed.

Finally, for dashboard to work properly, you must
incorporate the following lines in the
project.properties of each of the
subprojects:

maven.dashboard.rungoals=true
maven.dashboard.includes=project.xml

Overall site generation

The site pages, as well as the overview of all the subprojects
and the aggregate dashboard report are generated based on
the material residing in the site
directory. Rationale: since we deviated from the canonical
project lay-out, the multiproject nor the dashboard plug-ins
"know" how to handle this situation, and special
adaptations have been incorporated to cope with this new situation.

Let us first focus on the overview of all subprojects.
This page is generated from the projects-overview.xml
file in the xdocs-templates directory:

<?xml version="1.0"?>
<document>
  <properties>
    <title>Overview of projects</title>
  </properties>
  <body>
    <section name="Overview of projects">
      <table>
      #foreach ($reactorProject in $reactorProjects)
        <tr>
          <td><b><a href="${reactorProject.file.parentFile.name}/index.html">${reactorProject.artifactId}</a></b></td>
          <td>${reactorProject.shortDescription}</td>
        </tr>
      #end
      </table>
    </section>
  </body>
</document>

Using velocity, this template is transformed into a
projects-overview.xml in the xdocs
directory, which in turn is transformed into an HTML document
by the xdoc plug-in.

This transformation process is controlled by the
maven.xml in the site directory:

<project
  default="allSite"
  xmlns:j="jelly:core"
  xmlns:maven="jelly:maven"
  xmlns:x="jelly:xml"
  xmlns:u="jelly:util"
  xmlns:a="jelly:ant"
  xmlns:velocity="jelly:org.apache.commons.jelly.tags.velocity.VelocityTagLibrary">

  <goal name="allSite">
    <attainGoal name="subSite"/>
    <attainGoal name="dashboard:xdoc"/>
    <attainGoal name="site"/>
  </goal>

  <goal name="subSite"
        description="Build the overview for all subprojects"
        >
    <maven:reactor
               basedir="${basedir}/../.."
               includes="${maven.multiproject.includes}"
               excludes="${maven.multiproject.excludes}"
               goals="pom"
               sort="true"
               banner="Building subprojects overview"
               postProcessing="true"
               ignoreFailures="false"/>

    <attainGoal name="dashboard:aggregate"/>

    <!--
     |
     | At this point because 'postProcessing' was turned on we have all
     | the processed POMs available to us in the ${reactorProjects} variable.
     |
     | The overall project page can now be generated.
     |
     -->

    <fileScanner var="templates">
      <fileset dir="${basedir}/xdocs-templates">
        <include name="*.xml"/>
      </fileset>
    </fileScanner>
    <j:set var="templatesX" value="${templates}X"/>
    <j:if test="${!templatesX.equals('X')}">
      <j:forEach var="template" items="${templates.iterator()}">

        <j:set var="templateName" value="${template.name}"/>
        <echo>Processing ${templateName} template ...</echo>

        <velocity:merge
          name="${basedir}/xdocs/${templateName}"
          basedir="${basedir}/xdocs-templates"
          template="${templateName}"/>

      </j:forEach>
    </j:if>

    <!--
     |
     | The sites of the various subprojects are now copied over to the main
     | site.
     |

    -->
    <j:forEach var="reactorProject" items="${reactorProjects}">
        <echo>Copying target dirs from project ${reactorProject.artifactId} to ${basedir}</echo>
        <copy todir="${basedir}/target/docs/${reactorProject.artifactId}">
           <fileset dir="${basedir}/../../${reactorProject.artifactId}/target/docs">
               <include name="**/*" />
           </fileset>
        </copy>
     </j:forEach>
  </goal>
</project>

Note how

  • The reactor merely builds the POM, in order to have
    all subprojects available, so that we can iterate over
    them. With this iteration, the projects overview page
    and the dashboard report can be constructed.
  • The subSite goal generates the overview page
  • Where and how the dashboard aggreation takes place
  • The dashboard aggregate data (in
    dashboard-data.xml) is transformed into
    a report (in dashboard-report.xml), which
    will eventually be transformed into a
    dashboard-report.html.
  • How the site material of the subprojects is copied over
    to the overall destination.

In order for the generation of the overview page to
work properly, the following dependencies have to be
incorporated in the project.xml in the
site directory:

<?xml version="1.0" encoding="UTF-8"?>
<project>
  <extend>../bibtexml-top-level-project.xml</extend>
  <pomVersion>3</pomVersion>
  <id>site</id>
  <name>BibTeXML Site</name>
  <dependencies>
    <dependency>
      <id>commons-jelly+tags-velocity</id>
      <version>SNAPSHOT</version>
    </dependency>

    <dependency>
      <id>velocity</id>
      <version>1.3</version>
    </dependency>
  </dependencies>

  <reports>
    <report>maven-multiproject-plugin</report>
    <!--report>maven-multichanges-plugin</report-->
    <!--report>maven-tasklist-plugin</report-->
  </reports>
</project>

Moreover, if you want the multiproject convergency report,
you have to enable it here. Do not enable
the dashboard report here: it won’t work, and what’s even
worse, it will overwrite our so carefully aggregated
report with an empty one!

Share.

About Author

16 Comments

  1. Pingback: AdSense Money Maker

  2. There is a typo at:
    #
    # Multiproject
    #
    maven.mu”tli”project.basedir=.

    by the way, when i try to keep the master/root project.xml in a dir no subproject is constructed in the end :(

  3. Zeger Hendrikse on

    The restructuring of our project is completed. However, we decided not to continue (temporarily) with Maven 1.1-beta. Some reasons were specific for our project, while others are more generic:

    1. We use velocity templates to generate a list of projects on our Maven generated web site. We get weird errors when going to Maven 1.1, which we don’t have time for to solve. Even more so, these errors are likely to be solved in the final Maven 1.1. release.
    2. Maven 1.1 requires JDK 1.4. As our project is not yet 100% JDK 1.4 compliant, we have to do some very dirty things to make Maven build successfully. This problem should be solved in october.
    3. There seem to be some (serious) quircks with Maven 1.1 and dashboard. More specifically, I couldn’t get dashboard properties right (maven.dashboard.includes, maven.dashboard.excludes), nor could I stop dashboard from starting a reactor in each subproject (and no, I’m not too stupid to do so, because I get it to work in Maven 1.0.2 ;-) ).

    All this made us decide to wait for the official release.

    Some positive remarks to conclude though: Maven 1.1 has a much lower memory footprint indeed, and seems to be much faster, a typical site generation for our project went down from 45 to 30 minutes (approx.).

    In fact, we cannot use the Maven reactor right now, because of “out of memory” problems, whereas Maven 1.1 did fine, except for the Dashboard and Velocity template generation.

  4. Zeger Hendrikse on

    Currently we are migrating to Maven 1.1, while at the same time restructuring our project. I’ll collect our experiences and describe them in a forthcoming follow-up. I’ll leave a comment with a link to this new post here.

  5. Thank you Rupert for your suggestion as it helps me make my multiproject dashboard work.
    I have made a my own improvements though, I think it is better to separate the “multiproject/project.xml” from the “base/project.xml” where all subcomponents inherited. This way you don’t have to edit each subprojects dashboard settings

    base/project.properties
    –>maven.dashboard.basedir=${pom.getArtifactId()} (this value depends on where you’ve put the base/project.xml file)

    I had to add this line to make the multiproject dashboard work:
    –>maven.dashboard.includes=${maven.multiproject.includes}

  6. Zeger Hendrikse on

    I still did not have time to check out your solution in details, but it looks promising!

    The reason I had my project structure laid out as described above was that I started working on a real-life instance of such a project the first of January, so the structure was already completely fixed when I started. I then had to come up with a Dashboard solution for the existing project structure.

    But again, your solution looks much neater and should be the to be preferred configuration for new projects.

    Thanks for your feedback.

  7. Rupert Smith on

    And my structure is actually:

    basedir
    |
    |- project_1
    |  |
    |  |-project.xml
    |  |-project.properties
    |  |-maven.xml
    |
    |- ...
    |
    |- mavenbuild
    |  |
    |  |- project.xml
    |  |- project.properties
    |  |- maven.xml
    
  8. Rupert Smith on

    I forgot something else. To each sub projects project.properties you need to add (for project_n):

    # Tell dashboard to run in on this project only when run in this directory. This relative directory trick will work when called from the
    # main build directory or from this directory.
    maven.dashboard.basedir=../project_n
    maven.dashboard.includes=project.xml

    Otherwise you get the full multi-line dashboard report in each sub project because in the main project.properties you told dashboard to run in the parent directory and include all sub-projects and these properties are inherited by all the sub-projects. So you need to override this in each sub-project to create the dashboard just for it when dashboard is run in its individual directory or called from the main script.

    Sorry for posting all this in bits in pieces. I should have planned my post out a little better. Hope its still usefull.

  9. Rupert Smith on

    Like this:

    basedir
    |
    |- project_1
    | |
    | |-project.xml
    | |-project.properties
    | |-maven.xml
    |
    |- …
    |
    |- mavenbuild
    | |
    | |- project.xml
    | |- project.properties
    | |- maven.xml

  10. Rupert Smith on

    My diagram of teh project structure hasn’t come out right. The mavenbuild directory contains a project.xml, project.properties and maven.xml. They are not in the parent directory as the diagram has come out.

  11. Rupert Smith on

    Zeger, Thanks for your post. It was helpful in pointing me in the right direction. I have a project structure with the master build project.xml in one of the sub-project as described in the root case above. However, I think the way you are running the dashboard is overly complicated and also inneficient. I have managed to get it working in a far simpler way with such a project structure.

    My structure is:

    basedir
    |
    |- project_1
    | |
    | |-project.xml
    | |-project.properties
    | |-maven.xml
    |
    |- …
    |
    |- mavenbuild
    |
    |- project.xml
    |- project.properties
    |- maven.xml

    The projects in project_1, … all inherit from the master script in mavenbuild.

    I have my reports set up in the mavenbuild/project.xml:

    maven-checkstyle-plugin
    maven-jdepend-plugin
    maven-pmd-plugin
    maven-jxr-plugin
    maven-simian-plugin
    maven-javadoc-plugin
    maven-clover-plugin
    maven-junit-report-plugin
    maven-tasklist-plugin
    maven-dashboard-plugin

    In the mavenbuild/project.properties I set up multiproject and the dashboard. I also set up clover to output the raw .xml report so that the dashboard can re-use it:

    # Set up the dashboard to run in the parent directory and to ignore this project.
    maven.dashboard.basedir=..
    maven.dashboard.excludes=mavenbuild/project.xml
    maven.dashboard.rungoals=false
    maven.dashboard.aggregators=csall,junitpassrate,junittests,juniterrors,junitfailures,clovertpc,cloverncloc

    # Tell multiproject to work in this directory and to ignore this project.
    maven.multiproject.basedir=..
    maven.multiproject.excludes=mavenbuild/project.xml

    # Tell clover to output the raw .xml report so taht the dashboard can re-use it without running clover again.
    maven.clover.report.xml=true

    Then in the mavenbuild directory I run the multiproject:site goal and it correctly generates the dashboard for all projects. This is much simpler than what you describe above. It is also more efficient because I have been able to use the rungoals=false setting to re-use the output of the other reports that I am generating. If this were set to true as in your example it would run all my tests, clover, checkstyle and so on two more times! Once when generating the single line dashboard for each individual sub-project and once when generating the full multi-line dashboard for the multi-project parent site.

    My project layout is a little different to what you have described here as I don’t have a sub-project within my mavenbuild directory for generating the site. Possibly, I have not fully understood your reasons for doing it that way?

    It seems to me that the important trick with my layout is telling the dashboard to work in the parent directory and to exclude the mavenbuild directory with the settings in the project.properties file.

    I hope you find this helpful.

  12. Zeger Hendrikse on

    By the way, I forgot to mention that I got an excellent introduction into Maven myself from my colleague Marcel de Koster. Consequently, this example (code and structure) is also partly based on his work.

  13. Zeger Hendrikse on

    As a matter of fact, I did so for the concrete instantiation of the "template" project presented in this post.

    Basically, you set up the project structure exactly as discussed in this post, that is, with the root directory on the same level as your subprojects.

    You have to play a bit with the project settings in Eclipse, to get the source and conf directories in your classpath, but then everything integrates seamlessly.

    You may want to use the mevenide eclipse plugin to keep the dependencies section of your project.xml synchronized with your Eclipse classpath.

    If this isn’t enough to get you started, just leave another comment, and I’ll seriously consider a follow-up!