A Maven-friendly Pattern for Storing Dependencies in Version Control

I feel that I need to start this post with a disclaimer – this is for exceptional use cases only! In particular, take caution:

In any of these cases, it’ll result in a transitive closure that won’t be resolvable if the project is deployed into a repository without copying the repository module around

Over the course of time, the question about why not to store dependencies in version control when using Maven has faded as users have recognised the benefits of having a repository (and particularly a repository manager) in place to maintain their artifacts in a specialised way for the purposes of reuse and reproducibility.

But what happens when that case occurs where that is not practical, such as Greg pointed out in this issue, where you can’t get a repository installed yet but need those dependencies?

A commonly used solution is to have a series of install:install-file executions that manually install the files into the local repository for the build, either using a shell script or very convoluted POM. However, it breaks out of the Maven model of being able to build a multi-module project in one command (or makes it unnecessarily complicated to do so). An example of what results can be seen in the defunct NMaven incubator podling’s bootstrap build.

A better solution is to create a file-based repository stored in version control. By using the native repository mechanism, it ensures dependencies are treated consistently – even if they were to be available from a remote repository in the future. However there can be further problems if you are in a multi-module project. Since it is not recommended to use relative paths outside of the current project, having a single repository for a multi-module build could be complicated again.

The solution I proposed is to create a repository module specifically for housing these special dependencies. This can be included in the build like any other module, and simply needs to be listed first to guarantee that it is built before any other modules. Let’s see how this might look.

Say we have a parent pom.xml:

<project>
  <groupId>com.example</groupId>
  <artifactId>parent</artifactId>
  <version>1.0-SNAPSHOT</version>
  ...
  <modules>
    <module>repository</module>
    <module>modules</module>
  </modules>
</project>

The repository/pom.xml file would then be similar to this:

<project>
  <parent>
    <groupId>com.example</groupId>
    <artifactId>parent</artifactId>
    <version>1.0-SNAPSHOT</version>
  </parent>
  <artifactId>repository</artifactId>
  ...
  <dependencies>
    <dependency>
      <groupId>ancient-artifact</groupId>
      <artifactId>ancient-artifact</artifactId>
      <version>3.1.0.2</version>
    </dependency>
  </dependencies>
  <repositories>
    <repository>
      <id>local</id>
      <url>file:${basedir}/src/repository</url>
    </repository>
  </repositories>
</project>

Here we can see that the dependency (stored in repository/src/repository/ancient-artifact/ancient-artifact/3.1.0.2/ancient-artifact-3.1.0.2.jar) is self-contained to the repository module, and guaranteed to be installed when that project is built. Since it is listed first in parent build, the local repository will be correctly populated before any other modules are built. What’s more, if the dependencies are already present locally this module will skip by extremely quickly.

Note: if you happen to use this technique and also have a repository manager configured using Maven’s mirrorOf settings directive, make sure that you use <mirrorOf>external:*</mirrorOf> to ensure that the file-based repository requests are not passed to the repository manager.

A more complete example can be seen in the project attached to MNG-3989.

Now, while this is likely to be of very limited use, I did find it an interesting exercise to illustrate how you can still map alternate workflows neatly into the Maven paradigm, in a way that doesn’t compromise its artifact flow or build lifecycle, and still gives you a path forward to a different configuration with a remote repository storing the artifacts simply by removing the repository module.

This technique may also have additional utility for those with a policy-driven requirement to use their version control to store dependencies for reproducibility purposes, if they can afford the verbosity of restating all those dependencies in the repository POM.

Of course, I would still recommend using a remote repository wherever possible to manage these artifacts, and to treat its contents on the same level as your version control in your infrastructure if you intend to have a long-lived Maven installation. As noted at the beginning, this becomes a requirement if you want to publish your project as something others may depend on. However, if you need to store your dependencies in version control, this technique may help you at least honour Maven practices in doing so and perhaps reduce an initial adoption curve.

About these ads

10 responses to “A Maven-friendly Pattern for Storing Dependencies in Version Control

  1. Brett,

    I really enjoyed your article. I tried to run the sample code. I modified the repository pom by adding a log4j dependency…

    4.0.0

    MNG-3989
    test
    1.0-SNAPSHOT

    repository
    repository

    ancient-artifact
    ancient-artifact
    3.1.0.2

    log4j
    log4j
    1.2.12

    local
    file:${basedir}/src/repository

    I was expecting to see the log4j library to appear in the /src/repository directory. Is this an incorrect assumption?

    thanks!
    Brian

  2. Very creative idea!

  3. That is an interesting solution to this very common problem and definitely a step up from the abuse of the system scope maven users have been using.

    However, I downloaded your example project from MNG-3989 and it will not build with maven 3.0.3 (which is the current version as I write this). The error I see is “could not resolve dependencies”.

    Even with maven 2.0.x I am suspicious that under your solution if one were to try to build modules separately, which is very common, that it would probably not work, a possible show stopper for some. Furthermore, I have been unable to get the m2Eclipse plugin for Eclipse to work properly with your solution (which may be a problem with that plugin, I don’t know) but would also be a show stopper for many.

    As a Maven user and believer I am somewhat annoyed that MNG-3989 was closed as “Won’t Fix” after your solution was proposed. I would have like to have seen the Maven guys take this common problem more seriously and come up with a cleaner and less cumbersome solution. Until then I think we are stuck choosing between a bunch of half baked solutions.

    Cheers!

  4. I’ve updated it for Maven 3, which may get close to a more integrated solution by using extensions: http://jira.codehaus.org/browse/MNG-3989?focusedCommentId=261657&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_261657

    I’m not sure if this will work in m2e. I’d suspect probably not, or at least not very well.

    The same problems about building subprojects would be noted.

    The issue was actually closed by the reporter as won’t fix – we both recognise this is for a very specific use case of distributing a self-contained source code bundle. If you are actually developing against a project in a regular manner, you should move your artifacts into a repository manager.

  5. The suggested solution is fine, however I’d say it is not very elegant (because of the necessity to manage a maven repository in the local file structure). I’ve recently released a maven plugin which provides an easier way to add local jars to project’s classpath:

    com.googlecode.addjars-maven-plugin
    addjars-maven-plugin
    1.0.1

    add-jars

    ${basedir}/lib

    In the given example, all jars from the ‘lib’ folder will be added to the classpath.

    • Basil, the management of the repository in the local file system is not a complicated task. It just entails creating the write directory structure and file names. Or you can use install:install-file to put jars in there, just like you would to install them in any other repo. All you need is one additional parameter, localFileRepository, to tell it to install it to that directory.

      Your solution however missed the whole multi-module nature of the problem. If I had multiple child projects that all depend on the same libraries that I need to be stored in my project, where would I put the library? Duplicate it in a lib directory in each child project? Your solution doesn’t work.

      Brett’s solution keeps the library in only one place.

      • > Your solution however missed the whole multi-module nature of the problem.

        I’m using the addjars plugin in a multimodule project, so not sure why you consider it problematic.

        You can put the jar files to the parent module’s lib directory. Then add the same addjars-maven-plugin declaration to all modules, which depend on the jar. The tag should point to ${basedir}/../lib in this case. Or, alternatively, you can create a separate module for libs, and make other modules dependent on it.

        With Brett’s solution you would need to do similar things.
        Either duplicate the same list of dependencies for each module which depends on them. Or, alternatively, create a separate module for libs, and make other modules dependent on it :)

        If you have plenty of jars, the addjars-maven-plugin declarations would be much shorter than the plain list of dependencies. That’s why I prefer using the plugin.

      • “The tag should point to ${basedir}/../lib in this case. ”

        read it as

        “The ‘directory’ tag should point to ${basedir}/../lib in this case. “

  6. “The tag should point to ${basedir}/../lib in this case. ”

    read it as

    “The ‘directory’ tag should point to ${basedir}/../lib in this case. “

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s