Using a GPG agent for signing Maven releases on Mac OS X

Using the Maven GPG Plugin makes it easy to sign a large number of artifacts when performing a release with Maven.

However, one of the annoying parts is that interactive password entry is not particularly easy, and you often have to put it on the command line or into your settings file in plaintext, which is not very comfortable. The better alternative is to use gpg-agent, and if you’re not using GPG 2.0 this needs to be configured in the POM:

  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-gpg-plugin</artifactId>
    <version>1.1</version>
    <configuration>
      <useAgent>true</useAgent>
    </configuration>
  </plugin>

GPG will fallback to the same password entry if it is not running or if it is provided via the property, so there’s no particular downside to enabling the option.

The problem on Mac is that the situation with GPG is a little confusing. There seem to be several different projects floating around to provide it.

I’m a big fan of Homebrew, and previously I’d been using that to install GnuPG and gpg-agent. It worked quite well, but the password entry required curses, and also didn’t work in an interactive session – so you’d need to make sure to have entered it manually before starting a release. If it failed mid-release, you’d need to restart the agent. None of this was Homebrew’s fault – I just hadn’t been able to find a native pinentry app for the Mac that stood alone and didn’t crash on use.

After fiddling with it for a while, I decided to try one of the native Mac GPG installers again. This is where it got confusing, since there are several efforts floating around. Others may work, but the one I had success with is the revived GPGMail project. After uninstalling all the packages using Homebrew, I installed GPGTools from the download page. Since this includes a native pinentry application, and pre-configures gpg and the agent to run once on login, it worked much better.

The only catch if you’re trying this is that you need to logout, or in your shell run this until you have:

open /usr/local/libexec/start-gpg-agent.app/
. ~/.gpg-agent-info
export GPG_AGENT_INFO

You can test that it is working outside of Maven by running: gpg -ab (ending with Ctrl-D),

Creating a Custom Build Extension for Maven 3.0

In the process of testing the Maven 3.0 release that’s being voted on, I wanted to try out the additional extension capability, and have a little fun with it in the process.

The result is an old friend of mine:

[INFO] Scanning for projects...
[INFO]  __  __
[INFO] |  \/  |__ _Apache__ ___
[INFO] | |\/| / _` \ V / -_) ' \  ~ intelligent projects ~
[INFO]   |_\__,_|\_/\___|_||_|  v. 3.0
[INFO] 
[INFO] --------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] retro-example
[INFO] retro-example-1
[INFO] retro-example-2
[INFO]                             

This is a trivial example of course, but it shows that you can jump into the startup process much earlier than in the past. This is useful for those that want to extend Maven in a consistent way for a given project or type of project. I believe it is already in use by Tycho, and is a likely mechanism to simplify and enhance NPanday in the future.

While plugins and extensions could already offer additional components, it is now possible load components from within a project that can be set up before the build starts at all. The feature would most often be used to perform additional validation or processing of an entire set of projects in the reactor before they are executed. It is possible that it might also be used to adjust settings, execution properties or dependency resolution – though bearing in mind that these may already have been used to load the projects.

The above example makes use of the afterProjectsRead method (the whole project is in Subversion):

package org.apache.maven.examples.retro;

import org.apache.maven.AbstractMavenLifecycleParticipant;
import org.apache.maven.execution.MavenSession;
import org.apache.maven.execution.RuntimeInformation;
import org.codehaus.plexus.component.annotations.Component;
import org.codehaus.plexus.component.annotations.Requirement;
import org.codehaus.plexus.logging.Logger;

@Component( role = AbstractMavenLifecycleParticipant.class, hint = "retro" )
public class RetroMavenExtension
  extends AbstractMavenLifecycleParticipant
{
  @Requirement
  private Logger logger;

  @Requirement
  private RuntimeInformation runtime;

  public void afterProjectsRead( MavenSession session ) {
    logger.info( " __  __" );
    logger.info( "|  \\/  |__ _Apache__ ___" );
    logger.info( "| |\\/| / _` \\ V / -_) ' \\  ~ intelligent projects ~" );
    logger.info( "|_|  |_\\__,_|\\_/\\___|_||_|  v. " + runtime.getApplicationVersion() );
    logger.info( "" );
  }
}

There are a couple of things to note here, particularly if you are familiar with writing components for Maven. Firstly, there are now some real annotations to use instead of the Javadoc-based version (though both would continue to work). This example still uses the Plexus annotations and the generated descriptor from the corresponding POM, though in the future it could use the JSR-330 equivalents.

Obtaining a logger is much easier than before, using a simple @Requirement to inject it instead of the LogEnabled interface or abstract class. Other requirements (in this case, the legacy RuntimeInformation class) continue to be injected as they have before.

The rest is self explanatory – based on the “role” and the abstract class, the method is called by the Maven core after the projects have been assembled but before they have been sorted and executed. We just output the ASCII art, however the method has the MavenSession object available to it to obtain the projects, as well as execution properties, repository access and settings.

To use the extension in a project, the following would need to appear somewhere in the POM hierarchy:

<build>
  <extensions>
    <extension>
      <groupId>org.apache.maven.examples</groupId>
      <artifactId>retro-maven-extension</artifactId>
      <version>1.0-SNAPSHOT</version>
    </extension>
  </extensions>
</build>

It’s worth noting that Maven 3.0’s extension handling and classloading is significantly better than Maven 2.x – you can now feel relatively free to use extensions and plugin dependencies in a multi-module project without conflicts.

That’s it! Every Maven command on a project with the extension will carry the banner lost for the last 5 years.

Copying a Codeplex subversion repository with rsvndump

As part of the process of migrating NPanday to the Apache Incubator, I had to find a way to extract the Subversion repository from Codeplex. The challenge here is that it isn’t actually a Subversion repository, but a TFS server running SVNBridge to appear like one. Its occasional quirks and timeouts had been part of the reason we decided to move. Here is what I’ve learned.

Available Tools

The aim was to get it as a dump of a Subversion repository so that it could be loaded into the ASF Subversion repository, retaining the full history. I tried everything to get it out of there – svnsync, cloning it as a Git repository, cloning it as a Mercurial repository, and other similar tools. All would timeout or freak out at some point due to the nature of SVNBridge. I made some progress with tfs2svn (which seems to be what Codeplex is using to migrate repositories to Mercurial when needed), but I started to find that not only was it needing regular manual intervention, it wasn’t quite the same (eg. Subversion properties came in as ..svnbridge hidden folders, and every comment appended the original TFS revision number).

I had tried rsvndump earlier without a lot of success, but eventually gave it another try. While it needed some help, it was making the most progress and was the one that ended up being successful.

Patching rsvndump

It wasn’t anywhere near smooth sailing, and rsvndump needed some modifications to handle the task. So I brushed off my dusty C skills and made the required changes, which can be found in my github fork of the project (pull request pending).

The main problem was that rsvndump expected to be able to do the repository all in one hit. Given that Codeplex would timeout on several requests, this made it impossible. It did allow selecting a subset of revisions, but even then it would both do a full svn log (which wouldn’t succeed), and beyond that would traverse revisions to construct a path hash (I believe for detecting moves, etc.).

To assist with these, I added a --log-window-size option, similar to git-svn. This retrieves the logs in multiple requests, avoiding problems with timeouts. Next, I added a --first-rev argument, which would start retrieving logs and content from a later revision than 0. While it introduced some risk of crashing due to missing revisions, in many circumstances it allowed restarting a dump from a later revision in a much faster manner.

The next problem was that Codeplex shares a single TFS repository between several projects, so your own revision numbers are not sequential. NPanday started at revision 21102, and ended at 60509 with a lot of gaps, having only 1427 revisions of its own. This wasn’t too much of a problem – because rsvndump was designed to deal with subdirectories of a Subversion repository it expected the gaps. The --first-rev argument helped deal with the big gap to the start. But another SVNBridge quirk was that svn copy operations copied from the (current revision - 1) – even when it didn’t exist! To correct this, I had to adjust the code to search backwards through the revision numbers until it found one that existed to make the copy operations correct.

Finally, rsvndump added padding revisions into the dump file when a revision number was missing. This is helpful if you want to maintain the same numbers, but due to my use of --first-rev they were already out and I was importing to an existing repository, so I decided to strip these out. For that, I added another flag --omit-padding-revnums.

Running rsvndump

I ended up running a command like the following:

rsvndump --omit-padding-revs --adjust-missing-revnums \
  --first $FIRST --log-window-size 1000 -v --incremental \
  -r $REV1:$REV2 https://npanday.svn.codeplex.com/svn \
  3>&1 >&2 2>&3 3>&- >$REV1-$REV2.dump | tee $REV1-$REV2.log.txt

The first few arguments are the customisations described above (and --adjust-missing-revnums to make the dumped revisions sequential). The next are the traditional svnadmin dump arguments that rsvndump honours. Finally, I redirected the output so that I could channel stdout to the dumpfile and stderr to a log file that I could also tee.

Other Codeplex SVNBridge Issues

With these changes in place, I was getting moderately successful dumps – but a few frustrating issues remained.

Firstly, many svn copy operations (such as creating a tag) were tracked file by file by Codeplex instead of at the top level directory. This resulted in further timeouts that I couldn’t work around. We had seen this manifest on the Codeplex repository, being unable to even list the /tags/ directory. I didn’t attempt to correct this, instead manually applying the copy operation again after the preceding dump, then continuing.

For example:

svn cp \
  -m '[maven-release-plugin]  copy for tag npanday-project-1.2.1' \
  $REPO/trunk $REPO/releases/npanday-project-1.2.1

svn ps svn:author "SND\jocaba_cp" --revprop -rHEAD $REPO
svn ps svn:date \
  2010-09-07T07:15:46.723000Z \
  --revprop -rHEAD $REPO

If that revision appeared in the dump file (either incomplete or not able to be applied), I’d delete it by searching for Revision-number: xyzxyz and deleting the lines up until the next revision.

Between tags and a few other stubborn revisions that wouldn’t come across (including one where even svn log wouldn’t succeed), I manually reconstructed 100 revisions like that. The upside was that it provided an opportunity to clean out some botched releases (due to the SVNBridge /tags/ issues) and branches that had never been used.

So the process was to dump as many revisions as possible, then apply to a test repository, check it out, make required modifications, and repeat. I captured all of this in a shell script so that at any time I could recreate the work repository and reapply all of the dumps and modifications to date. This because useful a few times as I gradually identified inconsistencies with a checkout from the same revision in Codeplex from having missed something.

This still uses a lot of bandwidth – starting at a given revision will both reconstruct the path hash for the whole repository at that revision, and fetch the “base revision”, which is a checkout of an entire revision, tags and all. So the process took a few days running intermittently. I also had to start the --first-rev at least one revision earlier and sometimes more, to avoid getting a cryptic Subversion error message about the “editor drive”.

Properties were also quite quirky on SVNBridge, due to the way they are apparently stored in TFS as described earlier. Some could not be removed (eg, bogus svn:mime-type), and some were set oddly (svn:ignore on a file, svn:eol-style on a directory). I chose to leave these alone and correct them after the import.

Some properties went missing, which was part of a larger problem on SVNBridge with copying from an existing revision. If you attempt to copy in the working copy and then make a modification before committing, this doesn’t show up as A+ in the svn log result later, but simply M. The dumps know it was copied, but not that it was added, so attempt to modify a non-existant file when being applied. What’s more, this step wipes out some properties that are set on directories.

In some cases, I manually applied the revision, in others I made an edit in the dump file from:

Node-action: change

to

Node-action: add

Deleting directories hit snags as well. I’m unsure if this was a problem in SVNBridge or rsvndump, but it would dump deletions for every path and file like so:

/tags/npanday-1.2-RC1
/tags/npanday-1.2-RC1/pom.xml
/tags/npanday-1.2-RC1/dotnet
...

When applying the dump, it would successfully delete the first then fail on the others that were already deleted in the first step. I ended up removing the nodes for all the later entries manually in these instances. You would take these out 4 lines at a time (including trailing whitespace of 2 lines):

Node-path: tags/npanday-1.2-RC1/pom.xml
Node-action: delete

Final manipulations

Aside from Codeplex, for NPanday we needed to make some more manipulations. First, changing the usernames to line up to their final accounts on the ASF using repeated changes to the svn:author revision properties.

The dump was also loaded onto another partial subversion repository that contained some intermediate history between leaving the incubator originally and arriving at Codeplex.

Loading to an existing Subversion repository and path

After all this was eventually done, and there was a repository that was matched with the history of the Codeplex one, it needed to be dumped to load into the ASF repository.

Normally, this would be a simple:

svnadmin dump --incremental --deltas work-repo >npanday.dump

However, the objective was to load this onto a path that already existed. This was because we sought to have continuity with the history from the point at which the project was forked from the incubator originally.

To achieve this, I identified the revision in the dump that matched the content in the ASF repository, which due to the initial creation of branches and tags, was revision 4. I then dumped it using:

svnadmin dump --incremental -r5:HEAD work-repo >npanday.dump

Originally, --deltas had been included to reduce the size, but we found that this caused checksum problems, possibly due to different line endings between r4 and the original in the ASF repository.

At last, this yielded a dump that could be loaded into the ASF repository, and the results can be seen here: http://svn.apache.org/viewvc/incubator/npanday. You can now see the historical continuity in files such as http://svn.apache.org/viewvc/incubator/npanday/trunk/pom.xml.

Conclusion

This took considerably more work than anticipated when we originally thought it would be a good idea to retain the history.

I found that there wasn’t a lot of information about these topics on the web, so I hope this post will help to expand that for those that might face this challenge in the future. I’ve also found that editing Subversion dump files (when not in delta-mode) is reasonably straightforward.

Interestingly I’ve learned that Subversion 1.7 will include the ability to do a remote svnadmin dump, however I don’t believe this will work when svnsync is not supported (as was the case here), or support sub-paths as rsvndump does.

Apache Maven 2: Effective Implementation Sample Chapter

When I announced the release of the book last week, I neglected to mention that there is a sample chapter online. It is available from the book’s site, but can be directly downloaded as a PDF:

• Chapter 6 – Useful Maven Plugins

There is also an excerpt online if you don’t like PDF.

While it does stand alone, the examples are also worked in to the context of the sample project used throughout the book where appropriate.

This chapter was a late inclusion, but turned out to be one of my favourites. While it doesn’t cover all the useful Maven plugins (many such as the Enforcer and Assembly plugins are picked up elsewhere in the book), it highlights some that aren’t always well known or used:

• The Remote Resources plugin – and particularly the right pattern for including a package of reusable resources
• The Build Number plugin
• The Shade plugin – a very useful way to handle redistributing JARs and “uberJARs” and reducing dependency and classloading issues
• The Build Helper plugin – useful bits and pieces like additional artifacts to deploy and reserved network ports for tests
• The AntRun plugin (and other scripting languages) – how to still use Ant without the guilt of violating The Maven Way!
• The Exec plugin – fork an external process in your build, or configure it for running your standalone application easily from the POM

I do think if there was one plugin I would have liked to cover more in this centralised location it would have been the Enforcer – perhaps pulling some entries together would make a good blog topic in the future though.

I hope you enjoy the sample chapter and consider buying the book!

Book Released – Apache Maven 2: Effective Implementation

After being available in “RAW” (draft) form for the last few months, the final release of Apache Maven 2: Effective Implementation is now available online! It is available in both eBook and printed + eBook versions.

We had some specific goals in writing this that I think we’ve achieved:

• It is intended to build on top of knowledge from the free books that have gone before it with minimal duplication – though still enough information to stand alone.
• The book should be of most value to intermediate Maven users, but also useful to beginners. Everyone should learn something from it. It should update Maven 2.0 users on the latest available technology such as Maven 2.2, the newer Archetype creation from a project mechanism, and under-utilised plugins like the Enforcer or Shade plugins.
• We wanted to focus on “best practices” and tying everything together in a way that shows how Maven was meant to be used. Hopefully readers will experience the occasional “aha!” moment.
• The book works through the issues by a gradual example application, like building up (or applying Maven to) your own project. It intends to show how a reasonably complete project structure is best worked with, and the example application should be relatively interesting in its own right. It gets built from scratch, up to an assembly, building it in CI, deploying it to the repository, and releasing it.
• We wanted to give some coverage to Archiva and Continuum (projects that we’ve both been involved in for some time) to illustrate team concepts, but also convey the concepts in a way that translates to other equivalent tools.

You can see what was covered in the Table of Contents.

The book eventually weighed in at 450 pages – far more than we’d intended when we set out, though still with plenty of potential topics to cover. When we started this just over a year ago, my thoughts had initially been around simply covering the content from my series of Maven presentations and training content in book form, but soon found we could expand on many of the topics.

I had the good fortune to work with Deng Ching on the book (her announcement is on her blog), who poured a number of weekends and evenings into writing half of the content and reading (and re-reading) my writing.

We had some great help from our reviewers – Carsten Ziegler, Wendy Smoak and Emmanuel Venisse, as well as the encouragement of several others who wanted to help but couldn’t commit the time. Thank you all!