Tag Archives: npanday

Copying a Codeplex subversion repository with rsvndump

As part of the process of migrating NPanday to the Apache Incubator, I had to find a way to extract the Subversion repository from Codeplex. The challenge here is that it isn’t actually a Subversion repository, but a TFS server running SVNBridge to appear like one. Its occasional quirks and timeouts had been part of the reason we decided to move. Here is what I’ve learned.

Available Tools

The aim was to get it as a dump of a Subversion repository so that it could be loaded into the ASF Subversion repository, retaining the full history. I tried everything to get it out of there – svnsync, cloning it as a Git repository, cloning it as a Mercurial repository, and other similar tools. All would timeout or freak out at some point due to the nature of SVNBridge. I made some progress with tfs2svn (which seems to be what Codeplex is using to migrate repositories to Mercurial when needed), but I started to find that not only was it needing regular manual intervention, it wasn’t quite the same (eg. Subversion properties came in as ..svnbridge hidden folders, and every comment appended the original TFS revision number).

I had tried rsvndump earlier without a lot of success, but eventually gave it another try. While it needed some help, it was making the most progress and was the one that ended up being successful.

Patching rsvndump

It wasn’t anywhere near smooth sailing, and rsvndump needed some modifications to handle the task. So I brushed off my dusty C skills and made the required changes, which can be found in my github fork of the project (pull request pending).

The main problem was that rsvndump expected to be able to do the repository all in one hit. Given that Codeplex would timeout on several requests, this made it impossible. It did allow selecting a subset of revisions, but even then it would both do a full svn log (which wouldn’t succeed), and beyond that would traverse revisions to construct a path hash (I believe for detecting moves, etc.).

To assist with these, I added a --log-window-size option, similar to git-svn. This retrieves the logs in multiple requests, avoiding problems with timeouts. Next, I added a --first-rev argument, which would start retrieving logs and content from a later revision than 0. While it introduced some risk of crashing due to missing revisions, in many circumstances it allowed restarting a dump from a later revision in a much faster manner.

The next problem was that Codeplex shares a single TFS repository between several projects, so your own revision numbers are not sequential. NPanday started at revision 21102, and ended at 60509 with a lot of gaps, having only 1427 revisions of its own. This wasn’t too much of a problem – because rsvndump was designed to deal with subdirectories of a Subversion repository it expected the gaps. The --first-rev argument helped deal with the big gap to the start. But another SVNBridge quirk was that svn copy operations copied from the (current revision - 1) – even when it didn’t exist! To correct this, I had to adjust the code to search backwards through the revision numbers until it found one that existed to make the copy operations correct.

Finally, rsvndump added padding revisions into the dump file when a revision number was missing. This is helpful if you want to maintain the same numbers, but due to my use of --first-rev they were already out and I was importing to an existing repository, so I decided to strip these out. For that, I added another flag --omit-padding-revnums.

Running rsvndump

I ended up running a command like the following:

rsvndump --omit-padding-revs --adjust-missing-revnums \
  --first $FIRST --log-window-size 1000 -v --incremental \
  -r $REV1:$REV2 https://npanday.svn.codeplex.com/svn \
  3>&1 >&2 2>&3 3>&- >$REV1-$REV2.dump | tee $REV1-$REV2.log.txt

The first few arguments are the customisations described above (and --adjust-missing-revnums to make the dumped revisions sequential). The next are the traditional svnadmin dump arguments that rsvndump honours. Finally, I redirected the output so that I could channel stdout to the dumpfile and stderr to a log file that I could also tee.

Other Codeplex SVNBridge Issues

With these changes in place, I was getting moderately successful dumps – but a few frustrating issues remained.

Firstly, many svn copy operations (such as creating a tag) were tracked file by file by Codeplex instead of at the top level directory. This resulted in further timeouts that I couldn’t work around. We had seen this manifest on the Codeplex repository, being unable to even list the /tags/ directory. I didn’t attempt to correct this, instead manually applying the copy operation again after the preceding dump, then continuing.

For example:

svn cp \
  -m '[maven-release-plugin]  copy for tag npanday-project-1.2.1' \
  $REPO/trunk $REPO/releases/npanday-project-1.2.1

svn ps svn:author "SND\jocaba_cp" --revprop -rHEAD $REPO
svn ps svn:date \
  2010-09-07T07:15:46.723000Z \
  --revprop -rHEAD $REPO

If that revision appeared in the dump file (either incomplete or not able to be applied), I’d delete it by searching for Revision-number: xyzxyz and deleting the lines up until the next revision.

Between tags and a few other stubborn revisions that wouldn’t come across (including one where even svn log wouldn’t succeed), I manually reconstructed 100 revisions like that. The upside was that it provided an opportunity to clean out some botched releases (due to the SVNBridge /tags/ issues) and branches that had never been used.

So the process was to dump as many revisions as possible, then apply to a test repository, check it out, make required modifications, and repeat. I captured all of this in a shell script so that at any time I could recreate the work repository and reapply all of the dumps and modifications to date. This because useful a few times as I gradually identified inconsistencies with a checkout from the same revision in Codeplex from having missed something.

This still uses a lot of bandwidth – starting at a given revision will both reconstruct the path hash for the whole repository at that revision, and fetch the “base revision”, which is a checkout of an entire revision, tags and all. So the process took a few days running intermittently. I also had to start the --first-rev at least one revision earlier and sometimes more, to avoid getting a cryptic Subversion error message about the “editor drive”.

Properties were also quite quirky on SVNBridge, due to the way they are apparently stored in TFS as described earlier. Some could not be removed (eg, bogus svn:mime-type), and some were set oddly (svn:ignore on a file, svn:eol-style on a directory). I chose to leave these alone and correct them after the import.

Some properties went missing, which was part of a larger problem on SVNBridge with copying from an existing revision. If you attempt to copy in the working copy and then make a modification before committing, this doesn’t show up as A+ in the svn log result later, but simply M. The dumps know it was copied, but not that it was added, so attempt to modify a non-existant file when being applied. What’s more, this step wipes out some properties that are set on directories.

In some cases, I manually applied the revision, in others I made an edit in the dump file from:

Node-action: change

to

Node-action: add

Deleting directories hit snags as well. I’m unsure if this was a problem in SVNBridge or rsvndump, but it would dump deletions for every path and file like so:

/tags/npanday-1.2-RC1
/tags/npanday-1.2-RC1/pom.xml
/tags/npanday-1.2-RC1/dotnet
...

When applying the dump, it would successfully delete the first then fail on the others that were already deleted in the first step. I ended up removing the nodes for all the later entries manually in these instances. You would take these out 4 lines at a time (including trailing whitespace of 2 lines):

Node-path: tags/npanday-1.2-RC1/pom.xml
Node-action: delete

Final manipulations

Aside from Codeplex, for NPanday we needed to make some more manipulations. First, changing the usernames to line up to their final accounts on the ASF using repeated changes to the svn:author revision properties.

The dump was also loaded onto another partial subversion repository that contained some intermediate history between leaving the incubator originally and arriving at Codeplex.

Loading to an existing Subversion repository and path

After all this was eventually done, and there was a repository that was matched with the history of the Codeplex one, it needed to be dumped to load into the ASF repository.

Normally, this would be a simple:

svnadmin dump --incremental --deltas work-repo >npanday.dump

However, the objective was to load this onto a path that already existed. This was because we sought to have continuity with the history from the point at which the project was forked from the incubator originally.

To achieve this, I identified the revision in the dump that matched the content in the ASF repository, which due to the initial creation of branches and tags, was revision 4. I then dumped it using:

svnadmin dump --incremental -r5:HEAD work-repo >npanday.dump

Originally, --deltas had been included to reduce the size, but we found that this caused checksum problems, possibly due to different line endings between r4 and the original in the ASF repository.

At last, this yielded a dump that could be loaded into the ASF repository, and the results can be seen here: http://svn.apache.org/viewvc/incubator/npanday. You can now see the historical continuity in files such as http://svn.apache.org/viewvc/incubator/npanday/trunk/pom.xml.

Conclusion

This took considerably more work than anticipated when we originally thought it would be a good idea to retain the history.

I found that there wasn’t a lot of information about these topics on the web, so I hope this post will help to expand that for those that might face this challenge in the future. I’ve also found that editing Subversion dump files (when not in delta-mode) is reasonably straightforward.

Interestingly I’ve learned that Subversion 1.7 will include the ability to do a remote svnadmin dump, however I don’t believe this will work when svnsync is not supported (as was the case here), or support sub-paths as rsvndump does.

Advertisements

NPanday 1.2.1 Released

In the midst of the move to Apache that I’ve just written about, Joe has also released NPanday 1.2.1 from Codeplex to wrap up the outstanding issues that were in SVN before we started the migration process.

The main improvements of the release were to dramatically improve the performance of SNAPSHOT resolution, and to support building with Maven 3. In NPanday 1.2, we started to bring the resolution back to the Maven standard APIs to improve compatibility. This release further streamlines the process, and allows you to take advantage of additional performance improvements of the Maven 3 betas.

You can download NPanday 1.2.1 from the Codeplex project, or from the NPanday Maven repository. You can find more information in the installation instructions.

Thanks to those that submitted issue reports and patches for the release!

The full list of changes is reproduced below:

  • 13590 Update/Create all three “types” of assembly version during compile when it is not there (AssemblyInfo.cs) initially
  • 13758 Resync references may cause timeouts after successive snapshot metadata retrievals
  • 13948 Performance Issue when Building projects with many snapshot dependencies
  • 13984 NPanday not building resource files correctly
  • 11651 Remove for all but the compile and aspx plugins
  • 12946 Create Code Coverage Plugin for NPanday
  • 13179 Document Workaround: GAC install fails with access denied when UAC is turned on
  • 13198 compile-plugin has two deploy phases for ArtifactType.NETPLUGIN
  • 13394 Include the steps on how to execute the ITs correctly from NPanday trunk
  • 13395 Release process notes is outdated
  • 13461 npe building gac artifact
  • 13462 add missing dependencies to dotnet poms
  • 13477 support jar style custom manifest entries in assemblies
  • 13575 Update build to use the NPanday 1.2 release
  • 13615 Configure Emma plugin
  • 13616 Configure Code Coverage plugin for .Net
  • 13635 Snapshot updates are not recognised due to the copies in UAC
  • 13652 image missing in Addin docs
  • 13655 Fix APT syntax errors on the main site
  • 13669 Improve the plugin sites
  • 13846 POM files incorrectly updated when including Web References in VS
  • 14049 Error message contains unnecessary text if project not supported.
  • 14093 Resync Reference doesn’t update SNAPSHOT artifact from local repository that already exist in .references folder
  • 14102 duplicate web reference entry in POM when including web reference

NPanday has moved to the Apache Incubator

Just short of two years ago, NPanday was established at Codeplex. It picked up where one of the branches of the NMaven podling had left off, and has since grown into the most stable, popular and comprehensive solution for Maven users looking to build .NET projects.

I wrote more about what NPanday does in the previous post on its 1.2 release, and there is plenty of information in the incubation proposal if this is an area you are interested in.

There were several reasons in my mind for returning to Apache.

The most pressing reason was repeated problems with infrastructure at Codeplex, and in particular the SVN interface they provide over the top of TFS. Failures with creating tags, altering properties, and general timeouts and weird errors were holding up releases and development (extracting the history from this was also an “interesting” exercise, and the subject of a future post).

Apache provides not only the infrastructure but the governance structure to best support and “future proof” a growing community. We had recently added two committers in Lars and Craig, and have been seeing regular patches from other contributors. The Codeplex stats showed that we had thousands of downloads and visitors, not including traffic directly to the Maven repository. This growing community has given me confidence that the project will move quickly towards a successful graduation.

Moving isn’t a quick process (now at two months since the original proposal), and thanks go to our users for bearing with us. We’re still in the process of moving infrastructure, with the following set up:

We’re still working on migrating the web site and issue tracker.

Back to the code, there’s plenty of work to do, and discussions on the next features and changes have started on the new list. If you’re interested in a Maven-like solution for building .NET projects, or working in a hybrid Java-.NET environment, we’d like to hear from you! We’re particularly interested in working with folks that can help improve the C# portions of the code, including the Visual Studio Add-in.

If you’re one of the many users out there, what do you think of the move?

NPanday 1.2 Released – .NET integration for Apache Maven

Over the last month, the NPanday developers have been doing a considerable amount of work to ship the 1.2 release, which is now available at Codeplex. There are over 50 improvements included, with the main items of note being:

  • Improved type names to better distinguish .NET artifacts
  • An MSI-based installer for the Visual Studio Addin and Maven plugins
  • MVC project support
  • Rewrite of documentation

NPanday primarily provides two capabilities: a set of Maven plugins, for constructing builds in Maven that use the .NET command-line tools; and a Visual Studio Addin that keeps a Visual Studio project in sync with the Maven POM and adds reference resolution from Maven artifact repositories. Together this allows you to use a single tool across .NET, Java or any other Maven-based projects, including the same benefits of dependency management, automated release and source control management, and so on.

NPanday has emerged as the defacto solution for those looking to build .NET projects using Maven, and it has been great to see the community growing and active over the last year, with many more users participating and submitting patches. Earlier in the year we added Lars Corneliussen as our newest committer, and he has been leading the charge to NPanday 2.0 and cleaning up some of the historical cruft in the codebase. We’re looking forward to a lot more activity in that area in the near future that will make NPanday even easier to work with in the future.

If you’re building .NET projects and are familiar with Maven, or looking for Maven-like features, I hope you’ll to check it out and let us know what you think!