cvs2svn-migration/trunk: . README builder invoke_cvs2svn macros migrator

[saq]

Author: saq
Date: Tue Sep 13 18:48:27 2005
New Revision: 6373

Added:
   cvs2svn-migration/trunk/
   cvs2svn-migration/trunk/README
   cvs2svn-migration/trunk/builder   (contents, props changed)
   cvs2svn-migration/trunk/invoke_cvs2svn   (contents, props changed)
   cvs2svn-migration/trunk/macros
   cvs2svn-migration/trunk/migrator   (contents, props changed)
Log:
- initial checkin

Added: cvs2svn-migration/trunk/README
==============================================================================
--- (empty file)
+++ cvs2svn-migration/trunk/README	Tue Sep 13 18:48:27 2005
@@ -0,0 +1,70 @@
+===
+
+Steps to migrate the PLD repo cvs->svn
+
+1. Disable write access to the CVS repo
+2. Use migrator to reorganize the repo according to the dir-per-package scheme
+3. Perform some manual clean-ups.
+4. Use cvs2svn to generate svn entries and revisions.
+5. Perform some manual clean-ups.
+6. Celebrate.
+
+After step 3 you can try working with CVS on the new structure (read-only),
+you will have to `mkdir CVSROOT` for that.
+
+===
+
+migrator quick docs
+
+What it does: creates a copy of the existing PLD repo (or its rsync copy),
+reorganizing the files on the fly. The result is a flat structure: each
+package has a directory of its own, S{PEC,OURCE}S are no more. Don't worry
+about disk space, it's a cheap (hardlink-based) copy. Do place the source
+and target on the same filesystem.
+
+For configuration, see the sources. For testing purposes you can limit
+yourself to a subset of the specs; again, see the sources.
+
+A source is considered to belong to a given package if it's a Source or a
+Patch of at least one revision of a given foo.spec,v (malformed revisions
+are discarded). rpm parsing of each single release is probably the most
+costly operation. Converting a*.spec took 30 minutes.
+
+The above rule means that if a source belongs to various specs, it will
+get duplicated in the resulting repo. Sources which don't belong anywhere
+go to a common directory (that is, no file gets omitted, except for files
+that weren't valid CVS-controlled files in the first place).
+
+===
+
+migrator open issues:
+
+- There are files in SPECS which aren't valid specs (builder,v for
+  example). Find a cosy place for them.
+- There are files in SOURCES which apparently don't belong to any package.
+  Find a good place for them (bitbucket is an option).
+
+===
+
+Other open issues:
+
+- How to write %changelog under svn? What to do with the existing entries?
+- What tasks are necessary in steps 3 and 5 above?
+- What needs to be done to adapt src builder?
+- What about the filter_tags.sh rules?
+- What directory structure for tags and branches? cvs2svn gives:
+trunk/MathReader/MathReader.spec
+tags/auto-ac-MathPlanner-3_1_3-1/MathPlanner/MathPlanner.spec
+branches/RA-branch/MathPlanner/MathPlanner.spec
+
+===
+
+Changes to the development environment:
+
+- topdir/S{PEC,OURCE}S are no more. topdir/{BUILD,{,S}RPMS} will stay there
+- topdir/foo looks like the right checkout spot for foo.spec and its sources
+- topdir/foo/foo-0.00.tar for distfiles
+- need new macros to find the sources in the new place
+- builder will never be the same again
+
+===

Added: cvs2svn-migration/trunk/builder
==============================================================================
--- (empty file)
+++ cvs2svn-migration/trunk/builder	Tue Sep 13 18:48:27 2005
@@ -0,0 +1,43 @@
+#!/bin/sh
+
+# Note: it's only useful for the conversion proof-of-concept.
+
+# Hint: -r branches/AC-branch  or  -r tags/auto-ac-bofh-0_2-4
+
+SVNROOT="file:///home/users/saq/tmp/svn"
+TAGDIR="trunk"
+PACKAGE=""
+
+BUILD="yes"
+
+while test $# -gt 0; do
+	case "$1" in
+		-g )
+			BUILD="no";shift;;
+		-r )
+			TAGDIR="$2";shift;shift;;
+		* )
+			PACKAGE="$1";shift;;
+	esac
+done
+if [ "$PACKAGE" = "" ]; then
+	echo "Usage: builder [-g] [-r foo] package"
+	exit 1
+fi
+cd $(rpmbuild --eval %_topdir 2>/dev/null)
+svn co $SVNROOT/$TAGDIR/$PACKAGE
+cd $PACKAGE
+dump="$(rpmbuild --nodigest --nosignature --nodeps --define 'prep %dump' $PACKAGE.spec 2>&1)"
+source0url=$(echo "$dump"|awk '/SOURCEURL0/ {print $3}')
+source0=$(echo "$dump"|awk '/SOURCE0/ {print $3}')
+#echo S $source0url $source0
+have_md5=$(md5sum $source0|awk '{print $1}')
+need_md5=$(cat $PACKAGE.spec|awk '/ Source0-md5/{print $3}')
+#echo M $have_md5 $need_md5
+if [ "$have_md5" != "$need_md5" ]; then
+	rm -f $source0
+	wget http://distfiles.pld-linux.org/by-md5/$(echo $need_md5|sed -e 's|^\(.\)\(.\)|\1/\2/&|')/$(basename $source0)
+fi
+if [ "$BUILD" = "yes" ]; then
+	rpmbuild -bb $PACKAGE.spec
+fi

Added: cvs2svn-migration/trunk/invoke_cvs2svn
==============================================================================
--- (empty file)
+++ cvs2svn-migration/trunk/invoke_cvs2svn	Tue Sep 13 18:48:27 2005
@@ -0,0 +1,14 @@
+#!/bin/sh
+
+OPTS="--encoding=ISO-8859-2"
+
+# Checking for tag/branch mismatches...
+# ERROR: The following symbols are tags in some files and branches in others.
+# yees, the beauty of PLD
+OPTS="$OPTS --force-branch=DEVEL --force-branch=RA-branch --force-branch=AC-branch"
+OPTS="$OPTS --force-branch=RA-branch_general"
+OPTS="$OPTS --force-branch=LINUX_2_6 --force-branch=LINUX_2_4 --force-branch=LINUX_2_2_DEVEL"
+OPTS="$OPTS --force-branch=GNOME_2_0 --force-branch=rpm3"
+# you'll find many others
+
+exec cvs2svn $OPTS -s /home/users/saq/tmp/svn /home/users/saq/tmp/repo

Added: cvs2svn-migration/trunk/macros
==============================================================================
--- (empty file)
+++ cvs2svn-migration/trunk/macros	Tue Sep 13 18:48:27 2005
@@ -0,0 +1,7 @@
+# The macros that need to be modified. Not a very long list.
+
+# A cool feature: rpmbuild --rebuild foo...src.rpm mkdirs those if they
+# don't exist
+
+%_sourcedir	%{_topdir}/%{name}
+%_specdir	%{_topdir}/%{name}

Added: cvs2svn-migration/trunk/migrator
==============================================================================
--- (empty file)
+++ cvs2svn-migration/trunk/migrator	Tue Sep 13 18:48:27 2005
@@ -0,0 +1,164 @@
+#!/usr/bin/python
+
+# requires: cvs2svn, rcs, rpm-build
+
+# source
+cvsdir="/home/users/saq/PLD/repo"
+# target
+tmpcvsdir="/home/users/saq/tmp/repo"
+# rpm can't read a spec from stdin
+tmpspecfile="/home/users/saq/tmp/foo.spec"
+# reasons of failures
+logdir="/home/users/saq/tmp/migrator-log"
+
+# only those SPECS/ files will be copied
+spec_re="^c.*"
+#spec_re=".*"
+# only those SOURCES/ files will be considered for SOURCES.old
+# (files for specs are sought among all sources)
+source_re="^c.*"
+#source_re=".*"
+
+# internals follow
+
+import cvs2svn_rcsparse
+import os
+import re
+from subprocess import Popen,PIPE
+
+spec_re=re.compile(spec_re)
+source_re=re.compile(source_re)
+
+def cvspackagedir(package):
+	dir=tmpcvsdir+"/"+package
+	return dir
+
+specsdir=cvsdir+"/SPECS"
+sourcesdir=cvsdir+"/SOURCES"
+specsolddir=tmpcvsdir+"/SPECS.old"
+sourcesolddir=tmpcvsdir+"/SOURCES.old"
+
+# utilities
+
+class Sink(cvs2svn_rcsparse.Sink):
+	def __init__(self):
+		self.revs=[]
+	def define_revision(self,rev,*args):
+		self.revs.append(rev)
+
+def get_revisions(f):
+	"""Get all revision numbers from a ,v file"""
+	sink=Sink()
+	cvs2svn_rcsparse.parse(f,sink)
+	return sink.revs
+
+def mkdir(dir):
+	# might exist already
+	try:
+		os.makedirs(dir)
+	except OSError:
+		pass
+
+def ln(oldpath,newdir):
+	lastdir,file=oldpath.split("/")[-2:]
+	if lastdir=="Attic": newdir+="/Attic"
+	newpath="%s/%s"%(newdir,file)
+	# the noble exception of nps.spec
+	try:
+		link=os.readlink(oldpath)
+		oldpath+="/"+link
+	except OSError:
+		# not a symlink
+		pass
+	os.link(oldpath,newpath)
+
+def listdir(dir):
+	"""For a given CVS module directory, get a mapping:
+	version controlled file->path to its ,v"""
+	# ERROR: A CVS repository cannot contain both /tmp/cvs/SPECS/Mis.spec,v and /tmp/cvs/SPECS/Attic/Mis.spec,v
+	# PLD has always been exceptional :)
+	# Having no better idea, discard the Attic version. As well as all non-,v files.
+	livefiles=os.listdir(dir)
+	atticfiles=os.listdir(dir+"/Attic")
+	files={}
+	for f in atticfiles:
+		if not f.endswith(",v"): continue
+		files[f[:-2]]="%s/Attic/%s"%(dir,f)
+	for f in livefiles:
+		if not f.endswith(",v"): continue
+		files[f[:-2]]="%s/%s"%(dir,f)
+	return files
+
+def spec2sources(f):
+	"""
+	Get a sequence of the sources that belong to the given spec.
+	This means checking each revision to follow the sources' list changes.
+	"""
+	assert f.endswith(",v")
+	all_sources=set()
+	for rev in get_revisions(file(f)):
+		# This part would have been easier in another language
+		p1=Popen(["co","-p","-r"+rev,f[:-2]],stdout=PIPE,stderr=file("/dev/null","w"))
+		# post-processing to maximize the chance of a successful parse
+		p2=Popen(["sed",r"/^Icon:/d;/^Serial:/d;/^%define.*_kernel_ver/d;s/^Copyright:/License:/"],
+			stdin=p1.stdout,stdout=file(tmpspecfile,"w")
+		)
+		if p1.wait()!=0 or p2.wait()!=0:
+			print "WARN: %s@%s: checkout failed"%(f,rev)
+			continue
+
+		cmd1=["rpmbuild","--nodigest","--nosignature","--nodeps",
+			"--define","_kernel_ver_str 2.0.28",
+			"--define","prep %dump",tmpspecfile]
+		p1=Popen(cmd1,stderr=PIPE)
+		p2=Popen(["awk",'/SOURCE[0-9]+|PATCH[0-9]+/ {sub(".*/","",$3);print $3}'],stdin=p1.stderr,stdout=PIPE)
+		# strip trailing \n
+		sources=[line[:-1] for line in p2.stdout]
+		if p1.wait()!=0 or p2.wait()!=0:
+			what="%s@%s"%(f.split('/')[-1],rev)
+			print "WARN: %s: unable to parse, see log for details"%(what,)
+			Popen(cmd1,stderr=file("%s/%s.log"%(logdir,what),"w")).wait()
+			continue
+		all_sources.update(sources)
+	return all_sources
+
+# let's go
+
+specs_all=listdir(specsdir)
+specs_used=set()
+sources_all=listdir(sourcesdir)
+sources_used=set()
+
+print "0. Cleanup"
+os.system("rm -rf %s"%(tmpcvsdir,))
+os.system("rm -rf %s"%(logdir,))
+mkdir(logdir)
+
+print "1. Reorganizing CVS repo"
+for spec,path in specs_all.iteritems():
+	if spec_re.match(spec) is None: continue
+	if not spec.endswith(".spec"):
+		print "WARN: what the hell is SPECS/%s?"%(spec,)
+		continue
+	package=spec[:-5]
+	dir=cvspackagedir(package)
+	mkdir(dir+"/Attic")
+	ln(path,dir)
+	sources=spec2sources(path)
+	for source in sources:
+		if not source in sources_all: continue
+		ln(sources_all[source],dir)
+	sources_used.update(sources)
+	specs_used.add(spec)
+
+print "2. Keeping remaining SPECS and SOURCES"
+mkdir(specsolddir+"/Attic")
+mkdir(sourcesolddir+"/Attic")
+for spec,path in specs_all.iteritems():
+	if spec in specs_used: continue
+	if spec_re.match(spec) is None: continue
+	ln(path,specsolddir)
+for source,path in sources_all.iteritems():
+	if source in sources_used: continue
+	if source_re.match(source) is None: continue
+	ln(path,sourcesolddir)