Difference between revisions of "Import from SVN"

From Gcube Wiki
Jump to: navigation, search
(→‎Fix authors info)
 
(37 intermediate revisions by 5 users not shown)
Line 2: Line 2:
 
This guide explains how to migrate a project from SVN to Git.
 
This guide explains how to migrate a project from SVN to Git.
  
−
= Download migration utilities =
+
= Configure your system to allow Migration Facilities to properly work =
 +
 
 +
== Download migration utilities ==
 
For an easy reference in this guide, the migration scripts is downloaded in the home folder:
 
For an easy reference in this guide, the migration scripts is downloaded in the home folder:
 
<source lang="bash">
 
<source lang="bash">
−
  cd $HOME &&  wget https://bitbucket.org/atlassian/svn-migration-scripts/downloads/svn-migration-scripts.jar  
+
  cd $HOME &&  wget https://code-repo.d4science.org/gCubeSystem/Configs/raw/branch/master/SVN/svn-migration-scripts.jar  
 
</source>
 
</source>
  
−
= Install git-svn =
+
== Install git-svn (Ubuntu) ==
−
== Ubuntu==
+
 
−
On Ubuntu, if you you get this output:
+
On Ubuntu, if you get this output:
  
 
<source lang="bash">
 
<source lang="bash">
Line 38: Line 40:
 
Subversion: using version 1.9.7
 
Subversion: using version 1.9.7
 
git-svn: using version 2.17.1
 
git-svn: using version 2.17.1
−
</source>  
+
</source>
  
−
= Mount a case-sensitive disk image (for Mac OS) =
+
== Mount a case-sensitive disk image (for Mac OS) ==
 
Check if this step is needed by running:
 
Check if this step is needed by running:
 
<source lang="bash">
 
<source lang="bash">
Line 63: Line 65:
 
</source>
 
</source>
  
−
= Extract the author(s) information =
+
= Extract the author(s) information=
 +
 
 +
SVN uses the username to associate the commit, instead, Git uses the email. The author file is required to properly associate the history of commits to the right person.
 +
 
 +
It is possible to get the author list for the current repository only or use the global author list (extracted for your convenience).
 +
 
 +
== Get authors info from the desired repository ==
  
 
<source lang="bash">
 
<source lang="bash">
 
  cd ~/GitMigration
 
  cd ~/GitMigration
−
  java -jar ~/svn-migration-scripts.jar authors https://svn.d4science.research-infrastructures.eu/gcube/trunk/Common/gxREST  > authors.txt
+
  java -jar ~/svn-migration-scripts.jar authors http://svn.research-infrastructures.eu/public/d4science/gcube/trunk/Common/gxREST  > authors.txt
 
About to create the authors file.
 
About to create the authors file.
 
</source>
 
</source>
Line 79: Line 87:
 
</source>
 
</source>
  
−
== Edit the authors file ==
+
=== Edit the authors file ===
 
You need to edit the file and add the correct email address (i.e. the address configured in the Git service) for each listed author:
 
You need to edit the file and add the correct email address (i.e. the address configured in the Git service) for each listed author:
 
<source lang="bash">
 
<source lang="bash">
−
> cat authors.txt
+
$ cat authors.txt
 
luca.frosini = luca.frosini <luca.frosini@mycompany.com>
 
luca.frosini = luca.frosini <luca.frosini@mycompany.com>
 
lucio.lelii = lucio.lelii <lucio.lelii@mycompany.com>
 
lucio.lelii = lucio.lelii <lucio.lelii@mycompany.com>
 
manuele.simi = manuele.simi <manuele.simi@mycompany.com
 
manuele.simi = manuele.simi <manuele.simi@mycompany.com
−
> vi authors.txt
+
$ vi authors.txt
  
−
> cat authors.txt
+
$ cat authors.txt
−
luca.frosini = luca.frosini <luca.frosini@isti.cnr.it>
+
luca.frosini = Luca Frosini <luca.frosini@isti.cnr.it>
−
lucio.lelii = lucio.lelii <lucio.lelii@isti.cnr.it>
+
lucio.lelii = Lucio Lelii <lucio.lelii@isti.cnr.it>
−
manuele.simi = manuele.simi <manuele.simi@isti.cnr.it>
+
manuele.simi = Manuele Simi <manuele.simi@isti.cnr.it>
 
</source>
 
</source>
  
−
= Clone the SVN repository =
+
You can also do that with just one bash command
  
 
<source lang="bash">
 
<source lang="bash">
−
git svn clone --authors-file=authors.txt --follow-parent https://svn.d4science.research-infrastructures.eu/gcube/trunk/Common/gxREST --username manuele.simi gxRest
+
$ sed -i s/@mycompany.com/@isti.cnr.it/g authors.txt
 +
</source>
 +
 
 +
== Use the global authors list ==
 +
 
 +
You can get the global authors mapping file as following
 +
 
 +
<source lang="bash">
 +
$ wget -O authors.txt https://code-repo.d4science.org/gCubeSystem/Configs/raw/branch/master/SVN/all-svn-authors.txt
 +
</source>
 +
 
 +
 
 +
'''BEFORE USE THE FILE PLEASE DOUBLE CHECK IF YOUR INFORMATION ARE CORRECT. IF YOU NOTICE SOME ERRORS FOR ANY AUTHOR PLEASE CORRECT THEM ON THE SOURCE FILE'''
 +
 
 +
= SVN Repository Layouts=
 +
Depending on the structure of your SVN repo, the git-svn (see below) command needs to be configured differently. There are two possible layouts.
 +
 
 +
== Standard Layout ==
 +
The SVN project uses the standard /trunk, /branches, and /tags directory layout. This is the recommended way to organize a repository:
 +
* a ''trunk'' directory to hold the “main line” of development,
 +
* a ''branches'' directory to contain branch copies,
 +
* and a ''tags'' directory to contain tag copies.
 +
 
 +
In the standard layout, these are top-level directories.
 +
 
 +
[[File:SNVStardardLayout.png|400px]]
 +
 
 +
== Non-Standard Layout ==
 +
The SVN project uses a custom layout.
 +
 
 +
[[File:SVNNonStandardLayout.png|400px]]
 +
 
 +
= Clone the SVN Repository =
 +
The git svn clone command transforms the trunk, branches, and tags in your SVN repository into a new Git repository. Depending on the structure of the SVN repo, the command needs to be configured differently.
 +
 
 +
<source lang="bash">
 +
> git svn clone --authors-file=authors.txt --follow-parent   http://svn.research-infrastructures.eu/public/d4science/gcube/trunk/Common/gxREST \
 +
  --username manuele.simi gxRest
  
 
Note: --follow-parent makes it slower, but it’s needed if the SVN folder has been moved around.
 
Note: --follow-parent makes it slower, but it’s needed if the SVN folder has been moved around.
Line 134: Line 179:
 
creating empty directory: gxJRS/src/main/resources
 
creating empty directory: gxJRS/src/main/resources
 
creating empty directory: gxJRS/src/test/java/org/gcube/common/gxrest/response
 
creating empty directory: gxJRS/src/test/java/org/gcube/common/gxrest/response
 +
</source>
 +
 +
Do note that the command above DO NOT automatically import the SVN branches because the repo does not have the standard SVN [[Import_from_SVN#SVN_Repository_Layouts|layout]] .
 +
 +
If the SVN repository doesn’t have a standard layout and you want to import everything, you need to provide the locations of your trunk, branches, and tags using the --trunk, --branches, and --tags command line options. See [https://git-scm.com/docs/git-svn git-svn] for further options.
 +
 +
= Change dir in the cloned folder =
 +
The last parameter of the "git svn clone" command is the name of the folder ("gxRest" in the previous example) where the SVN repository is cloned and converted to Git. All the git commands in the next sections are executed in the cloned folder.
 +
So you need to:
 +
<source lang="bash">
 +
> cd <cloned repo>
 
</source>
 
</source>
  
 
= Rename “trunk” branch to “master” (if needed) =  
 
= Rename “trunk” branch to “master” (if needed) =  
 
<source lang="bash">
 
<source lang="bash">
−
>git branch
+
> git branch
 
* trunk
 
* trunk
−
>git branch -m trunk master
+
> git branch -m trunk master
 
> git branch
 
> git branch
 
* master
 
* master
Line 190: Line 246:
 
</source>
 
</source>
  
−
= Clone the repository and create a working copy =
+
= Change the SCM Connection =
 +
The SCM section of the POM must be changed to reflect that we are now working with a Git repository.
  
−
<source lang="bash">
+
For instance, the following section:
−
> cd ~/Work/CNR/Projects/Git/
+
<pre>
−
> git clone https://code-repo.d4science.org/manuele.simi/gxRest.git
+
  <scm>
−
Cloning into 'gxRest'...
+
<connection>scm:svn:http://svn.d4science.research-infrastructures.eu/gcube/trunk/distributions/${project.artifactId}</connection>
−
remote: Counting objects: 168, done.
+
<developerConnection>scm:svn:https://svn.d4science.research-infrastructures.eu/gcube/trunk/distributions/${project.artifactId}</developerConnection>
−
remote: Compressing objects: 100% (81/81), done.
+
<url>http://svn.d4science.research-infrastructures.eu/gcube/trunk/distributions/${project.artifactId}</url>
−
remote: Total 168 (delta 44), reused 168 (delta 44)
+
  </scm>
−
Receiving objects: 100% (168/168), 39.47 KiB | 0 bytes/s, done.
+
</pre>
−
Resolving deltas: 100% (44/44), done.
+
−
</source>
+
  
−
= Configure authors info =
+
must be changed to:
  
−
== Single repo configuration ==
+
<pre>
−
In the root folder of the repository:
+
<scm>
−
<source lang="bash">
+
    <connection>scm:git:https://code-repo.d4science.org/gCubeSystem/${project.artifactId}.git</connection>
−
> git config user.email "manuele.simi@isti.cnr.it"
+
    <developerConnection>scm:git:https://code-repo.d4science.org/gCubeSystem/${project.artifactId}.git</developerConnection>
−
> git config user.name "Manuele Simi"
+
    <url>https://code-repo.d4science.org/gCubeSystem/${project.artifactId}</url>
−
</source>
+
</scm>
 +
</pre>
  
−
== Global configuration (all repos on this computer) ==
+
'''NB'''. In this case, the ${project.artifactId} declared in the pom is equal to the repository name declared on Git.
−
In any folder:  
+
 
−
<source lang="bash">
+
Otherwise, you must declare the Git repository URL like is it:
−
> git config --global user.email "manuele.simi@isti.cnr.it"
+
 
−
> git config --global user.name "Manuele Simi"
+
<pre>
−
</source>
+
<scm>
 +
    <connection>scm:git:[YOUR_GIT_REPOSITORY_URL].git</connection>
 +
    <developerConnection>scm:git:[YOUR_GIT_REPOSITORY_URL].git</developerConnection>
 +
    <url>[YOUR_GIT_REPOSITORY_URL]</url>
 +
</scm>
 +
</pre>
 +
 
 +
 
 +
''Back to the [[Continuous_Integration_procedure_(2019) | CI guide]].''
  
−
Back to the [[Continuous_Integration_procedure_(2019) | CI guide]].
+
[[Category:Continuous_Integration]]

Latest revision as of 21:31, 16 January 2020

This guide explains how to migrate a project from SVN to Git.

Configure your system to allow Migration Facilities to properly work

Download migration utilities

For an easy reference in this guide, the migration scripts is downloaded in the home folder: 

 cd $HOME &&  wget https://code-repo.d4science.org/gCubeSystem/Configs/raw/branch/master/SVN/svn-migration-scripts.jar

Install git-svn (Ubuntu)

On Ubuntu, if you get this output: 

$ java -jar ./svn-migration-scripts.jar verify  
svn-migration-scripts: using version 0.1.56bbc7f 
Git: using version 2.17.1  
Subversion: using version 1.9.7
git: 'svn' is not a git command. See 'git --help'.
The most similar commands are
fsck
mv
show
git-svn: ERROR: Unable to determine version.

You must install the git-svn package by running: 

$ sudo apt install git-svn

Now, you will get something like this: 

$ java -jar ./svn-migration-scripts.jar verify
svn-migration-scripts: using version 0.1.56bbc7f
Git: using version 2.17.1
Subversion: using version 1.9.7
git-svn: using version 2.17.1

Mount a case-sensitive disk image (for Mac OS)

Check if this step is needed by running: 

java -jar ~/svn-migration-scripts.jar verify
 
svn-migration-scripts: using version 0.1.56bbc7f
Git: using version 2.11.0
Subversion: using version 1.9.4
git-svn: using version 2.11.0
You appear to be running on a case-insensitive file-system. This is unsupported, and can result in data loss.

Following the warning, we create a disk image dedicated to the migration activities: 

java -jar ~/svn-migration-scripts.jar create-disk-image 5 GitMigration
 
created: /Users/manuelesimi/GitMigration.sparseimage
/dev/disk2     		 GUID_partition_scheme     		 
/dev/disk2s1   		 EFI                       		 
/dev/disk2s2   		 Apple_HFS                 		 /Users/manuelesimi/GitMigration
The disk image was created successfully and mounted as: /Users/manuelesimi/GitMigration

Extract the author(s) information

SVN uses the username to associate the commit, instead, Git uses the email. The author file is required to properly associate the history of commits to the right person.

It is possible to get the author list for the current repository only or use the global author list (extracted for your convenience).

Get authors info from the desired repository

 cd ~/GitMigration
 java -jar ~/svn-migration-scripts.jar authors http://svn.research-infrastructures.eu/public/d4science/gcube/trunk/Common/gxREST  > authors.txt
About to create the authors file.


Alternative (pure SVN): 

svn co https://svn.d4science.research-infrastructures.eu/gcube/trunk/Common/gxREST
cd gxREST
svn log -q | awk -F '|' '/^r/ {sub("^ ", "", $2); sub(" $", "", $2); print $2" = "$2" <"$2">"}' | sort -u > authors.txt

Edit the authors file

You need to edit the file and add the correct email address (i.e. the address configured in the Git service) for each listed author: 

$ cat authors.txt
luca.frosini = luca.frosini <luca.frosini@mycompany.com>
lucio.lelii = lucio.lelii <lucio.lelii@mycompany.com>
manuele.simi = manuele.simi <manuele.simi@mycompany.com
$ vi authors.txt
 
$ cat authors.txt
luca.frosini = Luca Frosini <luca.frosini@isti.cnr.it>
lucio.lelii = Lucio Lelii <lucio.lelii@isti.cnr.it>
manuele.simi = Manuele Simi <manuele.simi@isti.cnr.it>

You can also do that with just one bash command 

$ sed -i s/@mycompany.com/@isti.cnr.it/g authors.txt

Use the global authors list

You can get the global authors mapping file as following 

$ wget -O authors.txt https://code-repo.d4science.org/gCubeSystem/Configs/raw/branch/master/SVN/all-svn-authors.txt


BEFORE USE THE FILE PLEASE DOUBLE CHECK IF YOUR INFORMATION ARE CORRECT. IF YOU NOTICE SOME ERRORS FOR ANY AUTHOR PLEASE CORRECT THEM ON THE SOURCE FILE

SVN Repository Layouts

Depending on the structure of your SVN repo, the git-svn (see below) command needs to be configured differently. There are two possible layouts.

Standard Layout

The SVN project uses the standard /trunk, /branches, and /tags directory layout. This is the recommended way to organize a repository:

  • a trunk directory to hold the “main line” of development,
  • a branches directory to contain branch copies,
  • and a tags directory to contain tag copies.

In the standard layout, these are top-level directories.

SNVStardardLayout.png

Non-Standard Layout

The SVN project uses a custom layout.

SVNNonStandardLayout.png

Clone the SVN Repository

The git svn clone command transforms the trunk, branches, and tags in your SVN repository into a new Git repository. Depending on the structure of the SVN repo, the command needs to be configured differently. 

> git svn clone --authors-file=authors.txt --follow-parent   http://svn.research-infrastructures.eu/public/d4science/gcube/trunk/Common/gxREST \
  --username manuele.simi gxRest
 
Note: --follow-parent makes it slower, but it’s needed if the SVN folder has been moved around.
 
Initialized empty Git repository in /Users/manuelesimi/GitMigration/gxRest/.git/
 
This may take a while on large repositories
Checked through r173000
 
Checked Ahrough .classpath
 
    A    pom.xml
    A    gxJRS/.classpath
    A    gxJRS/.project
    A    gxJRS/distro/profile.xml
    A    gxJRS/distro/LICENSE
    A    gxJRS/distro/changelog.xml
    A    gxJRS/distro/README
    A    gxJRS/src/test/java/org/gcube/common/gxrest/request/GXWebTargetAdapterRequestTest.java
    A    gxJRS/src/test/java/org/gcube/common/gxrest/request/GXHTTPStringRequestTest.java
    A    gxJRS/src/test/resources/logback-test.xml
    A    gxJRS/src/main/java/org/gcube/common/gxrest/methods/package-info.java
    A    gxJRS/src/main/java/org/gcube/common/gxrest/request/GXHTTPStreamRequest.java
    A    gxJRS/src/main/java/org/gcube/common/gxrest/request/package-info.java
    A    gxJRS/src/main/java/org/gcube/common/gxrest/request/GXWebTargetAdapterRequest.java
 
    [omitted output]
 
    A    gxJRS/src/main/java/org/gcube/common/gxrest/response/entity/SerializableErrorEntityTextWriter.java
    A    gxJRS/src/main/java/org/gcube/common/gxrest/response/entity/SerializableErrorEntityTextReader.java
    M    gxJRS/src/main/java/org/gcube/common/gxrest/response/entity/SerializableErrorEntity.java
r178787 = ac04855b00de818f2095d0784eb68c51a6ec9f77 (refs/remotes/git-svn)
Checked out HEAD:
  https://svn.d4science.research-infrastructures.eu/gcube/trunk/Common/gxREST r178787
creating empty directory: gxHTTP/src/main/resources
creating empty directory: gxJRS/src/main/resources
creating empty directory: gxJRS/src/test/java/org/gcube/common/gxrest/response

Do note that the command above DO NOT automatically import the SVN branches because the repo does not have the standard SVN layout .

If the SVN repository doesn’t have a standard layout and you want to import everything, you need to provide the locations of your trunk, branches, and tags using the --trunk, --branches, and --tags command line options. See git-svn for further options.

Change dir in the cloned folder

The last parameter of the "git svn clone" command is the name of the folder ("gxRest" in the previous example) where the SVN repository is cloned and converted to Git. All the git commands in the next sections are executed in the cloned folder. So you need to: 

> cd <cloned repo>

Rename “trunk” branch to “master” (if needed)

> git branch
* trunk
> git branch -m trunk master
> git branch
* master

Check the imported history

> git log -10
commit ac04855b00de818f2095d0784eb68c51a6ec9f77
Author: manuele.simi <manuele.simi@isti.cnr.it>
Date:   Sun Mar 31 03:39:06 2019 +0000
 
	Add JAX-RS MessageBodyWriter/Reader responsible for converting SerializableErrorEntity to/from a stream.
 
	git-svn-id: https://svn.d4science.research-infrastructures.eu/gcube/trunk/Common/gxREST@178787 82a268e6-3cf1-43bd-a215-b396298e98cf
 
commit 6eb3f608dbe31c50578e46c53e16c469a0cc7f0c
Author: manuele.simi <manuele.simi@isti.cnr.it>
Date:   Sat Mar 30 19:58:21 2019 +0000
 
	Tweak some javadoc.
 
	git-svn-id: https://svn.d4science.research-infrastructures.eu/gcube/trunk/Common/gxREST@178786 82a268e6-3cf1-43bd-a215-b396298e98cf

Add code-repo’s Git repository as new remote

Do note that, before running this step, you need to create a new Git repository. 

> git remote add origin https://code-repo.d4science.org/manuele.simi/gxRest.git
> git remote -v
origin    https://code-repo.d4science.org/manuele.simi/gxRest.git (fetch)
origin    https://code-repo.d4science.org/manuele.simi/gxRest.git (push)

Push the local repository to the new remote

> git push --set-upstream --force origin master
Counting objects: 168, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (125/125), done.
Writing objects: 100% (168/168), 39.47 KiB | 0 bytes/s, done.
Total 168 (delta 44), reused 0 (delta 0)
remote: Resolving deltas: 100% (44/44), done.
To https://code-repo.d4science.org/manuele.simi/gxRest.git
 + 4ec6b48...ac04855 master -> master (forced update)
Branch master set up to track remote branch master from origin.

Change the SCM Connection

The SCM section of the POM must be changed to reflect that we are now working with a Git repository.

For instance, the following section: 

  <scm>
	<connection>scm:svn:http://svn.d4science.research-infrastructures.eu/gcube/trunk/distributions/${project.artifactId}</connection>
	<developerConnection>scm:svn:https://svn.d4science.research-infrastructures.eu/gcube/trunk/distributions/${project.artifactId}</developerConnection>
	<url>http://svn.d4science.research-infrastructures.eu/gcube/trunk/distributions/${project.artifactId}</url>
  </scm>

must be changed to: 

<scm>
    <connection>scm:git:https://code-repo.d4science.org/gCubeSystem/${project.artifactId}.git</connection>
    <developerConnection>scm:git:https://code-repo.d4science.org/gCubeSystem/${project.artifactId}.git</developerConnection>
    <url>https://code-repo.d4science.org/gCubeSystem/${project.artifactId}</url>
</scm>

NB. In this case, the ${project.artifactId} declared in the pom is equal to the repository name declared on Git.

Otherwise, you must declare the Git repository URL like is it: 

<scm>
    <connection>scm:git:[YOUR_GIT_REPOSITORY_URL].git</connection>
    <developerConnection>scm:git:[YOUR_GIT_REPOSITORY_URL].git</developerConnection>
    <url>[YOUR_GIT_REPOSITORY_URL]</url>
</scm>


Back to the CI guide.

Retrieved from "https://wiki.gcube-system.org/index.php?title=Import_from_SVN&oldid=33237"