Migrating Subversion (SVN) to Git

We decided to move from a local subversion we were using to git using Bitbucket. Why moving from svn to git is a matter of taste imho. Whatever suits you best, there would definitely be pros and cons for each. We did it cause the programmers seemed to like working with git and we didn’t want to maintain an svn ourselves.  Bitbucket vs Github is all about the cost, in the end they are both git. In our case Bitbucket was cheaper since we are a small group of developers with large amount of repos, so the pricing of Bitbucket (per developer) is just right for us.

Bitbucket offers migration through their webpage but that was no option for us because we needed our commit history intact. So we were off for alternatives. Also our svn was pretty old so we had to do some extra steps before migrating. So let’s go through them.

Step 1

Install svn to a local machine.

sudo yum install mod_dav_svn subversion

For the rest you can follow this guide up to the point of creating a testrepo. Instead of creating testrepo we are going to take our own repos from the remote svn and put them in /var/www/svn.

Step 2
On the remote server.

tar -zcvf svnbackup.tar.gz /var/www/svn

Then get this to your local svn and extract accordingly.

Step 3

Give proper user rights

chown -R apache.apache *
chcon -R -t httpd_sys_rw_content_t /var/www/svn/*

And then goto your backuped svn folders and perform and svnadmin upgrade like this

 svnadmin upgrade /var/www/svn/myproject 

Step 4

Get a list of the authors so our commits get linked to them. Using this trick.

svn log --quiet http://localhost/svn/myproject \
| grep '^r' | awk '{print $3}' | sort | uniq > authors.txt

Step 5

Now the rest is simple using this guide.

git svn clone http://localhost/svn/svn.myrepo/ -A authors.txt my_repo
git init --bare my_bare_repo
cd my_bare_repo
git symbolic-ref HEAD refs/heads/trunk
cd ../my_repo
git remote add bare ../my_bare_repo
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ../my_bare_repo
git branch -m git-svn master
git remote add origin https://me@bitbucket.org/me/myrepo.git
git push origin master

Bumps and how you can manage them!

    1. After git svn clone you end up with this error
fatal: refs/remotes/trunk: not a valid SHA1
update-ref HEAD refs/remotes/trunk: command returned error: 128

In most cases the below should do the job. Else do a git branch -a see your remotes and set master to follow one of them.

 git update-ref refs/heads/master refs/remotes/git-svn 

#bitbucket, #git-2, #github, #migration, #svn-2

Migrating VMware ESXi

Hello.
Recently one of our VMware ESXi servers started acting up weird.
Situation was that we had no means of accessing ESXi through SSH ( user/pass got refused although it was the right one) and the console was also unaccesible for each VM.
After invastigating a bit through the Vsphere Client it appearead that the system couldn’t find the necessary pam modules and thus no authentication for us.
Hopefully at least we had access through the VSphere Client and also to the Web-Based datastore Browser.
Anyway to make long story short we had to find a way of getting our VMs from there and moving them to another machine.
Typically and afaik we would simply login on our ESXi console and scp our files through the servers. In our case that was no option since we had no ssh access at all. So we came up with two solutions.

General Prerequisites 

  1. Power off your VM while transferring.
  2. Make sure there are no snapshots.
  3. If you got snapshots  take care of them and delete. You can’t move snapshots and you have to have only “the current working branch” .

Transferring (Having a shell access)

  1. scp [[user@]from-host:]source-file [[user@]to-host:][destination-file]  
  2. Example: scp root@myno1.esx.com:/vmfs/volumes/datastore1/linux/linux.vmdk root@myno2.esx.com:/vmfs/volumes/datastore1/linux

Transferring (Alternative way, no ssh required)

    1. I’ve found this very handy little script http://blogs.vmware.com/vsphere/2010/01/scripting-datastore-access.html . What it does is accessing your datastore through the web interface and downloads to local .
    2. So to make things faster we can have this script download the files on a remote box and then transfer from there to our new ESXi server.
#!/bin/bash

CURL_ARGS="--insecure"
# Change if you want to use an alternate user
# (you'll be prompted for the password each time)
USER=root
if ! $(which curl > /dev/null); then
echo "ERROR: curl not found in your path" >&2
echo "" >&2
echo "You'll need to install curl on your system for this script to work." >&2
exit 1
fi
usage() {
echo "USAGE: $0 <get|put> <hostname> <ds> <ds path> <local_path> [thread_count]" >&2
echo "" >&2
echo "one of source or target must be a datastore path" >&2
echo "Example: $0 put hostname datastore /file.iso ./file.iso'" >&2
echo "" >&2
echo "If you set the thread_count then this script will" >&2
echo "use that many parallel threads when downloading the file." >&2
echo "Warning: you'll need ~2x the file size in available space locally for this approach" >&2
}
urlescape() {
TMP=$(echo $1 | sed \
-e ' {
s/%/%25/g
s/ /%20/g
s/</%3C/g
s/>/%3E/g
s/#/%23/g
s/{/%7B/g
s/}/%7D/g
s/|/%7C/g
s/\\/%5C/g
s/\^/%5E/g
s/~/%7E/g
s/\[/%5B/g
s/\]/%5D/g
s/`/%60/g
s/;/%3B/g
s|/|%2F|g
s/?/%3F/g
s/:/%3A/g
s/@/%40/g
s/=/%3D/g
s/&/%26/g
s/\$/%24/g
}' )
echo ${TMP}
}
if [ $# -lt 5 ] ; then
usage
exit 1
fi
OPERATION=$1
HOSTNAME=$2
DATASTORE=$(urlescape "$3")
REMOTE_PATH=$(urlescape "$4")
LOCAL_PATH=$5
if [ $# == 6 ] ; then
THREADS=$6
else
# Default to 4 threads for better performance
THREADS=4
fi
URL="https://${HOSTNAME}/folder/${REMOTE_PATH}?dcPath=ha-datacenter&dsName=${DATASTORE}"
if [ "${OPERATION}" == "get" ]; then
if [ ${THREADS} -gt 1 ] ; then
echo -n "Enter password for ${USER}@${HOSTNAME}: "
stty -echo
read PASSWORD
stty echo
echo ""
# Note: This is somewhat insecure as the password will show up on the
#    command line. Consider switching to use netrc or SSPI
#	   see the curl man page for more details.
#
#    This first curl invocation grabs just the header to get the size
LENGTH=$(curl ${CURL_ARGS} -s -u "${USER}:${PASSWORD}" "$URL" -I | awk '/Content-Length:/ { print $2 }'|sed -e "s/\r//g")
CHUNK=$((LENGTH / THREADS + 1))
START=0
echo "Starting download (${THREADS} parallel threads)..."
COUNT=1
while [ ${COUNT} -le ${THREADS} ] ; do
END=$((START + CHUNK - 1))
# Progress reporting with multiple threads gets jumbled up, so be silent
curl -s ${CURL_ARGS} -u "${USER}:${PASSWORD}" --create-dirs --range ${START}-${END} "$URL" -o "${LOCAL_PATH}.${COUNT}" &
COUNT=$((COUNT + 1))
START=$((END + 1))
done
wait
# Merge the files back together
rm -f "${LOCAL_PATH}"
touch "${LOCAL_PATH}"
COUNT=1
while [ ${COUNT} -le ${THREADS} ] ; do
cat "${LOCAL_PATH}.${COUNT}" >> "${LOCAL_PATH}"
rm -f "${LOCAL_PATH}.${COUNT}"
COUNT=$((COUNT + 1))
done
echo "Done"
else
curl ${CURL_ARGS} -u ${USER} --create-dirs "$URL" -o "${LOCAL_PATH}"
fi
elif [ "${OPERATION}" == "put" ]; then
if [ ! -f ${LOCAL_PATH} ] ; then
echo "ERROR: ${LOCAL_PATH} does not exist" >&2
echo "" >&2
usage
exit 1
fi
curl ${CURL_ARGS} -u ${USER} -T "${LOCAL_PATH}" "$URL"
else
usage
exit 1
fi

Configuration of the VM to the new ESXi

  1. After having transfer all our files ( *-flat.vmdk *.vmdk *.vmx) we now have to create a new VM on the new server.
  2. Create a new VM using the exact same options as on the old ESXi.
  3. Choose Advanced options and select an existing hard disk.
  4. Finsh and boot.

Final Configuration

  1. Since you are now propably on a new network you need to reconfigure your network and your VMAC address.
  2. Create a VMAC address from your host panel.
  3. On CentOS goto /etc/udev/rules.d/70-persistent-net.rules and edit making sure eth0 uses the newly created VMAC.
  4. Goto /etc/sysconfig/network-scripts and edit ifcfg-eth0 accordingly with your new ip, gateway and hwaddr.
  5. Edit /etc/sysconfig/network-scripts/route-eth0 and configure your gateway.

 

Et voila! You have now succesfully migrated ! 🙂

Suggestions or alternative methods are really appreciated 🙂

#centos, #datastore, #esxi, #esxi4, #esxi5, #migration, #moving, #vmware-2