distcc - Compiling With Multiple Machines

For our final lab for the semester, we are required to compile a program with a program called distcc.

This program allows us to distribute the task of compiling packages to multiple machines so that we can reduce compiling time, compared to compiling packages on one machine.

To start, I picked the machine Australia as my client machine, which will run the rpmbuild command. And I chose Scotland, HongKong, and Germany as my servers, they will be the machines I will distributing my work to.

I followed the instructions that my professor posted up: Link which really helped a lot.

To see if the server boxes are helping me compile my package, which I choose abiword.
I sshed into them and typed in the command top which displays all the tasks going on in real time. So far everything was working fine, till I noticed that Scotland was not receiving any work.

The way you can tell if ditcc is working on all machines is check for either an instance of cc1, distccd, or ccplus



So instead of stopping and rebuilding, I let it finish to see how long it took to compile with 3 machines. The J value I set up for this build was 15. You can read back to my previous blog about what the J value does: Link
The time it took to compile the package with 3 computer is 7 mins 43.183 s


After the completion, I edited the J value this time to 25 and ran the rpmbuild command again.
This time, I noticed that all 4 machines are now compiling as you can see in my next screen shot.



Below is the final screen shot of the final compiling time.
The time it took this time to compile with 4 machines is 5 min 47.615s


As you can see, the more machines you have distruted with your work the faster it gets. As my professor Chris Tyler mentioned they use distcc to compile Fedora

Backporting GCC 4.4 with 4.5 Plugin Capabilities

Last Week, I was Invited To Help Dave Humphrey one of the Open Source Professors at Seneca, my Professor Chris Tyler, Diego from Google, Ehren a Student at Seneca, and Taras from Mozilla with a project to backport GCC 4.5 features to 4.4.

Here's a Link to Wikipedia describing what backporting is: LINK

So far, I was asked to download two branches ( or otherwise I think they mean directories) one for GCC 4.4 plugins the directory where we will be patching GCC with 4.5 patches and trunk a directory to help create the patches needed.

Diego was kind enough to provide us links to download these two branches.

svn co svn://gcc.gnu.org/svn/gcc/branches/gcc-4_4-plugins -- This creates the directory gcc-4.4 with gcc 4.4 source codes

svn co svn://gcc.gnu.org/svn/gcc/trunk -- This will create the directory trunk where we will run a script that will help create patches that we will run against the gcc-4._4-plugins directory

Here is a link about what SVN is: LINK

Since we are downloading a lot of source codes, it did take a while to download both the branches.

After completion, I ran a script Diego also provided us to create the patch, here is a copy of the script

What I did was save that script in the trunk directory, and changed the permission so its executable.

Next Running the Script...

In one of the emails that was sent to me, there was a document containing a bunch of information regarding what changes has been made to the revision.

So with little guidance at first, I did what Diego asked me to do, execute the script with the revision number

So my syntax would be: ./svn-get-rev "revision number" i.e. ./svn-get-rev 153472

But today, I found out I was doing it wrong. With guidance later with Diego via email, and chatting with Dave via IRC and in person they set me on the right path.

I was suppose to redirect the output of the command to the file, and that file would become the patch I would use.

so the proper syntax is:

./svn-get-rev "revision number" > /directory/"filename".diff

i.e. ./svn-get-rev 153472 > ~/gcc-logs/rev-153472.diff

So this command will run the script against that revision number, and create a patch file. I was told by Dave if you use the extension .diff or .patch most developers will know what to do with it.

After creating all the patch files, its time to patch the gcc4_4-plugins directory

A tip Dave gave me while I was at this step.

For patching, before committing to execute the patch, I should do a test run to see if there would be any errors

and the syntax for that is

patch -p0 --dry-run < /directory/"patch file".diff

i.e. patch -p0 --dry-run < ~/gcc-logs/rev-146274.diff

Here is some information on patch: Link

What this will do is run the command patch with the patch file and run it test mode.

After you see the outcome from running the script you can run the command regularly

Syntax: patch -p0 < /directory/"patch file".diff

i.e. patch -p0 < ~/gcc-logs/rev-146059.diff

If your lucky, you might not get any errors.
Sadly for me, I did receive some errors...here is the error I got.

So what I would have do to is view the file the script created for the file that its having issues patching (ends with a .rej extension) and compare it with the original file, and add or remove some of the lines needed to make it work.

If you made some changes to the patched file and wanted to undo the changes, Dave provided me some commands that I can do to revert the changes.
They are

svn revert "file name" -- This will basically undo all changes to the file and put it back to the original form.
i.e. svn revert gcc/Makefile.in

patch -p0 -R < /directory/"patch file".diff -- This command will undo the changes for that particular patch, and not undo all the work you have done.
i.e patch -p0 -R < /usr/gcc/gcc-patch/rev-146059.diff So far, I only did a test run with the patch files, hopefully tomorrow I can get all the patching done. Until then, Bye World