pld-builder.new: doc/ARCHITECTURE - ws cleanup

[glen]

Author: glen                         Date: Tue Nov 16 09:14:14 2010 GMT
Module: pld-builder.new               Tag: HEAD
---- Log message:
- ws cleanup

---- Files affected:
pld-builder.new/doc:
   ARCHITECTURE (1.5 -> 1.6) 

---- Diffs:

================================================================
Index: pld-builder.new/doc/ARCHITECTURE
diff -u pld-builder.new/doc/ARCHITECTURE:1.5 pld-builder.new/doc/ARCHITECTURE:1.6
--- pld-builder.new/doc/ARCHITECTURE:1.5	Tue Nov 16 10:13:50 2010
+++ pld-builder.new/doc/ARCHITECTURE	Tue Nov 16 10:14:09 2010
@@ -1,7 +1,7 @@
 0. ABSTRACT
 
-Below is a description of the way the builder infrastructure works. You should 
-read it in order to be able to understand what's exactly going on when you try 
+Below is a description of the way the builder infrastructure works. You should
+read it in order to be able to understand what's exactly going on when you try
 to build a package.
 
 (It assumes you have a basic understanding of how RPM packages are created.)
@@ -13,32 +13,32 @@
 - the source builder
 - the binary builders
 
-The basic idea is to allow developers (requesters) to easily specify which 
-package they want built, and have the rest of the system take care of building 
-and upgrading packages, uploading them where appropriate and reporting back to 
-the developers on the results (were there errors, how long it took to build, 
+The basic idea is to allow developers (requesters) to easily specify which
+package they want built, and have the rest of the system take care of building
+and upgrading packages, uploading them where appropriate and reporting back to
+the developers on the results (were there errors, how long it took to build,
 etc.).
 
-What follows is a more detailed description of what each part is supposed to 
+What follows is a more detailed description of what each part is supposed to
 do.
 
 1.1 NOTES ON UPLOADING
 
-When talking of uploading packages and logs it is meant as placing them 
-somewhere on the original system (via a simple 'cp'; this also includes copying 
-to /dev/null), or placing them on a remote system via the scp or rsync 
+When talking of uploading packages and logs it is meant as placing them
+somewhere on the original system (via a simple 'cp'; this also includes copying
+to /dev/null), or placing them on a remote system via the scp or rsync
 protocols.
 
 1.2 TOOLS USED
 
-99% of the system is written in python, with remaining parts using shell 
-scripts (POSIX shell is assumed, bashisms are to be shot at sight and their 
-authors beheaded in a public display). The presence of various helper tools is 
+99% of the system is written in python, with remaining parts using shell
+scripts (POSIX shell is assumed, bashisms are to be shot at sight and their
+authors beheaded in a public display). The presence of various helper tools is
 assumed. They are: gpg, rsync and scp command line utilities.
 
 2. SHORT DIAGRAM
 
-Below are complete basics of how data is tossed around. Only the most relevant 
+Below are complete basics of how data is tossed around. Only the most relevant
 transfers are presented.
 
          status info          status info
@@ -54,111 +54,111 @@
 
 Main task: sending build requests to source builders.
 
-This is the simplest part. It consists of a shell script (make-request.sh), 
-which, based on the way it's configured and on the cmdline parameters given to 
-it, generates an xml-formated build request, which it then signs with the 
-requester's PGP key (using the gpg utility) and then sends it to the source 
-builder via a sendmail compatible command line application (by default invoking 
+This is the simplest part. It consists of a shell script (make-request.sh),
+which, based on the way it's configured and on the cmdline parameters given to
+it, generates an xml-formated build request, which it then signs with the
+requester's PGP key (using the gpg utility) and then sends it to the source
+builder via a sendmail compatible command line application (by default invoking
 "sendmail -t").
 
 Two modes of operation are:
 - sending requests to build chosen package(s) on a specified group of builders
 - sending a chosen command to be executed on a specified group of builders
 
-See HOWTO-CLIENT for a hands-on approach to setting yourself up for being able 
+See HOWTO-CLIENT for a hands-on approach to setting yourself up for being able
 to send out build requests.
 
 4. SOURCE BUILDER (AKA SRC.BUILDER)
 
 Main task: building src.rpm packages and serving them to binary builders.
-Additional tasks: reporting back to the requester about the status of the build 
+Additional tasks: reporting back to the requester about the status of the build
 and uploading generated src.rpm packages where appropriate. Also -- tagging,
 access control, uploading buildlogs and keeping the 'queue.html' file in sync.
 
-The source builder is the central place for managing access rights for 
-requesters. Upon receiving a request (via email), it first checks the pgp 
-signature with which the request was signed, against the pgp key currently 
-assigned to the requester (all done with the 'gpg' command). After verifying 
-that the requester name/requester pgp signature pair match, it then checks if 
-the requester is actually allowed to perform the actions she wants to perform.  
-That is: does she have access to a specified binary builder, is she allowed to 
-specify the 'upgrade' flag or not, is she allowed to use the priority she set 
-for the request and even if she is allowed to request building of a given 
+The source builder is the central place for managing access rights for
+requesters. Upon receiving a request (via email), it first checks the pgp
+signature with which the request was signed, against the pgp key currently
+assigned to the requester (all done with the 'gpg' command). After verifying
+that the requester name/requester pgp signature pair match, it then checks if
+the requester is actually allowed to perform the actions she wants to perform.
+That is: does she have access to a specified binary builder, is she allowed to
+specify the 'upgrade' flag or not, is she allowed to use the priority she set
+for the request and even if she is allowed to request building of a given
 package.
 
-Once a request passes all verifications, the src.builder uses its own chroot 
-equipped with a basic build environment to fetch the sources of a given 
-package, tag them if configured to do so and create a src.rpm out of them (all 
-done by invoking the 'builder' script as found in the SPECS module of the PLD 
+Once a request passes all verifications, the src.builder uses its own chroot
+equipped with a basic build environment to fetch the sources of a given
+package, tag them if configured to do so and create a src.rpm out of them (all
+done by invoking the 'builder' script as found in the SPECS module of the PLD
 cvs repository (see http://www.pld-linux.org/Repositories for details)).
 
-After finishing the build procedure the src.builder reports (via email) to the 
-requester on the status of the build (OK/FAIL), updates the 'queue.html' file 
-with that information (urls for PLD2.0 and 3.0 queues: 
+After finishing the build procedure the src.builder reports (via email) to the
+requester on the status of the build (OK/FAIL), updates the 'queue.html' file
+with that information (urls for PLD2.0 and 3.0 queues:
 http://ep09.pld-linux.org/~buildsrc/queue.html and
-http://src.th.pld-linux.org/queue.html) and, if configured to do so, 
-uploads the complete log of the whole procedure (called the buildlog) 
-somewhere, most likely to our PLD buildlogs system 
+http://src.th.pld-linux.org/queue.html) and, if configured to do so,
+uploads the complete log of the whole procedure (called the buildlog)
+somewhere, most likely to our PLD buildlogs system
 (http://buildlogs.pld-linux.org).
 
-If the build procedure was successful, the resulting src.rpm file is most 
-likely uploaded somewhere (if configured to do so; in our case it's our FTP 
-server) and, what is most important, it's being served via a http server for 
-the binary builders to download. An xml-formated (and pgp signed) 'queue.gz' 
-and a small 'max_req_no' files are also updated with relevant information, in 
-order for the bin.builders to be able to notice new requests and figure out 
+If the build procedure was successful, the resulting src.rpm file is most
+likely uploaded somewhere (if configured to do so; in our case it's our FTP
+server) and, what is most important, it's being served via a http server for
+the binary builders to download. An xml-formated (and pgp signed) 'queue.gz'
+and a small 'max_req_no' files are also updated with relevant information, in
+order for the bin.builders to be able to notice new requests and figure out
 which are addressed to which builders.
 
-The last matter taken care of by the src.builder is receiving (gpg signed) 
-emails from bin.builders with status information regarding what happened to 
-various builds (whether they succeeded or failed) and updating the 'queue.html' 
+The last matter taken care of by the src.builder is receiving (gpg signed)
+emails from bin.builders with status information regarding what happened to
+various builds (whether they succeeded or failed) and updating the 'queue.html'
 file accordingly.
 
-See the HOWTO-BUILDERS file for a hands-on approach on setting up builders. All 
+See the HOWTO-BUILDERS file for a hands-on approach on setting up builders. All
 of the technical details are there.
 
 5. BINARY BUILDERS (AKA BIN.BUILDERS)
 
 Main task: building arch.rpm packages and uploading them where appropriate.
-Additional tasks: reporting back to the requester and the source builder on the 
+Additional tasks: reporting back to the requester and the source builder on the
 status of the build. Also -- uploading buildlogs.
 
-Upon noticing that a new request has been published by the src.builder and 
-verifying the appropriate gpg signatures, binary builders download the relevant 
-src.rpm files and use their build-environment-equipped chroots to start 
-building them (by invoking the 'rpmbuild' command). Once a build is completed 
-(meaning arch.rpm files are produced, or the whole procedure is interrupted for 
-whatever reason), the status of the build (OK/FAIL) is sent via email both to 
+Upon noticing that a new request has been published by the src.builder and
+verifying the appropriate gpg signatures, binary builders download the relevant
+src.rpm files and use their build-environment-equipped chroots to start
+building them (by invoking the 'rpmbuild' command). Once a build is completed
+(meaning arch.rpm files are produced, or the whole procedure is interrupted for
+whatever reason), the status of the build (OK/FAIL) is sent via email both to
 the requester and to the src.builder.
 
-Procedure for uploading the produced arch.rpms and buildlogs is the same as 
+Procedure for uploading the produced arch.rpms and buildlogs is the same as
 with the src.builder.
 
-Bin.builders however need to deal with one thing src.builders are mostly not 
-affected by -- managing the build environments inside their chroots. That means 
-installing new packages if required by a given build (using the 'poldek' 
-package manager; http://poldek.pld-linux.org/), upgrading older packages for 
-much the same reasons, downgrading from time to time and locally caching newly 
+Bin.builders however need to deal with one thing src.builders are mostly not
+affected by -- managing the build environments inside their chroots. That means
+installing new packages if required by a given build (using the 'poldek'
+package manager; http://poldek.pld-linux.org/), upgrading older packages for
+much the same reasons, downgrading from time to time and locally caching newly
 built files for short periods of times.
 
-All of this, along with lots of quirks and obscure details is explained in the 
+All of this, along with lots of quirks and obscure details is explained in the
 HOWTO-BUILDERS file.
 
 6. Q/A
 
 Q: What's this 'tagging' all about with the src.builder?
-A: When a src.rpm is created and arch.rpms built based on it, it is sometimes 
-useful to be able to check which sources were used to build them. Casual users 
-might be satisfied with just extracting everything from the src.rpm (that is 
-what they are uploaded to the ftp server for), but the old ones aren't always 
-available and even if they are, developers require source control in order to 
-work efficiently. Hence all sources used to build a given src.rpm for a given 
-line are tagged appropriately in the PLD CVS repository, for example trying to 
-build cthulu.spec on the 2.0 (codename Ac) line of our distro might result in a 
+A: When a src.rpm is created and arch.rpms built based on it, it is sometimes
+useful to be able to check which sources were used to build them. Casual users
+might be satisfied with just extracting everything from the src.rpm (that is
+what they are uploaded to the ftp server for), but the old ones aren't always
+available and even if they are, developers require source control in order to
+work efficiently. Hence all sources used to build a given src.rpm for a given
+line are tagged appropriately in the PLD CVS repository, for example trying to
+build cthulu.spec on the 2.0 (codename Ac) line of our distro might result in a
 tag looking something like this being set -- 'auto-ac-cthulu-6.6-6'.
-(Technically the previous sentence is not 100% correct due to (a) the way our 
-DISTFILES infrastructure impacts which sources are and aren't in the CVS repo 
-and hence are subject to tagging and (b) specific policies regarding tagging 
+(Technically the previous sentence is not 100% correct due to (a) the way our
+DISTFILES infrastructure impacts which sources are and aren't in the CVS repo
+and hence are subject to tagging and (b) specific policies regarding tagging
 for each distro line; for the latter refer to appropriate policies.)
 
 
================================================================

---- CVS-web:
    http://cvs.pld-linux.org/cgi-bin/cvsweb.cgi/pld-builder.new/doc/ARCHITECTURE?r1=1.5&r2=1.6&f=u