Yolinux.com Tutorial Cabie logo

Cabie Automated Build System

"Continuous Automated Build and Integration Environment"

Contents:

Related YoLinux Tutorials:

°Jenkins Build Server

°Web Server Configuration

°Subversion And Trac Server Installation / Configuration

°Subversion Commands

°Software development tools

°YoLinux Tutorials Index




Free Information Technology Magazines and Document Downloads
TradePub link image


   

    Bookmark and Share


Advertisements




Cabie Intro:

Cabie build server architectural diagram
Cabie is a build system with distributed build servers to perform software builds on various hardware platforms. Cabie supports "continuous integration" (build with each CM i.e. Subversion check-in) or "nightly builds", and automated regression testing. Cabie is "open source" and distributed under the terms of the GNU General Public License version 2.

Cabie is comprised of:

  • A client program which submits a build job (by inserting entries into the MySQL database) and can submit build requests for that job to the build servers.
  • A mySQL database which accepts job definitions and reporting from the build servers (build request status).
  • Numerous build servers which accept the build requests and performs the builds. There is one build server per platform for which builds are to be generated. (i.e. various flavors of Linux, MS/Windows and Unix) The build server will prepare the environment, perform an update and then submit four jobs (shell scripts):
    1. pre-build job: i.e. make clean or ant clean
    2. build job: compilation i.e. make or ant
    3. post-build job: verify make/build, does new build product exist
    4. post-post-build job: regression test build product (application)
    The build server supports the following CM systems: This tutorial presumes that the CM systems is already installed and configured. See the YoLinux.com Subversion and Trac Server Installation and Configuration tutorial.
  • An Apache CGI which interfaces with the database and presents build results logs. It auto-updates the web page to continually show the latest status.

Cabie will build with build monitoring "watchdog" threads to notify of overdue processes or insufficient resources generate email notifications, respond to build inquiries and test. Cabie is written in Perl.

Also see: Cabie process flow diagram.


Cabie Installation:

I'll start by saying that the Cabie perl scripts and configuration are specific to one particular installation and that it will take a lot of tweaking and debugging to get it to work for your specific configuration. It will fail if mis-configured. This tutorial is centric to a Subversion CM integration. A knowledge of Perl is helpful to get Cabie installed and working properly. Unlike other YoLinux.com Tutorials which have a cook book, quick start style, you may find Cabie more complex and difficult to configure. - Good luck!

System Preparation:

The following native Linux packages MUST be installed:

  • Web Server: perl, httpd (Apache), mysql, perl-DBI, perl-DBD, smtp server (any email server)
  • Build Server: perl, perl-DBI, perl-DBD
Additional Perl packages required by both the Cabie web and build server: Mail::Sendmail, FreezeThaw, Archive::Zip, Compress::Zlib

Perl:

Cabie requires a threaded version of perl. The version of perl included with RHEL4 is threaded. Use the command "perl --version". The response "i386-linux-thread-multi" usually indicates that the version of perl you are using is threaded (see RHEL3 exception and installation below). For more detail use: perl -V. A threaded version of perl is also available from ActiveState.com and CPAN.org.

Downloading / installing Perl with CPAN:

  • perl -MCPAN -e shell
  • force install threads
Reply "yes" if asked to build a threaded version.

Red Hat Enterprise 4 / CentOS 4: Cabie requires the installation of four perl modules (above and beyond the regular Red Hat Enterprise 4 release):

  • Mail::Sendmail - rpm -ivh perl-Mail-Sendmail-0.79-1.2.el4.rf.noarch.rpm
  • FreezeThaw - rpm -ivh perl-FreezeThaw-0.43-1.2.el4.rf.noarch.rpm
  • Archive::Zip - rpm -ivh perl-Archive-Zip-1.16-1.2.el4.rf.noarch.rpm
  • Compress::Zlib - rpm -ivh perl-Compress-Zlib-1.33-0.2.el4.rf.i386.rpm
Obtain these RPMS from the "Home made" APT/Yum "RPM forge" repository at http://pkgs.repoforge.org/

SuSE (9.3 pro): The SuSE 9.3 distribution comes with FreezeThaw, Archive::Zip and Compress::Zlib. Use CPAN to install Mail::Sendmail:

  • perl -MCPAN -e shell
  • Answer "No"
  • install Mail::Sendmail
  • quit

Also see the YoLinux SysAdmin Tutorial: Managing Perl

Installation on other platforms is covered below: SGI/IRIX, RHEL3, MS/Windows XP


Cabie Download/Install:

  1. Download Cabie source from http://cabie.tigris.org: cabieserver.tar
    Note: This tutorial is for release 0.8.1. Newer releases are available from the CVS repository but may differ slightly from this tutorial.
  2. Un-tar in the directory /opt/Cabie/: tar xf cabieserver.tar
    Creates:
    • /opt/Cabie/client/ (Client perl command program)
    • /opt/Cabie/server/ (Perl scripts and libraries, job bash scripts, ...)
    • /opt/Cabie/web/ (CGI, css and icons)
  3. Support for Subversion is available by downloading the latest files from the Tigris.org Cabie CVS repository.
  4. Remove "^M"s: find /opt/Cabie -name "*" -exec dos2unix {} \;

Update Database interface mysqlwdb.pl:

This system uses a MySQL perl CGI script to admin the database.

Download/Install:

  • Get the latest version from: MySqlWDB home page
    Download: mysqlwdb-1.5.4.tar.gz
  • Untar/compress: tar xzf mysqlwdb-1.5.4.tar.gz
  • cp mysqlwdb-1.5.4/mysqlwdb.pl /opt/Cabie/web/cgi-bin/
  • Edit the script and specify:
    • database host server
    • MySQL port: 3306
    • Name of database: builds (defined by SQL import file)
    • MySQL user/password (Note: Cabie by default is configured to use a NULL password.)
An older version of this script comes with Cabie.

Cabie Architecture/Topology and System Configuration:

The architecture and topology of the Cabie system requires a system to function as the web and database server. It can also host a build server. The remote build servers must have network mounted (NFS or SAMBA for MS/Win) file access which allows the web server access to the build logs and converted postbuild HTML pages. This can be achieved two ways:

1) Web server mounts the exported drives from build servers

Web server: /etc/fstab Build servers: /etc/exports
bldsrv1:/share/Cabie/BUILDS /share/Cabie/BUILDS/bldsrv1 nfs auto 0 0
bldsrv2:/share/Cabie/BUILDS /share/Cabie/BUILDS/bldsrv2 nfs auto 0 0
bldsrv3:/share/Cabie/BUILDS /share/Cabie/BUILDS/bldsrv3 nfs auto 0 0
...
bldsrv1 /etc/exports
/share/Cabie/BUILDS 192.168.1.0/255.255.255.0(rw)

bldsrv2 /etc/exports

/share/Cabie/BUILDS 192.168.1.0/255.255.255.0(rw)
...
bldsrvWIN: Export build area as a share.
C:\Cabie\BUILDS
...
This configuration allows one to use a local file system which increases the speed of a build and reduces network traffic.

or
2) Build servers mount the exported drive of web server.

Web server: /etc/exports Build servers: /etc/fstab
/share/Cabie/BUILDS 192.168.1.0/255.255.255.0(rw)
Can view subdirectories:
  • /share/Cabie/BUILDS/bldsrv1
  • /share/Cabie/BUILDS/bldsrv2
  • /share/Cabie/BUILDS/bldsrv3
  • ...
bldsrv1 /etc/fstab
webserver:/share/Cabie/BUILDS /share/Cabie/BUILDS nfs auto 0 0
Create subdirectory bldsrv1
(i.e. /share/Cabie/BUILDS/bldsrv1)

bldsrv2 /etc/fstab

webserver:/share/Cabie/BUILDS /share/Cabie/BUILDS nfs auto 0 0
Create subdirectory bldsrv2

...

The second option is favored if the web server is also a general file server and thus booted first. The clients systems would then be booted afterwards and mount the shared exported drive.

MS/Windows Build Servers: It is required that MS/Windows build servers build on a local drive. This is because Cabie uses the perl disk directive "GetDriveGeometry" which does NOT work on networked Microsoft filesystems. Thus the web server will use an SMB mount to mount the MS/Windows file server.

Cabie Web server SMB mount: /etc/fstab

//bldsrvWIN/bldsrvWIN /share/Cabie/BUILDS/bldsrvWIN smbfs credentials=/home/linux-user-id/.smbpassword,workgroup=MS-WINDOWS-DOMAIN,uid=linux-user-id,gid=linux-user-group-name 0 0
Where the credentials file: .smbpassword
username = ms-windows-login-name
password = super-secret
            

On MS/Windows, create the share from the directory: C:\Cabie\BUILDS\bldsrvWIN.
(Right click on the directory in MS/Windows Explorer.)

For details see: YoLinux Tutorial: Mounting a MS/Windows share

Another option is to install "Unix services for MS/Windows" and export an NFS partition from the MS/Windows build server. This will be necessary for very strict MS/Windows authentication configurations.

Check-out code from CM: Create the Subversion local repository on the build server. It must exist before starting the Cabie build server.

  • cd /share/Cabie/BUILDS/bldsrv1
  • svn co http://subversion-server/svn/projectX


MySQL Database Configuration for Cabie:

The MySQL database is at the heart of Cabie operations. Build definitions are registered in the database. The build servers register and report status and information to the database. The Cabie web CGI uses data stored in the MySQL database to generate a display of build information.

See the YoLinux MySQL tutorial to configure and start the MySQL database.

Create database user buildmgr:

  • Assign password: mysqladmin -u root password 'supersecret'
  • mysql -h localhost -u root -psupersecret
  • mysql> use mysql;
  • mysql> INSERT INTO user (Host, User, Password ) VALUES ('localhost','buildmgr',password('buildsupersecret'));
  • mysql> INSERT INTO user (Host, User, Password ) VALUES ('%','buildmgr',password('buildsupersecret'));
  • mysql> UPDATE user set Select_priv='Y',Insert_priv='Y',Delete_priv='Y',Create_priv='Y' where User='buildmgr';
  • mysql> FLUSH PRIVILEGES;
  • mysql> CREATE DATABASE builds;
  • mysql> GRANT ALL PRIVILEGES ON builds.* TO buildmgr@'%';
  • mysql> FLUSH PRIVILEGES;
  • mysql> quit

[Potential Pitfall]: The older 1.1 release of Cabie (tar archive) does not have a method for Cabie to use a password and I have found that assigning no password ("") works. I also have been using user "root" for everything. The database login is specified in /opt/Cabie/server/lib/host-name.pm The latest code in the CVS repository does support MySQL login/password.

Import the Cabie database schema:

  • cd /opt/Cabie/server/sql
  • mysql -h localhost -u login -pPASSWORD
  • mysql> use builds;
  • mysql> \. schema
  • mysql> show tables
  • mysql> quit

Tigris.org: Description of Cabie schema

You can now connect with: mysql -h localhost -u buildmgr -pbuildsupersecret


Cabie Build Server:

Build Server Process Flow:

  • Build Server: /opt/Cabie/server/buildserver.pl: Receives commands from build client program /opt/Cabie/client/build.pl
    • Read configuration from server/lib/host-name.pm
    • Call appropriate subroutine based on hash table of build server commands
      • Command: "createjob":
        • Reads arguments and load database.
        • Data checking performed to see if command arguments are valid.
        OR
      • Command: "build":
        • /opt/Cabie/server/bin/build.pl called by buildserver.pl
        • Gather data and initialize info.
          • Read configuration from server/lib/hostname.pm (98)
          • Store job start time.
          • Calls GetJobInfo() to read database table "configuration" and store values. (325)
          • Calls _initchangeno() and sets BuildNum (363)
        • Start Job:
          • Create Semaphore (blocks new runs till done) (384)
          • Updatejobs: Execute script updatejobs.sh (defined in server/lib/hostname.pm (392)
          • _findbuildscripts() (406)
          • Verify and set: prebuild, postbuild, postpostbuild, onfail, retail and debug.
          • MainBuildProc() (411)
            • Sleep 30 seconds to make sure build area has been updated from CM.
            • Get max build time from build history. (451)
            • Launch watchdog thread. (470)
            • Set environment variables. (486)
            • Update build area: (Call to server/lib/cmbroker.pm)
            • Read job version number. (503)
            • Read file /share/Cabie/BUILDS/buildserver/projectA_job.build (534)
            • Call pre-build script: server/jobs/projectA_job/prebuild.sh (669)
            • _runjob(): Calls server/jobs/projectA_job/rtlbuild.sh (730)
            • Notify via email if build fails.
            • If build OK, then run server/jobs/projectA_job/postbuild.sh
            • Release semaphores for build job.

Build Server Configuration:

[Potential Pitfall]: Cabie version 1.1 does not suport a build server where the host name includes a "-" as it will break the Cabie perl scripts. This is fixed in the current release. Note that the build server configuration file name (/opt/Cabie/server/lib/hostname.pm) will require you to strip the "-" from the host name if it exists.

Edit/Configure Perl Scripts:

(Also see list of Cabie bug fixes which may or may not make their way back into the Cabie release from Tigris.org.)
  • server/buildserver.pl: Good as is.

  • server/bin/builder.pl:
    • (810) In order for server/jobs/projectA_job/postbuild.sh to log results (i.e. log postbuild or postpostbuild test results), capture return value of the script and log to database:
      • Alter "if" block for "postbuild" (search file for "postbuild" in builder.pl) and use the call "system()" instead of "forkprocess()" to use a blocking call which waits till done:
          ..
        ...

        @proglist = ("./script","$top","$type","$title","$BuildNum", ...);
        $retVALUE = system(@proglist);

        ...
        ..
      • Test "$retVALUE" and log results:
          ..

        _logresults($BUILDNUM,$StartTime,$CompileTime,X);

        ..

        Logs to database table "jobs" where status=X.

        Web Display status
        Build Untested 0
        Build Passed Test 3
        Build Failed 2
        Failed Test 1
        These status values correspond to the status colors used.

  • server/lib/your-buildserver-host-name-goes-here.pm:
    • This is the Cabie configuration file for the build server.
    • Copy server/lib/unix_example.pm_orig to your-buildserver-host-name-goes-here.pm
      Edit values in this perl module to define your buildserver configuration.
    • Configuration arguments: (partial list)
      Variable Description
      BD Build Drive: Linux/Posix set to '/'
      MS/Win set to 'Z:\\'
      BSR Build Server Root: Name of directory containing buildserver.pl - /opt/Cabie/server
      MS/Win set to: C:\Cabie\server
      BC Build Command: Path to build command - /opt/Cabie/server/bin
      MS/Win: C:\Cabie\server\bin\buildserver.pl
      BTMP Temp directory. Note that the directory "/tmp" is set as the default destination for debug log files.
      MS/Win: 'C:\temp'
      CVSROOT Not used by Subversion.
      COMPANY Domain to use for email. i.e. your-company.com
      LOGOICON Icon for company logo. i.e. /CabieIcons/logo.gif
      DFCMD Command to display free disk space: /bin/df
      MS/Win: ''
      DEFGROUP Default group for CM system. Use command "id" to view your group.
      BUGDB URL of your bug tracking system (i.e. your Trac server)
      'http://cm-server/cgi-bin/trac.cgi/ticket/%s'
      WORKDRIVE Work Drive: Linux/Posix set to '/'
      MS/Win: '\\'
      NULL NULL device, Linux: '/dev/null' MS/Win: 'nul'
      JOBDIR Cabie local builds directory on build server: i.e. /share/Cabie/BUILDS/BuildServerHostname
      (Match name in database table "configuration" filed "top". Used for checking disk space using command "df")
      MS/Win: 'C:\Cabie\BUILDS\BuildServerHostname
      MOUNTROOT Local builds "root" directory on build server: i.e. /share/Cabie/BUILDS
      MS/Win: 'C:\Cabie\BUILDS
      MAILER Path to mailer.pl: Use BSR/MAILER: support/mailer
      Linux: '/opt/Cabie/server/support/mailer.pl
      MS/Win: C:\Cabie\server\support\mailer.pl'
      PARALLEL Boolean: set to "0" if jobs are to be run sequentially.
      RLCHG Obtain "Product Release Number" (corresponds to the "major" version number) from change management (CM) system (var: sccs).
      PRN Product Release Number: corresponds to the "major" version number.
      INCREMENT Boolean: Set to false ("0") - default. Set true ("1") to increment "minor" version number even if build fails.
      BLDNUM Boolean: Incremental build number - "minor" version number read from file build-name.build. Default: 0
      SHAREPOINT Used for email messages for file URL. 'http://web-server/Cabie-cgi-bin/genweb'
      SMTP Outgoing SMTP email server. i.e. 'smtp.company.com'
      SMTPDELIM Delimiter for SMTP addressing: ','
      RELEASE Name of directory used to copy released builds to. Specify path.
      WEBSERVER Cabie web server i.e. http://webserver/Cabie/BUILDS
      SQLSERVER Cabie database server i.e. localhost
      SQLID Cabie database server login for database connection i.e. buildmgr
      DBPASSWORD Cabie database server password i.e. supersecret
      POLLFILE Name of file containing list of build jobs to poll. Default: 'poll.txt'
      PREBUILD Name of pre-build script file. Default: 'prebuild.sh'
      MS/Win: 'prebuild.bat'
      RETAIL Name of retail build script file. Default: 'rtlbuild.sh'
      MS/Win: 'rtlbuild.bat'
      POSTBUILD Name of post-build script file. Default: 'postbuild.sh'
      MS/Win: 'postbuild.bat'
      POSTPOSTBUILD Name of post-post-build script file. Default: 'postpostbuild.sh'
      MS/Win: 'postpostbuild.bat'

  • server/updatejobs.sh: Called by "server/bin/builder.pl". Not required. Intended to update the Cabie build server job build scripts (/opt/Cabie/server/jobs). Used if Cabie itself is stored in a CM system. Updates Cabie job and build scripts from CM.
    #!/bin/bash

    exit 0
    Cabie includes an example command if using the CM system Perforce.

  • If adding a CM system not included as default, add support in the perl module server/lib/cmbroker.pm

Continuous Build Mode:

The list of polled jobs which will be monitored and built when CM is updated: /opt/Cabie/server/poll.txt
projectA_job
projectB_job

...
..
Note:
  • The file "poll.txt" is specified in the Cabie build server configuration file: server/lib/build-server-host-name.pm as specified by the configuration variable "POLLFILE" in build server configuration file: server/lib/your-host-name-goes-here.pm
  • Any job specified in the poll.txt file will be polled and Cabie will operate in "Continuous Automated Build" mode. Cabie will constantly check for any update to the CM repository. If a check-in is detected, then Cabie will start the automated build process. To specify builds manually (or through cron), remove the job in poll.txt (or comment it out with a leading "#") and use the Cabie client to submit the build job (shown below).
  • The poll interval is hard coded in server/buildserver.pl for variable "sleepinterval". This can be customized to suit your environment. This variable is used in two places for two different purposes. One in "sub pendingjobs" is for polling the CM repository for updates and inserts results into the "pending" table used only for the mouse-over display of the "I" information icon. The other "sleepinterval" in the "sub polljobs" function is used for the job poll interval for build jobs. This is generally of equal or larger value.

Job Build Scripts:

Create a directory for each job and create job scripts for prebuild.sh, rtlbuild.sh and postbuild.sh as well as the failure handling script, onfail.sh:
/opt/Cabie/server/jobs/projectA_job/.

The scripts postbuild.sh and onfail.sh should be generic and not job specific. You should be able to use the same postbuild.sh and onfail.sh scripts for different jobs. The scripts prebuild.sh and rtlbuild.sh are more likely to have job specific directives.

  • prebuild.sh:
    #!/bin/bash

    export PATH=$PATH:/opt/bin
    TOP=$1
    TYPE=$2
    TITLE=$3

    if [ ! -d "$top/projectX" ];
    then
    echo "Build directory $TOP/$TITLE does not exist!"
    exit 1
    fi

    cd $TOP/$TITLE

    if [ -f configure ];
    then
    ./configure --prefix=/opt
    fi
    if [[ $? != 0 ]];
    then
    echo "prebuild.sh configure ERROR!"
    exit 1
    fi

    echo "configure completed."

    if [ -f Makefile ];
    then
    make clean
    fi
    if [[ $? != 0 ]];
    then
    echo "prebuild.sh ERROR!"
    exit 1
    fi
    echo "make clean completed."
    arguments:
    • $1 = Top of the source tree (Directory path: /share/Cabie/BUILDS/BuildServerHostname)
    • $2 = The build type (retail, debug, or both)
    • $3 = Title of build (job name: i.e. projectA_job)

  • rtlbuild.sh:
    #!/bin/bash

    export PATH=$PATH:$JAVA_HOME/bin:/opt/bin
    export JAVA_HOME=/usr/java/jdk1.5.0_05
    export CLASSPATH=/usr/java/jdk1.5.0_05/lib/tools.jar:/usr/java/jdk1.5.0_05/jre/lib/rt.jar:./

    TOP=$1
    TITLE=$2
    NUM=$3
    export DIST_DIR=$TOP/$TITLE.$NUM

    export BUILD_NUM=$(cat $TOP/$TITLE.build)

    cd $DIST_DIR

    cat > $DIST_DIR/build.properties <<-EOF
    Build Numer=$BUILD_NUM
    Buildserver DIR=$(pwd)
    dist.dir=$DIST_DIR
    C++ compiler=`g++ --version`
    jdk=$JAVA_HOME
    Job Title=$TITLE
    EOF

    if [ -f Makefile ];
    then
    make
    fi
    if [[ $? != 0 ]];
    then
    echo "Failure to build"
    exit 1
    fi
    echo "make completed"

    exit 0
    arguments:
    • $1 = Top of the source tree (Directory path: /share/Cabie/BUILDS/BuildServerHostname)
    • $2 = Title of build
    • $3 = Unique build number
    Note absence of "type". This reflects the configuration of the current release of Cabie.
    This script generates the file /share/Cabie/BUILDS/BuildServerHostname/projectA_job/build.properties which is delivered by the build server to the web CGI when requested.

  • dbgbuild.sh:
    This script builds a "debug" version of the build based on the "type" specified with the "createjob" command. The "type" is stored as a field in the "configuration" table which stores the job description. Two types are allowed:
    • retail
    • debug

  • postbuild.sh:
    Analyze the build to see if it was successful.
    #!/bin/bash

    export PATH=$PATH:/opt/bin
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/lib
    export TOP=$1
    export TYPE=$2
    export TITLE=$3
    export NUM=$4

    # Create output directory

    export DATA_OUTPUT=$TOP/$TITLE/$NUM

    if [ ! -d $DATA_OUTPUT ]
    then
    mkdir -p $DATA_OUTPUT
    fi

    cd $TOP/$TITLE
    cp *.log $DATA_OUTPUT

    # Prepare log files for presentation via web

    BLDSRVDIR=/opt/Cabie/server
    BLDSRVDIR_SHARED=$BLDSRVDIR/jobs/shared

    sleep 10

    # This script is only executed id rtlbuild.sh succeeds.
    # Thus build is good - record this fact.

    echo $NUM > $TOP/builds/$TITLE.lastgood


    perl $BLDSRVDIR_SHARED/getlogs.pl `hostname` $TITLE $NUM
    perl $BLDSRVDIR_SHARED/buildfuncs.pl removecomments

    if [ -f "$DATA_OUTPUT/$TITLE.$TYPE.log" ];
    then
    perl -I$BLDSRVDIR/lib \
    $SRV_SHARED_DIR/log2html.pl -e $BLDSRVDIR/config/$TITLE.errors \
    -i $DATA_OUTPUT/$TITLE.$TYPE.log \
    -o $DATA_OUTPUT/build_log.html
    else
    perl -I$BLDSRVDIR/lib \
    $SRV_SHARED_DIR/log2html.pl -e $BLDSRVDIR/config/$TITLE.errors \
    -i $DATA_OUTPUT/$TITLE.prebuild.log \
    -o $DATA_OUTPUT/prebuild_log.html
    fi

    # Test if executable has been created

    if [ ! -f application.exe ];
    then
    exit 1
    fi

    if [ -f Makefile ];
    then
    make install
    fi

    exit 0
    arguments:
    • $1 = Top of the source tree (Directory path: /share/Cabie/BUILDS/BuildServerHostname)
    • $2 = The build type
    • $3 = Title of build
    • $4 = Unique build number
    Note: Requires error file build-title.errors used as input to log2html script described below.

  • onfail.sh:
    #!/bin/bash

    TOP=$1
    TYPE=$2
    TITLE=$3
    NUM=$4

    sleep 10

    export LOG_OUTPUT=$TOP/$TITLE
    export DATA_OUTPUT=$TOP/$TITLE/$NUM

    if [ ! -d $DATA_OUTPUT ]
    then
    mkdir -p $DATA_OUTPUT
    fi

    cd $TOP/$TITLE
    cp *.log $DATA_OUTPUT

    # Prepare log files for presentation via web

    BLDSRVDIR=/opt/Cabie/server
    BLDSRVDIR_SHARED=$BLDSRVDIR/jobs/shared

    perl $BLDSRVDIR_SHARED/getlogs.pl `hostname` $TITLE $NUM &
    perl $BLDSRVDIR_SHARED/buildfuncs.pl removecomments

    if [ -f "$DATA_OUTPUT/$TITLE.$TYPE.log" ];
    then
    perl -I$BLDSRVDIR/lib \
    $SRV_SHARED_DIR/log2html.pl -e $BLDSRVDIR/config/$TITLE.errors \
    -i $DATA_OUTPUT/$TITLE.$TYPE.log \
    -o $DATA_OUTPUT/build_log.html
    elif [ -f "$DATA_OUTPUT/$TITLE.prebuild.log" ];
    then
    perl -I$BLDSRVDIR/lib \
    $SRV_SHARED_DIR/log2html.pl -e $BLDSRVDIR/config/$TITLE.errors \
    -i $DATA_OUTPUT/$TITLE.prebuild.log \
    -o $DATA_OUTPUT/prebuild_log.html
    elif [ -f "$DATA_OUTPUT/$TITLE.sync.log" ];
    then
    perl -I$BLDSRVDIR/lib \
    $SRV_SHARED_DIR/log2html.pl -e $BLDSRVDIR/config/$TITLE.errors \
    -i $DATA_OUTPUT/$TITLE.sync.log \
    -o $DATA_OUTPUT/sync_log.html
    fi

    arguments:
    • $1 = Top of the source tree (Directory path: /share/Cabie/BUILDS/BuildServerHostname)
    • $2 = The build type
    • $3 = Title of build
    • $4 = Unique build number

Regression Testing Application:

This is performed by the script: /opt/Cabie/server/jobs/projectA_job/postpostbuild.sh
Test the application to see if it is operational.
#!/bin/bash

TOP=$1
TYPE=$2
TITLE=$3
NUM=$4

sleep 10

cd $TOP/$TITLE

if [ -f Makefile ];
then
make run
fi

/opt/bin/examine-test-results.pl - Write this script to suite your app
Read program output from test run, use JUNIT or CppUnit, ...
arguments:
  • $1 = Top of the source tree (Directory path: /share/Cabie/BUILDS/BuildServerHostname)
  • $2 = The build type
  • $3 = Title of build
  • $4 = Unique build number

The processing is invoked by the build server (server/buildserver.pl) which calls server/bin/builder.pl which in turn calls the above scripts in succession if no errors are encountered. An error in "prebuild.sh" or "rtlbuild.sh" ends the processing with a final call to the script "onfail.sh". If the build is successful then an install (postbuild.sh) and test (postpostbuild.sh) will occur.

Support Scripts:

Located in server/jobs/shared/
Script Description
getlogs.pl Generates records for the "comments" table. These comments are displayed by web/cgi-bin/genweb when the open folder icon is selected or on icon "mouse-over". If there are no comments, then a closed folder icon is displayed. Selecting the closed folder icon allows you to manually generate comments for the build.
log2html.pl Read build results (compile/link) and create a color coded html page. Used by postbuild.sh and onfail.sh. Command options:
  • -i log-file: Input build log
  • -e error-file: input file. This file is a list of error strings used for finding errors (pattern matching) in the build results. See server/config/example_job.errors.
  • -o html-file: Output file. If not specified, then stdout used.
  • -S buildserver
  • -P port
  • -j job-name
  • -b path to build command: i.e. /opt/Cabie/client
  • -r refresh: specified in seconds.
Example: /opt/Cabie/server/config/errStringPatterns.errors (or build-title.errors)
fatal error
Fatal error
error
Error
ERROR
Unexpected Error
1 error
cannot resolve symbol
Error while building files
Name assignment wrong
** Error
')' expected
'(' expected
gentoc.pl Perforce specific. Generates a complete list of files and versions used for a build.
buildfuncs.pl For use with Perforce and CVS CM systems. Get build configuration from database.
  • buildinfo: Generate html page (build.info.html)
  • removecomments: Remove "Information" comment records from database table "comments".
  • removegenerated: Archive build files. Supports Perforce and CVS sync with CM repository.

File permissions:

All perl scripts (.pl) and shell scripts (.sh) need execute file permissions. (i.e.: chmod +x *.pl) All other files require at least read privileges.

Build Numbers:

Build numbers can be supported in two ways:
  1. Cabie can obtain a build number by using the revision number supplied by Subversion by executing the command (in cmbroker.pm): "svn info repository-dir" Any attempt to re-run a build with the same build number will be aborted. Cabie first checks with the "jobs" table in the MySQL database for duplication with any previous build numbers. Duplication results in the refusal to perform a build. The absence of the file /share/Cabie/BUILDS/BuildServerHostname/projectA_job.build and the build server configuration file variable "RLCHG" in server/lib/your-host-name-goes-here.pm defines this behavior.
  2. Increment the build number based on a the major version number set in the build server configuration file server/lib/buildserver-host-name.pm variable "PRN" and a minor version number stored in a file. The major version number "PRN" can also be obtained from the file /share/Cabie/BUILDS/BuildServerHostname/projectA_job.version. This is used for CM systems which do not have a global revision number like Clearcase. The "minor" build number is stored in the file $TOP/$TITLE.build (i.e. /share/Cabie/BUILDS/BuildServerHostname/projectA_job.build). To use this methodology, create this file and insert the number "1". The existence of this file triggers its use. Removal of this file will cause Cabie to query CM for a build number. The build server configuration file boolean variable "INCREMENT" can be set to "1" to always increment the build number. The resulting build number will be "major-version-number.minor-version-number".

Job Descriptions:

The Cabie web interface provides a link to the job using the job name. The web CGI "genweb" will make a request to the build server for the build description. This will then be incorporated into the web page. The job descriptions are held in on the build server in /opt/Cabie/server/build_docs/projecA_job.txt

This can be interrogated later with a working system using the web interface and selecting the link with the job name (projectA_job) or the command: /opt/Cabie/client/build.pl instructions -n projectA_job

Example: /opt/Cabie/server/build_docs/projecA_job.txt
<h2>projectA_job Job Information</h2>

<b>Build and Development Info:</b> <a href="http://cm-server/cgi-bin/trac.cgi/wiki/projectA">Trac wiki</a>

<b>Subversions:</b> <a href="http://cm-server/svn/trunk/productA/">SVN productA Repository</a>
- <a href="http://cm-server/cgi-bin/trac.cgi/browser/trunk/projectA">Trac Subversion View</a>

<b>Bugs:</b> <a href="http://cm-server/cgi-bin/trac.cgi/report/1">Trac tickets</a>

<b>Permissions:</b> This repository is open for all in group "dev".

<b>Tree Etiquette"</b> Before you check in, please make sure that your code compiles and runs.

<b>Output:</b> Generated binaries and build logs are available
<a href="http://cm-server/Cabie/BUILDS/productA_job/">here</a>.

<b>Job description:</b> Build using Makefile.

<b>Questions:</b> <a href="mailto:dude@mail-server">Email Dude</a>

Job Configuration:

Set the build number on the build server in /share/Cabie/BUILDS/BuildServerHostname/projectX.build. This file contains a single integer number. Initialize to 1 to start. It will be incremented during each build.

[Potential Pitfall]: The build server process must be able to read and execute the scripts and read the information files:

chmod ug+x -R /opt/Cabie/server/jobs
chown buildmgr:buildmgr -R /opt/Cabie
This assumes that the server/buildserver.pl process is started as user "buildmgr".

Debugging:

  • The Cabie debug mechanism allows you to call _debug("Write your message here with $variable);. This will send a message to the debug log file specified in server/lib/hostname-of-build-server.pm. Look for the debug messages in /tmp/debug.out
  • The directory /opt/Cabie/proc/ is where Cabie writes STDOUT and STDERR along with connection and rejection logs. Look at these log files for errors.
  • Printing info using Perl:
    ...

    open(HNDL, ">>/tmp/cabiedbg.log");
    print HNDL "Debug message goes here. Variable= $var"
    close(HNDL);

    ...


SGI/IRIX Build Server Installation and Configuration:

I tried using the SGI and after market freeware CDs of open source software for MySQL and Perl but found that only the following worked:

  • Install support packages from SGI freeware CD's:
    (May 2004: http://freeware.sgi.com/)
    • fw_libz
    • fw_openssl
    • fw_readlines
  • MySQL: (client libraries required by perl module DBD::Mysql)
  • Perl: (Must compile from source for threads support required by Cabie.)
    • Download Perl source from http://CPAN.org
    • tar xjf perl-5.8.8-stable.tar.bz2
      (By contrast, RHEL4 uses Perl 5.8.5)
    • cd perl-5.8.8
    • sh Configure -Dcc='cc -n32' -Duseithreads -d
    • make
      (Uses native SGI IRIX make and cc compiler.)
      Note: fails test "stat" but seems to have had no apparent effect.
    • make install
      (Installs to /usr/local)
    • Set environment variables:
      • export PATH=/usr/local/mysql/bin:/usr/local/bin:/usr/freeware/bin:$PATH
        (Note: The command /usr/local/mysql/bin/mysql_config is required for the installation of perl module DBD::mysql)
      • export LD_LIBRARYN32_PATH=/usr/local/mysql/lib:/usr/local/lib:/usr/freeware/lib:$LD_LIBRARY32_PATH
    • Create Cabie perl path: ln -s /usr/local/bin/perl /usr/bin/perl
  • Perl Modules:
    • Download Perl modules tar packages from http://CPAN.org:
      (Install in this order.)
      • Mail-Sendmail-0.79.tar.gz
      • FreezeThaw-0.43.tar.gz
      • DBI-1.52.tar.gz
      • DBD-mysql-3.0006.tar.gz
      • Compress-Zlib-1.42.tar.gz
      • Archive-Zip-1.16.tar.gz
      • libwww-perl-5.805.tar.gz
        (Used to supply HTTP::Date Perl module)
    • Install:
      • Mail::Sendmail: (typical)
        • tar xzf Mail-Sendmail-0.79.tar.gz
        • cd Mail-Sendmail-0.79
        • perl Makefile.PL
        • make
        • make install
      • ...
      • DBD::mysql:
        • tar xzf DBD-mysql-3.0006.tar.gz
        • cd DBD-mysql-3.0006
        • perl Makefile.PL --libs="-L/usr/local/mysql/lib -lmysqlclient -lz -lgen -lnsl -lm"
          Where the arguments can be obtained from: /usr/local/mysql/bin/mysql_config --libs
          or
          perl Makefile.PL
          Where the environment variable PATH includes the MySQL command mysql_config
        • make
        • make install
      • ...


SuSE (9.3) Installation and Configuration:

See installation and configuration instructions above.


Red Hat Enterprise 4 Build Server Installation and Configuration:

See installation and configuration instructions above.


Red Hat Enterprise 3 Build Server Installation and Configuration:

The version of perl included with Red Hat Enterprise Linux 4 works with Cabie. The version of perl included with Red Hat Enterprise Linux 3 (RHEL3) does not! It is not threaded in the traditional manner as a new process is generated for each thread. If using RHEL3, download the perl RPM from Active State Perl. (The tar.gz bundle does not include many of the basic modules you need, download the RPM as this includes additional Perl modules.)

  • Install: rpm -ivh ActivePerl-5.8.8.817-i686-linux-2.2.17-gcc-257965.rpm
  • Remove old perl: rpm -e perl
  • Create link so "/usr/bin/perl" works: ln -s /opt/ActivePerl-5.8/bin/perl /usr/bin/perl

If using "Active State Perl" you will have to download the Perl database, network, email and file compression interfaces from CPAN.org. Install in the following order:

  • DBI:
    • Download DBI-1.52.tar.gz
    • tar xzf DBI-1.52.tar.gz
    • cd DBI-1.52
    • perl Makefile.PL
    • make
    • make install
  • DBD-mysql 3.0006
  • FreezeThaw 0.43
  • Mail-Sendmail 0.79
  • Compress-Zlib 1.10
  • Archive-Zip 1.16
The version numbers are given because the latest version of each will not work as some are too new and thus incompatible with this older release of RHEL.


MS/Windows Build Server Installation and Configuration:

Install MySQL client, Subversion client, Perl and the appropriate modules to support the Cabie build server.

MySQL Client:

Development only. Not required. Perl module DBD-mysql will give you the database access required.
Download MySql for MS/Windows (version 4.1 to be compatible with RHEL4) the installer from MySQL.com (Full install: mysql-4.1.21-win32.zip) During the installation you will be queried for the components to install. Choose "Client software".

Subversion Client:

Download from http://subversion.tigris.org (svn-win32-X.X.X.zip)

Unzip archive into C:\SVN\. Add C:\SVN\bin to your system path. Path must match that used in server/lib/cmbroker.pm for svn.exe if path is fully qualified.

Perl:

Download Active State Perl ".msi" package for MS/Windows:
ActivePerl-5.8.8.819-MSWin32-x86-267479.msi
This MS/Windows installation package will include the "Perl Package Manager" (ppm)

Additional modules are required and can be (1) downloaded as PPM's and installed or (2) installed from an internet connection.

  1. Download and install PPMs:

    Unzip the ppm packages and install: Start the Perl Package manager console (command "ppm") and enter the command:

    install C:\....\package-name.ppd
    or from the command line console (cmd):
    ppm install C:\....\package-name.ppd

  2. Install from internet archive:
    Start the Perl Package manager console (command "ppm") and enter the commands:
    • ppm> install DBI
    • ppm> install DBD-mysql
    • ppm> install FreezeThaw
    • ppm> install Mail-Sendmail
    • ppm> install http://www.roth.net/perl/packages/win32-daemon.ppd
    • ppm> install http://www.roth.net/perl/packages/win32-adminmisc.ppd
    The first four packages will be obtained from the default archive - ActiveState.com.

    If using a proxy, set the environment variable http_proxy to http://name-of-proxy-server:80

    Check installation with the following commands:

    • View list of all installed packages: ppm> query *
    • View list of family of installed packages: ppm> search mysql

Job scripts:

Sample CMD batch scripts for MS/Windows:

Environment variables for use with the MS/Visual Studio compiler can be found in:

  • MS/Visual Studio .NET 2003: C:\Program Files\Microsoft Visual Studio .NET 2003\Common7\Tools\vsvars32.bat
  • MS/Visual Studio 8.0: C:\Program Files\Microsoft Visual Studio 8\VC\vcvarsall.bat
Also see the YoLinux DOS / bash command comparison

MS/Windows build server:

The build server must first be installed as a service:
(You must be logged in as an Administrator to perform this command.)
C:\Cabie\server\buildserver.pl -s 8091 -m 8092 -i
Once installed as a service, and you find that the build server needs to be started manually, select the following:
  1. Start
  2. Control Panel
  3. Administrator tools
  4. Services
    • There will be two build server services. Select each service individually.
    • Right click and select "Start"
To Remove the buildserver from the list of services, run the command C:\Cabie\server\buildserver.pl -s 8091 -m 8092 -r


Apache Web Server Configuration and Cabie CGI:

The Cabie web interface is a CGI interface to the Cabie data residing in a MySQL database (in this case on the same server: localhost)

Cabie CGI:

Edit the following files:
  • Main interface: /opt/Cabie/web/cgi-bin/genweb
    (Also: server/buildserver.pl, web/cgi-bin/: status.pl, gencomment, defectfix, jobinfo and server/lib: cabbieaddon.pm, hostname-of-buildserver.pm)
    • Edit/change lines which refer to "/icons" to "/CabieIcons/"
    • Edit/change lines which refer to "/cgi-bin/" to "/Cabie-cgi-bin/"
    • Edit/change lines which refer to "/main/static" to "/Cabie/static/"
    • See bugs by running at the command line. Genweb will complain as there are sure to be loads of undefined parameters which can be set in /opt/Cabie/server/lib/host-name.pm for an element of "%fields".
  • /opt/Cabie/web/cgi-bin/status.pl
    • Define reference to PWEBSERVER in "%fields"..
      (Used by Perforce CM system. Even if not used set it to something.)
  • /opt/Cabie/web/cgi-bin/jobinfo
    • Change path of "/opt/bin/build" to reflect actual path: /opt/Cabie/client/build.pl and arguments "-s name-of-build-server -p 9190 describe -n nameOf_job"
    • Set /opt/Cabie/web/icons/sct_logo.gif to your corporate logo.
  • Files missing from Cabie tar file: I noticed that some of the files missing from the Cabie tar file are available from the Cabie CVS repository on Tigris.org. (i.e. web/icons/logo.gif, web/icons/cabie.gif, ...)
Allow Apache web server access to CGI and files:
  • chown apache:apache -R /opt/Cabie/web
  • chmod ug+x -R /opt/Cabie/web/cgi-bin

Tip: CGI can be debugged by looking at the errors in : /var/log/httpd/error_log

File:

  • Red Hat/Fedora Core/CentOS4: /etc/httpd/conf.d/cabie.conf
  • SuSE 9.3: /etc/apache2/conf.d/cabie.conf
Alias /Cabie/BUILDS "/share/Cabie/BUILDS/"
<Directory "/share/Cabie/BUILDS/">
Options Indexes FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>

Alias /main/ "/opt/Cabie/web/htdocs/main/"
<Directory "/opt/Cabie/web/htdocs/main">
Options Indexes FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>

Alias /CabieIcons/ "/opt/Cabie/web/icons/"
<Directory "/opt/Cabie/web/icons">
Options Indexes FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>

ScriptAlias /Cabie-cgi-bin/ "/opt/Cabie/web/cgi-bin/"
<Directory "/opt/Cabie/web/cgi-bin">
Options Indexes FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>

Restart web server: service httpd restart

The Cabie URL: http://name-of-cabie-web-server/Cabie-cgi-bin/genweb
(to view jobs)

Cabie directories:

The build server and web server view the shared file system from two perspectives.
  • Build server: Constructs path based on data held in the database. It uses the table "configuration" and the fields "top" (top directory for build) and "title" (project title). It generates results files based on the path constructed from putting together /"top"/"title"/BuildNumber
    The build server can have a unique path for each build as each build has a unique database record.
  • Web server: Constructs the a path for each build but must use the same formula for each build. The path is constructed from: /MOUNTROOT(specified in /opt/Cabie/server/lib/hostname.pm)/buildserver-hostname/"title"/BuildNumber
The build server will create files in this location and the web server must view this location to serve files from it. Sometimes when complex directory paths are used, a soft link must be created so the view of the file system from the web server can be constructed properly.
ln -s /top/title /MOUNTROOT/hostname/title
This link allows the web server cgi program genweb to find files in the expected location. Use actual text values represented by the variable placeholders. If the two are the same, then obviously no link is needed.

The same effect may be achieved by selecting a mount point to meet this criteria.

[Potential Pitfall]: If using SELinux, you will have a lot to configure. I suggest turning this mode off. See YoLinux Apache Configuration: SELinux


Using Cabie:

  1. Add authorization privileges to database: Chicken-Egg problem - The Cabie build server can not run to accept the "authorize" command if it is not first authorized to run. Use a SQL insert to make the table entry:
    mysql -h localhost -u root
    mysql> use builds
    mysql> insert into authtable values('build-server','client-machine');
    mysql> quit

    You must also include an entry for the Cabie web server for each build server as it acts as a client to each build server.

    Test: /opt/Cabie/client/build.pl -s build_server -p 9190 authorized

    [Potential Pitfall]: You may have to use the fully qualified name of the computer. i.e. "buildserver1" will not be authorized if the network defines the computer as "buildserver1.megacorp.com".

  2. Start the build server daemon on the build server:
    • Linux/Unix:
      • cd /opt/Cabie/server
        (Run from server directory so that perl paths to library work.)
      • /opt/Cabie/server/buildserver.pl -s 9190 -m 9192 -d
      or use system init script shown below: service cabie start
    • MS/Windows:
      • cd C:\Cabie\server
      • C:\Cabie\server\buildserver.pl -s 8091 -m 8092 -i
      [Potential Pitfall]: The server and monitor ports are hard coded in /opt/Cabie/server/lib/winsys.pm On MS/Windows you must use the hard coded values. We never got user specified values to work on MS/Windows.
    • -s: Server port (default 8091)
    • -m: Monitor port (default 8092)
    • -d: Create server daemon (Linux/Unix)
    • -i: Install MS/Windows service. (You must have admin privileges.) Service registration allows for a permanent configuration where the build server will start/stop upon system boot/shutdown.
    • -r: Remove MS/Windows service.
    Upon starting the build server, it will add a record to the table "buildservers".
    Note: Look for progress and errors in the buildserver log files located in /opt/Cabie/proc/.

  3. Test build server using the following client commands:
    • List supported commands: /opt/Cabie/client/build.pl -s build-server -p 9190 commands -l -w
    • Show server daemon process id: /opt/Cabie/client/build.pl getid
      (Use environment variable or specify "-s" and "-p" as above.)
    • Show server configuration: /opt/Cabie/client/build.pl dumpconfig
    Note: You can specify the build-server and port using an environment variable: export BLDSERVER=build-server:port

  4. Define a build job using Cabie client /opt/Cabie/client/build.pl:
    • Define system configuration:
      • /opt/Cabie/client/build.pl authorize -c server-1
      • /opt/Cabie/client/build.pl authorize -c server-2
      • ...
    • One must first create the job build scripts (prebuild.sh, rtlbuild.sh, ...) as shown above before a build job can be operational.

    • One must then define the build job:
      /opt/Cabie/client/build.pl -s build-server -p9190 createjob -n projectA_job -p http://cm-server:80/svn/ -c projectA -r /share/Cabie/BUILDS/BuildServerHostname -t retail -d "/opt/bin" -k 10 -s subversion -b CM-URL -m 1 -C "Job Description"
      Command argument Description
      -n job-name Name of job. i.e.: projectA_job
      -p port-number

      -p svn-sub-dir

      build-server:port

      Subversion: Adds this info to mouse-over data.

      -c p4-port
      or
      -c projectA
      Subversion: Subdirectory under "source root" /share/Cabie/BUILDS/BuildServerHostname which is the Subversion local buildserver repository.
      -r source-root CM repository root directory. Subversion local repository parent directory.
      -t retail Build type: debug | retail | both.
      -d /path/to/tools Path to build tools to be included in path: ie: /opt/bin (Win32: C:\SVN\bin)
      -k keeplevel How many to keep in database: i.e. 10
      -s subversion CM repository type: perforce, cvs or subversion
      -b URL CM repository URL: http://cm-server/...
      -m 1 Build server failure notification: 1 or 0
      -C "comment" Comment to describe build.
      Note:
      • Command arguments MUST be given in the same order as above or you will get an error complaining.
      • Verify entry using the Cabie client command: /opt/Cabie/client/build.pl describe -n projectA_job
      • Defining a job with "createjob" is run once. It can not be re-run with the same job name. Once can:
        • Delete the job definition and then create a new job definition.
        • Edit an existing job definition in the mySQL database.
        • Create a new job definition with a new job name.

      The build job definition will be inserted into the "configuration" table:

      FieldExample value
      servernameOfBuildserverHostname
      titleProjectX_job
      porthttp://webserver.com:80/svn/stuff/
      clientProjectX
      top/opt/Cabie/BUILDS/buildserver/svn/stuff/local/working/dir/projectX
      typeretail
      toolsdir/opt/bin
      isimakeno
      keeplevel10
      commentProjectX comments goes here
      sccssubversion
      browserlinkhttp://webserver.com/trac/browser/trunk/working/dir/projectX
      state0
      spam1
      buildsize1

      [Potential Pitfall]: The error "Unsupported request" can result from a client which is not authorized to submit the "createjob" command (or any other command). See step above to create an entry into the mySQL authorization table "authtable".

    • The build request must then be defined in the database. If you are not in polling mode, you can manually issue a build command to the build server by issuing the client command:
      /opt/Cabie/client/build.pl -s build-server -p9190 build -n projectA_job

      [Potential Pitfall]: If you find or suspect that the client can not connect to the build server, test the connection with the telnet command: telnet builServerHostname 9190

      • OK:
        Connected to builServerHostname
        Escape character is '^]'.
                
      • No connection:
        telnet: connect to address XXX.XXX.XXX.XXX: Connection refused
        telnet: Unable to connect to remote host: Connection refused

      Solution: Remove the host name assignment to the localhost IP address "127.0.0.1

      • Linux/Unix: /etc/hosts
      • MS/Windows: C:\WINDOWS\system32\drivers\etc\hosts

      Hosts file: (even if using DNS)

      • OK:
        127.0.0.1        localhost.localdomain  localhost
        192.168.XXX.XXX buildserver1
      • Not OK:
        127.0.0.1        buildserver1 localhost.localdomain  localhost
        192.168.XXX.XXX buildserver1
      • Not OK: (MS/Windows default)
        127.0.0.1        localhost
      Kill and restart the buildserver.pl process if you have performed this change.

      The network information command: netstat -punta | grep 9190 can also show if the build server is attached to 127.0.0.1 only and thus not accepting network connections.

  5. View progress via the web:
    • See status and interact with Cabie: http://web-server/genweb
    The web page will show the nine stages of progress in a color coded scheme:
    Job Idle
     
    Not listed in database.
    Syncing

    Creating build environment and performing CM update (cmbroker.pm::sccs_update()). Includes 30 sleep.
    Building

    Script server/jobs/projectA_job/rtlbuild.sh launched.
    Overdue

    Builds lasting over 150% of average build time.
    Generating Email


    Build Untested

    Build complete. Next: check with postbuild.sh. Results of "make" OK
    Table: jobs, Field: status=0
    Build (compile/link) Passed Test

    No build test errors according to postbuild.sh
    Table: jobs, Field: status=3
    Build Failed

    Errors (compile or link) found in build by postbuild.sh
    Table: jobs, Field: status=2
    Failed Test

    Run postpostbuild.sh regression test on application. Test results not good.
    Table: jobs, Field: status=1

    Note: The first few builds generated by Cabie will be listed as "overdue" for part of the duration of the build. This is because there are no build times or an insufficient number of build times stored in the database. Once data on a build is stored in the system, Cabie will have build times which can be used for judging build duration.

    Look for CGI errors in /var/log/httpd/error_log.

The Cabie build server can be terminated with the command: /opt/Cabie/client/build.pl shutdown


[Potential Pitfall]: The case (upper vs lower) of the hostname MUST be consistent throughout the configuration. The host name used in the job definition will be used to define the directory path of the builds directory as viewed by the web server. Entries in the database and the hostname used to define the configuration file /opt/Cabie/server/lib/hostname.pm and Perl module are all tied together in a single configuration. Consistency is important.

[Potential Pitfall]: Database cleanup for a build which gets stuck or for a build in an inconsistent state (i.e. if a build server is shut down during a build, the database will be in an inconsistent state. Cleanup as follows:
Connect to the MySQL database: mysql -h localhost -u buildmgr -p

  • delete from joblog where server='buildsrvrx' and title='ProjectX';
  • delete from jobs where server='buildsrvrx' and title='ProjectX' and job='1027';
  • delete from semaphores where server='buildsrvrx' and title='ProjectX';
  • delete from restore where server='buildsrvrx' and title='ProjectX';
  • delete from proctree where server='buildsrvrx' and title='ProjectX';
This cleanup can also be performed with the Cabie client commands:
  • cd /opt/Cabie/server
  • /opt/Cabie/client/build.pl -s buildsrvrx -p 9190 cleanjob -n ProjectX_job -j job-number-to-purge -f


Cabie and email notification:

Create a list of users who will receive email notification:

login-name-1:email-1@domain.com:FirstName1 LastName1
login-name-2:email-2@domain.com:FirstName2 LastName2

...
..
CM path is set in server/lib/cmbroker.pm in subroutine "subversion_useraddress".
This file (users.txt) will be under CM control and access or use: /usr/bin/GET http://subversion-web-server/users.txt to access the file in subroutine "subversion_useraddress".


Configuring for a nightly build:

This is when "Continuous Automated Build" mode is turned off by removing the reference to a job from server/poll.txt.
The cron job can manually submit the job.

Cron script: /etc/cron.daily/cabie

#!/bin/sh
/opt/Cabie/client/build.pl -s build-server -p9190 -n projectA_job /opt/Cabie/client/build.pl -s build-server -p9190 -n projectB_job
This could also be added to a CM check-in trigger for a continuous build system.

See the YoLinux.com Tutorial on scheduling jobs.


Init Script:

The build server can be started upon system boot with the following init script. See YoLinux Init Tutorial for configuration.

File: /etc/init.d/cabie
#!/bin/bash
#
# cabie This starts and stops cabie build server.
#
# chkconfig: 345 98 35
# description: The Cabie build server is an automated distributed build system.
#

# Source function library.
. /etc/init.d/functions

# Get config.
test -f /etc/sysconfig/network && . /etc/sysconfig/network

# Check that networking is up.
[ "${NETWORKING}" = "yes" ] || exit 0

[ -f /opt/Cabie/server/buildserver.pl ] || exit 1
[ -f /opt/Cabie/client/build.pl ] || exit 1

prog="Cabie build server"

start(){
echo -n $"Starting $prog: "

ulimit -S -c 0 >/dev/null 2>&1
RETVAL=0
initlog $INITLOG_ARGS -c \
"/opt/Cabie/server/buildserver.pl -s 9190 -m 9192 -d"
RETVAL=$?
[ "$RETVAL" -eq 0 ] && success $"Cabie build server startup" || \
failure $"cabie start"
echo
[ "$RETVAL" -eq 0 ] && touch /var/lock/subsys/cabie
return $RETVAL
}

stop(){
echo -n $"Stopping $prog: "
initlog $INITLOG_ARGS -c \
"/opt/Cabie/client/build.pl -s `hostname` -p 9190 shutdown >/dev/null 2>&1"
RETVAL=$?
[ "$RETVAL" -eq 0 ] && success $"Cabie build server shutdown" || \
failure $"Cabie buildserver shutdown"
echo
[ "$RETVAL" -eq 0 ] && rm -f /var/lock/subsys/cabie
return $RETVAL
}

# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
restart|reload)
stop
sleep 3
start
;;
condrestart)
if [ -f /var/lock/subsys/cabie ]; then
stop
sleep 3
start
fi
;;
status)
status buildserver.pl
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart|reload}"
RETVAL=1
esac

exit $RETVAL

Register the service: chkconfig --add cabie
or: chkconfig --level 35 cabie on

Manually start service: service cabie start

[Potential Pitfall]: The above init script does not work for Red Hat Enterprise 3. (Works for RHEL4) For some unknown reason, this script when executed by the "service" command (service cabie start) on RHEL3, will cause the build server to launch four processes instead of the regular two buildserver processes (server and monitor process) thus use the script linked below instead.

SGI IRIX init script: /etc/init.d/cabie
SuSE Linux init script: /etc/init.d/cabie
RHEL3 Linux init script: /etc/init.d/cabie

Be sure to set execute privileges on the script:

chmod +x /etc/init.d/cabie


Cron Jobs To Recycle Build Server:

I have found it necessary to recycle the build scripts weekly using a cron job. Be sure to set execute privileges on the script:

chmod +x /etc/cron.weekly/cabie.sh

File: /etc/cron.weekly/cabie.sh (Red Hat Enterprise 4)

#!/bin/bash
/etc/init.d/cabie stop

# Ensure jobs have terminated even if stubborn
ps axwww | grep "buildserver" | awk '{print $1 " " $3} | \
grep Sl | awk '{print $1}' | \
( while read pid; do kill -9 $pid; done )

/etc/init.d/cabie start

File: /etc/cron.weekly/cabie.sh (Red Hat Enterprise 3)

#!/bin/bash
/etc/init.d/cabie stop

# Ensure jobs have terminated even if stubborn
ps axwww | grep "buildserver" | awk '{print $1}' | \
( while read pid; do kill -9 $pid; done )

/etc/init.d/cabie start


Perl Tips:

  • Check perl script syntax: perl -c perl-script.pl
    Should respond with: ./perl-script.pl syntax OK

  • Verify Perl modules required for Cabie with the following scripts:
    • testFreezeThaw.pl
      #!/usr/bin/perl
      eval "use FreezeThaw"; $hasModule = $@ ? 0 : 1;
      printf "FreezeThaw". ($hasModule ? "" : " not") . " installed";
      printf "\n";
    • testMailSendmail.pl
      #!/usr/bin/perl
      eval "use Mail::Sendmail"; $hasModule = $@ ? 0 : 1;
      printf "Mail::Sendmail". ($hasModule ? "" : " not") . " installed";
      printf "\n";
    • etc ...

  • Perl library paths: One may add the fully qualified path name:
    BEGIN {
    push @INC, "/opt/Cabie/server/lib";
    }
    or specify the path: /usr/bin/perl -I/opt/Cabie/server/lib perl-script.pl

    MS/Windows Active Perl path:

    BEGIN {
    push @INC, "C:/Perl/site/lib";
    }

  • When debugging the Perl CGI script web/cgi-bin/genweb, the text delivered to the web browser is in 512k blocks from the buffered I/O. The latest statements printed may never show up in the browser because they are stored in the I/O buffer until the buffer is full, at which time it is flushed. This makes it hard to diagnose problems with print statements. Flush the buffer after writing out debugging statements with the Perl directive "$|=1". This will flush the stdout buffer.

  • YoLinux.com Tutorial: SysAdmin - Perl Administration


Links:


Books icon Books:

publisher cover Better Software Magazine

Free subscription for software professionals who care about quality. Each issue brings you relevant, timely information to help you build better software. Continuing to deliver in-depth articles on testing, tools, defect tracking, metrics, and management.

Free
Subscription

Copyright © 2006 - 2009 by Greg Ippolito