Learn Python: Automate your routine with a python script

In two of the previous posts, show Git branch on your Linux prompt and Git colored output, we’ve learned how to make work on your Debian/Ubuntu Linux with Git more usable.  It would be useful to have some script that does this for you to save you or someone else some time in the future. My idea is to script it with Python!

This article is a tutorial with an example of the usage of various basic Python modules. You can use parts of it as building blocks for your own automation needs.

Step 1: Create dist files to append .bashrc and .gitconfig

First, we need to create two empty files and put in a few lines of code:

# Add git branch if its present to PS1
parse_git_branch() {
 git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/(\1)/'
}
if [ "$color_prompt" = yes ]; then
 PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[01;31m\] $(parse_git_branch)\[\033[00m\]\$ '
else
 PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w $(parse_git_branch)\$ '
fi

bashrc.dist – to display Git branch name on your command prompt.

[alias]
lg=log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit --date=relative
st=status
ci=commit
co=checkout

[color]
branch=auto
diff=auto
status=auto

[color "branch"]
current=yellow reverse
local=yellow
remote=green

[color "diff"]
meta=yellow bold
frag=magenta bold
old=red bold
new=green bold

[color "status"]
added=yellow
changed=green
untracked=cyan

gitconfig.dist – to use shorter commands when operating Git.

Step 2: Display Git branch on command prompt

Let’s write a simple Python script. We will read/write files, search with regular expressions  and install package (colordiff) using three modules: os (miscellaneous operating system interfaces)re (regular expressions) and subprocess (subprocess management) .

The example below displays the first task for script. It opens your local .bashrc (if it exists), executes method show_branch_in_shell , which modifies bashrc content with code we stored in bashrc.dist beforehand. If any content changes, the script will overwrite the existing file.

#!/usr/bin/python -tt

import re
import os
import subprocess

# get user home directory
home_dir = os.path.expanduser('~')
# path to distfiles
bashrc_dist_filename = 'bashrc.dist'
gitconfig_dist_filename = 'gitconfig.dist'
# We will identify if our addon exists by this comment
git_prompt_regexp = r'# Add git branch if its present to PS1'
# flag for content change
content_changed = False

def show_branch_in_shell(bashrc_content):
    global content_changed

    # modify bashrc_conent only if regexp is not match
    if not re.search(git_prompt_regexp, bashrc_content):
        bashrc_dist_handler = open(bashrc_dist_filename, 'r')
        dist_content = bashrc_dist_handler.read()
        bashrc_dist_handler.close()
        bashrc_content = bashrc_content + '\n' + dist_content
        content_changed = True

    return bashrc_content

def main():
    path_bash = os.path.join(home_dir, '.bashrc')
    if not os.path.isfile(path_bash):
        print('bashrc is not found')
        exit(1)

    # read local .bashrc file
    file_handler = open(path_bash, 'rU')
    bashrc_content = file_handler.read()
    file_handler.close()

    # execute changes
    bashrc_content = show_branch_in_shell(basrc_content)

    if content_changed:
        # save changed bashrc
        file_handler = open(path_bash, 'w')
        file_handler.write(bashrc_content)
        file_handler.close()
if __name__ == '__main__':
    main()

Step 3: Enable Git color output

Now we have script with one method implemented. Our next step is to enable color prompt in bashrc. You can do it manually by setting the force_color_prompt parameter value to ‘yes’. Let’s modify our script. It is good to use regular expressions to find if it is enabled or not:

def color_prompt_enable(bashrc_content):
    # Searches for commented out "force_color_prompt" and enables it
    return re.sub(r'#\s*force_color_prompt\s*=\s*yes', r'force_color_prompt=yes', bashrc_content)

After that, we need to have colordiff package installed on our Linux. Here is implementation of install_colordiff method:

def install_colordiff():
    # check if colordiff is installed
    out = subprocess.check_output(['apt-cache', 'policy', 'colordiff'])

    if re.search(r'Installed:\s*\(none\)', str(out)):
        # install with "sudo" if not exists
        subprocess.call(['sudo', 'apt-get', 'install', 'colordiff'])
        print('colordiff has been installed')

Step 4: Add useful Git aliases

We’re almost finished. Last step is to add a few useful Git aliases. For this, we will need to read content from our gitconfig.dist and append it in your ~/.gitconfig file.

def add_git_aliases():
    # make absolute path for new .gitconfig in user home directory
    home_gitconfig_path = os.path.join(home_dir, '.gitconfig')
    if not os.path.exists(home_gitconfig_path):
        # create .gitconfig file if it doesn't exists
        subprocess.call(['cp', gitconfig_dist_filename, home_gitconfig_path])
        print('.gitconfig file has been created')
        return

In this example, I assume I do not have ~/.gitconfig or have it empty. You might want to perform additional logic to merge ditconfig.dist with your existing config. In this case we will use Python module called configparser (configuration file parser). Let’s import this module to our script and modify add_git_aliases() function:

# in case of using Python2 use ConfigParser instead of configparser
import configparser

def add_git_aliases():
    # make absolute path for new .gitconfig in user home directory
    home_gitconfig_path = os.path.join(home_dir, '.gitconfig')
    if not os.path.exists(home_gitconfig_path):
        # create .gitconfig file if it doesn't exists
        subprocess.call(['cp', gitconfig_dist_filename, home_gitconfig_path])
        print('.gitconfig file has been created')
        return

    # optional (if exists, merge 2 configs and write the result)
    config = configparser.ConfigParser()
    config.read(home_gitconfig_path)
    config.read(gitconfig_dist_filename)

    # set up config file name
    config.filename = '.gitconfig'

    # write merged config to disk
    file_handler = open(home_gitconfig_path, 'w')
    config.write(file_handler)
    file_handler.close()

    print('.gitconfig file has been updated\n')

Step 5: Complete script

Now we have all the methods implemented we need to finish the script. Add those methods in a sequence execution and it is complete.

# execute changes
install_colordiff()
add_git_aliases()
bashrc_content = show_branch_in_shell(
    color_prompt_enable(bashrc_content)
)

Basic implementation is now finished. Feel free to add your own improvements to the script, such as more Linux distributives support, version control, custom useful aliases, and share this experience with other users!

With you newly gained knowledge and experience in Python you can now create your own Linux system administration scripts. Let me know what you came up with, please share them using the comments.

Source references:

  1. Script source on GitHub
  2. Python operating system interfaces module documentation
  3. Python regular expressions module documentation
  4. Python subprocess module documentation
  5. Python configparser module documentation
Share

Testing your project with PHPUnit and Selenium

PHPUnit is a very handy tool you can use to test and detect possible mistakes in your PHP project.

Functional testing is also possible with PHPUnit using default headless browser (crawler). It works perfect as long as you don’t need to test functionality which uses JavaScript. For this purpose, you can use PHPUnit and Selenium.

Selenium is a powerful test tool that allows you to perform tests for web applications in any programming language using any mainstream browser.

Installation requirements

Before you start using Selenium to test your PHP project, you need to install the following:

  1. Selenium Server
  2. PHPUnit_Selenium package

1. Installing Selenium Server

Perform the following steps to install the Selenium Server:

    1. Download a distribution archive of Selenium Server.
    2. Unzip the distribution archive and copy selenium-server-standalone-2.35.0.jar (check the version suffix) to a folder from where you will run the server (such as /usr/local/bin).
    3. Start the Selenium Server by running the following command:
java -jar /usr/local/bin/selenium-server-standalone-2.35.0.jar

2. Installing PHPUnit_Selenium package

      The PHPUnit_Selenium package is necessary for natively accessing the Selenium Server from PHPUnit.
      To install it, run the following command:
pear install phpunit/PHPUnit_Selenium

Using Selenium in PHPUnit tests

There are two Selenium test cases:

    • PHPUnit_Extensions_Selenium2TestCase
    • PHPUnit_Extensions_SeleniumTestCase

PHPUnit_Extensions_Selenium2TestCase test case allows you to use the WebDriver API (partially implemented).

<?php
class WebTest extends PHPUnit_Extensions_Selenium2TestCase
{
    protected function setUp()
    {
        $this->setBrowser('firefox');
        $this->setBrowserUrl('http://www.example.com/');
    }

    public function testTitle()
    {
        $this->url('http://www.example.com/');
        $this->assertEquals('Example WWW Page', $this->title());
    }

}
?>

PHPUnit_Extensions_SeleniumTestCase test case implements the client/server protocol to talk to Selenium Server as well as specialized assertion methods for web testing.

<?php
require_once 'PHPUnit/Extensions/SeleniumTestCase.php';

class WebTest extends PHPUnit_Extensions_SeleniumTestCase
{
    protected function setUp()
    {
        $this->setBrowser('*firefox');
        $this->setBrowserUrl('http://www.example.com/');
    }

    public function testTitle()
    {
        $this->open('http://www.example.com/');
        $this->assertTitle('Example WWW Page');
    }
}
?>

Using different browsers with WebDrivers

For running tests with different browsers you should have WebDriver of that browser. As an example, we’ll try to use the Google Chrome browser and its WebDriver.

  1. Download WebDriver.
  2. To make Selenium Server aware of this WebDriver, perform one of the following tasks:
    • Store the Chrome WebDriver binary in the system path
    • Start the Selenium Server with -Dwebdriver.chrome.driver=path/to/your/chromedriver

You can now set Chrome as a browser for your functional test:

...
$this->setBrowser('chrome');
...

Running headless Selenium Server with Xvfb

Sometimes you don’t want the browser to be launched at your desktop during testing, or you may not be using Xserver. To make it work, one of the easiest solutions is to use Xvfb.

Xvfb is an X11 server that performs various graphical operations in memory, without displaying any screen output.

So, let’s try it. At first ensure that you have Xvfb installed on your server. If not, you can install it from your OS repository. For example, to install it on Ubuntu, run the following command:

sudo apt-get install xvfb

Once installed, to run your Selenium Server in Xvfb, run the following command:

DISPLAY=:1 xvfb-run java -jar selenium-server-standalone-2.35.0.jar

Once the server starts, run any of the test case examples. The output should display as follows:

PHPUnit 3.8.0 by Sebastian Bergmann.

F

Time: 4 seconds, Memory: 6.25Mb

There was 1 failure:

1) WebTest::testTitle
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'Example WWW Page'
+'Example Domain'

/your/project/Tests/Functional/WebTest.php:14

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

That’s all for now. Enjoy testing!

Source references:

  1. PHPUnit and Selenium (official documentation)
  2. Selenium Server downloads list
Share

Continuous Integration for Symfony2 using Jenkins

Today I will show how to set up Continuous Integration (CI) for Symfony2 using Jenkins and the PHP-template for Jenkins jobs by Sebastian Bergmann.

This configuration works successfully with Symfony ver. 2.1 and Jenkins ver. 1.480.1.

Installing requirements

Before creating template jobs you need to install the Required Jenkins Plugins and Required PHP Tools.

Using the Symfony2 PHP Jenkins job template

I based the first steps on http://jenkins-php.org/, but I will use config.xml, build.xml, phpunit.xml and additional files from this website: https://github.com/xurumelous/symfony2-jenkins-template.

  1. Fetch the jenkins-cli.jar from your Jenkins server
    wget http://localhost:8080/jnlpJars/jenkins-cli.jar
  2. Download and install the job template
    curl https://github.com/xurumelous/symfony2-jenkins-template/blob/master/config.xml | \
    java -jar jenkins-cli.jar -s http://localhost:8080/jenkins create-job symfony2-php-template

    or add the template manually:

    cd $JENKINS_HOME/jobs
    mkdir symfony2-php-template
    cd symfony2-php-template
    wget https://github.com/xurumelous/symfony2-jenkins-template/blob/master/config.xml
    cd ..
    chown -R jenkins:jenkins symfony2-php-template/
  3. Reload Jenkins’ configuration, for instance using the Jenkins CLI:
    java -jar jenkins-cli.jar -s http://localhost:8080 reload-configuration
  4. Click on “New Job”
  5. Enter the “Job name”
  6. Select “Copy existing job” and enter “symfony2-php-template” into the “Copy from” field.
  7. Click “OK”
  8. Configure your new job with version control and whatever other fields you need.

Project configuring and issues fixing

  1. Make changes from the basic Jenkins-PHP config to Symfony2-Jenkins-PHP as described here: https://github.com/xurumelous/symfony2-jenkins-template
    • Move the Jenkins folder to [SYMFONY2_ROOT]/app/Resources/ inside your Symfony2 project
    • Move build.xml to the root folder of your Symfony2 application
    • Move phpunit.xml to [SYMFONY2_ROOT]/app folder or update the existing one. The logging node is needed!
  2. If you get the following message: PHP Fatal error:  Class ‘XSLTProcessor’ not found in /usr/share/php/TheSeer/fXSL/fxsltprocessor.php on line 58, you can fix  it with 
    sudo apt-get install php5-xsl
  3. You might also get this error message, because we run Symfony 2.1, which needs a composer: PHP Warning:  require(/var/lib/jenkins/jobs/TestJob/workspace/app/../vendor/autoload.php): failed to open stream: No such file or directory in /var/lib/jenkins/jobs/TestJob/workspace/app/autoload.php on line 5 . You can fix this problem like this:
  4. Sometimes you encounter a problem on the “vendors” step, saying something like this:

    [exec] The deps file is not valid ini syntax. Perhaps missing a trailing newline?
    [exec] PHP Warning:  parse_ini_file(/var/lib/jenkins/jobs/TestJob/workspace/deps): failed to open stream: No such file or directory in  /var/lib/jenkins/jobs/TestJob/workspace/bin/vendors on line 69

    You can fix this by removing “vendors” from build dependency and the “vendors” target in build.xml

  5. If you get a Status: 2 during the codesniffer step (phpcs) you need to install Symfony2 coding standard https://github.com/opensky/Symfony2-coding-standard
  6. Enable checkbox “Poll SCM” and write: */5 * * * * (This step makes Jenkins check every 5 minutes if there are changes in the repository. If any changes are found, it will automatically build a project)

That’s it! You are now ready to build your PHP Symfony2 project with Jenkins! If you find bugs (or fixes), links, or have other suggestions, do not hesitate to comment below.

Enjoy the result!

Share