Learn Python: Automate your routine with a python script

In two of the previous posts, show Git branch on your Linux prompt and Git colored output, we’ve learned how to make work on your Debian/Ubuntu Linux with Git more usable.  It would be useful to have some script that does this for you to save you or someone else some time in the future. My idea is to script it with Python!

This article is a tutorial with an example of the usage of various basic Python modules. You can use parts of it as building blocks for your own automation needs.

Step 1: Create dist files to append .bashrc and .gitconfig

First, we need to create two empty files and put in a few lines of code:

# Add git branch if its present to PS1
parse_git_branch() {
 git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/(\1)/'
}
if [ "$color_prompt" = yes ]; then
 PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[01;31m\] $(parse_git_branch)\[\033[00m\]\$ '
else
 PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w $(parse_git_branch)\$ '
fi

bashrc.dist – to display Git branch name on your command prompt.

[alias]
lg=log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit --date=relative
st=status
ci=commit
co=checkout

[color]
branch=auto
diff=auto
status=auto

[color "branch"]
current=yellow reverse
local=yellow
remote=green

[color "diff"]
meta=yellow bold
frag=magenta bold
old=red bold
new=green bold

[color "status"]
added=yellow
changed=green
untracked=cyan

gitconfig.dist – to use shorter commands when operating Git.

Step 2: Display Git branch on command prompt

Let’s write a simple Python script. We will read/write files, search with regular expressions  and install package (colordiff) using three modules: os (miscellaneous operating system interfaces)re (regular expressions) and subprocess (subprocess management) .

The example below displays the first task for script. It opens your local .bashrc (if it exists), executes method show_branch_in_shell , which modifies bashrc content with code we stored in bashrc.dist beforehand. If any content changes, the script will overwrite the existing file.

#!/usr/bin/python -tt

import re
import os
import subprocess

# get user home directory
home_dir = os.path.expanduser('~')
# path to distfiles
bashrc_dist_filename = 'bashrc.dist'
gitconfig_dist_filename = 'gitconfig.dist'
# We will identify if our addon exists by this comment
git_prompt_regexp = r'# Add git branch if its present to PS1'
# flag for content change
content_changed = False

def show_branch_in_shell(bashrc_content):
    global content_changed

    # modify bashrc_conent only if regexp is not match
    if not re.search(git_prompt_regexp, bashrc_content):
        bashrc_dist_handler = open(bashrc_dist_filename, 'r')
        dist_content = bashrc_dist_handler.read()
        bashrc_dist_handler.close()
        bashrc_content = bashrc_content + '\n' + dist_content
        content_changed = True

    return bashrc_content

def main():
    path_bash = os.path.join(home_dir, '.bashrc')
    if not os.path.isfile(path_bash):
        print('bashrc is not found')
        exit(1)

    # read local .bashrc file
    file_handler = open(path_bash, 'rU')
    bashrc_content = file_handler.read()
    file_handler.close()

    # execute changes
    bashrc_content = show_branch_in_shell(basrc_content)

    if content_changed:
        # save changed bashrc
        file_handler = open(path_bash, 'w')
        file_handler.write(bashrc_content)
        file_handler.close()
if __name__ == '__main__':
    main()

Step 3: Enable Git color output

Now we have script with one method implemented. Our next step is to enable color prompt in bashrc. You can do it manually by setting the force_color_prompt parameter value to ‘yes’. Let’s modify our script. It is good to use regular expressions to find if it is enabled or not:

def color_prompt_enable(bashrc_content):
    # Searches for commented out "force_color_prompt" and enables it
    return re.sub(r'#\s*force_color_prompt\s*=\s*yes', r'force_color_prompt=yes', bashrc_content)

After that, we need to have colordiff package installed on our Linux. Here is implementation of install_colordiff method:

def install_colordiff():
    # check if colordiff is installed
    out = subprocess.check_output(['apt-cache', 'policy', 'colordiff'])

    if re.search(r'Installed:\s*\(none\)', str(out)):
        # install with "sudo" if not exists
        subprocess.call(['sudo', 'apt-get', 'install', 'colordiff'])
        print('colordiff has been installed')

Step 4: Add useful Git aliases

We’re almost finished. Last step is to add a few useful Git aliases. For this, we will need to read content from our gitconfig.dist and append it in your ~/.gitconfig file.

def add_git_aliases():
    # make absolute path for new .gitconfig in user home directory
    home_gitconfig_path = os.path.join(home_dir, '.gitconfig')
    if not os.path.exists(home_gitconfig_path):
        # create .gitconfig file if it doesn't exists
        subprocess.call(['cp', gitconfig_dist_filename, home_gitconfig_path])
        print('.gitconfig file has been created')
        return

In this example, I assume I do not have ~/.gitconfig or have it empty. You might want to perform additional logic to merge ditconfig.dist with your existing config. In this case we will use Python module called configparser (configuration file parser). Let’s import this module to our script and modify add_git_aliases() function:

# in case of using Python2 use ConfigParser instead of configparser
import configparser

def add_git_aliases():
    # make absolute path for new .gitconfig in user home directory
    home_gitconfig_path = os.path.join(home_dir, '.gitconfig')
    if not os.path.exists(home_gitconfig_path):
        # create .gitconfig file if it doesn't exists
        subprocess.call(['cp', gitconfig_dist_filename, home_gitconfig_path])
        print('.gitconfig file has been created')
        return

    # optional (if exists, merge 2 configs and write the result)
    config = configparser.ConfigParser()
    config.read(home_gitconfig_path)
    config.read(gitconfig_dist_filename)

    # set up config file name
    config.filename = '.gitconfig'

    # write merged config to disk
    file_handler = open(home_gitconfig_path, 'w')
    config.write(file_handler)
    file_handler.close()

    print('.gitconfig file has been updated\n')

Step 5: Complete script

Now we have all the methods implemented we need to finish the script. Add those methods in a sequence execution and it is complete.

# execute changes
install_colordiff()
add_git_aliases()
bashrc_content = show_branch_in_shell(
    color_prompt_enable(bashrc_content)
)

Basic implementation is now finished. Feel free to add your own improvements to the script, such as more Linux distributives support, version control, custom useful aliases, and share this experience with other users!

With you newly gained knowledge and experience in Python you can now create your own Linux system administration scripts. Let me know what you came up with, please share them using the comments.

Source references:

  1. Script source on GitHub
  2. Python operating system interfaces module documentation
  3. Python regular expressions module documentation
  4. Python subprocess module documentation
  5. Python configparser module documentation

Git tip: Beautiful colored and readable output

Git is a powerful tool that is very user friendly when used from the Command Line Interface (CLI). There is no need to add Graphical User Interface (GUI) tools to simplify or visualize the output. What can help to make the CLI output more usable is to enable colors and to allow for some command abbreviations (aliases). Below you see the output we get when we run our commands.

Git colored output

git-st-colored

As you can see, we configured the Git alias for “status”, so that it can be abbreviated with “st”. You can also see that the colored output has been enabled.

git-diff-colored

In Ubuntu colored diffs can be achieved be installed by using the “colordiff” tool. The “colordiff” tool is a replacement for the normal diff tool and can be used with the same arguments. It can be installed using “apt-get”, like this:

sudo apt-get install colordiff

However, Git does not require this because it has support for the colored diff built in.

git-lg

This is the most useful output. It is an enhanced Git “log” output that we alias as “lg”. By adding color it becomes dramatically more usable.

Instructions

We assume you have enabled the colored prompt in “.bashrc” by uncommenting the “force_color_prompt=yes” line as described in the previous Git post. Next you should add the following in your “~/.gitconfig” file:

	[alias]
		lg = log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit --date=relative
		st = status
	[color]
		branch = auto
		diff = auto
		status = auto
	[color "branch"]
		current = yellow reverse
		local = yellow
		remote = green
	[color "diff"]
		meta = yellow bold
		frag = magenta bold
		old = red bold
		new = green bold
	[color "status"]
		added = yellow
		changed = green
		untracked = cyan

Save the file and try the commands. Enjoy!

Git tip: Show your branch name on the Linux prompt

git-branch-prompt-colored

Git is one of the most popular Source Control Management (SCM) software packages to provide revision control. Subversion (aka SVN) and Mercurial are other popular systems. These systems organize source code in branches and revisions that can be named by means of tagging. Branches can be used to allow you to work on different versions of the software concurrently. A popular way of organizing is to create a branch for every major feature you are building. Another organization can be used to distinguish between a “development” and “production” version, so that you can easily apply a small code change in case of any bugs.

In any of the above schemes it is important to know which branch you are working on, when committing code to the repository. This post explains how you can add the branch name in red to the prompt.

Firstly, we have to turn on the colored prompt in “.bashrc”:

# uncomment for a colored prompt, if the terminal has the capability; turned
# off by default to not distract the user: the focus in a terminal window
# should be on the output of commands, not on the prompt
#force_color_prompt=yes

Now add the following code to “.bashrc” for Git branch information:

# Add git branch if its present to PS1
parse_git_branch() {
 git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/(\1)/'
}
if [ "$color_prompt" = yes ]; then
 PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[01;31m\]$(parse_git_branch)\[\033[00m\]\$ '
else
 PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w$(parse_git_branch)\$ '
fi
unset color_prompt force_color_prompt

With this change you will be less likely to commit your code to the wrong branch. Enjoy!

PS: You can also show the branch name on the Linux prompt when using Mercurial (tip by Răzvan Tudorică).

Git version information in Symfony2 WDT

At LeaseWeb we are migrating from SVN to Git. Not only because we like to work with cutting-edge technology, but also because Git allows different ways of working, like “pushing to production“.

We mentioned before that we have the LswVersionInformationBundle that shows Subversion information in Symfony2’s Web Debug Toolbar. We are proud to tell you that the same bundle now supports Git as well! Screenshots:

git_info

Picture 1: Git information in the Symfony2 web debug toolbar

git_pane

Picture 2: This information is shown when you click the Git icon in the toolbar

svn_info

Picture 3: Subversion information in the Symfony2 web debug toolbar

svn_pane

Picture 4: This information is shown when you click the SVN icon in the toolbar

The code can still be found on Github and installed using composer.

PHP deployment using SVN and Git

Traditionally we use Subversion in LeaseWeb. Recently we started to move to Git. This is not an easy move; It requires education, integration and backwards compatibility tricks to make sure productivity is not impacted. Now that the most problems are solved we are beginning to see the advantages and opportunities that Git offers us. One of the nice things has to do with the deployment flow.

If you use SVN you could do “svn up” on the production server. This is a very simple way of deploying that works if you follow two important rules: 1) do not commit config files and 2) do not change any non-config files on production. When you follow these rules you know that you will not run into merge conflicts during deployment.

Another approach is to create a deployment script that deploys and configures into a seperate directory and then changes the target location of the symlink that is configures as the document root for the webserver. This allows you to keep directories with version numbers and do easy rollback (assuming the database is not impacted). The actual symlink change is atomic (according to Mike Zupan) and I believe he is right.

The first approaches requires SVN access from the production server. In the second approach the code could be checked out using “svn export” and then packaged using “tar -cvzf” and copied over using “scp” and then unpacked and as a last step the symlink can be switched (make sure you set the right permissions). Obviously one can easily write a small bash script that automates all this.

With Git we have another very stylish option: “git push production”. We can setup a second remote called “production”. The first remote is called “origin” and this is the location where you downloaded the repository from, hence the name. With the command “git remote add” you can add the production server. You must create a bare repository on the production server to push to. The push operation to the production server uses SSH to connect and transfer the files. Logging in can be secured and automated using key-based authentication just like with “scp”.

After the push you can tell Git to automatically deploy using a “post-receive” hook. With Git you can also choose for either a “svn up” like solution using “git pull”, or a symlink flip after issuing a “git clone” and configuring the project. Don’t forget to remove access to the hidden “.git” directory in your project. Accidentally serving it is probably not a smart idea.