PyCon Australia 2014: conference videos online


PyCon Australia 2014 was held last week (1st – 5th August) at the Brisbane Convention & Exhibition Center.

PyCon Australia is the national conference for the Python Programming Community, bringing together professional, student and enthusiast developers with a love for developing with Python.

For all of you that did not go, most of the conference is available on YouTube (39 videos):

  1. Graphs, Networks and Python: The Power of Interconnection by Lachlan Blackhall
  2. PyBots! or how to learn to stop worrying and love coding by Anna Gerber
  3. Deploy to Android: Adventures of a Hobbyist by Brendan Scott
  4. How (not) to upgrade a platform by Edward Schofield
  5. Caching: A trip down the rabbit hole by Tom Eastman
  6. Verification: Truth in Statistics by Tennessee Leeuwenburg
  7. Record linkage: Join for real life by Rhydwyn Mcguire
  8. Command line programs for busy developers by Aaron Iles
  9. What is OpenStack? by Michael Still
  10. Software Component Architectures and circuits? by James Mills
  11. IPython parallel for distributed computing by Nathan Faggian
  12. A Fireside Chat with Simon Willison
  13. Accessibility: Myths and Delusions by Katie Cunningham
  14. Python For Every Child In Australia by Dr James R. Curran
  15. Lightning talks
  16. How to Read the Logs by Anita Kuno (HP)
  17. Serialization formats aren’t toys by Tom Eastman
  18. Django MiniConf: Lightning talks
  19. What in the World is Asyncio? by Josh Bartlett
  20. Try A Little Randomness by Larry Hastings
  21. Building Better Web APIs by HawkOwl
  22. devpi: Your One Stop Cheese Shop by Richard Jones
  23. Learning to program is hard, and how to fix that by Jackson Gatenby
  24. Lesser known data structures by Tim McNamara
  25. The Quest for the Pocket-Sized Python by Christopher Neugebauer
  26. Sounds good! by Sebastian Beswick
  27. Running Django on Docker: a workflow and code by Danielle Madeley
  28. Software Carpentry in Australia: current activity and future directions by Damien Irving
  29. The Curse of the Django Podcast by Elena Williams
  30. Grug make fire! Grug make wheel! by Russell Keith-Magee
  31. (Benford’s) Law and Order (Fraud) by Rhys Elsmore
  32. The Curse of the Django Podcast by Elena Williams
  33. Lightning talks
  34. The Curse of the Django Podcast by Elena Williams
  35. Seeing with Python by Mark Rees
  36. Descriptors: attribute access redefined by Fraser Tweedale
  37. How do debug tool bars for web applications work? by Graham Dumpleton
  38. Continuous Integration Testing for Your Database Migrations by Joshua Hesketh
  39. Closing

Have fun watching!


EuroPython 2014: full conference on YouTube


EuroPython 2014 was held last week (July 21 – 27) at the Berlin Congress Center (BCC) in Germany.

The EuroPython conference is the second largest global conference for the popular programming language Python, after the PyCon US conference in North America.

For all of you that did not go, the full conference is available on YouTube (120 videos):

  1. Welcome [20:56]
  2. pymove3D Winner Announcement [21:05]
  3. One year of Snowden, what’s next? [58:00]
  4. What can python learn from Haskell? [41:20]
  5. Lightning Talks [40:18]
  6. Amanda: A New Generation of Distributed Services Framework [24:50]
  7. Will I still be able to get a job in 2024 if I don’t do… [46:47]
  8. Cutting-edge APIs using hypermedia at BSkyB [32:30]
  9. Mobile Games to the Cloud With Python [29:45]
  10. Statistics 101 for System Administrators [29:45]
  11. The Magic of Attribute Access [26:47]
  12. Pythonista: A full-featured Python environment for iOS … [40:32]
  13. Brain Waves for Hackers [33:36]
  14. Rethinking packaging, development and deployment [40:21]
  15. The Sorry State of SSL [44:56]
  16. Traversing Mazes the pythonic way and other Algorithmic… [44:50]
  17. The Cython Compiler for Python [48:01]
  18. Message-passing concurrency for Python [42:02]
  19. Extending Python, what is the best option for me? [32:22]
  20. Marconi – OpenStack Queuing and Notification Service [24:37]
  21. Solution oriented error handling [28:19]
  22. Web Scraping in Python 101 [20:18]
  23. Jigna: a seamless Python-JS bridge to create rich HTML … [23:36]
  24. How to make a full fledged REST API with Django OAuth T… [23:59]
  25. Documenting your project with MkDocs. [22:09]
  26. VPython goes to School [25:17]
  27. Full Stack Python [25:47]
  28. Using python, LXC and linux to create a mass VM hosting… [23:09]
  29. log everything with logstash and elasticsearch [18:29]
  30. Gamers do REST [24:20]
  31. Teaching Python [28:09]
  32. PyPy status talk (a.k.a.: no no, PyPy is not dead) [31:38]
  33. pymove3D – Python moves the world – Attractive programm… [23:10]
  34. Designing NRT(NearRealTime) stream processing systems :… [36:35]
  35. GNU/Linux Hardware Emulation with Python [20:34]
  36. Lightning Talks [1:22:47]
  37. Our decentralized future [47:06]
  38. Writing multi-language documentation using Sphinx [24:46]
  39. How we switched our 800+ projects from Apache to uWSGI [28:20]
  40. Introduction to pytest [29:16]
  41. Embedding Python: Charming the Snake with C++ [28:06]
  42. SQLAlchemy Drill [28:37]
  43. Design considerations while Evaluating, Developing, Dep… [40:39]
  44. Using All These Cores: Transactional Memory in PyPy [42:01]
  45. gevent: asynchronous I/O made easy [44:01]
  46. Scaling with Ansible [40:37]
  47. Don’t fear our new robot overlords! [38:43]
  48. Systems Integration: The OpenStack success story [40:00]
  49. Performance Python for Numerical Algorithms [48:28]
  50. DevOps Risk Mitigation: Test Driven Infrastructure [47:25]
  51. Compress Me, Stupid! [35:45]
  52. Stackless: Recent advancements and future goals [44:41]
  53. Design Your Tests [25:32]
  54. Automatic code reviews [20:49]
  55. Supercharge your development environment using Docker [22:36]
  56. An HTTP request’s journey through a platform-as-a-service [24:13]
  57. 3D sensors and Python: A space odyssey [22:40]
  58. How to Setup a new Python Project [24:45]
  59. Graph Databases, a little connected tour [25:33]
  60. Identifying Bugs Before Runtime With Jedi [25:04]
  61. Python refactoring with Rope and Traad [25:02]
  62. I want to help! How to make your first contribution to … [19:14]
  63. Managing the Cloud with a Few Lines of Python [25:37]
  64. How to become an Agile company – case study [22:30]
  65. For lack of a better name(server): DNS Explained [24:54]
  66. Python in system testing [23:10]
  67. Lightning Talks [1:28:23]
  68. Advanced Uses of py.test Fixtures [26:32]
  69. Multiplatform binary packaging and distribution of your… [20:20]
  70. Conversing with people living in poverty [29:12]
  71. Eve – REST APIs for Humans™ [30:44]
  72. The Shogun Machine Learning Toolbox [19:47]
  73. RISCy Business: Development of a RNAi design and off-ta… [25:05]
  74. How Disqus is using Django as the basis of our Service … [30:59]
  75. Metaprogramming, from decorators to macros [39:03]
  76. Support Python 2 and 3 with the same code [39:58]
  77. The Return of “The Return of Peer to Peer Computing”. [39:35]
  78. Python Debugger Uncovered [31:12]
  79. Writing Awesome Command-Line Programs in Python [41:25]
  80. Elasticsearch from the bottom up [36:54]
  81. Scikit-learn to “learn them all” [40:17]
  82. How Pony ORM translates Python generators to SQL queries [44:55]
  83. Probabilistic Programming in Python [45:35]
  84. Morepath: a Python Web Framework with Super Powers [27:46]
  85. Python for Zombies: 15.000 enrolled in the first Brazil… [26:11]
  86. The inner guts of Bitbucket [28:50]
  87. Learning Chess from data [21:58]
  88. Red Hat Loves Python [23:28]
  89. Jython in practice [25:51]
  90. packaging and testing with devpi and tox [26:46]
  91. Non Sequitur: An exploration of Python’s random module [29:13]
  92. Packaging in packaging: dh-virtualenv [31:34]
  93. The Continuum Platform: Advanced Analytics and Web-base… [24:55]
  94. Farewell and Welcome Home: Python in Two Genders [31:42]
  95. Lessons learned from building Elasticsearch client [24:41]
  96. Pioneering the Future of Computing Education [25:55]
  97. Lightning Talks [1:31:25]
  98. How to become a software developer in science? [24:53]
  99. Python’s Role in Big Data Analytics: Past, Present, and… [49:45]
  100. Combining the powerful worlds of Python and R [25:54]
  101. Scientific Visualization with GR [23:19]
  102. Scalable Realtime Architectures in Python [28:31]
  103. Python Driven Company [35:12]
  104. Out-of-Core Columnar Datasets [22:32]
  105. Twisted Names: DNS Building Blocks for Python Programmers [27:51]
  106. Event discrete simulation with SimPy [25:16]
  107. Advanced Database Programming with Python [31:00]
  108. Using asyncio (aka Tulip) for home automation [26:57]
  109. Concurrent programming with Python and my little experiment [21:12]
  110. Big Data Analytics with Python using Stratosphere [20:26]
  111. Practical PyBuilder [23:01]
  112. Building Realtime Web Applications with WebRTC and Python [20:18]
  113. Ganga: an interface to the LHC computing grid [24:28]
  114. Extending Scikit-Learn with your own Regressor [25:54]
  115. Post-Mortem Debugging with Heap-Dumps [26:54]
  116. Fun with cPython memory allocator [28:36]
  117. Everything You Always Wanted to Know About Memory in Python But Were Afraid to Ask [29:04]
  118. Conference Closing [15:52]
  119. Lightning Talks [1:16:09]
  120. Sponsoring von Open Source [28:02]

If you manage to watch all of them… you deserve a medal! 😉



Linux context switching costs of inotify in Python

“A context switch is the process of storing and restoring the state (context) of a process or thread”, it is also “an essential feature of a multitasking operating system” according to Wikipedia.

In this post we go a little deeper into detail on why context switching might be expensive and how you can spot these hidden costs. First I read the fantastic article on Tsuna’s blog titled: “How long does it take to make a context switch?” In the end Benoit Sigoure writes:

Context switching is expensive. My rule of thumb is that it’ll cost you about 30µs of CPU overhead.

Okay, that is quite clear 🙂 Now how do we measure the context switches of an application? I wrote “” in bash:

T="$(date +%s)"
$@ > /dev/null &
watch -n1 grep ctxt /proc/$pid/status
V="$(cat /proc/$pid/status | grep ^voluntary_ctxt_switches | cut -f2)"
N="$(cat /proc/$pid/status | grep ^nonvoluntary_ctxt_switches | cut -f2)"
T="$(($(date +%s)-T))"
echo "voluntary_ctxt_switches: $((V/T))/sec"
echo "nonvoluntary_ctxt_switches: $((N/T))/sec"
kill -9 $pid

This scripts gets the time in seconds. Then it starts whatever you put as arguments and pipes the output to /dev/null. It backgrounds the process and stores the PID. We watch the context switches in real time until you press Ctrl-C. After that we get the amount of context switches and we get the difference in amount of seconds. We calculate some average and then we finally (force) kill the application.

Inotify in Python

After the last post we are particularly interested in the context switching overhead of inotify in Python. I tested a simple Python inotify script using:

bash python2 /var/log/nginx/access.log

And in a separate terminal I ran:

siege -c 100 http://localhost/

I’ve found that the inotify script does about:

voluntary_ctxt_switches: 125/sec
nonvoluntary_ctxt_switches: 23/sec

148 x 30 microseconds = 4.44 ms CPU just for the context switching each second. 4 ms out of 1000 ms = 0.4% CPU load. And note that this is expected to linearly increase with a higher load on the Nginx server, because this is now doing “only” 200 requests per second.

Nginx buffered access logging

My colleague Carlos pointed out that I was wrong in my last post and that inotify is great choice for looking at Nginx access logs (and he is right). The only thing you need to change is that you need to reduce the amount I/O operations that Nginx does for the logging. By default the access log is not buffered, but you can easily change that in /etc/nginx/nginx.conf:

access_log /var/log/nginx/access.log;

To enable buffered logging change this into:

access_log /var/log/nginx/access.log combined buffer=1024k flush=5s;

Now Nginx will only write every 1024k to disk or every 5 seconds, whatever comes first. You can easily verify the improved behavior using the “” bash script.


Iterators and 10 other Python 3 upgrade reasons

I ran into a very interesting presentation from Aaron Meurer titled:

10 awesome features of Python that you can’t use because you refuse to upgrade to Python 3 – source

Aaron gave the presentation on April 9, 2014 at the Austin Python User Group (APUG). Austin is a bit too far away from Amsterdam to just go there and to see the talk, but I am glad we get access to his slides, because they are awesome! Below I quote the slides on “iterator” improvements of Python 3 that he explains exceptionally well.

Why iterators are good

Aaron Meurer lists the advantages of using iterators (and generators) on slide 58:

  • Only one value is computed at a time. Low memory impact (see range example below).
  • Can break in the middle. Don’t have to compute everything just to find out you needed none of it. Compute just what you need. If you often don’t need it all, you can gain a lot of performance here.
  • If you need a list (e.g., for slicing), just call list() on the generator.
  • Function state is “saved” between yields.
  • This leads to interesting possibilities, a. la. coroutines…

Why Python 2 was not so good

Then he states that in Python 2 things were not that good (slide 42-50), since you could accidentally forget to use iterators:

  • Don’t write range or zip or dict.values or …. If you do…
  • python_iterator
    (click to enlarge)
  • Instead write some variant (xrange, itertools.izip, dict.itervalues, …).
  • Inconsistant API anyone?

Why Python 3 is much better

In  slide 51 Aaron Meurer states that in Python 3 things have improved and iterators are used by default. He explains that you can still get a list if you need one:

  • In Python 3, range, zip, map, dict.values, etc. are all iterators.
  • If you want a list, just wrap the result with list.
  • Explicit is better than implicit.
  • Harder to write code that accidentally uses too much memory, because the input was bigger than you expected.

The 10 other features

Interested in the 10 other features “you can’t use because you refuse to upgrade to Python 3”? Look at awesome presentation by Aaron Meurer on his GitHub account:


Minesweeper game for iOS in Python (with Pythonista)

In one of my previous posts on Python, I wrote about how I like to program games when learning a new language. In my initial post, I have written “tic-tac-toe” with some AI. In my second post, I went for a very ambitious AI on the famous “Connect 4” game. In this third post, I will release a “minesweeper” game for iOS written using the “Pythonista” App. The picture below shows the game on my iPad mini:

minesweeper pythonista

A rainy day

Last week I was on holiday, and since it rained almost the entire week (I stayed in the Netherlands) I had plenty of time to sharpen my Python skills. The house we were staying in had a good Wi-Fi Internet connection and although I did not bring my laptop, I did bring my iPad.

Normally I use the iPad for reading websites on the couch while listening to some streaming music. I also like to play small games or type drafts for blog posts (like this one) on it. For the latter, I use the (free) WordPress iOS App. But now I had to do some Python Minesweeper coding.

I expected having to connect to one of my Linux boxes using a terminal application and then coding remotely using a text mode editor, like Vim. But when I explored the possibilities I ran into the Pythonista App. It is a full featured run-time and IDE for Python on iOS. It allows to both program (type) and execute the code on the iPad.

Although it is a paid app, I liked the idea of coding Python on the iPad so much that I decided to buy it. Pythonista comes with some sample applications that will get you up to speed in no time. Pythonista also has a fairly good help system so that you can get context sensitive help on Python functions.

After many hours of typing, Googling, and thinking, I came to the point that the code was ready. I ran into a few nice Python best practices: generators, enums, namedtuple and genuinly had a good time. I especially liked the “yield” (generator) construct a lot and as was happy to find yield is included in PHP (since version 5.5).

Windows XP graphics

Minesweeper X is a nice minesweeper clone written by Curtis Bright. On the site you find a set of skins for his game. Instead of inventing my own skin system, I decided to implement his. To load the skin I converted the file from BMP to PNG using Gimp. After that I converted the file to a base64 string to be able to paste it (with minimal length) in the code. For the game, I chose the familiar “Windows XP” skin, shown below:


Pythonista is not free

Pythonista” is a paid app (6 eur/7 usd). It is comparable to the Lua programming app “Codea” (9 eur/10 usd). For people that like PHP there is “Kodiak” (9 eur/10 usd). And finally, for people that code JavaScript there is “Procoding” (4.5 eur/5 usd). All four applications allow you to run scripts on your iOS device. Although geeking out on your iOS device with your favorite programming language is lots of fun, typing on-screen is not. That’s why I recommend a blue-tooth iPad keyboard for any serious programming effort.

There also is a very innovative app called “ScriptKit” (11 eur/12 usd), that allows for touch-enabled “drag-and-drop” programming in a Lua-like language.

Download the source code

Downloading the source code is as simple as following the link below and copy-pasting the code into your Pythonista app:

Further reading

If you want to dive deeper into Pythonista, you may want to check out the following links:

Python games on GitHub

With this last post I have finished the three games I normally write when learning a new language:

  1. Tic-tac-toe (github) (blog)
  2. Connect 4 (github) (blog)
  3. Minesweeper (github) (blog)

I have published all three games on my GitHub account and discussed each of them in a blog. I hope it inspired you to pick up Python as well.