Software Overview
Updated May 11, 2009 - Click here or scroll to bottom for links to older versions of this page.
The software deliverables of the project fall into two categories: platform software and application software. The platform software is the package called "rp-mt-scripts", which installs and runs a variety of third-party multi-touch software (blob tracker, etc) useful on a Linux platform, specifically Ubuntu 9.04. The application software is tactus-navigator which builds upon the software provided by rp-mt-scripts and the underlying Linux platform to provide task management functionality - that is, replacing a traditional taskbar or dock.
These two packages each have their own version control repository hosted on GitHub, which can be viewed and cloned without registration, or downloaded as a snapshot. Issues can also be reported through that interface, and the components that are useful in the community following the conclusion of the Spring 2009 semester will likely continue on these pages and grow to include wiki pages at GitHub as well.
These pieces of software can be understood in the context of the entire software system with the following diagram:
As you proceed down the diagram, each row is a higher-level of software. The rp-mt-scripts ensure that the correct packages provided by Ubuntu Linux are installed in addition to those components it directly manages. The two pieces of original software, rp-mt-scripts and tactus-navigator, have their own project description page aimed primarily at potential users. The page for the tactus-navigator includes information on the interaction design, both the implemented portions and extensions. The remainder of this page will discuss the overall software architecture, design and implementation considerations, and third-party software used.
Tools
- DrPython development IDE for Python - While an IDE is not necessary for Python (it's an interpreted language, so you can just use a text editor then run "python yourfilename.py" or "./yourfilename.py" in a command line), having a Python-aware IDE like this that does function finding, has a terminal pane, etc. is convenient. I used no IDE for the shell scripts.
- 'Git' version control - This is a very, very useful tool that allows me to record each change that I wish to preserve in an atomic way with full history, and the ability to create a "branch" of development for new/risky features, etc. Combined with remote hosting (see GitHub below), it allowed me to easily conduct development across multiple machines and virtual machines. Additionally, "committing" a change to the version control system is an energizing thing and serves as good motivation for further work, because it feels like an accomplishment. I was used to using a variety of version control systems (primarily Subversion and CVS) due to my work in open-source software, but had dabbled in using Git before. I used Git due to its exceptional branching/merging support and the availability of GitHub hosting that provided a lot of functionality for minimal administrative work. I have been very pleased with this tool.
- GitHub version control hosting and project management - I use this as a remote repository to hold the version control history, so that I can synchronize changes made on my various computers and virtual machines with each other and have a backup. It also produces zipped copies of the latest source code snapshot on request, which simplifies software release: I can just link to those for the latest version, which are updated every time I run "git push" to publish changes, rather than manually updating a released file on the web site.
Base level - Distribution and Desktop Environment
- Ubuntu Linux 9.04 - This is the latest version of Ubuntu, a distribution of the Linux operating system and a wide variety of software. It was released April 20, 2009, and I have used it or a pre-release version since April 14. Prior to that, I used Ubuntu 8.10, the previous version, with some packages from 9.04 prerelease ported manually back to the older system. The new system includes suitable versions of all the major libraries needed, so for a standard install, only PyMT (a very new, cutting edge framework) needs to be built and installed system-wide by the rp-mt-scripts from source rather than a standard package. The installation process of Ubuntu is very easy, especially if using the standard "Desktop CD" combination live CD environment and installer, and can peacefully coexist with Windows on a single computer. In my setup, an additional 20gb hard drive was added to the system for Ubuntu, since the existing 20gb hard drive was almost full due to the Windows and application install. On startup of the computer used, it now offers a choice between Windows and Ubuntu Linux, with Ubuntu as the default.
- Gnome Desktop Environment, as included in Ubuntu Linux - This powerful, open source, general purpose computing environment (comparable to closed-source Windows and Mac OS X plus third party software), was used as the implementation environment due to the ease of replacing the desired desktop component (window switcher) and the available open-source libraries. I am comfortable and familiar with the Gnome environment, which has a strong focus on usability and ease of use. Additionally, the Gnome libraries are available in a wide variety of languages, including Python. I opted not to use Windows because it was not modular enough to have ease interfaces that I could replace the taskbar with, and I already knew which libraries I would use to get this done on Linux.
Lower Level Libraries and Programming Language
- OpenCV - This is a computer vision library, an open source project originally started by Intel, that is used transparently by Community Core Vision (a.k.a. tbeta) to perform webcam image processing for blob tracking.
- Linux OpenGL Support (test platform uses nVidia binary drivers) - OpenGL is a standard for 3D graphics supported on Windows, MacOS X, and Linux. The link goes to nVidia - the test hardware uses an nVidia graphics card and so uses the nVidia drivers for that card to provide OpenGL support. However, any card capable of 3D accelerated rendering under Linux should work to some degree, though higher performance cards, especially if they support 'hardware frame buffer objects', a relatively new OpenGL extension. Our GeForce 8400GS does support this, however, the Intel i810 video card built into the motherboard of that system corrupts all images in PyMT and so should be avoided. The hardware page has information and a video on how we set up our nVidia card for dual-screen support to run tactus-navigator.
- libwnck (pronounced lib-wink) - This library, whose name stands for "Window Navigator Construction Kit", comes with Gnome and is used by the default taskbar to perform its tasks. It provides access to abstractions of windows, information on their status, and methods to change them (minimize, restore, maximize, move, etc). Through PyGTK's Gnome bindings, this library is available in Python as well, providing an object-oriented, if somewhat low-level, interface to window navigation. The tactus-navigator contains a container class wrapping wnck.Window objects and providing a higher-level interface suitable for direct use or subclassing (as I do to connect each window with a PyMT multi-touch widget). The link goes to the C documentation, but the Python interface is similar, but less boilerplate and better readability. Documentation can be accessed by starting Python then entering the commands "import wnck" and "help(wnck)" or using the "Browse PyDoc..." menu item in DrPython.
- Python programming language - Python is an very-high-level, object-oriented, cross-platform, free programming language and standard library, that is included by default with the vast majority of Linux distributions, MacOS X, and is available for Windows as well. It is an interpreted language offering an interactive mode useful for testing. I selected Python for tactus-navigator because it is very-high-level and allows me to concentrate on the logic of the application rather than housekeeping. It is well suited for this sort of application creation. It can also be used for rapid prototyping and can connect to libraries written in C or C++ if you wish to migrate performance-critical modules. Many libraries common on modern Linux systems include native Python bindings, which are a great help. In addition to producing tactus-navigator (in its entirety: base classes, the GTK+ partial test application, and the final multi-touch interface) with Python, I have started an interface to the rp-mt-scripts in GTK+ that provides a graphical launcher for the shell scripts.
- "bash" Linux/Unix shell scripting - This is used for the rp-mt-scripts to automate installation, management, and running of Linux multitouch software. It is the natural choice since it replicates command-line statements used to set the software up, and includes flow control, variables, and functions as well to produce a more sophisticated package. While I knew basic shell scripting (essentially just common shell commands, 'grep' regular expressions, pipes, and variables) before the project, I learned increasingly advanced shell scripting to produce a complete project that can set up a blank install of Ubuntu Linux 9.04 to run demos or tactus-navigator in a single, automatic step (requiring only password entry).
Mid-level Dependencies
- Pyglet - This is a Python OpenGL environment that provides an event loop and event handling constructs as well as the 3D operations required to build an OpenGL interface in Python. PyMT extends this framework to add high-level multi-touch widgets and gesture recognition - I use Pyglet directly only to handle dual-screen details that are not wrapped by PyMT.
- tbeta (now known as Community Core Vision) - This is blob recognition and tracking software (a "tracker") from the NUI Group community that accepts input from a webcam, applies configurable filters, and extracts "blobs" corresponding to areas of touch, then sends TUIO messages over a network (usually just to the local computer 127.0.0.1) using UDP, which can be used by most multi-touch software. (Flash apps require the FLOSC gateway, and some apps designed for the Windows WM_TOUCH api need a similar gateway.) The software for the project all uses tbeta 1.1 - CCV 1.2 was offered in a limited release, with limited publicity, starting April 28th and does not yet have binaries available for easy testing. For instructions, see the Downloads page for the Quick Guide and Manual, or for third-party information, see Getting Started guide and this forum post for filter tuning tips.
- Flash Player and FLOSC Gateway (gateway included with tbeta) - The standalone Flash player from Adobe is used to run Flash/ActionScript 3 applications. With the current (1.1) version of tbeta, the FLOSC gateway software is required to pass TUIO/OSC messages to the Flash applications, though the new release of Community Core Vision (formerly tbeta, only available as source code right now) supports direct connection to Flash in addition to standard TUIO.
While none of my project software is written in Flash, a lot of existing demos and multi-touch software are, so rp-mt-scripts can install and run the standalone Flash player and FLOSC to expand the array of supported user software on the hardware and to provide opportunities for additional development in the future (other students may prefer to code in Flash/ActionScript 3). - PyMT Multi-touch Framework - This is a very recent Python multi-touch framework that extends Pyglet to permit creation of high-level tactile interfaces. I used it to produce tactus-navigator, whose main classes multiply inherit from PyMT interaction classes and the lower-level tactus-navigator classes that I wrote to wrap libwnck. This use of multiple inheritance allows me to start with the best-fit PyMT widget and extend it to directly interact with the window navigation - I instantiate classes that are simultaneously an icon on the navigation display and the corresponding window in use by the user. PyMT also offers more sophisticated gesture recognition and a lot of room for extension, even though it is still very fresh, young software. I always use a very recent version from their central version control (installed using rp-mt-scripts) rather than a "release version" due to the pace of development. While there is only basic documentation at this time for PyMT, the Python source for the library and the examples included helped me to understand how to use it, and the Pyglet documentation helped understand those shared elements, such as the event system.
Note: I originally planned on using a different gesture recognizer written in Java with C++ bindings (Sparsh UI), which would have expanded my project scope to include Python-C++ bindings for that software. It would not have been a very difficult task (Python is good at this kind of thing), but it would have introduced more opportunities for problems than using the pure Python system I ended up using.