Improving Code – The Open Source Way

The latest CDT offering has just been released. Today CDT is generally acknowledged as the de-facto C/C++ IDE in the industry. The extensive code navigation features are amongst the highlights that the framework has to offer. Not wanting to rest on their laurels, work is already well underway on the next release, CDT 6.1. In a recent blog article, CDT lead Doug Shaefer highlights that there will be a big focus on fixing up the scanner discovery mechanism.

Scanner discovery mechanism is the feature in CDT that supports code navigation by automatically discovering build information such as include paths and build macros. The build information is normally retrieved in one of two ways. The first involves running a fixed gcc command to query the known include paths and macros. The second way involves parsing the build output to identify relevant build information.

This is a great feature that enhances code navigation, and in particular creates a very good start up impression. It means users can navigate to code libraries even before they build a project. In our experience this has been a very desirable thing, particularly when our clients provide specialist libraries such as peripheral driver code. Scanner discovery means their customers can easily navigate the code and link to header files to browse the api for the peripherals. However, as it stands today the scanner discovery mechanism is both architecturally flawed and buggy. We have spent significant time fixing it up for clients in order to give the desired experience. However, there’s only so many times we can patch it up before wanting to help fix it up once and for all.

As a result we will be working with the CDT team to help fix scanner discovery for the next release. In fact we will be part of the ‘rag-tag group’ Doug refers to in his blog. The first step was to have an initial kick-off meeting, followed by all capturing the problem and trying to define the architecture. This has been done using the CDT wiki. While one or two CDT committers have put down details of what the high-level architecture should be, we have added our thoughts on the specific issues we have seen with the implementation and low-level architecture based on our experiences.

The next step will be the interesting one as we work out how best to proceed. In the meantime, Kichwa Coders will also be working on some low-hanging fruit by working on scanner-related bug fixes such as using variables in dialog boxes. All in all, it will be an interesting experience working on a relatively far-reaching feature with several other developers spread out across the globe, yet ultimately very rewarding to be part of fixing up this key feature of the CDT. In particular for us, it means yet another feature our clients can take for granted and keep focusing on their core competences.

Eclipse Summit 2009 – Short Talk Proposal

The planning for this year’s Eclipse Summit in Ludwigsburg, Germany is underway.
We’ve put in a submission for a short talk to present the work we did at Diamond Light Source. You can view the abstract here. The talk will focus on taking Eclipse into a new scientific domain, with the potential of forming the basis of an open source platform for synchrotron data acquisition. We envisage this would meet the criteria of getting people excited and engaged at the potential for Eclipse to conquer this new territory. Whether the committee agrees with this remains to be seen. The level of talks at Eclipse conferences and summits is always very high so it would be great to make the cut. However competition is stiff and at the time of writing there were 152 submissions to contend with. The talks will be chosen by August 25th, so we await the decision with bated breath…

SWT_AWT Bridge and Code Reuse

Moving an existing application to a new platform can be a huge undertaking, often requiring re-writing large portions of the code base to work in the new framework. However, moving a legacy AWT/Swing application to the Eclipse platform does not mean having to abandon the existing GUIs to take advantage of what Eclipse has to offer.

Take Diamond Light Source’s GUI for Data Acquisition (GDA) as an example.

Diamond’s GDA application was originally developed as a monolithic application using Swing for rendering GUIs. As the number of users and developers grew, Diamond decided that Eclipse could quickly provide new features, scalability and improved component interoperability. However, there was a problem with transitioning to a new platform: a large investment had already been made in Swing.

The Eclipse developers had already solved this problem though. They created an SWT_AWT bridge that allows a Swing/AWT component to be added as a component to an SWT composite. This bridge significantly simplified how Diamond was able to transition, by reusing existing Swing components, but still taking advantage of the new capabilities that Eclipse/SWT has to offer. A prime example of a feature that Diamond GDA users and developers desired was more control over the layout of the GUI. By embedding the existing Swing components into individual Eclipse views in a short period of time all the previously mostly static components were draggable, dockable and closeable.

The SWT_AWT bridge lowered the barriers that Diamond had to transitioning GDA to the Eclipse platform. Instead of having to re-write all the necessary component in the new framework, Diamond developers were able to take advantage of Eclipse very quickly and roll out those improvements to the users in record time.

For the technical details on how to use the bridge, see the excellent article by Gordon Hirsch Unfortunately, integrating the SWT_AWT bridge has suffered from some headaches when using Xlib (as used on Linux/GTK systems). Some of the Xlib APIs are inherently not thread safe in their design. For example, XSetErrorHandler, which returns the old error handler when setting a new one. Without some application level locking, it is easy to get the error handler into a state when the currently installed error handler is incorrect. When using the SWT_AWT bridge, two separate threads are both accessing Xlib APIs. This can cause the AWT thread to have a stack overflow. The underlying problem will remain due to the Xlib APIs, however, SWT and AWT can avoid using XSetErrorHandler, and therefore avoid this issue. These issues are discussed more fully in Eclipse Bug 171432. The AWT issue has already been fixed in JDK7 build 60, but the fix has not yet been back ported to JDK 6. See Bug and Revision with fix.

Leveraging the Power of the Plug-in

The beauty of the Eclipse platform lies in its plug-in architecture. This extensible architecture was the key motivation for migrating Diamond Light Source’s existing IDE (GDA) to the Eclipse platform. In doing so it was hoped that the GDA would benefit hugely by being able to leverage open source plug-ins. Work began on developing a rich-client platform (RCP) prototype to try to achieve this goal.

In developing the prototype, this goal was quickly demonstrated. The GDA had a custom log panel which was used to display log messages. In Eclipse, the logback plug-in was dropped into the prototype and this instantly gave an equivalent log console that was able to display the log messages – without requiring a single line of code change. In addition, the new log console came with additional features the original lacked, such as the ability to configure the message properties. It was not all plain sailing though, as one issue with using the logback plug-in was managing the versioning. On upgrading to a newer version of the plug-in the console failed to work, meaning that GDA could not take advantage of the latest version until the issue was fixed. When incorporating third-party plug-ins it is up to the user to ensure they contend with handling the versioning and compatibility of the plug-ins. The more they are in touch with the plug-in providers, the easier issues can be resolved.

Following on from the quick success of the logback plug-in, the next step was to attempt to use the Pydev plug-ins to provide a Jython editor and Jython console to replace those of the existing IDE. This was trickier to do, but nonetheless after going through the readily-available code and understanding how things worked, we were able to provide a fully-featured Jython editor and console in a relatively short time. In addition we were able to integrate the plug-ins with the existing codebase in such a way that it allowed auto-completion on not just Jython keywords but also GDA system specific functions. It was a great example of leveraging open source plug-ins out there. Some of the issues in general with using the Pydev plug-ins revolved around the dependencies – for example using the editor plug-in relied on pulling in the JDT plug-ins which then introduced unwanted functionality into the GDA RCP prototype. Once again these issues can be resolved, even more so by working closely and contributing back to the Pydev project.

In addition, we were also able to demonstrate integration with the Desy project. Using their plug-ins we were able to use their Signal Probe view and connect up the GDA back-end to it to generate the data. This was done after alot of time understanding the code-base and done to a prototype level, but nonetheless showed how two independent plug-ins developed within the same field could be used together thanks to the Eclipse plug-in architecture.

Thanks to the nature of Eclipse’s plug-ins, IDE providers are able to take advantage of existing code-bases to obtain functionality that would take months to develop in-house. Also code bases can be shared between different institutions, thereby helping pave way for setting standards within the field. Of course to receiving the maximum benefit comes by working closely with those who produced the code in the first place – but the more you put in the more you get out.

Features vs. Product Files: A Case Study

Diamond Light Source’s synchrotron features many beamlines. A beamline is an experimental station where beams of light are used for a wide variety of scientific applications. Each beamline is controlled via client software from the beamline control room. The software is produced by the Generic Data Acquisition (GDA) team, and is called GDA Client. The GDA Client historically was written as a Swing GUI Java Application. In 2008, the GDA group began transitioning GDA to the Eclipse platform to take advantage of the many benefits Eclipse has to offer, such as:

  • moving from a monolithic implementation to well partitioned bundles,
  • better software reuse,
  • easier integration of third-party components and tools and
  • use of the many features already written into Eclipse, such as editors, views, and a consistent extensible menu structure.

One of the goals for improving GDA by migrating it to Eclipse was to streamline and improve the deployment procedure for the Client. The requirements for this were:

  • Build both interactively within the Eclipse IDE and from the command line
  • Incorporate the non-Java files like python scripts and configuration files
  • Support the individual configurations of each beamline

A number of alternative solutions were considered for solving this problem. Following general Eclipse recommendations, they all centred on using a combination of features and product files.

Eclipse features are a type of project providing a description to deploy and distribute an entire set of plug-ins that work together. A feature allows you to define what plug-ins make up the set, describe the dependencies and the versions of those dependencies. Features are used by Eclipse when creating update sites for use by the update manager. They are generally the top level item deployed if your deployment is expected to be added to an already installed instance of Eclipse.

Eclipse product xml files are the main definition of an install for a customized eclipse deployment. The product definition contains a list of plug-ins or features that make up the deployment. The product also defines the splash screen and other branding items, platform specific launch options, and which eclipse defined application to start with. An Eclipse product is used if the deployment is the entire application, eg a standalone RCP application.

Prior to complete product definition using the .product xml file, features would be used to build one or more sets of plug-ins, and something outside of eclipse, such as an ant build script, would assemble all the features, their corresponding plug-ins, an eclipse.exe – possibly renamed – and other necessary parts of an eclipse installation.

With the product xml file, it is possible to create a full deployment in one step using either the Export wizard in eclipse, or at the command line with the available ant tasks. It is possible to create an entire branded deployment in one step.

Defining a deployment using a product is a big step forward making it much easier to configure and deploy. However, there are still some drawbacks that make it more difficult to use that it could be.

One of them is that a product file can only list a set of features or a set of plug-ins, not a combination of the two. If most of your project is defined as features, any plug-ins not in a feature already will need to be trivially wrapped by a feature to be included in a product. The alternative is to list all the plug-ins in the product and not take advantage of the already defined features, leading to duplicated effort and maintaining multiple copies of the same information. Another problem is the only way to tell a plug-in to unpack on installation is to define that attribute in a feature.

For Diamond’s GDA Client, the main requirements mean that using product files are the correct option; however, features are still used due to the drawbacks mentioned above. In the GDA environment, each beamline requires its own customized deployment. Therefore, each beamline requires its own .product file. Since a feature is always needed due to the drawbacks listed, each beamline has its own feature project with its feature.xml. The feature.xml contains the list of plug-ins needed for this beamline, either as a list or plug-ins, included features or a combination of the two. For simplicity the product file is stored in the feature project so that all the beamline specific build and configuration files are located in the same location.

In conclusion products are a big step forward as a method for deploying eclipse products. However for this particular instance, they would need a couple of bugs fixed to be the ideal solution.