“Hello World!” – One Small Step for Python Scripting in Eclipse

We are excited to be able to run a ‘hello world’ command using EASE and CPython in Eclipse (quickly followed by running the ‘Zen of Python’ for good measure :). It is the first key stage of proving our approach of using Py4J as the enabling technology to getting scripting available with EASE for Eclipse. It is a small step, but a significant leap towards unprecedented automation, dramatically easier Eclipse extensions and powerful third-party library integration using Python.

Why Scripting?

Since we first started working with Eclipse back in 2003, scripting has always been one of the big missing pieces, especially when users would make comparisons to Visual Studio’s macro support. There was no established way to automate the IDE or record commands for playback later. Over the years, many have crafted their own solutions (including us) but in our experience these usually work for subsets of use-cases and run out of steam when used more generally or for large-scale efforts.

Now the time is right to solve this and solve it well for the entire community. We envision a scripting framework that can automate all aspects of the IDE, provide easy extension of the user interface and the ability to share and integrate scripts with third-parties. All with the end goal of increased developer and user productivity. We wrote about some ways we envisage this in this article: ‘How to use Python to hack your Eclipse IDE‘.

Why EASE?

The Eclipse Advanced Scripting Environment (EASE), led by Christian Pontesegger, came along as a concerted effort to address scripting scenarios platform-wide. It is a language-independent framework for scripting, designed to work hand-in-hand with language tools (like PyDev) and interpreters. In addition to automation, EASE also enables easy extension of Eclipse with just a couple of lines of script. So all of a sudden users could add a button or a menu item without having to write Java, restart the IDE or build a plug-in. This opens up Eclipse to a whole new field of programmers. No surprises EASE was voted ‘Most Innovative Project‘ at the Eclipse community awards this year.

However, initially the only languages used were those that ran in the Java Virtual Machine (JVM) which included Jython, Javascript and Groovy. While these are great languages and provide good scripting options, they are no match for Python and its extensive ecosystem.  Additionally, it was important to know if the EASE framework could extend to non-JVM languages.

Why Python?

Python is the de facto scripting language in many domains. In scientific circles, Python along with the powerful numpy and scipy libraries dominates.  In the embedded world, Python is the scripting language of choice for gdb, as well as several utility libraries for Linux. Also the dynamic and easy-to-use nature of the language lends itself well to the exploring necessary for data exploration work. It is an excellent language for those new to programming to get to grips with and has a terrific community to go with it.

There are hundreds and hundreds of valuable open source Python packages out there. Imagine being free to play with everything in the biggest Lego store in the world. Imagine what you would create. This would be waay better than that: any Python library fully integrated with any Eclipse functionality. The possibilities are breath-taking.

Why Py4J?

Python and Java are have some fundamental differences, including their respective threading and memory management models. So we need technology to bridge these differences. Fortunately there exist a number of potential solutions out there, so it was a matter of evaluating and choosing the most suitable one.

One of the key requirements is to fully support CPython and c-extension libraries like NumPy. Additionally we required multi-platform (Linux, Windows & Mac) support and an Eclipse compatible licence for any bridging technology used.  There are a number of technologies we reviewed, and here are the main contenders that were not suitable.

• Jython – does not support CPython compiled extensions, so can’t run NumPy, etc.
• JyNI – Jython extensions for Java. Only has initial NumPy support, lacks Windows support and has licensing issues (GNU LGPL).
• JEP – a solution that embeds CPython directly in Java through JNI. This embedded interpreter approach results in some CPython libraries not being fully compatible, in part as Python cannot be fully reset/reloaded without restarting Eclipse itself.

Py4J is a bridge between Java and Python and designed to allow Python to access Java classes. It is the technology behind the Python interface to Apache Spark (PySpark). We were familiar with it through our work with DAWNSci, even having submitted patches for autocompletion functionality. Early prototypes using EASE looked promising. However, we ran into a major issue trying to workout how to do recursive calls between Python & Java and involving the main SWT thread.  The solution to that is worth its own blog post, but the summary was that we worked closely with the architect of Py4J, Barthelemy Dagenais to come up with a new ‘pinned’ threading model. This has just been released as part of Py4J 0.10.0.

What’s Next?

So with the new threading model implemented and integrated with EASE, we were ecstatic to see hello world working and then get on with testing all the various scenarios. A first working version can be reproduced from this repository.

We are working to a deadline now, with 6 weeks to go to a workshop on EASE and Python at EclipseCon France. The next steps will be to get the new version of Py4J available in Orbit, make the EASE-Py4J plugin more readily available and implement many of the language features. With the biggest risk of Java & Python interaction mitigated, we will see if it is smooth sailing from here (though probably not). Either way we’re sure we will enjoy the journey and keep you posted.

This work is being sponsored by forward-looking companies who share our vision, want to achieve tremendous productivity gains for their users and want to make Eclipse tools better for the entire community. We are actively looking for more companies like that who would like to get involved, so please get in touch if you can help or know someone who would be interested.

Likewise, if you are a developer who can envisage using this technology to solve your real world problems, do get in touch so we know your use-cases and benefit from your input to work together to build the best solution.  Or even better, join us at EclipseCon France!