CDT has been reformatted!

Eclipse CDT has just had all its source code reformatted. That means I just pushed a massive set of commits with a huge git shortstat:

$ git diff --shortstat  3cf0297769..HEAD
 10806 files changed, 697885 insertions(+), 593225 deletions(-)

That also means that any commit/patch you have on CDT before the reformat will almost certainly not apply cleanly.

Therefore I have created this guide for you to reformat your commit and make it easier for you to submit your updated patch to gerrit.

To be able to rebase your commit onto the current master you need to create a version of it with the new formatting. At a high-level the steps we are going to do are:

  1. Reformat your commit with CDT’s new coding style
  2. Create a commit against the pre-formatted CDT that has only the files your commit changed formatted – but without your other changes

The diff between 1 and 2 above is an updated version of the diff for your real change.

Step-by-step this breaks down as:

Pre-requisites

  • Modern bash shell with sed, awk, grep, git, python, etc
    • I am using Ubuntu 18.04 LTS and python 2.7
  • Eclipse EASE with Py4J Python language support
  • Eclipse workspace setup for CDT development, follow the standard setup on the wiki and perform these extra steps:
  • Copy releng/scripts/cleanup.py somewhere outside of your gitroot so you can access it as the branches change later
  • Set ECLIPSE environment variable to path to eclipse on your machine to make some of the commands below easier to copy/paste

Step 1: Checkout the commit to rebase

First step is to checkout in CDT repo the commit you want to rebase onto current master. For example if you are updating a commit in gerrit, you could press Download in the top right and copy/paste the Checkout command:

download

Step 2: Reformat commit to new style

Start by creating a new branch to work on:

git checkout -b commit_to_rebase
Then rebase your change onto commit 35996a5c5ca, where all the per-project formatter settings have been applied, but the code has not been formatted yet. If your original commit is not too far behind this will finish without conflicts. If you have any conflicts, resolve them now.
git rebase 35996a5c5ca5c254959ba48241eaada6dbf8628d

Reformat the code to current standard using the following steps:

Step 2a: Close open editors in Eclipse

The EASE script will run on all open editors, so make sure to start with no open editors.

Step 2b: Open all the Java files modified in your commit

Using git’s diff-tree command we can get all the Java files as a list and then open them with Eclipse.
git diff-tree --no-commit-id --name-only -r commit_to_rebase -- *.java | xargs $ECLIPSE

Step 2c: Run cleanup.py

With Eclipse EASE run cleanup.py:

  1. Create a new Run Launch Configuration of type EASE Script
  2. In Script Source browse the filesystem to where you saved cleanup.py earlier
  3. Ensure Execution Engine is set to Python (Py4J)
  4. Press Run

Each open editor will have the clean up actions run, and then be closed.

Step 2d: Remove trailing whitespace in files

CDT’s code standards require trailing whitespace to be removed from all files marked with remove trailing whitespace in .gitattributes. To do this, run this bit of shell:

git show master:.gitattributes | awk '/# remove trailing whitespace/{getline; print $1}' |
    while read i ; do
        echo "Removing trailing whitespace on $i files"
        git diff-tree --no-commit-id --name-only -r commit_to_rebase -- "$i" | xargs --no-run-if-empty sed -i 's/[ \t]*$//'
    done

Step 2e: Save/commit cleanup changes

Update the change to now contain the formatted files.
git add -u
git commit --amend --reuse-message=HEAD

Step 3: Create a base commit to compare against

In this step create a commit that has only the files modified in the original commit cleaned up, but no other ones. This will be the base for the diff we create later.

Step 3a: Create a new branch

Start by checking out to a new branch the same commit as above with the formatter settings

git checkout 35996a5c5ca5c254959ba48241eaada6dbf8628d -b commit_to_format

Step 3b: Cleanup the modified files

Repeat Step 2a to Step 2d above to cleanup the files.

Step 3c: Save/commit cleanup changes

Save the files which are now formatted, but without the change you wish to rebase in a commit
git add -u
git commit -m"formatted files"

Step 4: Create a new commit that can be cherry-picked to master

Apply the change so you have a history on commit_to_format branch that is two ahead of 35996a5c5c, the first being the formatted files created in step 3, the second being the change we are trying to get onto master, which will be created now

Step 4a: Diff and apply the change

Diff the two branches we just made, that diff is the real work you are trying to get on master, and apply that diff
git diff commit_to_format..commit_to_rebase | git apply

Step 4b: Commit the change

Save the edit, reusing your original commit message (and therefore Change-Id!)
git add -u
git commit --reuse-message=commit_to_rebase

Step 5: Cherry-pick the new commit onto master

You now have a commit, marked with commit_to_format, that can be cherry-picked onto master. If other commits have made a conflicting edit since the reformat you may have some conflicts to resolve when you do this cherry-pick
git checkout master
git cherry-pick commit_to_format

Step 6: Push commit to gerrit

Push your commit now that it is ready, if you are using gerrit you can do:

git push origin master:refs/for/master

Step 7: Clean up your branches

Once you are done make sure to remove the branches you don’t need anymore.

git branch -D commit_to_rebase
git branch -D commit_to_format

All done!

And that is it. Hopefully you have now recreated your patch with just a few automated steps, instead of a painful manual copy+paste job. These instructions are also in the CDT repo in releng/scripts/rebase_helper.sh.

This set of steps shows that there are still some more automatable things that could be done. In particular being able to run Code Clean Ups from the command line in Eclipse would be really nice. Eclipse’s Java code formatter can be run from the command line.

And finally, join in on the cdt-dev mailing list!

The Eclipse Debug Perspective just got a lot better

A Debug Menu!
The next major release of Eclipse (4.8 aka Photon) due in June 2018 has a small – but significant – new feature. The next version will have a Debug menu!

DebugMenu

This improves things in two ways:

  • No more going to awkwardly names Run menu when you want to Debug
  • A less full Run menu as the sometimes overlong Run menu is now split into two distinct and logical parts.

Better Default Layout in Debug Perspective

The first thing I do whenever I create a new workspace is change to the Debug Perspective and do most of my work in that perspective, rarely changing to other perspectives. I re-position the views in the perspective in my own special way, or so I thought until I found out that lots of other developers also prefer roughly the same layout as I have. The new default layout gives more place to the editor and more vertical space to the Debug view (the one with the stack traces).

Here is what the new layout looks like with a few additional views open:

debug_extra_opened_views

So what?

“So what?” I hear you say, “these are small changes, not earth shattering new features.” Changes like this are a big deal, thanks to the marginal gains principle. Eclipse has been a very stable top class IDE for years, but you may have noticed that things haven’t been changing as much in the last number of years. Part of this is down to it being a great IDE already, but other stuff is at play here. Making fundamental changes like the default layout to probably the most commonly used perspective and changing the top level debug menu is a sign that the Eclipse community is alive and well, that the ideas are still coming in. But it also means that the leaders in the community are open to change and evolution. If we don’t evolve we die.

What this means for extenders and plug-in writers?

If you, like me, are developing and extending Eclipse there is some impact on you to. For example, if you are contributing a clearly debug related task to the Run menu, you probably want to start contributing it to the Debug menu going forward. Both so your users can find it and so it does not stand out as being in the wrong place. You probably also want to consider the default opening location for any additional views you provide to the Debug perspective to ensure they pop-up in the logical place.

Of course it goes further than this. You should also take a step back and consider how your tools work. Perhaps it is time to do a Lean Design Critique?

Can’t wait until June?

If you can’t wait until June to get a hold of these improvements, you can pick them up in the latest nightly build, or wait until the Photon M4 milestone build in a few weeks. You can download the full packages from https://www.eclipse.org/downloads/index-developer.php or the Eclipse Platform/SDK on its own from http://download.eclipse.org/eclipse/downloads/

Thanks!

Finally thank you to the Eclipse Platform committers and contributors who are taking the bold steps to change and evolve. A particular thanks to Andrey Loskutov, from whom I borrowed the screenshots from (via Bug 464898 and Bug 513355), see Andrey announcement on the Eclipse mailing list: https://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg14986.html

Woohoo! Java 9 has a REPL! Getting Started with JShell and Eclipse January

With Java 9 just around the corner, we explore one of its most exciting new features – the Java 9 REPL (Read-Eval-Print Loop). This REPL is called JShell and it’s a great addition to the Java platform. Here’s why.

With JShell you can easily try out new features and quickly check the behaviour of a section of code. You don’t have to create a long-winded dummy main or JUnit test – simply type away.  To demonstrate the versatility of JShell, I am going to use it in conjunction with the Eclipse January package for data structures. Eclipse January is a set of libraries for handling numerical data in Java, think of it as a ‘numpy for Java’.

Install JShell

JShell is part of Java 9, currently available in an Early Access version from Oracle and other sources. Download and install Java 9 JDK from http://jdk.java.net/9/ or, if you have it available on your platform, you can install with your package manager (e.g. sudo apt-get install openjdk-9-jdk).

Start a terminal and run JShell:capture1

As you can see, JShell allows you to type normal Java statements, leave off semi-colons, run expressions, access expressions from previous outputs, and achieve many other short-cuts. (You can exit JShell with Ctrl-D.)

Using JShell with Eclipse January

To use Eclipse January, you need to:

1. Download January:

Get the January 2.0.2 jar ( or older version January 2.0.1 jar).

2. Download the dependency jars:

The January dependencies are available from Eclipse Orbit, they are:

3. Run JShell again, but add to the classpath all the jars you downloaded (remember to be the in the directory you downloaded the jars to):

Windows:

"c:\Program Files\Java\jdk-9\bin\jshell.exe"  --class-path org.eclipse.january_2.0.2.v201706051401.jar;org.apache.commons.lang_2.6.0.v201404270220.jar;org.apache.commons.math3_3.5.0.v20160301-1110.jar;org.slf4j.api_1.7.10.v20170428-1633.jar;org.slf4j.binding.nop_1.7.10.v20160301-1109.jar

Linux:

jshell --class-path org.eclipse.january_2.0.2.v201706051401.jar:org.apache.commons.lang_2.6.0.v201404270220.jar:org.apache.commons.math3_3.5.0.v20160301-1110.jar:org.slf4j.api_1.7.10.v20170428-1633.jar:org.slf4j.binding.nop_1.7.10.v20160301-1109.jar

Some notes:
Some version of jshell the command line argument is called -classpath instead of --class-path
If you are using git bash as your shell on Windows, add winpty before calling jshell and use colons to separate the path elements.

capture2

Then you can run through the different types of January commands. Note JShell supports completions using the ‘Tab’ key. Also use /! to rerun the last command.

Import classes

Start by importing the needed classes:

import org.eclipse.january.dataset.*

(No need for semi-colons and you can use the normally ill-advised * import)

Array Creation

Eclipse January supports straightforward creation of arrays. Let’s say we want to create a 2-dimensional array with the following data:

[1.0, 2.0, 3.0,
 4.0, 5.0, 6.0,
 7.0, 8.0, 9.0]

First we can create a new dataset:

Dataset dataset = DatasetFactory.createFromObject(new double[] { 1, 2, 3, 4, 5, 6, 7, 8, 9 })
System.out.println(dataset.toString(true))

This gives us a 1-dimensional array with 9 elements, as shown below:

[1.0000000, 2.0000000, 3.0000000, 4.0000000, 5.0000000, 6.0000000, 7.0000000, 8.0000000, 9.0000000]

We then need to reshape it to be a 3×3 array:

dataset = dataset.reshape(3, 3)
System.out.println(dataset.toString(true))

The reshaped dataset:

 [[1.0000000, 2.0000000, 3.0000000],
 [4.0000000, 5.0000000, 6.0000000],
 [7.0000000, 8.0000000, 9.0000000]]

Or we can do it all in just one step:

Dataset another = DatasetFactory.createFromObject(new double[] { 1, 1, 2, 3, 5, 8, 13, 21, 34 }).reshape(3, 3)
System.out.println(another.toString(true))

Another dataset:

 [[1.0000000, 1.0000000, 2.0000000],
 [3.0000000, 5.0000000, 8.0000000],
 [13.000000, 21.000000, 34.000000]]

There are methods for obtaining the shape and number of dimensions of datasets

dataset.getShape()
dataset.getRank()

Which gives us:

jshell> dataset.getShape()
$8 ==> int[2] { 3, 3 }

jshell> dataset.getRank()
$9 ==> 2

Datasets also provide functionality for ranges and a random function that all allow easy creation of arrays:

Dataset dataset = DatasetFactory.createRange(15, Dataset.INT32).reshape(3, 5)
System.out.println(dataset.toString(true))

[[0, 1, 2, 3, 4],
 [5, 6, 7, 8, 9],
 [10, 11, 12, 13, 14]]


import org.eclipse.january.dataset.Random //specify Random class (see this is why star imports are normally bad)
Dataset another = Random.rand(new int[]{3,5})
System.out.println(another.toString(true))

[[0.27243843, 0.69695728, 0.20951172, 0.13238926, 0.82180144],
 [0.56326222, 0.94307839, 0.43225034, 0.69251040, 0.22602319],
 [0.79244049, 0.15865358, 0.64611131, 0.71647195, 0.043613393]]

Array Operations

The org.eclipse.january.dataset.Maths provides rich functionality for operating on the Dataset classes. For instance, here’s how you could add 2 Dataset arrays:

Dataset add = Maths.add(dataset, another)
System.out.println(add.toString(true))

Or you could do it as an inplace addition. The example below creates a new 3×3 array and then adds 100 to each element of the array.

Dataset inplace = DatasetFactory.createFromObject(new double[] { 1, 2, 3, 4, 5, 6, 7, 8, 9 }).reshape(3, 3)
inplace.iadd(100)
System.out.println(inplace.toString(true))

[[101.0000000, 102.0000000, 103.0000000],
 [104.0000000, 105.0000000, 106.0000000],
 [107.0000000, 108.0000000, 109.0000000]]

Slicing

Datasets simplify extracting portions of the data, known as ‘slices’. For instance, given the array below, let’s say we want to extract the data 2, 3, 5 and 6.

[1, 2, 3,
 4, 5, 6,
 7, 8, 9]

This data resides in the first and second rows and the second and third columns. For slicing, indices for rows and columns are zero-based. A basic slice consists of a start and stop index, where the start index is inclusive and the stop index is exclusive. An optional increment may also be specified. So this example would be expressed as:

Dataset dataset = DatasetFactory.createFromObject(new double[] { 1, 2, 3, 4, 5, 6, 7, 8, 9 }).reshape(3, 3)
System.out.println(dataset.toString(true))
Dataset slice = dataset.getSlice(new Slice(0, 2), new Slice(1, 3))
System.out.println(slice.toString(true))

slice of dataset:

[[2.0000000, 3.0000000],
 [5.0000000, 6.0000000]]

Slicing and array manipulation functionality is particularly valuable when dealing with 3-dimensional or n-dimensional data.

Wrap-Up

For more on Eclipse January see the following examples and give them a go in JShell:

  • NumPy Examples shows how common NumPy constructs map to Eclipse Datasets.
  • Slicing Examples demonstrates slicing, including how to slice a small amount of data out of a dataset too large to fit in memory all at once.
  • Error Examples demonstrates applying an error to datasets.
  • Iteration Examples demonstrates a few ways to iterate through your datasets.
  • Lazy Examples demonstrates how to use datasets which are not entirely loaded in memory.

Eclipse January is a ‘numpy for Java’ but until now users have not really been able to play around with it in the same way you would numpy in Python.

JShell provides a great way to test drive libraries like Eclipse January. There are a couple of features that would be nice-to-have such as a magic variable for the last result (maybe $_ or $!) and maybe a shorter way to print a result (maybe /p :-). But overall, it is great to have and finally gives Java the REPL and ability to be used interactively that many have gotten so used to with other programming languages.

In fact we will be making good use of JShell for the Eclipse January workshop being held at EclipseCon France, see details and register here:  https://www.eclipsecon.org/france2017/session/eclipse-january

eclipseConV2

Launching with EASE

Launching with EASE - using auto completion
Launching with EASE – using auto completion

I have been doing some experiments with Eclipse EASE in preparation for determining the suitability – and creating a prototype – of using EASE with non-JVM scripting languages, in particular Python (CPython specifically).

To achieve this end goal I am working on a new module, /System/Launch, to allow me to explore that functionality. As a practical goal I am trying to use EASE to solve a long term problem of how to launch more complicated systems, especially those that run out of steam with the Launch Groups. For instance, multiple launches for debugging a multi-core system (especially with customizations and interdependencies) or parametrization of job launches.

Scripted Launch Group

As a starting point I want to simply re-create what Launch Groups can do in Javascript. So with EASE installed, including my new module, this is what a simple Launch Group replacement looks like.

The Launch Module

The Launch Module is a new module under consideration for EASE (Bug 478397). Its key method is the “launch” method which takes a Launch Configuration Name and an optional Launch Mode. The method then loads the named launch configuration and launches it, finally returning an ILaunch to allow future interaction with.

By using the launch method multiple times, with some additional control around it, allows complex launch sequences to be created.

Examples

For these examples I have already created three launch configurations for my individual tools (the name of the launch configuration is the quoted string in the bullet list):

  • “Prepare” – An External Tools configuration which prepares my test environment
  • “Server” – A PyDev supplied, Python Run configuration to launch my Python server
  • “Client” – A Java configuration to launch my Java client

Running The Examples

  1. Create three launch configurations named as above.
  2. Clone/Import “JavaScript Snippets” project. The examples are located in org.eclipse.ease.scripts git repository (or will be soon, follow Bug 478397 to find out more).
  3. Open the scripts in “Launch Module Examples” folder.
  4. Right-click on the desired script
  5. Choose Run As –> EASE Script

Alternatively paste the lines from the examples into the Rhino Script Shell console (as in the screenshot above). You get auto-completion of the names of the launch configurations and the launch modes too!

Example 1 – Fire and forget

In this first example, we simply launch the Client in Debug mode.

loadModule("/System/Launch")

launch("Client", "debug")

Line 1: load the Launch module, this populates the namespace with all the methods defined in the Launch module.
Line 3: launch the existing launch configuration “Client” in debug mode.

Example 2 – Replicate Functionality from Launch Groups

In this example, we prepare our environment with the “Prepare” configuration, then launch the “Server” and “Client” configurations.

loadModule("/System/Launch")

prepare = launch("Prepare")
while (!prepare.isTerminated()) {
 java.lang.Thread.sleep(1)
}

launch("Server")
java.lang.Thread.sleep(3000)

launch("Client", "debug")

Line 1: load the Launch module
Line 3: launch the Prepare configuration
Line 4-6: Busy-wait until the Prepare launch has terminated[1]
Line 8: launch the server
Line 9: Wait 3 seconds for the server to be ready
Line 11: launch the client in Debug mode

Example 3 – Terminating the Server Automatically

Example 1 and 2 are a promising start, but do not yet add any new functionality to Eclipse. So what do you do if you want the server to stop automatically when you finish debugging your client. Well that is really easy now, just monitor the client launch and terminate the server.

loadModule("/System/Launch")

prepare = launch("Program prepare")
while (!prepare.isTerminated()) {
 java.lang.Thread.sleep(1)
}

server = launch("Python Server")
java.lang.Thread.sleep(3000)

client = launch("Java Client", "debug")
while (!client.isTerminated()) {
 java.lang.Thread.sleep(1)
}
server.terminate()

Line 1-7: the same as Example 2
Line 8: launch the Server, but keep a handle of the ILaunch
Line 9: Wait 3 seconds for the server to be ready
Line 11: launch the Client
Line 12-14:Busy-wait until the Client debug session launch has terminated[1]
Line 15: terminate the server

This is a screenshot that shows what the Debug View looks like when we are busy-waiting on line 12.
At the top is the EASE Script launch of the example.
Then is the now terminated Prepare launch.
Followed by the still running Server in Run mode.
And finally, the Java Client, in Debug mode stopped at a breakpoint in main.

ease_launching_debug_view
Terminating the Server Automatically – Debug View

More Advanced Options

With the full power of the scripting language you can take these examples to the next step. A good place to start would be to remove the 3 second delay on Line 9 and replace that with some logic that actually determines if the server is ready to accept connections.

Other Functionality in the Launch Module

The Launch module is very new and I invite additional contributions to it to make it more useful. For now this is a quick overview of what it does:

String[] getLaunchConfigurationNames()

Returns an array of all the Launch Configuration Names known to the Launch Manager. These names can be used as the argument to the getLaunchConfiguration, launch and launchUI methods.

ILaunchConfiguration[] getLaunchConfigurations()

Returns an array of all the Launch Configurations known to the Launch Manager. These can be used as the argument to launch and launchUI methods.

ILaunchConfiguration getLaunchConfiguration(String name)

Return the launch configuration given by name parameter. The launch configuration can be edited or otherwise operated on. See ILaunchConfiguration.getWorkingCopy().

ILaunch launch(String launchConfigurationName, String mode)
ILaunch launch(ILaunchConfiguration configuration, String mode)

Launch the configuration either given by name or a launch configuration and return the ILaunch for further processing. This is the way to launch a configuration within a script which is itself launched.

ILaunch launchUI(String launchConfigurationName, String mode)
ILaunch launchUI(ILaunchConfiguration configuration, String mode)

Launch the configuration in the UI thread. This method respects the workspace settings for things like building before launching.

ILaunchManager getLaunchManager()

Obtain the platform launch manager. This allows access to the Eclipse debug core launch manager, allowing control over all non-UI aspects of launches. The most valuable of these should be wrapped for ideal script usage and made available in the module itself.[2]

Footnotes

1. A better API than busy-waiting is probably desired here, but that is for another day (and more Javascript knowledge).
2. Additional UI functionality is within the DebugUITools class, enabling access to this class directly from within the launch module is an option. Additionally, a Debug module would be very useful.