innovative user interfaces for the Android using Processing

Android programming can sometimes feel a bit clunky because of that gap of time between changing code and seeing the effect.  It takes some time to compile and upload the sketch to the phone, and even longer to get the virtual machine up and running.  This gap of time can slow the tinkering and exploration process.  Processing allows you to prototype interactive visualizations faster.  I’ve found Processing to be a great place to prototype ideas for Android UIs- especially new ways to interact with screens, using processing-android to export android applications.

In this post I’ll describe a bit how you can rapidly prototype gestural UIs in Processing and then hook them up to a larger Android application in Eclipse.  However, there are limitations.   As you’ll see, there are methods specific to android programming that work only in processing-android, and others that are easier to integrate in Eclipse.

The accompanying code to this post is here.

Whats kind of cool about writing this app in Processing is that my app can run on the Desktop, in the browser and (by changing two lines) on my Android phone.

Making the basic UI in Processing

The application I want to make is an interface to output low frequency waveforms.  I want to generate repeating strings of numbers, which I can then hook up to something else, perhaps an LED, or pitch parameter for a synthesizer.  I build the interface in Processing, using mousePressed to perform actions when the mouse is clicked or not.  The location of the mouse is given by mouseX and mouseY.  Basically the position along the Y axis maps to a value.  When I have the mouse pressed it begins recording numbers and I can move along the Y axis.  When I let go, it repeats that string over and over.

During the development process I just push the processing ‘play’ button to see the results of my code. I can make changes in the color, background, font, whatever, and then push play again to see the effects. Closing the time in the development loop makes it much easier to settle to a UI I like.

Above you can see the app. It is rendered in the browser with processingjs. Basically I produced a single sketch from the multiple processing (.pde) files with cat *.pde > processing_js_script.pde, and then linked to that file in a canvas element.

Moving onto the Android

Once I get that UI, I switch into Android mode and resave the sketch adding “_android”.

I do this to make changes specific to the android version of the UI. I want to first make some layout changes. I want the app to always be in landscape mode (horizontal), and I want it to take up the whole screen on my phone. I replace

size(800, 400);

in the Processing sketch to

size(screenWidth, screenHeight);

in the android sketch. At this point you can plug in your phone and run it on your phone with Sketch>>Run on Device. When you do that you’ll find that mousePressed is called when you touch the screen, and that mouseX and mouseY is given by the x,y coordinates of your finger.

There are other methods implemented for processing-android that you can include to further develop your user interfaces. If you wanted to include multitouch (I used this when making Duel Cities) you would include extra code in the processing-android sketch instead of the Processing sketch since Processing would not recognize the multitouch methods. Similarly you can include sound events for games in the processing-android sketch. I find the forums are a great resource to see what was been implemented in processing-android.

Now there are some features that you may want to use in your application that are not yet implemented with Processing. I found this recently when I was trying to include Bluetooth in my app. I tried a few and my processing apps crashed.

Moving into Eclipse for more Android functionality
With the basic UI completed, I can export the app into android. This is the right arrow on the processing-android controls:

This introduces a new folder in your sketch called ‘android’. You can open up these files as an Android project in Eclipse. You’ll notice all of the processing code is included as a class extending PApplet. I found it pretty useful to see this, as it gave me an idea of how processing-android wrapped my code to get it working on the droid. Now you can integrate your UI from processing with other libraries or into larger Android applications.

You can check out the code for the gesture looper on github.