Archive for the ‘Programming’ Category

Batch Table Operations – A QtiPlot plug-in/script

Thursday, January 30th, 2014

I’ve been taking lots of data at work. Each data file collected has to be imported to QtiPlot as a table and then several columns are added to the table and calculated from the imported data. Some days I have over 100 data sets, which means > 100 tables to manipulate. It was becoming impractical to perform all of the necessary table operations by hand. This script was written to automate the table operations I needed.

Batch Table Operations

To use one simply selects the tables to be affected from the list of existing tables. Then one selects and configures the operation to perform. Finally, clicking the “apply” button will perform the operation on the chosen tables.

As of today (1/30/14) the script allows adding and removing columns based on name, changing column types (x, y, z, etc.), and applying formulas to columns. Obviously, alot more could be done, but at the moment, this fits my needs. As I require more features I will add them. If you would like to request a feature add a comment or send me an email and I’ll see what I can do.

Download:

batchTable.zip

Edit 8/5/14:

I just tried to install this in version 0.9.8.9 of QtiPlot running on Windows 7 but was unable to get the script interface to load. The particular version of QtiPlot installed on the target machine came from an unofficial Windows binary (https://www.cells.es/old/Members/cpascual/docs/unofficial-qtiplot-packages-for-windows) that I’ve used before. That QtiPlot package comes with parts of PyQt4 but not the entire library. As such it is necessary to install both python 2.7 and PyQt4 as well. Then, the …/PyQt4/uic module folder has to be copied into the PyQt4 directory of the the QtiPlot installation.

At least, that was the solution in the past. Now this no longer works. The uic module is found but fails to load. I expect it’s because the version of Qt against which QtiPlot was complied and the version for which the uic module were written are now too different. Or it may be that the version of Python is too new (2.7.8). I got this to work in the past with Python 2.7.5. Anyway, under Windows at least, I no longer expect for the batchTable plugin to be usable because it draws its interface using the uic module I can’t get working. Things should still work on Linux installations. The long term solution would be to rewrite the plugin to use a Tkinter interface instead of Qt thereby removing the dependency problem.

Extract Y at X – A QtiPlot plug-in/script

Saturday, September 7th, 2013

I’ve written and posted a QtiPlot plug-in/script (python) I call “Extract Y at X” that extracts the y-values at a given x-value for a series of curves contained in a graph and generates a new table with those values.

The files can be downloaded using the link below. The .zip file contains three files: the script (.py) file, an xml file defining the Qt inteface (.ui), and a README file. Refer to the README file for notes on installation and use.

I wrote this script to analyze a large group of datasets that I had collected. Each dataset was measurement of current as a function of voltage for a device I had constructed and each was collected at a different temperature. In a single dataset the current grew exponentially and across datasets the current grew exponentially with temperature. In order to quantitatively determine how the current changed with temperature I needed to extract the current at some voltage from each dataset. The usual way to do this is to look through the data manually in each dataset and find the point I needed, e.g. the current at +1 V for each dataset. Given that I had so many datasets, it was useful to write a script to do this for me.

Download:

extractYatX.zip

Update 7/18/14:

I found and fixed a bug that limited the number of extracted points to 30.

Update  8/5/14:

As described in another post this plug-in may not work in Windows versions of QtiPlot without the proper dependencies satisfied. The trouble is not the script itself but its reliance on the uic module of Qt which may not have been packed with a particular QtiPlot version when it was compiled and is therefore unavailable to the script. Linux users should be fine, so long as Qt libraries are installed.

An improved key-reader library for the SainSmart LCD Keypad Shield (for Arduino)

Monday, August 5th, 2013

The LCD Keypad shield from SainSmart is a bargain. For ~$8 (and free shipping, at the time of this post) you get 32 character (16×2) blue/white LCD panel with a built-in keypad with 5 momentary switches. This provides a quick solution for interaction with Arduino projects. To control the LCD part of the shield, one can use the Arduino supplied LiquidCrystal library (included with Arduino IDE). In the Key_Grab demo (v. 0.2) on the SainSmart website is included a library, DFR_Key, for reading which keys are pressed on the keypad. However, the library is not written well. This post describes both the problem and the solution.

First, a little background is necessary about how the keypad signals a key is pressed. Only a single analog-in pin is used and the shield uses a voltage divider to send a voltage that corresponds to a given key. To detect a key, one has only to read the voltage at the analog pin and determine if it falls within a certain range. For example, a voltage of 0 to 0.7 V signals the UP key was pressed. The DFR_Key library contains a getKey() function that does this and returns an integer from 0 to 5 that represents the key pressed (0 is supposed to mean no key, but as written means the last key was just released).

You’d expect that when you call getKey() it would read the voltage at the pin, figure out what key that represented, and send back the integer representing the particular key. So really, it should just be translating the voltage to an integer from 0 to 5 (none, select, left, up, down, right, respectively). But, there is some “extra functionality” the library provides that is the problem. Here’s how the logic of the getKey() function works:

Has a minimum time passed since the last time I checked for a key press?

If not, do nothing – return no value at all!

If so, read the voltage at the pin. Is this voltage different than the voltage I read the last time I checked for a keypress?

If not, return -1.

If they’re different, it means a key was pressed right? Determine which key was pressed and return an integer from 0 to 5 representing that key.

The first bit of “extra functionality” is what the library refers to as the refreshRate. The idea is that you don’t want to check for a key press too quickly, else the function might return that the key is being held down when in fact it’s just a normal press. This makes some sense if you simply dump a getKey() call in your Arduino sketch’s loop() function. You would then check the returned value to see if it actually did a key press. If it decided enough time hadn’t pass it would return….oh wait! It doesn’t return anything in this case. That’s not appropriate behavior. When the program says getKey(), it should expect a key code sent back. Therefore, it should be the job of the programmer to only call getKey() at appropriate intervals (say every 100 ms).

That brings up the next problem. There are five keys on the keypad. Add in an extra code for “no key being pressed” and you get the six integers from 0 to 5. Perfectly reasonable. So what’s this -1 return value for? The library defines the alias SAMPLE_WAIT for this value (as opposed to NO_KEY, and UP_KEY, etc.). I’m not sure what the meaning was supposed to be, but this value is returned when the getKey() function determines that the key being pressed is the same as the key that was detected upon the last call to getKey(). At first, it would seem to be letting you know that the last key pressed is being held down. But 1: you don’t know what the last key pressed was, unless you’re tracking it yourself. And 2: that may not even be true – if you make two getKey() calls 60 s apart, whose to say whether the key was being held the whole time; all we know is that the same key was pressed as last time. So it’s telling us the same key was pressed as before and according to point 1, I’m keeping track of what that key was. If it had just returned the actual key code, I could have figured this out for myself by comparing it to my stored value of the last key pressed.

Now to the final problem. If you follow the logic stated above carefully, you’ll see that to determine if the same key was pressed, the getKey() function compares the voltage read at the analog pin to the voltage it read last time it detected a key was pressed. But the voltage does not directly correspond to a given key – a key press can be signaled by a range of voltages. For example, assume on the first call to getKey() it reads a value of 200 mV. That corresponds to the UP_KEY and it returns 3. On the next call to getKey() assume it reads a value of 205 mV. The user is pressing the up key again (or holding it down) but the voltage is slightly different. The getKey() function compares the new value, 205 mV, to the old value, 200 mV, and thinks a new key is being pressed. So instead of returning -1 for “same key is being pressed” it returns 3 for “up key is being pressed”.

Actually, this should be the behavior! Upon every call, the getKey() function should just return an integer representing the key that is being pressed at the time the call is made. But because the authors meant for it to return -1 when there was no change in the key being pressed, and meant for it to be called on every loop() iteration, the Key_Grab demo flickers the screen like crazy. Actually, it works fine when the UNO board is removed from all sources of noise and interference. But put it next to a piece of electronics (e.g. your computer) and the noise causes the voltage supplied to the analog pin to fluctuate, new key presses to be registered, and the demo to update the screen on every loop() iteration. It’s the constant updating that causes the flicker which makes it appear as though you got ripped off $10.

How can we fix this mess? Actually, the fixes are simple and actually reduce the code in the library.

  1. Remove all of the logic relating to the “refreshRate”. The programmer will be forced to decide how often to call getKey().
  2. Remove the comparison of the current key press to the previous key press. The programmer can decide if the user is pressing the same key by recording the last pressed key.

Here is the derived library which I’ve called LCD_Keypad_Reader.cpp (feel free to rename). The original SainSmart DFR_Key library didn’t come with a license so I’m not putting one on this derived work either. I don’t really care what you do with this code, but it’s possible that SainSmart could impose some kind of limitations.

#include "Arduino.h" 
#include "LCD_Keypad_Reader.h" 

static int DEFAULT_KEY_PIN = 0;  
static int DEFAULT_THRESHOLD = 5; 

// The Sainsmart keypad uses a voltage divider to deliver a voltage  
// between 0 and 5 V that corresponds to the key being pressed in  
// order to use only a single input pin. The values below are from 0 to  
// 1023 because the Arduino uses a 10 bit resolution. 
static int UPKEY_ARV = 144; // 0.720 V, that's read "analogue read value" 
static int DOWNKEY_ARV = 329; // 1.645 V 
static int LEFTKEY_ARV = 505; // 2.525 V 
static int RIGHTKEY_ARV = 0; // 0 V 
static int SELKEY_ARV = 742; // 3.710 V 
static int NOKEY_ARV = 1023; // 5.115 V 

LCD_Keypad_Reader::LCD_Keypad_Reader() 
{     
  _keyPin = DEFAULT_KEY_PIN; 
  _threshold = DEFAULT_THRESHOLD; 
  _curInput = NO_KEY; 
  _curKey = NO_KEY; 
} 

int LCD_Keypad_Reader::getKey() 
{ 
  _curInput =  analogRead(_keyPin); 
  _curKey = categorizeKey(_curInput); 
  return _curKey; 
} 

int LCD_Keypad_Reader::categorizeKey(int analogKeyValue){ 
  int categorizedKeyValue = 0; 

  if (analogKeyValue > UPKEY_ARV - _threshold && analogKeyValue < UPKEY_ARV + _threshold ){ 
      categorizedKeyValue = UP_KEY; 
  } 
  else if (analogKeyValue > DOWNKEY_ARV - _threshold && analogKeyValue < DOWNKEY_ARV + _threshold ){ 
      categorizedKeyValue = DOWN_KEY; 
  } 
  else if (analogKeyValue > RIGHTKEY_ARV - _threshold && analogKeyValue < RIGHTKEY_ARV + _threshold ){ 
      categorizedKeyValue = RIGHT_KEY; 
  } 
  else if (analogKeyValue > LEFTKEY_ARV - _threshold && analogKeyValue < LEFTKEY_ARV + _threshold ){  
      categorizedKeyValue = LEFT_KEY; 
  } 
  else if (analogKeyValue > SELKEY_ARV - _threshold && analogKeyValue < SELKEY_ARV + _threshold ){ 
      categorizedKeyValue = SELECT_KEY; 
  } 
  else{ 
    categorizedKeyValue = NO_KEY; 
  } 

  return categorizedKeyValue; 
}

And here is the header file, LCD_Keypad_Reader.h

#ifndef LCD_Keypad_Reader_h 
#define LCD_Keypad_Reader_h 

#include "Arduino.h" 

#define SAMPLE_WAIT -1 
#define NO_KEY 0 
#define UP_KEY 3 
#define DOWN_KEY 4 
#define LEFT_KEY 2 
#define RIGHT_KEY 5 
#define SELECT_KEY 1 

class LCD_Keypad_Reader 
{ 
  public: 
    LCD_Keypad_Reader(); 
    int getKey(); 
    int categorizeKey(int); 
  private: 
    int _keyPin; 
    int _threshold; 
    int _curInput; 
    int _curKey; 
}; 

#endif

pyK-Control 2400 version 1.0 alpha released

Thursday, June 20th, 2013

pyK-Control 2400 is a serial port controller and communicator for the Keithley 2400 SourceMeter written in python using the GTK+3 framework.  It’s main purpose is to configure and run current or voltage sweeps from a convenient software interface. The current release, 1.0 alpha, has been tested in Debian GNU/Linux and Windows XP.

More information can be found on the pyK-Control 2400 project page.

GtkGrid alignment issues

Friday, May 17th, 2013

Recently I ran into a problem laying out some controls with a GtkGrid widget that took quite a while to figure out. My controls were contained within a vertical GtkBox with three rows. The first row contained a grid with two columns. The left column housed labels and the right housed the actual controls. The next two rows of the vertical box were similar except the grids within were contained within a frame. The main vertical box itself is in the left column of a 2 column horizontal box. The right column contained a GtkScrolledWindow with a GtkTextView. The scrolled window was set to expand to take up all of the remaining horizontal space in the window. Problem was, the controls on the left weren’t aligned properly. A picture should help the explanation.

Screenshot - 05172013 - 11:48:09 AM

In the image, the vertical box (red) has been enclosed within a GtkFrame (frame13) to show its borders. As you can see, the controls in the yellow grids are aligned to the left and do not fill the cells of the grid. I would like them to be either right aligned or to fill the cells of their parent grids so that the right edge of all controls are aligned. This turned out to be nearly impossible. Most solutions I tried would cause the scrolled window in the adjacent column to shrink for apparently no reason to about 1/3 of the total horizontal area. The image below is an example of one trial solution in which I set the horizontal expand property of the GtkLabel “Loop” to True. The label expands to push the neighboring control (a GtkComboText) to the right side of the vertical box extents. But for some reason, the space outside of the vertical box is allocated to the label’s parent grid instead of the scrolled window.

Screenshot - 05172013 - 11:43:50 AM

The behavior is the result of the GtkGrid inheriting the hexpand property its child, the “Loop” label, and so on up the chain. The vertical box doesn’t appear expanded, but it is allocated the space anyway. To override this reverse inheritance behavior, one must set the hexpand property of the parent container (the vertical box in this case) to False. I’m designing in Glade and by default, this property is “No” for all widgets. This is not the same as setting the property to False. The workaround is to add a line of code to force the property to be false. I have unfortunately not found a fix that can be applied within Glade. The line looks something like this:

    builder = Gtk.Builder()
    builder.get_object('verticalBox').set_hexpand(False)

From there, setting the horizontal expand on the labels pushes the controls to the right, but no space is allocated outside of the vertical box parent. The successful alignment, with frame13 removed and some column spacing added between the controls and the scrolled window looks like this:

Screenshot - 05172013 - 12:11:45 PM

This project is being developed in Python 2.7+ using the Gtk+3 framework. The UI is designed in Glade 3.

Saving the image drawn with Cairo on a GtkLayout or GtkDrawingArea using a pixbuf

Wednesday, May 8th, 2013

In my previous post, I discussed how to get the draw() event to fire for Gtk3 widgets in order to draw on them using cairo, specifically the GtkLayout and GtkDrawingArea widgets. Since that time I’ve had some success writing code to plot data from a connected source meter (through the RS232 port) in real time. I got stuck again trying to save the image drawn on the GtkLayout to a file. Most of the material I found on the internet wasn’t up to date and not relevant for the GTK3 framework. Finally I figured out how to save the image and it’s pretty easy to do too.

The solution is to get a GdkPixbuf from the widget’s GdkWindow and use the GdkPixbuf’s .savev() method to write the image to disk. For some reason, when I use the .get_window() method of GtkWidget’s, I don’t get a GdkWindow returned but rather a GdkX11Window object (gtk.gdk.X11Window). I’m not sure why there’s a difference but it seems to be ok to feed this into the Gdk.pixbuf_get_from_window() function. Please note that you must use the .savev() method of the pixbuf. Most sources online use the .save() method but that seems to be gone in Gtk3 (or perhaps there is no python binding?).

For example, assume you have a GtkLayout stored in the variable layout (I find mine with layout = self.builder.get_object(‘mylayout’) – because I’m using Glade). To capture an image of the layout and whatever you’ve drawn on it, you would do the following:

pixbuf = Gdk.pixbuf_get_from_window(layout.get_window(), 0, 0, width, height)
pixbuf.savev("foobar.png","png", [], [])

In this example, width and height need to be the width and height of the widget. The rectangular area defined by (0,0) to (width, height) is saved. In my case, I did something like this:

rect = layout.get_allocation()
width = rect.width
height = rect.height

In the .savev() method, the first argument is the file name, the second is the type, and the third and forth options are arrays of options and their settings. I pass empty arrays because I don’t set any options.

Note that this works with GtkLayout and should work with GtkDrawingArea and GtkWindow. For other widgets, the window returned by .get_window() seems to be that of the parent GtkWindow. If you want to capture an image of those wigdets, you’ll need to specify the x,y position of their top left corners in the Gdk.pixbuf_get_from_window() call.

layout

Drawing to GtkLayout and GtkDrawingArea with Python

Monday, May 6th, 2013

This one took me several hours to figure out. I’m writing an application that will include a graph of some data collected from the serial port (using the pySerial library). I’m writing the application in Python (2.7.2+) using the GTK 3 framework. UI design is being done in Glade 3. To make the graph, I wanted the base object to be a GtkLayout widget because this widget can be scrolled (inside a GtkScrolledWindow), can contain other widgets, has a fixed layout, and can be directly drawn to.

The problem was, I couldn’t get the draw event for the GtkLayout to trigger no matter what I did. The problem was worsened by the fact that most of the information online is not relevant for Gtk 3, or if it is, it uses the GtkDrawingArea widget instead. But, even when I gave up and went with a GtkDrawingArea widget, I still failed to trigger the draw event.

Finally, after reading everything ever written on the internet, I found the answer in a forum (link below). I was missing the python-gi-cairo package (Debian package) which has the python cairo bindings. After installing this package, the draw event triggers as expected and I’m able to draw to both the GtkLayout and GtkDrawingArea widgets. I didn’t even have to add another “import” line to my script, which is nice, but the dependency still exists.

Link: http://python.6.x6.nabble.com/pygi-gtk-drawingarea-doesn-t-work-td4981920.html

A GtkDrawingArea example: https://git.gnome.org/browse/pygobject/tree/demos/gtk-demo/demos/drawingarea.py