Archive for the ‘Computer’ Category

Finding the area under a curve with a spreadsheet program (Excel, Libreoffice Calc) or QtiPlot

Tuesday, October 28th, 2014

For performing elemental analysis by x-ray photoelectron spectroscopy, I needed to integrate and find the area under the peaks in some spectra. One approach is to fit each peak with a Gaussian function and use the area of the Gaussian. For some convoluted peaks, I wanted something simpler. I wanted to get the total area under each peak without fitting it in any way.

I use QtiPlot almost exclusively for plotting and fitting (fityk is my other go-to program) and one of the menu options for plots is Analysis->Integrate. That option brings up a tiny dialog where you can select the curve and range over which to integrate.

Screenshot - 10282014 - 09:13:32 PM

In the results log, an area is printed for the integrated curve. I wasn’t sure what QtiPlot was doing so I did an experiment with a simple set of data. I now know conclusively the algorithm used by QtiPlot and am able to replicate it in a spreadsheet program. I use LibreOffice Calc, but the instructions here should work in Excel as well.

Given a set of n points, P1 through Pn, a curve is generated by connecting each point with straight lines.qtiplot-integration-plot1

The area under the curve between two points, (x1, y1) and (x2, y2), is a trapezoid with area A = 1/2 |x1 – x2| (y1 + y2).qtiplot-integration-plot2

To find the area under the curve defined by a set of points, QtiPlot iterates through the points from P0 to Pn-1. For each point Pi, it calculates the area using Pi as (x1, y1) and Pi+1 as (x2, y2) and sums all of the trapezoid areas.

The above process is easy to replicate in LibreOffice Calc (or Excel) with the x,y data that define the curve. Assuming the x coordinates are in column A starting at row 1 and the y coordinates are in column B starting row 1, the areas of the individual trapezoids can be generated in column C. The formula in the first row is: “=0.5*abs(A1-A2)*(B1+B2)”. That formula is then copied down column C in all of the rows but the last. Finally, the areas are summed to give the total area under the curve.

Screenshot - 10282014 - 09:48:15 PM

Update apt public keys in Debian to fix package authentication errors

Thursday, May 29th, 2014

The apt program for Debian uses public and private keys to validate packages downloaded from the repositories. You can learn all about it at Public keys are stored in /etc/apt/trusted.gpg. If keys are missing, outdated, corrupted, etc. apt will complain when trying to install packages with an error like “The following signatures couldn’t be verified because the public key is not available”. You can choose to ignore the error. If so, when it reoccurs in the future you will see warnings like “WARNING: The following packages cannot be authenticated!”.

The solution to this problem is to update your public keys for the Debian repositories. These keys are installed via the package debian-archive-keyring. A simple “apt-get install debian-archive-keyring” is probably enough to update the keys are remove the errors/warnings during package installs.

Accessing status and diagnostic information for a Motorola Surfboard cable modem

Wednesday, May 28th, 2014

My ISP (to protect its identity, let’s call it Flomflast) drops my internet connection frequently. During the latest occurrence, I noticed from my router’s status page that the WAN address had been assigned to I was surprised to see this since I believed there was no connection to the ISP via the cable line. How could the ISP DHCP server assigned an ip address to the router? And why the private network address?

I guessed that if the DHCP server was assigning addresses in the 192.168.100.x range then itself or another server (the gateway?) might be located at I typed the address in the address bar of my favorite web browser and was greeted with a web page being served by my Motorola Surfboard cable modem.

Screenshot - 05282014 - 11:14:33 PM

On the configuration page I found information that the cable modem has a built-in DHCP server that takes over when the internet connection is dropped. As configured, it begins assigning addresses at Since my router is the only device on that side of the network with the cable modem, it was assigned the first ip address in the range.

The status page for the cable modem is especially handy. From the comfort of my chair I can see that the internet connection is indeed lost without the usual process of several rounds of pinging followed by a walk to another room to check the modem lights. Even with the internet connected, the modem is still accessible at

auth.log errors: missing

Thursday, May 1st, 2014

I had an authorization log (/var/lib/auth.log) was filling up with errors related to a missing library:

May  1 01:25:01 servername CRON[1906]: pam_unix(cron:session): session opened for user root by (uid=0)
May  1 01:25:01 servername CRON[1906]: pam_unix(cron:session): session closed for user root
May  1 01:25:16 servername auth: PAM unable to dlopen(/lib/security/ /lib/security/ cannot open shared object file: No such file or directory
May  1 01:25:16 servername auth: PAM adding faulty module: /lib/security/

…over and over. I tried upgrading winbind to the latest version in the Debian stable repositories. I installed libpam-winbind (2:3.6.6-6+deb7u3 for both). Neither of those actions fixed the problem. The library,, was not present in /lib/security.

After reading this thread:

I found that the library was simply in the wrong location. It was the only file under /lib/powerpc-linux-gnu/security/. This machine is a powerpc. Other architectures will have a different architecture name in the /lib directory.

The simple solution was to make a symlink under at /lib/security/ that points to the existing library at /lib/powerpc-linux-gnu/security/

sudo ln -s /lib/powerpc-linux-gnu/security/ /lib/security/

No more errors in the auth.log.

How libxi broke my system

Thursday, April 3rd, 2014

After some package updates I could no longer log into my Debian/XFCE system. The display manager (gdm3) would start to load but show only a black screen and rotating/spinning cursor. There was no login box – no way for me to get to a desktop. Digging through the logs I saw no errors in Xorg.0.log.  Dmesg and the kernel logs were no help either.  My first clue was found in the /var/log/gdm3/:0-greeter.log.* logs. Each of these logs had just one line:

*** Error in `/usr/bin/gnome-session’: malloc(): memory corruption: 0x000000000207c620 ***

A memtest run showed no errors. Reinstalling gnome-session didn’t fix things. Looking at ~/.xsession-errors I found lots of Gtk and GLib errors among strange things like “Discarding: 6 over 9″ and “** Message: applet now removed from the notification area”. Other errors that might have been related:

polkit-gnome-authentication-agent-1: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.0. Fatal IO error 11 (Resource temporarily unavailable) on X server :0.0.

orage: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.0.
nm-applet: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.0.

In some log (which I’ve forgotten now) I remember finding more “memory corruption” errors but for applications like nm_applet. I tried switching my display manager to lightdm but I got a similar result. Upon boot, I was greeted with a solid black screen. Logs showed that the x-session was started and the greeter as well. In /var/log/lightdm/x-0-greeter.log I found a familiar error:

*** Error in `/usr/sbin/lightdm-gtk-greeter’: malloc(): memory corruption: 0x00007fe0cd2bcd40 ***

I tried purging and resinstalling the x server but of course, it was working fine. From a console I was able to stop the display manager and invoke an x-session manually with “startx”. In the desktop environment, I noticed my wireless was not connecting automatically and the network manager (nm_applet) and orage icons were missing from by status tray.

I’d like to say I used these clues to determine that the problem was isolated to gtk apps (e.g. lightdm, gdm3, nm_applet, etc.) but I never noticed the connection. I was simply fortunate to stumble upon the solution in this post, courtesy of one dE_logics:

While the root cause isn’t described in detail, the simple statement at the end of the post provides the solution: “This’s a bug with libxi. Upgrading it…solves the problem.” I gave it a shot. I found libxi6 in synaptic and upgraded it to the highest version available in Debian’s stable repository. After a reboot, my display managers are working perfectly. My libxi6 version is now 2:1.6.1-1+deb7 according to synaptic.

Managing an iPod Shuffle (2nd gen.) in Linux

Monday, March 31st, 2014

I have an old second generation iPod Shuffle (model A1204). When I dug it out of storage it wouldn’t play any songs. I plugged it in to my mobile desktop running Debian and verified the file system could still be read. I transferred all of the existing songs off of the shuffle using Thunar and its handy renaming feature. In order to manage the tracks (i.e. add and remove songs) I needed a piece of Linux software with that capability. Unfortunately, the popular options, Banshee, Rhythmbox, and Amarok would all require that I update a significant amount of my system (mostly gnome and everything that depends on it). The best lite application I could find was gtkPod.

I first tried adding and removing tracks with gtkPod but kept getting errors. The process for adding songs was not straightforward and I couldn’t figure out how to fix those errors. Even when I got songs transferred to the device, it refused to play. It would only flash alternating green and orange indicating it couldn’t find any music. The database file wasn’t being written properly. So, gtkPod alone would not suffice (I may have an unsupported model, I couldn’t find it in the list of devices).

The solution relies on a python script “iPod shuffle database builder” which can be found on sourceforge. I used version 1.0-rc1. My process for adding tunes to the shuffle is now this (starting from scratch):

  1. Use file manager (Thunar) to permanently delete all files from the shuffle.
  2. Open gtkPod and use it to recreate the iPod Shuffle directories only.
  3. Use file manager to copy songs from desktop to Shuffle. Songs are stored in device root or under any arbitrary file hierarchy, just not in the normal /iPod_control/Music directory that iTunes uses.
  4. The python script mentioned above, named “” is copied to the root folder of the device.
  5. The python script is executed from the root folder of the device.

That’s it. Once the script rebuilds the Shuffle’s database it can be ejected and used as normal. Easy. Apple take notice.

How to change “about:home” to anything in Firefox (Icecat, Iceweasel)

Thursday, March 20th, 2014

Before reading, please note that I’ve written this post with respect to Icecat 17.0.1 which is a rebranded version of Firefox. I can’t guarantee that things will work the same with Firefox or Iceweasel (another rebrand) or other versions of any of these Mozilla browsers.

The concept of a home page is not well defined in Firefox. In Edit->Preferences there is a Home Page setting. When Firefox starts, this is the url that will be displayed. But it’s not the url displayed when a new tab is created or when visiting about:home. Those are two other, independent settings. The first is set from the about:config screen by changing the preference browser.newtab.url. It’s also necessary to set browser.newtabpage.enabled to false. The latter is not easily changed. There is no preference to change it.

The about:home page is a static page included in the Firefox distribution. In can be found in the file omni.jar (or omni.ja in my distribution). The exact location of this file will depend on the distribution, the operating system, and probably the Firefox version. It’s probably easiest to find it by doing a search of your file system. One my try entering “chrome://browser/content/findme” in the address bar of a Firefox tab. The resulting error message might indicate where omni.jar is to be found.

Once located, the omni.jar file can be opened and it’s contents browsed and edited. I use fileroller on my Debian based pc to browse the contents of the archive. Inside the archive, you need to find the aboutHome.xhtml file. In my version of Icecat (17.0.1) it is located within the archive at /chrome/browser/content/branding/. If you don’t have a rebranded version of Firefox, you might find it at /chrome/browser/content/browser/abouthome/.

Open this file for editing. Within the <head> element of the xhtml page, simply add the following script tag:

window.location.href = "";

Whenever about:home is accessed, the page will be automatically redirected to whatever url is set in the quotation marks. You can test this by typing “about:home” in the address bar of a Firefox tag. This shouldn’t require that you restart Firefox in order to work.

Playing around with the omni.jar file turns out to be a lot of fun. With my current configuration, I am able to browse the contents of the jar file using Firefox itself by typing in the following in the address bar of a tab: “jar:file:///home/Files/Programs/icecat-17.0.1/omni.ja!/”.

It also seems possible to access some components directly. For example, entering “chrome://browser/content/browser.xul” in the address bar of a tab, appears to load another instance of Firefox within that Firefox tab. It’s like a new Firefox window all contained within a tab. Very cool.

Screenshot - 03202014 - 03:35:10 PM

High quality figures with QtiPlot and GIMP

Sunday, March 9th, 2014

To make figures for publications and for general plotting I use Ion Vasilief’s QtiPlot. I find it allows excellent control of 2D plot formatting. By customizing some settings, it’s easy for me to import ASCII data from almost any file format which is important to me since I use a variety of instruments and each one exports its data in a different way. It has a fair set of data analysis tools, too, and is extensible using python, something I’ve written about before (see here and here).

Despite its many advantages, I was disappointed in the graph export implementation. For everyday general plotting and printing, I’ve been content to export my figures in the png (portable network graphic) format. Research journals, though, require plots to be created at high resolutions, usually 300 dpi or above. The export graph dialog allows export to many formats and allows one to set the desired size and resolution. For me, these settings do not work as expected and prevents me from directly exporting high resolution figures. I’ve developed a simple “work-around” to accomplish this so that I can continue using QtiPlot as my primary plotting software.

Below I describe the problems I’ve encountered with exporting graphs in QtiPlot and the solution I’ve found. Please note, I’m currently using version svn 2288. The bugs and solution presented my not be applicable to newer versions.

Problems with QtiPlot’s Export Graph Features

I haven’t bothered to browse the source code, so I won’t attempt to guess at the root causes of the “bugs” I’ve found. Instead, I’ll focus on describing what I see as the expected behavior and the actual behavior from QtiPlot. I will illustrate the problems with a few quick examples.

Example 1:

Screenshot - 03092014 - 11:26:50 AM

Let’s say I intend to produce a figure that is 3.25 inches wide (standard column width for ACS journals) and 3 inches tall. To ensure a high quality figure, I want to produce my figure at 600 dpi. That means the final image, if rendered to a raster type format (e.g. bmp, jpeg, or png) will have a final size of 1950 x 1800 pixels².

In QtiPlot, I make a new graph window and set its dimensions to 3.25″ x 3.5″. Why 3.5″? It turns out, the window size does not equal the final image size. Some of the window is occupied by the title bar, etc., and not by the graph. I’ve found that the final graph height will be about 1/2 inch less than the window height. The width does not have an easy relationship (though it does follow a y = mx + b relationship).

Screenshot - 03092014 - 11:25:40 AM

I add my curves and apply any desired formatting. Now I export the graph. I’ll try the png format first. The Export Graph dialog presents you with three options for the png format. I set the image quality to 90 and the resolution to 600 dpi. I’m ignoring the transparency setting for this plot. I save the graph.

Screenshot - 03092014 - 11:28:08 AM

Screenshot - 03092014 - 11:32:07 AM

The exported image seems to look ok, but the size, in pixels, is too small, only 304 x 278 pixels². The GNU image manipulation program, or GIMP, provides more details.

Screenshot - 03092014 - 12:50:40 PM

GIMP shows the correct resolution as 600 dpi but notice the image dimensions in inches are not correct. My image is only 0.507 x 0.463 inches². The math works out, though. An image with that geometry at 600 dpi would be 304 x 278 pixels². Perhaps there is an easy fix. On the Export Graph dialog is a second section allowing me to customize the print size. I set the custom size to 3.25 x 3 inches² and re-save the figure.

Screenshot - 03092014 - 11:39:41 AM

Screenshot - 03092014 - 11:40:12 AM

That fixed the size and resolution problem. I now have the correct dimensions in inches and pixels. The only trouble is, the graph is no longer rendered properly. All lines are too thin. All fonts are too small. In the Export Graph dialog, perhaps you noticed the “Scale Fonts Factor” setting. Since 3.25″ is roughly 6.4 times bigger than 0.507″, perhaps I only need to scale my fonts by a factor of 6.4. I make this change and export a new version of my figure.

Screenshot - 03092014 - 11:42:52 AM

This is closer. The fonts are closer to the correct size; the dimensions and resolution are correct. My text for “(a)” and “(b)” have gone missing. The lines and curves are still too thin. There are no other settings to change in the export graph dialog. A solution might be to increase all line thicknesses by a factor of 6.4 and try moving my text around until it exports in the right position. I’ll just say now that this is nearly hopeless. During design, such enormous line widths make your plots unreadable and no one wants to use trial and error to align text boxes.

There is one other approach left. My experiments tell me that to get the font and line widths correct, the resolution of the exported image needs to be set near the resolution of the image on the screen. For me, that is 96 dpi. That is, my 3.25 x 3 inches² figure occupies 304 x 278 pixels on my screen. If I want my plot to export at 1950 x 1800 pixels², I should set my graph window size to these dimensions in pixels. Now, in the QtiPlot interface, my graph window is so enormous I can’t see much of it on the screen at one time. But, by increasing my fonts and line thicknesses and exporting to png at 600 dpi with no custom size, I get something very close to what I wanted.


This isn’t bad, but makes working with graphs in QtiPlot extremely cumbersome, and there is still some trial and error left in adjusting font sizes and line widths. Also, the file size is 10 MB. This might be reduced by changing the png image quality on export but I haven’t experimented to be sure.

Example 2:

Now let’s say I have my same 3.25 x 3 inches² figure (window height is 3.5″, see above) but this time I try exporting to pdf, a vector image format. The Export Graph dialog lets me choose a resolution and a choice to export in color. A resolution setting doesn’t make sense for vector graphics and indeed, I’ve found it has no affect on the exported pdf. This setting might only apply to figures that contain embedded bitmaps. I set it to 600 dpi anyway and export.

Screenshot - 03092014 - 12:12:30 PM

Now we’re talking. The graph, at first glance, looks great. I was able to design my graph with a reasonable window size and didn’t have to tinker with font sizes or line widths. The pdf document dimensions don’t quite match the window dimensions. That’s an easy fix though, I can set the custom size in the Export Graph dialog to tweak it.

Screenshot - 03092014 - 12:15:53 PM

There. I now have the right dimensions. But wait a tick, something seems wrong with those zeros.

Screenshot - 03092014 - 12:17:56 PM

So close! The font choice seems to be the problem. In this case, I’ve been using Free Serif and Free Sans. If I change fonts to Serif and Sans I get a perfect looking figure.

Screenshot - 03092014 - 12:19:47 PM

What about resizing? That is, say I want to have big windows with which to work in QtiPlot, but then export my graphs in a different size? Or, what if I were to export my 3.25 x 3 inches² figure at a custom size of 10 x 8 inches²?

Screenshot - 03092014 - 12:23:21 PM

I’m back to the same problem I experienced with rastered formats. My fonts are too small and my lines too thin.

Conclusions and Solution:

From the above examples I’ve learned the following QtiPlot behaviors concerning designing and exporting graphs:

  1. Graph window dimensions do not equal the final figure dimensions. The final dimensions will be a bit smaller because the non-graph elements of the graph window will be excluded.
  2. To successfully export rastered images directly it is necessary to work with the graph window at the same size in pixels as the desired final image. For example, if I want a 3.25 x 3 inches² 600 dpi final image, my graph window dimensions should be 1950 x 1800 (approximately, due to the first point above).
  3. To successfully export vector images directly (like pdf) it is necessary to work with the graph window at the same size in physical units (inches) as the desired final image. For example, to get a 3.25 x 3 inches² final graph size my graph window should have these same dimensions in inches (approximately, due to the first point above).
  4. In either case, rastered or vector, the custom print size should be set to ensure the desired final size is correct, but note this only works if your graph window dimensions are set very close to the desired final size.

The best approach that I’ve found is to avoid rastered image formats because it is nearly impossible to work with graph windows of the same dimensions (in pixels) as the desired final image (point #2 above) unless you require only very low resolution. Instead, always work with graph windows close with dimensions (in inches) close to the desired final size and export to a vector image format like pdf. This will produce a document with the correct dimensions and an infinite resolution since it is not a rastered format. Check your fonts to be sure they will render correctly and set good ones as your defaults.

This approach makes great pdf images but sometimes you need a rastered format such as when inserting figures as pictures into word processing documents (e.g. LibreOffice Writer, MS Word). In this case, I use GIMP to open the pdf images and convert to png. Upon opening, GIMP will ask you to specify a desired resolution and based on the document size will render the image with the correct dimensions in pixels.

Screenshot - 03092014 - 12:42:48 PM

Of course, if you require any other kind of resizing, GIMP has this capability too, but it’s best to import the full size image into your word processing document and let the software scale it for you there.


Disable traceback on old WordPress posts and pages

Wednesday, February 26th, 2014

The WordPress traceback/pingback feature is a spam magnet for bloggers, including me. I long ago disabled the feature globally in the Discussion options. It turns out that this only affects future posts. All of the posts and pages written before it was disabled are not affected (WordPress v.3.8.1). I’m tired of moderating spam tracebacks so I looked into a way to disable the feature.

The proper way is probably to use an SQL query as described by this link:

Since that involves me remembering how to log into MySQL, the passwords, and associated commands, I wanted something a little quicker, if not dirtier. Another suggestion from the interweb was to delete the php code that handles the tracebacks and automatically generates the comments. This file is called wp-trackback.php and is in the top level WordPress directory. I can’t imagine that I’d ever want to turn this feature back on, but deleting the entire file may be overkill. A rename is probably sufficient, but I worried that this would lead to logs being filled with php errors.

My solution was to kill the script before it got anywhere. A quick perusal shows that after handling the traceback, the script uses a php die() call to exit. I simply slipped one of these in at the top of the file (wp-trackback.php, below the “<?php” initiator) and voila. Any time this script is called it dies quietly and without error. No response is returned to the traceback sender (spammer). If you want to be more creative, just below where the function trackback_response() is defined you can call that function with an error code of 1 and a message for the spammers like this:

trackback_response(1, “Go to Hell. Or Cleveland.”);

On second thought, it might be more effective if you were to explain in your message the detrimental effects of spam on communication via the internet, the associated ethical problems, and the ineffectiveness of spam advertising. Encourage them to change their ways and contribute to a healthy, open exchange of real, meaningful ideas.

I’m sure that can all be said with a few choice four letter words.

One thing to mention: editing, deleting, or renaming the wp-trackback.php file, or any other WordPress php scripts, probably won’t survive an update. The SQL method mentioned above is update-safe.

Batch Table Operations – A QtiPlot plug-in/script

Thursday, January 30th, 2014

I’ve been taking lots of data at work. Each data file collected has to be imported to QtiPlot as a table and then several columns are added to the table and calculated from the imported data. Some days I have over 100 data sets, which means > 100 tables to manipulate. It was becoming impractical to perform all of the necessary table operations by hand. This script was written to automate the table operations I needed.

Batch Table Operations

To use one simply selects the tables to be affected from the list of existing tables. Then one selects and configures the operation to perform. Finally, clicking the “apply” button will perform the operation on the chosen tables.

As of today (1/30/14) the script allows adding and removing columns based on name, changing column types (x, y, z, etc.), and applying formulas to columns. Obviously, alot more could be done, but at the moment, this fits my needs. As I require more features I will add them. If you would like to request a feature add a comment or send me an email and I’ll see what I can do.


Edit 8/5/14:

I just tried to install this in version of QtiPlot running on Windows 7 but was unable to get the script interface to load. The particular version of QtiPlot installed on the target machine came from an unofficial Windows binary ( that I’ve used before. That QtiPlot package comes with parts of PyQt4 but not the entire library. As such it is necessary to install both python 2.7 and PyQt4 as well. Then, the …/PyQt4/uic module folder has to be copied into the PyQt4 directory of the the QtiPlot installation.

At least, that was the solution in the past. Now this no longer works. The uic module is found but fails to load. I expect it’s because the version of Qt against which QtiPlot was complied and the version for which the uic module were written are now too different. Or it may be that the version of Python is too new (2.7.8). I got this to work in the past with Python 2.7.5. Anyway, under Windows at least, I no longer expect for the batchTable plugin to be usable because it draws its interface using the uic module I can’t get working. Things should still work on Linux installations. The long term solution would be to rewrite the plugin to use a Tkinter interface instead of Qt thereby removing the dependency problem.