Building open source C++ libraries on Windows for 32 bit and 64 bit

In my experience, most open source C/C++ library projects don't do a good job of providing the ability to build the library in all four builds, i.e. all combinations of release mode and debug mode with 32 bits and 64 bits. These days it is usually just 64 bit and sometimes it's just 64 bit release. To get all four build modes one has to start hacking but there is a little gotcha that nobbles me every now and then, so I thought I would blog about the solution so I never have to strain my brain to remember it in future. I can just go to my blog.

Add a new configuration from the configuration manager. Pick Win32 from the pick list and say you want to inherit from the 64 bit configuration (that's so you get all that the 64 bit configuration has). This configuration will claim to be 32 bit but there will be a problem. The linker will be set for 32 bit but the compilation will be in 64 bit. This is not apparent from the settings dialog. So when you build you will see an error like:

fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'X86'

To fix this, edit the Visual Studio project file, removing this line from the 32 bit sections:

<AdditionalOptions>%<AdditionalOptions> /machine:x64</AdditionalOptions>

That's it!

I have converted at long last to 1TBS

After decades of firm adherence to the Allman brace style I have finally changed my mind. I am now in the 1TBS camp.
Here is my reasoning, it is all a matter of my own personal opinion of course. The stuff below is not trying to make a logical reasoned argument for 1TBS in general, just why I changed my mind.

IMO Allman is useful to show where code blocks begin and end in legacy code where functions ramble on and on as they grow uncontrolled and undisciplined over the years. Such source often contains a random mixture of tabs and spaces.
It is quite hard to see where the scope blocks are under these conditions. Reformatting Allman style makes such code clearer than it would otherwise be. Using an Allman style on new code allows it to grow in an uncontrolled way where blocks just get bigger and bigger, rather than being refactored. When this happens the use of Allman means the blocks that would otherwise start to become much harder to see stay reasonably visible.

So that's why I used to think Allman was good. It allowed blocks to be more easily seen in ancient crufty code and it allowed new code to become crufty while still preserving some ability to see the blocks. But I have now decided that this is not a good reason to prefer the Allman style.

We all know that advocates of 1TB say it makes the code shorter, and it does. And shorter code has become much more fashionable over recent decades. The shorter the code the smaller the blocks and once a block becomes only a few lines the Allman style makes such blocks unnecessarily longer. I see this in Java code where 1TBS is the dominant layout style and shorter functions are practised much more than in C++. Maybe it's because alot of C++ code is ancient and in the dim and distance past it was more normal to write long, rambling unfactored functions. Java hasn't been around long enough for such cruft to accumulate to the same degree. Plus it came along later during which time shorter functions became more fashionable. When was the last time you saw a Java function that rambled on for hundreds or even thousands of lines. I wouldn't be surprised if you've never seen one. But we've all seen it in C and C++.

So, if I ever get the luxury of working on a new C++ project and I get any say in things like layout style, I would advocate 1TBS. The project would have short functions and the minute a function looks like growing to the point where the blocks start to become less visible it would have to be refactored. This is not just a matter of making it look pretty. The argument for refactoring would be that the result would be easier to test and it would be easier to reason about code coverage, as well as being easier to understand. This is already standard practise in Java, thank goodness.
It ought to be standard practise in any programming language.

During my conversion to 1TBS I have been working with python. This is a language where the issue has been designed away. How jolly sensible. Why don't all new languages learn from this? Python seems to be the only one. Every time some new language comes out it is inevitably based in C++ regarding layout. Java copied this and so has almost every other language since.

There is another factor which led to my 1TBS conversion: I have been working on a project where Allman is the standard, but with a twist. Single statements must not be surrounded with braces. We all know how potentially dangerous that is. It can lead to the dangling else problem, and has done on that very project, a fact revealed by a clang-tidy analysis.
It can also cause a problem when a developer changes the code to make the block more than one statement. These issues just don't arise when one uses 1TBS.

So, how is this change in belief going to affect my programming life? Hardly at all, unless I write stuff on my own (e.g. updating my sourceforge projects or creating new ones). After all, I am in an environment in which
1TBS is forbidden and its use would probably harm the code base as it would make the blocks in long rambling functions even harder to understand than they already are. I can't even use 1TBS in the java code, since Allman is mandated there as well.

A great C++ blog I've found

I've found a great C++ blog and I thought I just had to mention here it. There are lots of goodies about C++17 and C++20 and it keeps track of well known players in the industry and what they are up to, e.g. people like Barnje Stroustrup, Herb Sutter, Nicolai Josuttis and John Lakos. The blog mentions high profile features and plans for C++ including things like the inclusion of Howard Hinnant's date library into the standard and the adoption of contracts. I encourage everyone to take a look.

The insane OOM (out of memory) Killer

In the late nineties I worked on AIX for the first time. Back in those days there were several flavours of Unix available, all with their differences and idiosyncrasies. Linux was a fledging and fitted on just one CD. I came across a feature of AIX which I thought was crazy - the OOM (out of memory) killer. In this variant of Unix malloc always succeeded, even when there wasn't enough memory. The idea was that malloc returned a pointer to heap memory but wouldn't actually start to use it until the first reference was made. At the point at which it did then memory had jolly well better be available. If it was then all well and good. If not then the OOM killer came into play. The OOM killer would choose a victim process and kill it. The result was that memory would be freed and the access occurring at the time would succeed. Sounds insane, right? Right. I laughed and thought that this one feature rendered AIX useless compared to the other Unixes and would lead to its demise. How wrong I was. Fast forward a few years later. It was added to Solaris. Sigh. Fast forward to today. It has been added to Linux.

The OOM killer is a kernel development that mirrors what happens when banks try to innovate. It's what I call "the conspiracy of crappiness". It goes like this: some group or other tries to innovate but comes up with a really bad idea that doesn't work well and everyone hates it. The competition discover the move and for some inexplicable reason they copy it. Now everyone hates the competition as well and none of the players can be distinguished in this area. Bank charges on current accounts is an example. So is charging for withdrawals at ATMs (although customers have objected so vehemently to that one that there has been some back peddling). Well, in the world of Unix we now have the OOM killer.

There's a good article at LWN, that explains why this is insane. There's another article that gives tips on how to mitigate the nastiness, but surely that it yet another testimony to the fact that it is nasty. I also came across this article that discusses the nastiness and has an excerpt of .an amusing article that discusses the fairness, or otherwise, of how the victim is chosen. Here is the excerpt:

An aircraft company discovered that it was cheaper to fly its planes with less fuel on board. The planes would be lighter and use less fuel and money was saved. On rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thrown out of the plane. (When necessary, the procedure was repeated.) A large body of theory was developed and many publications were devoted to the problem of properly selecting the victim to be ejected. Should the victim be chosen at random? Or should one choose the heaviest person? Or the oldest? Should passengers pay in order not to be ejected, so that the victim would be the poorest on board? And if for example the heaviest person was chosen, should there be a special exception in case that was the pilot? Should first class passengers be exempted? Now that the OOF mechanism existed, it would be activated every now and then, and eject passengers even when there was no fuel shortage. The engineers are still studying precisely how this malfunction is caused.

Update: 27 March 2022

Since that aircraft analogy I have found an article on the perils of overcommit which gives a more dispassionate assessment, but still concludes it is a terrible idea: https://www.etalabs.net/overcommit.html

I can't stand the JBoss Application Server

I wonder which application server people chose when working on Java projects that need to publish dynamic web pages. I have used tomcat in the past and found it to be pretty good. But for the last few years I have been in an environment where JBoss was chosen. JBoss comes with all sorts of enterprisey EE things such as a JMS implementation and whilst initially this may seem attractive I have decided that I don't like it. I would now recommend that any project that needs JMS and dynamic web pages avoids an enterprise application offer. Instead I think it is better to chose the web page and JMS solutions separately.

Years ago I wrote a book review for ACCU on a JBoss tutorial book. I gave the book a bad review because it was largely XML fragments concerning JBoss configuration. But I now see that this is what struggling in a JBoss environment is all about. I still think the book was wrong to have such large XML sections though. The precise XML needed to make JBoss do what you want seems to wibble depending on the exact version of Jboss you have and also possibly on what colour socks you are wearing. But it gets worse. Recently (wrt the time of writing this, December 2018) JBoss went proprietary. Red Hat now calls it JBoss Enterprise Application Platform or JBoss-EAP for short. Not to be confused with the old open source version which was just called JBoss. In an attempt to deal with the confusion Red Hat renamed the old one to Wildfly and open source development is now done under that name. Wildfly does seem to be much better than JBoss but it's all relative; it is still derived from JBoss and so still suffers from the tremendous environmental difficulties caused by obscure and constantly changing XML configuration.


So, for people who want JMS and web pages with dynamic content, I recommend ActiveMQ and Apache Tomcat respectively.

Veracrypt instead of Truecrypt

Back in June 2014 Truecrypt died, but I and many others were able to build it from the source. I blogged about this before. Recently I had to access a couple of truecrypted volumes but found that my copy of truecrypt no longer worked. It relied on an old version of GTK that was no longer on my system. After some fruitless attempts to restore the required version of GTK2 I decided to try out VeraCrypt, which is the successor to Truecrypt.

Veracrypt is everything that Truecrypt was, and more. Fully open source, multi-platform strong encryption with optional plausible deniability and compatibility with Truecrypt. After installing yasm and libfuse I was able to build Veracrypt from source with no trouble at all. And it works. It was able to read my old truecrypt volumes. It also works on my Windows laptop. Wonderful! I have now switched over to Veracrypt.

How to find which dependent DLL can't be found

A couple of times in the last few years I have faced a knotty problem to do with DLLs on Windows. The first occasion was when a large complex program was performing that Win32 function ::LoadLibrary and it failed to find a sub-dependent DLL. A more recent case was where a java program called loadLibrary to load a shared C++ library used by a JNI interface. This library load also failed. Both failures were silent and mysterious. No details are given, just that the load failed.

I googled for help and asked friends and colleagues. The answer that came back again and again was to use the Dependency Walker at http://www.dependencywalker.com. Well, it turns out that every time I used that program to solve the riddle it was no help at all. I have now found a more reliable way, thanks to a tip via the ACCU general mailing list. I wrote a little C++ program. Before you run the program set PATH to the value it would have in your particular problem situation. The program waits for the user to hit return, then it calls ::LoadLibrary on the library name supplied. What you have to do is run the program (with PATH set appropriately) and while it is waiting for you to hit return, run Procmon from SysInternals. Enter the pid for the LoadLibrary program and set a filter for Operation to QueryOpen. Then hit return so it tries to load the library. The Procmon windows will then fill with all the file access attempts made to resolve the DLLs.Bear in mind that it is using PATH to locate the DLLs so there will be several access failures. The thing to do is check each leaf DLL name and find the case or cases where it failed to find the DLL no matter which PATH directories were searched. That's it, you have found which DLL failure(s) occurred!

It is a shame there is no more convenient way to deal with situation. If only the logic of calling ::LoadLibrary could be combined with the logic in Procmon that gets all the OpenQuery cases with the pathname and whether or not the access worked, all in one program. Maybe one day someone will write such a program, but in the meantime this solution will have to do.

Linux Mint 17 and scroll bar arrows

A while ago I did a complete reinstall of my desktop machine using Linux Mint 17. One of the first things I noticed after doing this was that the arrow bars that normally appear in conjunction with the scroll bar had disappeared. I deemed this a minor irritation and didn't do anything about it. But more recently I investigated why this was and what to do about it and found various blogs etc where people were complaining of the same thing and offering various solutions. I only found one solution that actually worked and give details on it below:

* Ensure that your changes are made to the Mint-X theme. You need access to the theme selector. Click on the Linux button (bottom left hand corner) and click on Settings. In the right hand menu pane click on Appearance (with the jacket and tie icon). This shows the theme selector when you pick the first tab, Style (which is the default tab). When you click on a theme it is immediately selected. There is no need to logout, reboot, or anything else. On selecting a theme the theme config files are read and processed. Therefore when you edit the theme files, use the selector to pick any theme *other* than Mint-X, then click on Mint-X again to pick up your changes.

* As root, edit the theme files. These are found under /usr/share/themes, so for Mint-X the directory is /usr/share/themes/Mint-X. There are sub-directories for gtk-2.0 and gtk-3.0. My edits were done to gtk-2.0. The file there is called gtkrc. Make a backup copy of the file first. Ensure your file contains the following:

GtkScrollbar::has-backward-stepper = 1
GtkScrollbar::has-forward-stepper = 1
GtkScrollbar::stepper_size = 18
GtkScrollbar::min-slider-length = 30
GtkScrollbar::slider-width = 18
GtkScrollbar::trough-border = 1
GtkScrollbar::activate-slider = 1

The crucial line turns out to be:

GtkScrollbar::stepper-size = 18

Without that line, no scroll bar arrows.
I found this tip on Linux Questions at http://www.linuxquestions.org/questions/linux-mint-84/question-how-to-enable-scrollbar-arrow-buttons-in-linux-mint-v17-3-a-4175580868/.

Using Joda time to handle date+time+timezone

I have been working on a client-server system where the client and server are in different timezones. The client is on London time, the server is in Los Angeles, a difference of 8 hours. This means that at the end of the business day in LA it already the next day in London. The API that I have to use contains functions with parameters of type Calendar. As we know, Calendar is a date+time+timezone triple, and the timezone defaults. This means that if the timezone is not explicitly specified then it will change its meaning as it goes over the wire.

I have been using the Joda datetime package to help me and after a bit of struggling eventually came up with the example program below which shows the construction of Joda DateTime objects for a specific date+time+timezone which is displayed correctly in both timezones. The program also shows how to construct Calendar objects from them for the correct timezone.

       
package jodaexample;

import java.text.SimpleDateFormat;
import java.util.Calendar;

import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import org.joda.time.format.DateTimeFormat;
import org.joda.time.format.DateTimeFormatter;

/**
 * 
 * @author marlowa
 * This example shows a time of 18:30 PST which is 8 hours behind UTC.
 * This means the date+time+timezone is 01:30 the previous day in UTC,
 * or 02:30 in BST.
 */
public class example {

    public static void main(String[] args) {
        System.out.println("Joda timezone example program.");
        DateTimeZone mytimezone = DateTimeZone.forID("America/Los_Angeles");
        DateTime mydatetime = new DateTime(2017, 3, 31, 18, 30, 0, mytimezone);
        String formatString = "yyyy-MM-dd HH:mm:ss z '('Z')'";
        DateTimeFormatter dtf = DateTimeFormat.forPattern(formatString);
        System.out.println("DateTime (local, i.e. behind UTC) = "+dtf.print(mydatetime));
        Calendar datetimeInAmerica = mydatetime.toGregorianCalendar();
        SimpleDateFormat sdfInAmerica = new SimpleDateFormat(formatString);
        sdfInAmerica.setCalendar(datetimeInAmerica); // to set the timezone.
        System.out.println("Calendar inAmerica                = "+sdfInAmerica.format(datetimeInAmerica.getTime()));
  
        long dateTimeMilliseconds = mydatetime.getMillis(); 
        int millisecondsOffset = mytimezone.getOffset(dateTimeMilliseconds);
        System.out.println(String.format("Milliseconds = %d, offset = %d", dateTimeMilliseconds, millisecondsOffset));
  
        long millisecondsInTimezone = dateTimeMilliseconds+millisecondsOffset;
        System.out.println("millisecondsInTimezone            = "+millisecondsInTimezone);
        long millisecondsInUTC = mytimezone.convertLocalToUTC(millisecondsInTimezone, false);
        DateTime dateTimeUTC = new DateTime(millisecondsInUTC, DateTimeZone.UTC);
        System.out.println("DateTime (UTC)                    = "+dtf.print(dateTimeUTC));
        Calendar datetimeInLondon = dateTimeUTC.toGregorianCalendar();
  SimpleDateFormat sdfInLondon = new SimpleDateFormat(formatString);
  sdfInAmerica.setCalendar(datetimeInLondon); // to set the timezone.
  System.out.println("Calendar inLondon                 = "+sdfInLondon.format(datetimeInAmerica.getTime()));
 }
}
 
 

The cause of MIDL error MIDL2398

Whilst working on a Windows Visual Studio (VS) project that uses MIDL (Microsoft IDL) I suddenly started getting the error MIDL2398 during the build. This was for no apparent reason. I tried the usual things, logging off, rebooting, cleaning the VS project, blowing my SVN checkout away and doing a fresh build. Sometimes this worked and sometimes it didn't. This was driving me nuts. Googling didn't yield much. Other people were also seeing this problem but no-one explained it and several people said that when they re-installed VS it went away. Yeah, right.

After a few hours of these failures I noticed something. The problem seemed to happen roughly on the hour. Then I remembered. I had recently set up a jenkins job to run on the hour. The jenkins job was building something else in an unrelated area but the coincidence seemd too great to ignore. I disabled the job. It turns out that the jenkins job was failing and during the failure logging it tried to log to a logfile without using double quotes around the filename. Since it was a jenkins job with the default jenkins install directory, the pathname started with "C:\Program Files (x86)\Jenkins". This was causing the logfile "C:\Program" to be created. The jenkins job, being run by jenkins, had administrator privileges so it was allowed to write this file to the root directory. When my VS got to the COM bit where it runs MIDL I got the error.

The fact is that the presence of the rogue file "C:\Program" kills MIDL with this weird error. Well I never. And it turns out to be easier to accidently create this rogue file than you might think.

The pesky capslock and inserts keys on Windows computers

It is a complete mystery to me why computer keyboards even have a capslock key. Old timers like me have a theory that in the old fashioned days of typewriters there was a practical reason, but surely there is no reason for it now. See capsoff for a history lesson. Nowadays it is just a key that you hit by accident. At home I use the Happy Hacking keyboard. That's right, I spent extra money to get a keyboard that doesn't have the capslock key!

For several years I have using registry hacks to disable the capslock key. It is one of the first things I do when setting myself up on a new machine. But until recently I didn't know what to do about another pesky key: the insert key. That's the key that makes typing either insert or overwrite according to the current setting. The current setting is not displayed so you only find out if you have hit it by accident when you notice that the last few characters you typed overwrote instead of inserting. Unlike the numlock key, the insert key has no feedback to tell you its current state. So this is another key you might want to disable. I couldn't find a registry hack for that but recently I found a great program for Windows called SharpKeys. This does allow you to turn off the insert key and, of course, the capslock key. So I now resolve to use SharpKeys whereever I go from now on. I hope you find it useful too.

The death of Purify

Purify is a memory debugger program used by software developers to detect memory access errors in programs, especially those written in C or C++. It was originally written by Pure Software.

My first experience of purify was way back in the days Motif programming around 1992. I used it to track down memory corruption and leakage bugs in my code for a complex oil and gas graphics program. After I had fixed my bugs I found that purify complained about loads of bugs in Motif. Over the next few years Motif got cleaned up dramatically, thanks in no small part to purify. I have been a keen user ever since and as time went on it was ported from Solaris to other flavours of UNIX and to Windows. A GUI was added, better support for multi-threading, it just got better and better.

Why was purify so good? Because at the time there was little else you could use that would do the same job in a completely comprehensive way. The other tools typically required access to the entire source of your product as recompilation was necessary. Other approaches included interposing special versions of new/delete and malloc/free which required special linking as sometimes special compilation as well. I saw one attempt at using a virtual machine, from IBM, but IMO it was a failure. I broke it with a simple 3 line program almost immediately. So I was very skeptical in the early days that emulation would ever work. Boy, was I wrong when it comes to valgrind. But valgrind wasn't around then. Remember, we are talking about how to debug legacy C++ that was written before valgrind was invented or linux was popular.

Pure Software acquired by Atria but the product continued to be good at that point and spread mainly by word of mouth. There were fully functional but time-limited trial versions. I used to say it was the next tool you should get right after the C++ compiler. But then it was acquired by Rational where it stayed for many years. It languished under the ownership of Rational who didn't seem particularly keen to sell it. One had to jump through hoops when one had finally won the argument to purchase licenses. These were not cheap but purify was so vastly superior to the other tools that the case could be made. Then the Rational purchase obstacles kicked in. One had to be determined. Then IBM acquired the product. If it was hard to buy from Rational it was almost impossible with IBM. And they neutered the demo/trail version, effectively making it so that it only spread by word of mouth. One could no longer use the trail version to evaluate it.

Fast forward to January 2015. IBM sold Purify to UNICOM. This sale has been disastrous for all users of purify. UNICOM no longer sell it. Instead they sell a product called PurifyPlus, which is a bundle of other tools developed by Pure Software and extended by subsequent owners. These tools are Quantify and PureCoverage, for performance and code coverage analysis respectively. These are and have always been good powerful tools. For some users it made sense to bundle them because if all three were desired the overall license fee was cheaper. Now there is no choice and buying all three is most definitely not for everyone. But there's more. You used to be able to purchase as many licenses as you wanted, from a single license to site-wide. Now UNICOM have made it so that the minimum number of licenses is FOUR. This makes it very expensive. Also a years support fees is compulsory. I recently got a sales quote for a client of mine and the quote was for over TEN THOUSAND dollars. Needless to say at that sort of price it was game over.

After discussion with some of my colleagues I have come to conclusion that UNICOM want to kill the entire product suite off. Why else would they only sell it to large enterprise outfits to whom tens of thousands of dollars for software purchases are as nothing? Effectively purify is dead. This is a serious problem for the development and maintenance of legacy C++ programs.

It's not a problem for any new C++ software development. Just start developing it on LINUX where valgrind is available. But valgrind will never be available for Windows. The problem is trying to purify a large Windows C++ program that cannot be ported to Linux (and where there may not be any need or desire to do so).

So I no longer recommend purify. It is consigned to the dustbin of history. What a pity, it was a fantastic tool right to the end.

The wonders of semantic versioning

Many years ago, in the dim and distant past, I used to work for Prime Computer Inc. They don't exist any more. They had a very good policy when it came to versions of their operating system. They used the familar major.minor.fix convention for denoting the version but were very strict about what this meant. The version numbers were always numbers, never strings, and you could and were supposed to infer things from the numbers. These inferences told you what versions were compatible with what other versions. They also told you about scale and kind of changes between versions. Sadly the industry as a whole doesn't do any of this in general. In fact, until recently, Prime was the only case I knew of that ever did this properly. Then I came across something called Semantic Versioning. See the web site at http://semver.org. This describes exactly what was done at Prime. How jolly sensible. Let's hope this catches on.

C++ code to dump memory in hex and ASCII

Every now and then I have the need to dump a block of memory as hex and ASCII in some C++ code I am working on. Each time I google for some code I can snaffle to do the trick. Each time I find something of very mediocre quality which will just about do for the occasion. Well, I finally got sick and tired of this and now I am publishing my own solution. I hope that people find it useful.

Many thanks to ECI training for this YouTube video on how to do file attachments in blogger.

TrueCrypt has gone! OMG!!!

My hard drive failed so this weekend I started to migrate my stuff over to a new machine which didn't have much stuff set up on it. One of the missing components was truecrypt so I thought I would just download and build from source. What a shock awaited me - truecrypt has gone! See the wikipedia page that describes how it went on 29th May 2014. There was also an article in the Guardian. I am writing this on Sunday 1st June 2014. This was not how I wanted to spend my Sunday!

Conspiracy theories to one side, the practical question remains, what does one do if one wishes to continue using truecrypt as it was? The answer seems to be to build from the source of the 7.1a. Download it from the final release repository. However, there is more to it than that. It doesn't build cleanly. I found someone else who was trying to do what I wanted to do, Reinhard Seiler. He blogged about his build experience. However, this was on a raspberry Pi and I had some different problems. Here's what I found:

• The build requires nasm, yasm won't do. No problem, I installed it via synaptic package manager.
• SecurityToken.cpp failed to compile due to missing PKCS11 header files. I followed Reinhard Seiler's instructions, placing the headers from ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-11/v211 into a sub-directory of my truecrypt source. This is so I could copy the entire directory if this ever happens to me again (i.e complete install on new machine needed).
• I got compilation errors due to missing macros such as CKR_NEW_PIN_MODE. Luckily, I found a blogger who had hit the same problem and posted a solution. Basically you ifdef out the offending lines. It is safe to do this since it is only error message handling.
• Once it got past the PKCS11 errors I found that it needs fuse. I installed libfuse-dev from synaptic package manager.
• The final compilation errors came from the GUI bits where it depends on wxWidgets. Synaptic to the rescue!
  
• Finally it built. But then I got an error at runtime along the lines of "Failed to communicate with kernel device mapped drive". I had done a rather large synaptic upgrade without bothering to reboot. Apparantly this kernel mode was affecting truecrypt so I was forced to reboot. Then it worked! Hurrah!

Once I had a working version of truecrypt I copied the entire build directory to my external USB backups directory, ready for the next time I need to install truecrypt on a new machine.

Now I will just put on my tinfoil hat briefly. I reckon that it is a conspiracy that truecrypt has gone. The developers say that the tool is not necessary now that Microsoft have BitLocker but this just doesn't wash. For a start I am on linux! And second, BitLocker is closed, secret, proprietary, so there is bound to be an NSA backdoor. Now I will remove my tinfoil hat and go and get a nice cup of coffee!

Software developer competency matrix

One of the ACCU mailing lists currently has a thread going where the challenges of hiring good C++ developers is being discussed. Someone posted a link to a software developer competency matrix. I thought it was rather good so decided to blog about so I didn't forget and so that I would have something to go back to for reference purposes.

Someone else in the thread posted that they had been inspired by another one but I found that was much more career/corporation oriented. The first one is much better for a techie to measure up against. It can be used to measure oneself or if you are involved the hiring game.

Reverse engineering a sybase database

The world seems to have decided on Oracle. But every now and then as a freelancer I come across a system that uses sybase. Usually it is an ancient legacy system that cannot be turned off for some reason, so sybase soldiers on. In such cases it is often useful to be able to reverse engineer the schema. This is where SchemaSpy comes in.

SchemaSpy requires graphviz, aka dot. The final generated output is a set of web pages and a set of dot files. I suspect that the dot is used to generate the image files used on the web pages, so dot is needed even if you never want to look at the dot files directly. Dot can be installed on windoze by merely unpacking the zip and adjusting PATH.

Some sybase installations have internationalisation set up in such a way that the URL needs to end in ?charset=iso_1. schemaSpy forms the URL based on a config it reads so the config must be extracted from the schemaSpy jar (a zip).

I first came across the charset issue when setting up database connections in DbVisualizer. I found
this web page which has the solution. This is also the solution for the same problem in schemaSpy.

Edit a copy of this config, appending ?charset=iso_1 to the name of the URL, then invoke schemaSpy with this command line (the sybase config filename is sybase.properties):

Note that the schema dbo must be specified, otherwise it tries to use the username as the schema name.

java -jar schemaSpy_5.0.0.jar -t sybase.properties -db <dbName> -u <userid> -p  <password> 
  -o <outputDir> -host <stringFromSybaseInterfaceFile> -port <portNumber>
  -dp <pathnameTojconn3.jar> -s dbo

a high-volume, low-latency market data processing system implemented with IBM middleware

Way back in 2003 I worked on a very interesting C++ CORBA project. It was to do with the near real-time distribution of market data to multiple subscribers. Most of the work was contracted out to that well known consultancy, Object Computing Inc. IMHO they did a very good job. One of the things that made it interesting was the way they used an XML configuration file to describe a pipeline of components through which the messages passed. This used the Configurator pattern described in POSA.

Back in January 2013 I came across a system that reminded me strongly of the one I worked on. I found an article in the Wiley Online Library (WOL). Unfortunately the WOL article is behind a paywall. Luckily, the authors also made it available in several other places including one where you get get it as a PDF.

This is a reminder to me that OCI aren't the only ones to develop a system like the one I saw and that there is very good write up about that other system. Here is what is says in the abstract:

A stock market data processing system that can handle high data volumes at low latencies is critical to market makers. Such systems play a critical role in algorithmic trading, risk analysis, market surveillance, and many other related areas. We show that such a system can be built with general-purpose middleware and run on commodity hardware. The middleware we use is IBM System S, which has been augmented with transport technology from IBM WebSphere MQ Low Latency Messaging. Using eight commodity x86 blades connected with Ethernet and Infiniband, this system can achieve 80 μsec average latency at 3 times the February 2008 options market data rate and 206 μsec average latency at 15 times the February 2008 rate.

Eclipse, CDT and the codan syntax checker

I have been using eclipse CDT recently on a legacy C++ project and it took a while for me to get the codan syntax checker to work. When it wasn't working it gave syntax errors for things that weren't syntax errors and claimed it couldn't find standard headers like stdlib.h, string.h etc. Use this command to invoke the GNU compiler so that it tells you where it is looking for includes:

g++ -E -x c++ - -v < /dev/null

These are the include directories that need to be configured. I got this tip from StackOverflow at http://stackoverflow.com/questions/11946294/dump-include-paths-from-g

You need to turn on full tracing when invoking eclipse (as explain my earlier post), then you will see if all the paths are resolved or not.

There are lots of posts of how to solve the codan problem but they are tend to be superficial and dismissive, often saying things like "put /usr/include" in the path. That is not good enough, you have to put in all the paths that the compiler uses, including the internal ones. Don't forget that when using cygwin and codan that the paths will have to be DOS-like, i.e. starting with the C:/cygwinDir..blahBlah.

How to install eclipse plugins on a machine that is cut off from the internet

Recently I have been using a proprietary eclipse plugin in a corporate environment. The plugin seems to assume java and windoze but the project was in unix-specific C/C++ and on linux. So I was faced with the task of setting up the plugin on linux. It was then I discovered that all the linux machines are cut off from the internet. What to do? I asked around and I amazed to discover that apparantly, no-one else has ever had to deal with this issue. Lots of people guessed that you just copy the jar files over from the plugins directory and restart eclipse. Wrong.

Copying files from the plugin directory plays a part in the solution but is by no means the be all and end all. For a start, sometimes the plugin is OS-specific, so it might contain DLLs (windoze) or shared libraries (UNIX). Any such plugin has to be obtained on the target machine somehow.

One of the main challenges of installing your own plugins without using a network connection is that you will have to chase down any dependencies. By default, eclipse does not tell you when there is a problem installing plugins but you really do need this information. Here is what you do:

create a file called .options which contains:

org.eclipse.equinox.p2.core/debug=true
.org.eclipse.equinox.p2.core/reconciler=true

then enter this command:

./eclipse -clean -console -consoleDebug -debug ./.options

This produces output that tells you what troubles it had loading plugins.

Under the eclipse directory create a directory called dropins. It should be a peer of plugins and features. Below dropins create eclipse and below that create plugins and features.

For any plugins that are missing, copy them from a windows environment that has them. They can be directories, files or a combination of the two. Be careful to copy to the correct target directory, either dropins/eclipse/features or dropins/eclipse/plugins depending on where it comes from in the source. To copy them, using WinCp. This is often allowed in a corporate environment.