Reverse engineering a sybase database

The world seems to have decided on Oracle. But every now and then as a freelancer I come across a system that uses sybase. Usually it is an ancient legacy system that cannot be turned off for some reason, so sybase soldiers on. In such cases it is often useful to be able to reverse engineer the schema. This is where SchemaSpy comes in.

SchemaSpy requires graphviz, aka dot. The final generated output is a set of web pages and a set of dot files. I suspect that the dot is used to generate the image files used on the web pages, so dot is needed even if you never want to look at the dot files directly. Dot can be installed on windoze by merely unpacking the zip and adjusting PATH.

Some sybase installations have internationalisation set up in such a way that the URL needs to end in ?charset=iso_1. schemaSpy forms the URL based on a config it reads so the config must be extracted from the schemaSpy jar (a zip).

I first came across the charset issue when setting up database connections in DbVisualizer. I found
this web page which has the solution. This is also the solution for the same problem in schemaSpy.

Edit a copy of this config, appending ?charset=iso_1 to the name of the URL, then invoke schemaSpy with this command line (the sybase config filename is sybase.properties):

Note that the schema dbo must be specified, otherwise it tries to use the username as the schema name.

java -jar schemaSpy_5.0.0.jar -t sybase.properties -db <dbName> -u <userid> -p  <password> 
  -o <outputDir> -host <stringFromSybaseInterfaceFile> -port <portNumber>
  -dp <pathnameTojconn3.jar> -s dbo

a high-volume, low-latency market data processing system implemented with IBM middleware

Way back in 2003 I worked on a very interesting C++ CORBA project. It was to do with the near real-time distribution of market data to multiple subscribers. Most of the work was contracted out to that well known consultancy, Object Computing Inc. IMHO they did a very good job. One of the things that made it interesting was the way they used an XML configuration file to describe a pipeline of components through which the messages passed. This used the Configurator pattern described in POSA.

Back in January 2013 I came across a system that reminded me strongly of the one I worked on. I found an article in the Wiley Online Library (WOL). Unfortunately the WOL article is behind a paywall. Luckily, the authors also made it available in several other places including one where you get get it as a PDF.

This is a reminder to me that OCI aren't the only ones to develop a system like the one I saw and that there is very good write up about that other system. Here is what is says in the abstract:

A stock market data processing system that can handle high data volumes at low latencies is critical to market makers. Such systems play a critical role in algorithmic trading, risk analysis, market surveillance, and many other related areas. We show that such a system can be built with general-purpose middleware and run on commodity hardware. The middleware we use is IBM System S, which has been augmented with transport technology from IBM WebSphere MQ Low Latency Messaging. Using eight commodity x86 blades connected with Ethernet and Infiniband, this system can achieve 80 μsec average latency at 3 times the February 2008 options market data rate and 206 μsec average latency at 15 times the February 2008 rate.

Eclipse, CDT and the codan syntax checker

I have been using eclipse CDT recently on a legacy C++ project and it took a while for me to get the codan syntax checker to work. When it wasn't working it gave syntax errors for things that weren't syntax errors and claimed it couldn't find standard headers like stdlib.h, string.h etc. Use this command to invoke the GNU compiler so that it tells you where it is looking for includes:

g++ -E -x c++ - -v < /dev/null

These are the include directories that need to be configured. I got this tip from StackOverflow at http://stackoverflow.com/questions/11946294/dump-include-paths-from-g

You need to turn on full tracing when invoking eclipse (as explain my earlier post), then you will see if all the paths are resolved or not.

There are lots of posts of how to solve the codan problem but they are tend to be superficial and dismissive, often saying things like "put /usr/include" in the path. That is not good enough, you have to put in all the paths that the compiler uses, including the internal ones. Don't forget that when using cygwin and codan that the paths will have to be DOS-like, i.e. starting with the C:/cygwinDir..blahBlah.

How to install eclipse plugins on a machine that is cut off from the internet

Recently I have been using a proprietary eclipse plugin in a corporate environment. The plugin seems to assume java and windoze but the project was in unix-specific C/C++ and on linux. So I was faced with the task of setting up the plugin on linux. It was then I discovered that all the linux machines are cut off from the internet. What to do? I asked around and I amazed to discover that apparantly, no-one else has ever had to deal with this issue. Lots of people guessed that you just copy the jar files over from the plugins directory and restart eclipse. Wrong.

Copying files from the plugin directory plays a part in the solution but is by no means the be all and end all. For a start, sometimes the plugin is OS-specific, so it might contain DLLs (windoze) or shared libraries (UNIX). Any such plugin has to be obtained on the target machine somehow.

One of the main challenges of installing your own plugins without using a network connection is that you will have to chase down any dependencies. By default, eclipse does not tell you when there is a problem installing plugins but you really do need this information. Here is what you do:

create a file called .options which contains:

org.eclipse.equinox.p2.core/debug=true
.org.eclipse.equinox.p2.core/reconciler=true

then enter this command:

./eclipse -clean -console -consoleDebug -debug ./.options

This produces output that tells you what troubles it had loading plugins.

Under the eclipse directory create a directory called dropins. It should be a peer of plugins and features. Below dropins create eclipse and below that create plugins and features.

For any plugins that are missing, copy them from a windows environment that has them. They can be directories, files or a combination of the two. Be careful to copy to the correct target directory, either dropins/eclipse/features or dropins/eclipse/plugins depending on where it comes from in the source. To copy them, using WinCp. This is often allowed in a corporate environment.

C++ bollocks

Every now and then I come across what I call "C++ bollocks". What do I mean? I am referring to various statements made about certain aspects of C++ that are, well, complete bollocks. The statements might have been true to some extent in the
dim and distant past, but anyone still trotting out these statements in this day and age is, in my arrogant opinion, talking bollocks. Here are some examples, divided into categories.

all too frequent bollocks

• C++ iostream are too slow. Use the printf family instead.
  As one person has pointed out via the comments, I do not have a direct refutation for this yet. The FAQ link merely explains why iostreams are generally superior.
• C++ exceptions are too slow. Use error codes or boolean return status instead.
  see the FAQ. Also see , "Why use exceptions?"
• Don't use std::endl, it's too slow.
  It is true that std::endl flushes the output buffer, which has a performance hit. This is even mentioned in the FAQ-lite. However, such concerns over micro-efficiency are usually misplaced. Premature optimisation is the root of all evil.
• C++ strings are too slow. Prefer C char arrays.
  Some people prefer C to C++. Those people should leave C++ alone and work in their preferred language. Also see , "What's wrong with arrays?".
• avoid virtual functions. The performance hit on virtual dispatch is too great.
  See the FAQ
• always implement getters and setters as inline. It's faster.
  See the FAQ. As ever, premature optimisation is the root of all evil. People making these claims never have the figures to back them up. Their response to being challenged is "everyone knows this".
• provide convenient implicit type conversion by implementing cast operators
  Actually you usually want to do the complete opposite of this. This is why many coding guidelines say that single arg ctors must be qualified by explicit. and that cast operators should not be written.
• Failing to make a base class dtor virtual is no big deal. It will just result in a small memory leak.
  Wrong. It is actually undefined behaviour. See the FAQ and the Jargon file
• saying delete mem instead of delete []mem (to go with the new type[count]) is no big deal. It will just result in a small memory leak.
  Wrong. It is actually undefined behaviour. See the FAQ and
  of course, the Jargon File
• Use Yoda notation to guard against accidental assignment. I have a rant about this one.

infrequent bollocks

• don't throw exceptions from constructors.
  this is from the same school of bollocks as "C++ exceptions are too slow".
  See the FAQ
• prefer pointers to references since references dont make it clear when the object can be modified.
  See the FAQ and this Parashift article.

A nasty .Net error in Visual Studio 2010 and how I fixed it

Every now and then I got a weird error whilst tinkering with some Visual Studio project files. The error was like this:

22>------ Build started: Project: myLibraryDll, Configuration: Release x64 ------
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(298,5): warning MSB8004: Intermediate Directory does not end with a trailing slash.  This build instance will add the slash as it is required to allow proper evaluation of the Intermediate Directory.
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018: The "VCMessage" task failed unexpectedly.
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018: System.FormatException: Index (zero based) must be greater than or equal to zero and less than the size of the argument list.
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at System.Text.StringBuilder.AppendFormat(IFormatProvider provider, String format, Object[] args)
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at System.String.Format(IFormatProvider provider, String format, Object[] args)
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at Microsoft.Build.Shared.ResourceUtilities.FormatString(String unformatted, Object[] args)
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at Microsoft.Build.Utilities.TaskLoggingHelper.FormatString(String unformatted, Object[] args)
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at Microsoft.Build.Utilities.TaskLoggingHelper.FormatResourceString(String resourceName, Object[] args)
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at Microsoft.Build.Utilities.TaskLoggingHelper.LogWarningWithCodeFromResources(String messageResourceName, Object[] messageArgs)
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at Microsoft.Build.CPPTasks.VCMessage.Execute()
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
22>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask, Boolean& taskResult)

This is quite obscure, I think you'll agree. I eventually tracked it down to a spurious trailing slash on OutputFile. Remove it and the error goes away.

I have been bitten by this several times now so I thought if I blogged about it, then I would remember.

How to build libxml2 on Windows using Visual Studio

Libxml2 depends on libiconv so that is the first thing to build. Like many GNU projects it is UNIX-centric, so building on Windows is not easy. However, some kind soul has created full instructions.

Once you have libiconv you will need to copy the built library from libiconv.lib to iconv.lib. This is due to a quirk in the libxml2 build where it assumes UNIX (whose linker removes the leading "lib" characters).

Once the libxml2 source tarball has been downloaded, unzipped and de-tar'd, you will see there are instructions in win32/Readme.txt. These instructions are quite good as far as they go, but there are still a few gotchas which are covered here.

The commands need to be issued from a DOS cmd prompt. Start one. Execute the following commands to set the window up for using the Visual Studio compiler. I am using Visual Studio 2005, aka VC8 (aka vc80).

set PATH=pathToLibiconvDLL;%PATH%

rem to pck up vcvarsall.bat
set PATH=C:\Program Files (x86)\Microsoft Visual Studio 8\VC;%PATH%

rem to pick up nmake and compiler
set PATH=C:\Program Files (x86)\Microsoft Visual Studio 8\VC\bin;%PATH%

cd win32
cscript configure.js compiler=msvc prefix=whereYouWantToInstall include=libiconvIncludeDir lib=libiconvLibDir debug=no

This will create an nmake file called Makefile.msvc. You need to edit it to set various compiler macros. Near line 65 you should see a line that looks like this:

CFLAGS = $(CFLAGS) /D_CRT_SECURE_NO_DEPRECATE /D_CRT_NONSTDC_NO_DEPRECATE

Follow it with this line:

CFLAGS = $(CFLAGS) /D_SECURE_SCL=0 /D_SCL_SECURE_NO_WARNINGS /D_SCL_SECURE_NO_DEPRECATE

This will ensure that those nasty checked iterators are turned off along with other string checking nastiness that noone wants.
Then enter these commands:

vcvarsall.bat
nmake
nmake install

You will now have an install directory with subdirectories include, bin and lib.

The bin directory will contain several test programs and the utility xmllint. But there is a problem. The exes do not have the manifests embedded. For each executable you need to run the mt command to add the manifest to the exe. The format of the command is:

mt -manifest pathToManifestFile -outputresource:pathToExe;1

Note the semicolon one which is part of the command.

If you dont merge the manifest with the executable the first error you get is that msvcr80.dll cannot be found. You can also get the error R6034.

problems with git svn clone on windoze using cygwin

I am back in a windoze development environment and am using subversion (svn) for source code control. I am trying to experiment with git so I thought I would try to use git in conjunction with svn. This is turning out to be quite a bit of trouble. It all depends on which version of svn you are using.

Basically version 1.7 is fine but version 1.6 gives horrendous problems. Version 1.6 is being used because the server backend is version 1.6.

Googling reveals that the git svn clone problems in 1.6 are acknowledged and reported as fixed in version 1.7. Deep sigh. The problems are due to missing components. Several people have made attempts to build or install the missing components but I have not come across any success stories yet. And all my attempts failed.

You might think why not just move to version 1.7. Well, there's a reason and it's to do with a bit of svn nastiness. It is to do with repo format and compatibility and is discussed in the svn release notes at http://subversion.apache.org/docs/release-notes/1.7.html. The notes says:

Subversion 1.7 servers use the same repository format as Subversion 1.6. Therefore, it is possible to seamlessly upgrade and downgrade between 1.6.x and 1.7.x servers without changing the format of the on-disk repositories.

People have quoted this and concluded that:

There is no need to do anything, your working copy will be upgraded, and will still be able to talk to the 1.6 server.

However, someone on stackoverflow responded with:

Note: TortoiseSVN will update the working copy format which will create problems for older clients. This is only a problem if you have an environment where multiple different clients are used to access the same working copy. E.g. if you access the working copy from TortoisSVN and from IDE (e.g. PHPStorm) that only supports 1.6 working copy format.

This is exactly the issue for me. Changing to a 1.7 command line client forces me to change all clients. So that'll be native WIN32 command line, cygwin command line and TortoiseSVN. And all because the client changes the working copy so it won't work with older clients. I reckon this is a nasty on the part of SVN.

I am getting quite desperate to move to git now since I am grappling with some merge issues that are decidedly more awkward in svn. I may have to bite the bullet and change all my svn clients to be version 1.7. I hope to add to this blog entry to report any issues when I get around to making the move.

The daily scrum - not!

More thoughts about the so-called daily scrum

Further to my moaning entitled "when is a scrum not a scrum" I have seen people moaning about a related phenomenon that is starting to be called "cargo cult scrum". Now after looking into this for a bit I have decided that my situation is not exactly "cargo cult scrum" but there are some similarities so I will start by talking about it.

The cargo cult

Cargo Cult Scrum is what happens when you adopt the practices,vocabulary, and artifacts of scrum but you don't understand why or how they work. It comes from the phrase "cargo cult", which I only came across a few years ago. Here is a brief summary culled from wikipedia, for those who haven't come across it yet.
The wikipedia article opens with:

"A cargo cult is a religious practice that has appeared in many traditional pre-industrial tribal societies in the wake of interaction with technologically advanced cultures. The cults focus on obtaining the material wealth the "cargo") of the advanced culture through magic and religious rituals and practices. Cult members believe that the wealth was intended for them by their deities and ancestors."

It talks about the behaviour of Melanesian islanders during and after WW2, living on islands occupied by the military. It says:

"With the end of the war, the military abandoned the airbases and stopped dropping cargo. In response, charismatic individuals developed cults among remote Melanesian populations that promised to bestow on their followers deliveries of food, arms, Jeeps, etc. The cult leaders explained that the cargo would be gifts from their own ancestors, or other sources, as had occurred with the outsider armies. In attempts to get cargo to fall by parachute or land in planes or ships again, islanders imitated the same practices they had seen the soldiers, sailors, and airmen use. Cult behaviors usually involved mimicking the day to day activities and dress styles of US soldiers, such as performing parade ground drills with wooden or salvaged rifles. The islanders carved headphones from wood and wore them while sitting in fabricated control towers. They waved the landing signals while standing on the runways. They lit signal fires and torches to light up runways and lighthouses."

Cargo cult scrum examples

Now I hope you understand where the phrase "cargo cult scrum" comes from and what it refers to. I am now seeing blogs appear where developers complain about it and describe it. I suppose I am just adding to that list. Here are some examples I found:

• I recently attended a meeting at a company that considers itself to be agile. It was a regular meeting that occurred every two weeks. It was not attended by a scrum team, but instead by a bunch of people from two different groups. The three questions were not used. The meeting was scheduled for, and took, 30 minutes. Yet the meeting was called a scrum. Other meetings at this company are called scrums, so much so that "scrum" has become a synonym for "meeting". This is a cargo cult. The true meaning of "daily scrum" has been lost, if it was ever apprehended in the first place.
• An example from a [paraphrased] conversation:
  Customer: Oh yes, we've been doing agile for a while.
  Coach: That's great! So you haven't had any trouble getting the Product Owner to work closely with the team?
  Customer: Well, we...uh...don't really have a Product Owner.
  Coach: Oh. Well, who keeps the Product Backlog in shape?
  Customer: Well, we...uh...don't really have a Product Backlog, per se.
  Coach: Oh. So how do you plan your sprints?
  Customer: Well, we really don't do that in a very formal way. But we do meet every morning. That's what it's all about, right?
  
• I saw a good summary by an agile coach which said:
  A "good" scrum implementation will be a wonderful experience only if you have people who want to be a part of a cross-functional team who trust and respect one another. You also need a spirit of self reflection, and a desire to always improve and serve the customer. You also have to have leadership that is there for support. If you hate being a part of a cross functional team, and just want to be left alone to code (or whatever) heads down, then any agile or lean
  methodology will be sheer torture. If you have a strong team with micro managers, then it will be sheer torture. If you have both...ouch, I feel bad for everyone there.
• Another person summed it up with:
  The problem with Scrum and daily scrums is:
  1) Organizations that are doing fine do not need to change anything
  2) Organizations that are failing want a quick fix. So they choose scrum
  3) Now they are doing a bunch of quick fixes, calling it progress, doing Cargo Cult Scrum, and making people miserable
  That's what I see in the field. Failed organizations choose scrum because they are failing, and then they fail harder with scrum but pretend they are on a path to salvation.
  

A name has yet to be invented

So, if I am not in a cargo cult scrum situation, what situation am I in? As far as I know it doesn't have a name yet. One thing it isn't is "ScrumBut", i.e. "we do scrum, but...". That's because there are no scrum practices at all.
The only reason scrum is mentioned is because the meeting is called a scrum instead of being called a meeting. Here are some of the symptoms:

• The daily meeting is called a scrum and consists of "what did you do since the last scrum, what are you doing now?" for each attendee. Everyone waits patiently and politely for the meeting to end, saying the minimum possible when it is their turn.
• The daily meeting gradually reduces in frequency until it is officially only once a week but in practice it is not even that. It is scheduled once a week but is frequently cancelled at point blank range ("we're too busy to have it").
• The meeting is attendees rather than team members. There are no teams in the sense of multiple people pulling in the same direction; rather, there are disparate solo workers that do not communicate with anyone. They're not
  anti-social, it's just that the nature of their work is isolated and communication is discouraged by the management.
• It lasts for the scheduled time, no more and no less. People are cut short if it looks like it is going to over-run. As attendees learn that it is has almost no value they learn to speak less and less to avoid the danger of overrun.
• The meeting does not talk about progress, how we are going to get there, the need to improve, remove obstacles etc. Instead it is about what time is being booked to (I worked on blah, today I will continue working on blah)..
• There is no allocation of work from the backlog; there is no backlog! There are no stories and no use-cases and no input from the business. There are no iterations, no releases, no burndown charts. Hence these things cannot be talked about in the meeting!
• People turn up late, the meeting starts late. The non-communication, lack of purpose or value and general lack
  of interest and the fact that the meeting is often cancelled at the last minute, means there is little reason to be there on time.
• They are not standups. No one stands up. Everyone is seated. The meeting typically lasts for one hour. Try standing up for that length of time. The word "scrum" is used not but they have not heard of "standups". Or if they have then they agree that one hour is a long time to be standing up! If you were to point out that standups should take around 15 minutes you will be told that with the number of attendees this is impossible. Plus it's impossible because of the time it takes for the management to say its piece.

Barf bags

Q. When is a brown bag not a brown bag?
A. When it's not held at lunchtime.

I have noticed a tendency to call training sessions "brown bags". I reckon it is part of the same tendency that causes people to say "scrum" when they mean "team meeting" (I use the word 'team' very loosely here).

First, here's what a brown bag is supposed to be according to wikipedia:

A brown bag seminar, session or lunch is generally a training or information session during a lunch break. "Brown bag" is representative of meals brought along by the attendees, or provided by the host. In the USA, those are often packed in brown paper bags. Brown bag seminars normally run an hour or two.

Brown bags are also described in Linda Rising's book "Fearless Change" as a way of introducing people to new ideas without it taking up official project time.

Now here's what they are not: compulsory sessions to disseminate information that is important for the project. You know the kind of thing: you are expected to attend and it is not held at lunchtime. It is not open the the general project but only to the people that are required to receive the information. It is supplied on a need-to-know basis, i.e. if they need you to know then they will tell you. There is a slideshow that is supposed to act as a substitute for attending if for some reason you cannot attend. I call these sessions 'barf bags'.

Building on Windoze with boost

Just recently I tried to build an app on Windoze that uses boost. I got an error at linktime to do with a missing symbol from boost. It was for system_category() in the boost system library. I eventually tracked down the cause, which is very non-obvious. It is to do with 32 bit versus 64 bit building. I am using Windoze 7, 64 bit. The problem was caused by the way I was using jam to do the boost build. The command line to use is:

bjam toolset=msvc-10.0 variant=release threading=multi link=static
--address-model=64 --build-type=complete --with-system

In particular, note that
you must say --address-model=64. the -- is very important. Some other reports of this issue on the web say that you have to say address-model=64.

Grid computing and high availability

One of the things I found out when working on a grid system is that Murphy's Law works overtime there. In fact I would go as far as to say that on a grid "even if nothing can go wrong then it will". So, for the next time, if ever, that I find myself working on a grid system, I will probably want to refer to this article about HA-OSCAR that I found in The Linux Journal.

Capturing a wiki

Trawling through my notes and ancient reminders to myself on this wet Sunday morning, I am reminded to publish a link I found in the Linux Journal. It is a very useful article on how to capture a wiki using wget.

Prettyprinting C++ code

After all these decades, there is still I need for prettyprinters. I am in shock. I was spoilt in java-land, where eclipse takes care of this for you. In C++ it's a different story.

I have to say that these days most C++ is formatted OK. Enough years have gone by that most people recognise that making the source readable to humans is important enough for it to be worth that little bit of extra effort. Sadly, not everyone is enlightened so poorly formatted C++ code is still being written today. C++ IDEs that take care of this issue are not universally used. IDEs themselves are still a bit of an issue since there is no multi-platform IDE that most C++ people can agree on. Eclipse CDT isn't there yet and in my opinion is several years away. So the question is, what to do?

I have given up on GNU indent. It was meant for C anyway rather than C++ so it can easily get confused and do the wrong thing when presented with C++. The GNU project had an appeal a few years ago for a new maintainer, but no-one stepped up to the plate. The project languishes in obscurity now. However, there is a new program called astyle, which stands for Artistic Style. I have been using this recently and find it to be OK; not brilliant but OK. So this is what I will use for now. I think it could do with some of the options/features that GNU indent has but beggars can't be choosers.

A while ago, when I was still in java-land, I thought that the eclipse formatter plugin might be used. It certainly works very well for java. There is a blog entry about this that discusses running the eclipse plugin from the command line for C++ code.

Java: separating unit tests from integration tests

I have been going through some old notes of mine and came across a link to a very informative blog about how to keep unit tests and integration tests separate when using maven. The link is onjavahell.blogspot.com. Enjoy...

Downloading large files on Windoze

Many websites that contain large files to download warn Windoze users that under Windoze the download may get silently truncated. I have seen these warnings often but paid no attention to them because I always assumed it was only users of Internet Exploiter that were vunerable. Well I was wrong. I got nobbled by this problem today using Firefox. I was trying to download a debian DVD image at 4.4GB. It downloaded 980MB and stopped. Grr!

I thought, oh well, I will just have to use wget. And then the trouble started. I was in a corporate environment so there was a proxy. It is possible to configure wget with a proxy but you do need to know the proxy details. They can be viewed from the control panel but then you can get a nasty problem. The information is shown in a non-resizeable window (duh!). So the proxy information I was after was clipped, due to the URL being quite long. I found that to get the value in a normal resizeable window you have to look it up in the registry. The key is Computer\KEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\AutoConfigURL. Armed with this information I set the environment variable http_proxy (lowercase) to the value and ran wget. It worked like a dream.

I think I must be the only person in the world who thinks that windows with variable content must always be resizeable. As far as I can see, all GUI environments seem to encourage people to create non-resizeable windows. I think MicrosoftWindows makes more use of them than any other GUI environment though. Other GUIs sometimes copy windows so these GUIs are also increasing the number of non-resizeable windows. Deep sigh.

When is a scrum not a scrum?

As the craze to claim that a project is agile gathers pace, I see more cases where the daily meeting is referred to as a scrum. Gimme a break. See this article at the Agile Forest, for what is wrong with the trend in status meetings.
Quoting from the article:

The key purpose of a Stand-up is the opportunity to collaborate, share and support each other in the delivery of valuable outcomes.

I reckon that over time we will all see more and more examples of where the language of agile is adopted without the actual corresponding practice. There will be talk of sprints, iterations, retrospectives, scrums, kanbans, etc etc whilst the waterfall approach continues to be used on the project as the actual main method. Of course this means the project will not be able to embrace change since it will not be able to move quickly and easily in any direction. But then no-one will see anything wrong with that because waterfall is always the way it has been done. And a slow, plodding speed of delivery with document+design,code,test,signoff in strict order with no feedback will continue to be the way that software development is expected to be done.

At the next "scrum" I am tempted to ask who is the scrum master. But I have a feeling that the irony will be lost: I will just be told it is the person that asks each person in turn, "what did you do yesterday, what will you do today?" whilst the rest of team waits for the meeting to finish.

Migrating my web hosting

I have been with FastHosts for years but this was because they were (relatively) cheap and UK-based. However, I have just moved to BlueHost, which (IMHO) is far far better.

Switching over

FastHosts has been ok but they do not (at the time of writing this) provide any Postgres facility. Databases are MySQL only. I actually want/need both. I use MySQL for Coppermine, but would like to use Postgres for DSpace. There are not many UK-based providers that have the Postgres option. The option is much more common with US hosting companies.

There is another reason for preferring my hosting to be done outside Europe. A new stupid law has been passed that forces web sites to declare if they use cookies. Yeah, right. Problem solved. My website is now outside Europe.

I did a bit of research and found that BlueHost has everything and is reasonably priced. I moved my website over and had a few slight technical issues at first. I used their online 24/7 technical help and found it to be excellent and free of charge. It is also available 24/7. Most UK support is only in office hours Monday to Friday and you have to pay for it via expensive phone calls with lengthly waiting times and obnoxious musak.

BlueHost do not allow a .co.uk domain to be transferred to them. However, they can still host such domains. I found this a bit confusing at first. Domain registration is handled by certain official bodies across the world. In the UK a common organisation that does this is called Nominet. Web hosting providers that offer to handle your domain registration do so by dealing with Nominet. BlueHost do not deal with Nominet since Nominet is for the UK. However, that doesn't matter. What matters is how requests that are aimed at your domain get directed to the appropriate IP address. This is done using nameservers. When Fasthosts first set up my domain they configured my domain to use their own nameservers. But their webpages have configuration that allows this to be changed for when you wish to change who is hosting your domain. I changed them and in less than 12 hours my web site had changed over. Marvellous!

Migrating Coppermine

I hit a few snags when migrating my photos over to BlueHost. I use Coppermine to manage my photos, so I got the latest version, installed it on my own machine, set up a MySQL database and Apache with PHP, and started up the Coppermine landing page to verify that it would work properly.

When you land on a Coppermine home page for the first time, the PHP starts to configure your environment. It did this for me on my own machine (eventually) so I reasoned that the time was right for me to FTP all the files to my BlueHost area and it should work in just the same way. It didn't. It gave a weird PHP scripting error. Googling for the error message gave hits that made me think the problem was to do with the use of SSI (Server Side Includes). However, with some help from BlueHost support it was determined that the problem was to do with file permissions. Once the folders were set to 0755 and the files were set to 0644 all was well.

Getting a TV card working with Debian

I made the mistake of buying a card without first checking that it would work with linux. How foolish. I was forced to replace it. I now have a DVICO FusionHDTV DVB-T Dual Digital 4. I updated my system to Debian Wheezy (testing), 64 bit.

dmesg reveals the error:

[ 1501.907761] xc2028 1-0061: Error: firmware xc3028-v27.fw not found.

this problem is described at http://www.linuxtv.org/wiki/index.php/Xceive_XC3028/XC2028#How_to_Obtain_the_Firmware
However, those instructions assume 32 bit. I am on 64 bit.

there seems to be a problem getting hold of the the firmware file.
I registered on ubuntuforums.org and searched for firmware file. There is a post that has the file as an attachment.

I unpacked it and copied it to /lib/firmware.

Then I rebooted. The dmesg error went away.

I used the scan command to find the channels:

scan -u /usr/share/dvb/dvb-t/uk-CrystalPalace >channel.conf

I then gave the command:

cp channels.conf /etc && cp channels.conf ~/.tzap

The command:

tzap -c /etc/channels.conf -r 'BBC ONE'

is interesting. It produced the following over and over again.

status 1e | signal c024 | snr fdfd | ber 00000000 | unc 00000000 | FE_HAS_LOCK
status 1e | signal c008 | snr fefe | ber 00000000 | unc 00000000 | FE_HAS_LOCK
status 1e | signal c014 | snr fefe | ber 00000000 | unc 00000000 | FE_HAS_LOCK

It is as if something else had the tuner resource.
However, as it turns out, this is what it is supposed to do. Starting another terminal session whilst tzap is still running enables me to give the command:

mplayer /dev/dvb/adapter0/dvr0 which now works!

So I now have TV working on my debian machine. All I need now is to get a PVR environment working. I will have a go with either Freevo or MythTV.

How to use lean/agile as a stick with which to beat the developer

This article, http://jimhighsmith.com/2011/11/02/velocity-is-killing-agility, by Jim Highsmith, says it all.