The CUDA compiler can go into a CPU loop

I am working on a project that uses CUDA to perform certain calculations at greater speed, using the machines NVIDIA graphics card. Today I thought I would try to move the project closer to Continuous Integration (CI) using the well-known CI tool, Jenkins (formerly known as Hudson). So I set up a script for Jenkins to run.
Mysteriously, it hung when it got to the bit where it runs the CUDA compiler. I tried building the project interactively in Visual Studio from the Jenkins sandbox. This also hung at the same point. Process Explorer revealed CUDA was consuming roughly 25% CPU but making no progress. It was stuck in a CPU loop. Process Explorer revealed that it was in that part of the compiler that does the preprocessing, so it should have been really fast.
After a lot of tinkering in Jenkins, renaming the project, rebuilding the sandbox, reinstalling CUDA etc etc I eventually fixed the problem. It turns out that CUDA seems to be sensitive to the path that it is running in. If any part of the path contains a directory whose name begins with a dot, then you get the CPU loop. This is unfortunate because Jenkins likes to run in $HOME/.jenkins. When I picked a different Jenkins home (setting JENKINS_HOME via the control panel) and called the directory jenkins rather than .jenkins, then CUDA started working again. Weird, huh? For good measure, I renamed the job so that it did not contain any spaces (Jenkins creates a directory named after the job). I am not sure if this has any effect or not but getting rid of the leading dot definately fixed the problem.

Boost for C++ developers has lost its shine

I have decided that I don't like the C++ boost libraries with just two minor exceptions. The exceptions are the smart pointers that are part of TR1 and boost threads (along with their mutexes and condition variables). They're ok. I haven\'t used all of the rest but I have used the following: --filesystem --named semaphores and named mutexes --sockets (asio) --POSIX date/time I have also looked briefly at the graph library but not actually used it. In the above cases my criticisms are the same:

• I have found several cases where the code does not work. Examples include:
  • filesystem: has difficulty with UNCs
  • named semaphores/mutexes: does not cleanup properly and does not use the underlying WIN32 primitives on MS-Windows.
  • sockets: Counterintuitive model that is contrary to the way that BSD sockets work. For example, async_accept does not do an asynchronous accept!

• The source code is template mania. It is VERY hard to understand. Compare this with Poco or ACE where the code does what you would expect, more-or-less how you would expect.

• The code does not have a clean separation for those parts of the implementation that are platform-dependant. This is very hard to do well but Poco manages to pull it off nicely. Boost fails dismally.

The more I look at boost, the less I like it. It seems to me that it is a prime example of what happens when you give a problem to academics to solve. You get a convoluted solution that only appeals to the Elite and is not very practical. In future, I hope that I get the chance to use Poco for my foundation C++ classes.

NTFS junctions are a poor substitute for symbolic links

Recently, I have been working in a Windoze-XP environment using multiple sandboxes checked out from CVS. Each sandbox needs a large collection of third party libraries (headers, lib and .dll files). Each third party directory takes up around 2GB. This space very quickly mounts up. In Unix we would use symbolic links to make all the third party directories point to a common place. But what to do in Windoze? Some people would say that NTFS junctions is the answer. Well, they can help and I am using them for this purpose but there are some nasties and I am documenting them here. I once saw a signature on USENET which goes: "Those who do not understand Unix are condemned to reinvent it, poorly." This is certainly true of Windoze. It doesn't understand symbolic links so it is no suprise to learn that NTFS junctions are a very poor imitation. Here are several points to bear in mind:

• They are a cross between hard links and symbolic links. Like the hard link, they can onloy refer to files on the same partition, but like symbolic links the path of the link is stored which means it can become invalid. This is because Windoze filesystems do not support links! Also, only directories can be linked.
• Windoze-XP has insufficient support for NTFS junctions despite them first appearing in Windoze-2000. There are no commands to manipulate them. You will have to download and install the SysInternals commands and use the junction command.
• More explicit support for NTFS junctions was added to Vista and is also in Windows 7. But even by these versions of Windoze, support is incomplete. When you use Windows Explorer to navigate to a directory that contains NTFS junctions, if you delete the directory it deletes the files that the junctions refer to!!! See http://shell-shocked.org/article.php?id=284 for the gory details.

The only way to delete a junction safely is to the use the SysInternals junction command. There is no safe way to delete a directory that may contain junctions. Currently, I do not know what the solution is to these problems. The cygwin rm command does not help, it follows the junction references just like Windoze Explorer does. I have a feeling a special delete command may have to be written. I will probably have to write it, in Python, and it will probably use the SysInternals junction command to delete any NTFS junctions it finds.

cron on Windoze using cygwin

There is quite a useful little document, http://csc.csudh.edu/kleyba/cygwin-cron.pdf, that explains how to install the cygwin cron, complete the cygwin setup screenshots. Very nice. But there is a little nasty when setting up cron in a corporate network. Typically the home directory is a network drive which will not be accessible. It took me a while to work out what was wrong and how to fix it. Some of the advice I found whilst googling turned out to be wrong. I tried setting HOME in the crontab. No effect. I tried seting HOME as an environment variable in the control panel. No effect. Finally I edited /etc/passwd and changed the value of the home directory field from /cygdrive/u to /cygdrive/c. Problem solved! Also, note that the pdf document says how to run the cron config command. This was very helpful because other web pages omit this and it turns out to be crucial to getting the service running. Without that config the service gets installed but dies as soon as it starts up.

Setting up PHP and Apache on Windoze

This link is very useful: http://www.thesitewizard.com/php/install-php-5-apache-windows.shtml I mention it here instead of adding it to my list of bookmarks. After all, it's not going to be done every day, is it?

IBM MQSeries queue viewer (Hermes)

At long last I managed to get JMS Hermes working with IBM MQSeries queues. These notes are about how I did it. First, there is a very useful install guide at

http://www.hermesjms.com/confluence/display/HJMS/Installing

A little detail it misses out is that you must set JAVA_HOME as well as HERMES_ROOT. Also, when you track down the jars for MQ, you might find it hard to find the PCF jar. I found it at

 http://www-01.ibm.com/support/docview.wss?rs=171&uid=swg24000668&loc=en_US&cs=utf-8〈=en

Finally, there is a flash tutorial at http://www.hermesjms.com/demos/demo_mq.html. It is a little out of date but still very helpful.

Computers and dates

Here follows a very useful link about Julian day numbers: http://www.hermetic.ch/cal_stud/jdn.htm. This explains why julian day numbers apply to a timeframe that has 7980 years. This is often referred to as the Julian period or Scaliger cycle.

The joys of Open Access

I have been clearing out my mailbox of old reference articles, useful links etc and came across this one about Open Access: http://www.earlham.edu/~peters/fos. This is of interest (to me anyway) after my work on Digital Libraries.

Searching PubChem

Here is a useful link for how to search PubChem. http://acdlabs.typepad.com/my_weblog/2008/02/dereplication-v.html. The focus is on having an unknown sample with some NMR data. Well, I thought it was interesting, anyway, especially in the light of my recent work on a Digital Library for a major publisher.

Free news server (NNTP) access

Most UK ISPs have never heard of newsgroups. They think the internet == the web. Duh! So it is very difficult these days to find an NNTP server. This note is to remind me of which one I am using at the moment. Currently it is nntp.aioe.org. I will update this page as this changes over time.Free newsgroup servers do tend to get shutdown eventually (sigh).

getting mySQL set up on ubuntu

It is too painful to setup mysql via a source code build so I used a non-RPM install. It didn't quite work out of the box. Here's what I did: groupadd mysql
useradd -g mysql mysql
cd /usr/local
gunzip < /path/to/mysql-VERSION-OS.tar.gz | tar xvf -
ln -s full-path-to-mysql-VERSION-OS mysql
cd mysql
chown -R mysql .
chgrp -R mysql .
scripts/mysql_install_db --user=mysql --basedir=/usr/local/mysql --force
chown -R root .
chown -R mysql data
bin/mysqld_safe --user=mysql &
This last step was an attempt to start up the mysql server. It failed, as can be seen by the next command and its output. bin/mysqladmin version
bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
On with an attempt at a fix: touch /var/run/mysqld/mysqld.sock
chown -R mysql /var/run/mysqld/
bin/mysqld_safe --user=mysql &
THIS DIDN'T WORK. Error log said:
[ERROR] Can't find messagefile '/usr/share/english/errmsg.sys'
fixed by saying:
bin/mysqld_safe --user=mysql --language=./share/english &
Now I can connect. All I need to do now is put this into the bootup sequence....

How to rollback a changeset in subversion

Suppose you want to rollback the changes in changeset 2340. You checkout the HEAD into some working directory, then do the following:
svn merge -c -2340 .
svn ci -m
Then you will probably want to set up the working copy so it has the changes you rolled back so you can fix whatever was wrong. Here's the command you use:
svn merge -c -2340 .
This is not very obvious to the beginner. It is done using merge rather than revert. The svn revert command is used to revert uncommitted changes.

How to solve a weird JAXB/xerces spring application context load error

I was trying to get an app working standalone a spring application context via the classpath. In eclipse it worked fine but at the command line I got:

Unable to validate using XSD: Your JAXP provider [org.apache.crimson.jaxp.DocumentBuilderFactoryImpl @148cc8c] does not support XML Schema. Are you running on Java 1.4 or below with Apache Crimson? Upgrade to Apache Xerces (or Java 1.5) for full XSD support.

I was using java 1.6 and the latest Xerces so this error message is not at all helpful. What was VERY helpful was finding the thread at http://forum.springsource.org/archive/index.php/t-21140.html which contained the answer. I need to define a couple of properties that force the factories for XML parsing. Just add the following as JVM parameters:

-Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl

How to have an iPod without using iTunes

I trashed Brenda's iPod today. I eventually managed to recover it. Here's what happened:
I ripped a couple of CDs and copied them to the iPod using gtkpod. This was an experiment. I had always used iTunes before, but I am trying to escape from Windoze so I wanted to see if I could use some open source software on my Debian machine. I found that although the iPod contained the new music, once the iPod was supposedly synch'd then disconnected, the iPod reckoned the music wasn't there. As I tried various approaches eventually the iPod reckoned there was no music on there at all. Even doing a factory reset didn't fix it. On my travels I found stories of people who had managed to get iTunes up and running on their system using wine. That didn't work for. For a start, the Etch version of Debian has a version of wine that is too old. I tracked down a backport and I used winetricks to fool the system into thinking I am on XP, which the iTunes installer requires. But iTunes could not fix the iPod with the state it had gotten into, and IMHO iTunes is rubbish anyway.
Eventually I found out that the problem is caused by Apple turning the iPod against me. They want it to work only with iTunes and they keep revising the firmware to make it not work with other software. This is behaviour worthy of the Evil Empire itself. Brenda's iPod is a fourth generation nano. That means it has all the latest stuff to prevent people being able to use the device without using Apple software.
I found that some kind people have backported a recent version of gtkpod to Debian Etch. The glibpod3 library required has also been backported. I downloaded and installed the Debian packages. The next and final trick was to update a crucial file on the iPod which is used to identify the firewire device. I mounted the ipod to /med/ipod. This made the pathname of the file to be /media/ipod/iPod_Control/Device/SysInfo. Run the command "sudo lsusb -v | grep -i Serial" (without the "") with your iPod plugged in, this should print a 16 character long string like 00A1234567891231. This is the id. Put it in that file in the form:
FirewireGuid: 0xffffffffffffffff.
The leading '0x' is important.
I found this help at http://mandrivausers.org/lofiversion/index.php/t50594.html. Phew. At last, I can now use the iPod without Windoze and without Apple software.

managing photos without facebook or flickr

I got tired of the unreliable photo upload of facebook and found the bandwidth and size limitations of flickr too, er, restrictive, so I looked around for something else. I came across Coppermine Gallery and IMHO it is brilliant.
The online manual tells you exactly how to set it up and it worked first time. I am pleased to say it works with FastHosts (my webspace provider). There is a slight deterioation in image quality that you don't get with facebook or flickr. I think this is combination of me making smaller jpgs and the fact that Coppermine is using GD. It can use ImageMagic where available. I am not sure which one is supposed to be the best.
So, all I need to do now is add menu options to my web site to refer to the various albums.

The pain of building subversion on solaris

I have eventually managed to get subversion 1.5.0 built on solaris (opensolaris running on VirtualBox actually). At the time of writing this, the most recent version of subversion is 1.6.0. However, I have a requirement to use an older version of subversion, 1.5.0. This is because subversion, subclipse and svnant need to all cooperate with each other. This means dropping back to 1.5.0. When I tried to build subversion 1.5.0 on solaris I got a load of link time errors to do with relocatable code in libneon.a. I eventually found a blog at http://sreekalyan.blogspot.com/2005/07/installing-subversion-on-solaris.html which gave me a hint on how to solve the problem. First, I found I had to use version 0.28.1 of neon. The subversion configure does not like a more recent version and says neon is not there! By default, libneon does not build shared libraries. You have to configure with --enable-shared=yes, an undocumented option no less! I then deleted /usr/local/lib/libneon.a so that it could not be picked up by accident. After a configure in subversion I had to hack the value for the neon library, changing it from libneon.la to libneon.so. Wow, what a pain this was!

DSpace, chemical structures, CML and jmol

I have started to put some chemical structures into my DSpace instance. They are going in as CML files. CML is Chemical Markup Language. There is an open source tool called jmol which allows structures to be viewed as 3D rotating images, given a CML file. JMol includes a servlet so that web pages can display these images. I would like to see this used by DSpace. But in the meantime I have set things up so that CML files invoke an external jmol viewer. This is a bit like putting up PDFs and getting the external acrobat reader to display them. Here's how I did it:

• Add a mimetype for XML to the list of registered types. Make the type known (as opposed to unknown or unsupported), and the mimetype chemical/x-cml.
• Download the jmol package from sourceforge and ensure that the script jmol.sh (or jmol.bat on Windoze) has execute permission. On Windoze the environment variable JMOL_HOME must also be defined. On Unix just add the jmol directory to your PATH.
• Test jmol by running the standalone viewer on a CML file. When that works you should be ready.
• Add a CML item to a collection in your Dspace. That's it. Clicking on it will now run the viewer!

It would be great if the 3D image could be in the web page directly, like PDFs can with the browser add-on. But AFAIK there is no such add-on. There is the servlet but this would involve codes changes to DSpace.

Customizing DSpace v1.5.1

Prior to version 1.5.1 of DSpace, customization was done using a separate project called Manakin. The name persists within DSpace, including DSpace documentation, but actually it is all integrated now and customization is done in the dspace tree, not some separate project.

Details on how to customize dspace are very thin on the ground and most talk about how to do it when it was the separate Manakin project. I have managed to find something that works for me. Here's how I did it:

• Stop tomcat.
• Delete the tomcat dir webapps/xmlui.
• go to the dir in which you unpacked dspace. My dir is called dspace-1.5.1-src-release.
• Go to the sub-dir where the themes are stored. The documentation I found leads you to the wrong place. I found I had to go to dspace-xmlui/dspace-xmlui-webapp/src/main/webapp/themes.
• Copy the dir that is closest to the theme you want. Copy the whole dir to a new target theme name. The one I copy is Reference.
• This next step is CRUCIAL - edit sitemap.xmap to make it refer to the new theme name. If you don't then it will still refer to Reference! You need to go to the map:components section near the start. There you will see a section that looks like this:

  <map:component-configurations>
      <global-variables>
          <theme-path>Reference</theme-path>
          <theme-name>put your theme description 
              here</theme-name>
      </global-variables>
  </map:component-configurations>
    

  Change Reference and update the comment.
• Edit dspace/config/xmlui.xconf. Go to the end where you will see the path defined for the default theme. It is Reference/. Note the trailing slash, it is important. Change Reference for your theme name.
• Change something in your theme dir so that the change will be visually obvious. For example you can change the style sheet to refer to a different background image.
• Rebuild dspace with these theme changes. Use the command 'mvn package'. This builds to [dspace-src]/dspace/target.
• Update the dspace installation (which is in c:\dspace for me). Use the command 'ant update'.
• copy the dir dspace/webapps/xmlui to the tomcat webapps dir.
• Restart tomcat.

importing into DSpace

I finally managed to import a text file into DSpace today. It was "Decline of Science in England" by Charles Babbage. Here's how I did it:

• create an import directory. I called mine test-import.
• Create a directory below the import directory. I named it decline, after the book.
• Put the text file into the directory 'decline'.
• Create the file dublin_core.xml, in the 'decline' directory. The file looked like this:

  <dublin_core>
      <dcvalue element="title" qualifier="none">
          Decline of Science in England
      </dcvalue>
  
      <dcvalue element="subject" qualifier="none">
         science
      </dcvalue>
  </dublin_core>
  

• Create a file called content. The file contains a list of filenames, one per line. In this case I had body (which was the name of my text file) and preface, which was a separate preface file I created.

To import the item I then created a community and a collection under it, and gave this command:

bin\dsrun org.dspace.app.itemimport.ItemImport -a -e emailAddress
-m mapfile.out -c 125050010/1469 -s test-import

setting up DSpace and tomcat on windoze

My attention has turned to digital repositories. I looked at Greenstone but had enough problems with even a simple setup (admittedly trying to build from source) that I gave up. So I am now looking at DSpace. I had terrible trouble getting it to work on Windoze with Tomcat. I got it working eventually. Here is the tale of my voyage of discovery.

Prerequisites

Postgres.

goto http://www.postgresql.org/download/ and click on link for windoze. this will take you to a one-click installer for 8.3 the one-click install runs a nice setup wizard. database superuser = postgres,1gandalf. listening port=5432 (default). locale = English, United Kingdom I got the popup "A non-fatal error occured during the cluster initialisation. Please check the installation log in /tmp for details." There did not seem to be a log in either c:\temp or c:\windows\temp. The wizard then went on to completion. The service says it is starting for quite a while then stops. this seems wrong. The windoze event log has a timeout waiting for server startup, preceeded by a fatal error "could not create lock file 'postmaster.pid', permission denied. Went into c:/Program Files/PostgreSQL/8.3 and did a recursive chmod 777. Then restarted the service. this fixed it.

tomcat.

download binary distro for tomcat 5.5 core, windows installer. connector port = 8030 (default is 8080). u,pw=admin,1gandalf service fails to start. turn on debug in tomcat configuration panel, then restart. catalina logfile shows a bind error, address already in use. this was because there was a leftover process that interferred. I found the process using tcpview. need to put tcpview on my web pages. it is much better than netstat. after the kill tomcat restarted just fine.

maven.

it does not seem to come with anything ready for windoze. I unpacked the binary zip and found it is an installation directory. mvn can be run from the bin directory. Add the full pathname of the bin dir to the PATH using the control panel.

postgres (continued)

run PG Admin III (start->PostgreSQL 8.3->pg Admin III. double-click on localhost to connect to the db, enter the password for the postgres user. double-click on loginRoles then rightMouse. Select 'New login role...'. u,pw=dspace,dspace, check 'can create database objects' and 'can create roles'. double-click on databases then rightMouse, select 'new database...'. name=dspace,owner=dspace, UTF8 encoding. privilege properties, add user/group public with connect rights.

dspace.

edit dspace/config/dspace.cfg.
change pathnames so they work for Windoze, e.g /dspace => c:/dspace
change port number to 8030, i.e the one used by tomcat.
db.name = postgres
db.driver = org.postgresql.Driver
db.username = dspace
db.password = dspace
mail.server=mail.company.co.uk
mail.from.address = amarlow@company.com
feedback.recipient = amarlow@company.com
handle.prefix = 123400009
authentication.password.domain.valid = company.com

create c:\dspace cd to where dspace was unpacked. mvn package. this starts by doing lots of downloading and takes a *long* time. it takes over 77 minutes! cd dspace/target/dspace-1.5.1-build.dir. ant fresh_install

using a DOS cmd window:

cd c:\dspace
bin\dsrun org.dspace.administer.CreateAdministrator
email: amarlow@company.com
first name: andrew
last name: marlow
password: 1gandalf

shutdown tomcat. cd to tomcat conf dir edit context.xml putting in the dspace stuff. the context file is new to tomcat v5. restart tomcat. http://localhost:8030/jspui does not work! could not get this to work. tried to copy webapps/jspui to tomcat webapps dir but this also failed, but for a different reason. this gave an error in the tomcat log because it couldn't find ${dspace.dir}\config\dspace.cfg also get a NPE trying to use log4j when not properly configured. problem seems to be to do with the config directory not being copied over. but where to put it?

See http://mailman.mit.edu/pipermail/dspace-general/2008-July/002103.html for someone else who had exactly the same problem as me.

There seems to be a servlet LoadDSpaceConfig that is supposed to manage this and presunably report if there are any problems. But it seems that this is not working. I found this little nuggest in the DSpace manual on page 111, way after the installation instructions!:

The org.dspace.app.webui.servlet.LoadDSpaceConfig servlet is always loaded first. This is a very simple servlet that checks the dspace-config context parameter from the DSpace deployment descriptor, and uses it to locate dspace.cfg. It also loads up the Log4j configuration. It's important that this servlet is loaded first, since if another servlet is loaded up, it will cause the system to try and load DSpace and Log4j configurations, neither of which would be found.

I removed the context stuff from context.xml. This is to pursue a working configuration using the jspui that is copied into the tomcat webapps dir.

I edited webapps/jspui/WEB-INF/web.xml, replacing the dspace.dir param value with the DOS path (C:\dspace.....). This did have an effect. this gave the following error:

log4j:WARN No appenders could be found for logger (org.apache.commons.digester.Digester.sax).
log4j:WARN Please initialize the log4j system properly.
INFO: Loading provided config file: c:\dspace\config\dspace.cfg
INFO: Using dspace provided log configuration (log.init.config)
INFO: Loading: c:/dspace/config/log4j.propertieslog4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: c:\dspace\log\dspace.log (Access is denied)
        at java.io.FileOutputStream.openAppend(Native Method)
        at java.io.FileOutputStream.(Unknown Source)
        at java.io.FileOutputStream.(Unknown Source)
        at org.apache.log4j.FileAppender.setFile(FileAppender.java:289)
        at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:167)

to fix this I did cd dspace;chmod -R 777 .

this gave

log4j:WARN No appenders could be found for logger (org.apache.commons.digester.Digester.sax).
log4j:WARN Please initialize the log4j system properly.
INFO: Loading provided config file: c:\dspace\config\dspace.cfg
INFO: Using dspace provided log configuration (log.init.config)
INFO: Loading: c:/dspace/config/log4j.properties

this looks alot healthier. Now when I visit http://localhost:8030/jspui/ I get the page I expect. Wow!