Tuesday, April 19, 2011

The CUDA compiler can go into a CPU loop

I am working on a project that uses CUDA to perform certain calculations at greater speed, using the machines NVIDIA graphics card. Today I thought I would try to move the project closer to Continuous Integration (CI) using the well-known CI tool, Jenkins (formerly known as Hudson). So I set up a script for Jenkins to run.
Mysteriously, it hung when it got to the bit where it runs the CUDA compiler. I tried building the project interactively in Visual Studio from the Jenkins sandbox. This also hung at the same point. Process Explorer revealed CUDA was consuming roughly 25% CPU but making no progress. It was stuck in a CPU loop. Process Explorer revealed that it was in that part of the compiler that does the preprocessing, so it should have been really fast.
After a lot of tinkering in Jenkins, renaming the project, rebuilding the sandbox, reinstalling CUDA etc etc I eventually fixed the problem. It turns out that CUDA seems to be sensitive to the path that it is running in. If any part of the path contains a directory whose name begins with a dot, then you get the CPU loop. This is unfortunate because Jenkins likes to run in $HOME/.jenkins. When I picked a different Jenkins home (setting JENKINS_HOME via the control panel) and called the directory jenkins rather than .jenkins, then CUDA started working again. Weird, huh? For good measure, I renamed the job so that it did not contain any spaces (Jenkins creates a directory named after the job). I am not sure if this has any effect or not but getting rid of the leading dot definately fixed the problem.