Was C++ really a good choice?

Is my mental flexibility decreasing? Am I getting old and start whining about the good old days? Or is this all just an excuse for laziness?

During the last days, we focused on getting the final software together for the data “fixing”. What does this mean? We need to “transform” the electronic signals seen in the detector into particles like electrons, muons, jets, neutrinos, each with an energy and a position, such that we can study the physics laws behind the interactions we’ve produced. To do this, we using some software, called the reconstruction program. Writing this software is one of the major tasks of physicists in an experiment, many people writing different small pieces of it. Since the beginning of this year, we’ve frozen a version of this reconstruction software to process all the data taken between 2001 and this autumn in a consistent manner. But naturally since then, we made progress in understanding our detector and its performances, so we want to further improve the particle reconstruction. Therefore we are putting together a “fixer”-program.

The last days, we aimed for having all the programs from all people together in order to produce the “fixer” version. You can probably imagine that this doesn’t go right in the first attempt: a bug slipped through somewhere, a change in one program, hampers another program run properly, compiling in one way is ok, but in another (the one you didn’t test) dependencies matter in a different way, and the whole thing crashes again. Then someone has to repair a piece, but he is on holidays etc. The reconstruction program as well as the fixer program depend on more the 300 modules – that is a lot, and a lot that can go wrong.

When I started my PhD, nearly 15 years ago, all the software was written in a language called FORTRAN, and code-management was done by a program called CMZ in a “linear” way. Since a couple of years, nearly all High Energy Physics experiments switched to C++ and we use CVS for code-management. This had really nasty side effects. First, C++ is to my taste a rather difficult language, much more involving than FORTRAN. I am sure you can do fantastic things with C++, but we are physicists not computing engineers. Many people who talked “fluently” FORTRAN never made it to C++, the inhibition threshold being just too high, so they may mumble around in C++, but I am sure we lost a lot of manpower. This could be justified if we would gain a lot in performance. But secondly, I am still not convinced that C++ is so much more adapted to what we are doing. My former office mate, who is rather a computing wiz and who worked on BARBAR, told me that on a C++ conference a speaker defending C++, stated, that C++ is not made for fast calculation, that’s why physicists use FORTRAN… And then “object orientation” – I can get the concept, when I write a graphical user interface, but I don’t get it for reconstruction software, where I still reason in terms of “functionality” and the one thing that I want to do after the other. What is our reconstruction program doing now, that reconstruction programs from the LEP, the HERA or the TEVATRON RUN I experiments were not doing? And developing several branches of software is just a nightmare: imagine the complexity added if 300 packages are living on three branches and for 2 different versions. Did we really gain something from this?

And why am I whining? Well in the “good old days” debugging was kind of fun: changing a line of code, compiling, testing. At my level, this sequence was pretty immediate, making the whole process a constant flow of thinking, maybe even a little bit like a video game. Now I change a line of code, compilation takes minutes, as does only the initialization of the program, just to run a test. There is the point where I lack mental flexibility: I should just use debugging software… yeah, yeah – but learning about new software is like using VCR’s – the famous example of people over 35 being unable to program them. No, it is not that we are so dumb, that we wouldn’t be able to do it, but it is just not exciting anymore to learn about another button to push, while simultaneously holding your big toe and rubbing your nose against your shoulder, in order to get a videotape. And I am taking ballet, so I am agile enough to do this!

Comments

[Michael Schmitt]

You are right on the mark! I got my Ph.D. a little earlier than you did, and am quick with Fortran which I still use whenever I can. To use C++, though, is a struggle and does not come naturally. I can see the power of the language, but it does not seem suited to data analysis or event reconstruction, just as you say. It's a little like putting gyros and sails on a Honda Civic.

[Aaron F.]

I admit I've never worked on a large programming project, but I've been writing in C++ since the seventh grade, and I've never found it too difficult. I've never used FORTRAN, on the other hand, but FORTRAN code perplexes me... any time I look at it, I'm overwhelmed with capital letters. I suppose some of FORTRAN's features, like built-in complex variables, might be useful, but there are surely C++ libraries for that sort of thing!

[ursula]

So the choice of C++ vs. Fortran is actualy about having lost either the young ones or the old ones? Looks like that is just the way things go: the need to cope with changes in the bigger picture, if we like it or not, if it's good for us or not.
C++ seems to be much wider spread now and so in High Energy Physics, students may participate in big computing projects and use this experience to get jobs in industry?

[ursula]

Now, we are chasing "memory leaks" - not even sure, if they are coming from our code, or somewhere else, etc. What a mess!

[Jochen Weller]

Hi Ursula

I made the switch from f77 to f95 a few years back. This is much more gentle then going straight away to C++. You have most of the functionalities of C++, like modules, pointers, arrays where you can change the size in the code etc. However I had to deal with memory leaks in the past and they can be very nasty because the debuggers do not pic them up so nicely.

Another observation I made is that most graduate students I work with use C++ or MATLAB. So the first thing they have to learn when working with me is to implement Fortran routines in C++. However, I am not sure for how long I can maintain this.
I wrote little programs in C++ in the past, but the problem is that I grew up with Fortran and basically write a C++ program how I would write a Fortran program, which is probably not the right way.
I also work with people on Astrogrid and they actually prefer Java :)

[Lou Keeble]

Hi Ursula,

A long time ago I was part of a team at Fermilab (CDF) that evaluated switching to C++ from Fortran 77. With the benefit of hindsight I think that switching was the wrong choice from an end-user (physicist's) perspective. C++ has a much higher learning curve than FORTRAN for most people and doesn't necessarily add any extra value. The complexity and trouble (e.g. memory leaks) diverts attention from physics.

The only tangible benefit to a physicist doing analysis that I can see is that C++ is more applicable outside of the physics world, so knowing C++ will make it easier to get a software engineering job. Good for the physicist - but maybe not so good for the high energy physics field!

Also, note that corporations in the U.S.A. are moving away from C++ and towards languages like Java and Microsoft dotNet. These languages avoid some of the problems of C++ and are easier to learn.

[Russ Acker]

Professor Bassler,

What do you think would be the best programming language(s) to learn for a young physicist-in-training at this point? Does it depend on your field?

[ursula]

Dear Russ,
frankly I couldn't tell you! In our field C++ is certainly the most widely used language. My real advice to you would be: learn what you are interessted in first and what is used by the people you are working with second - during you life as a physicists, you will certainly have to learn different languanges - not all of them perfect, but at least to a level, where you can write the programs to solve your problems. And try to find out, how you like to learn something new and how you learn it best and easiest.

[Gordon T Watts]

I used to ride a train from Chicago to Fermilab. I met a bunch of people that were working on video conferencing software & hardware from AT&T. Their claim was that worst job, and the worst time of the project was system integration. When you finally bring all the little bits together. It would always take more time than you thought, and small bugs that didn't show up before would suddenly become a crisis. ;-) Sounds like our fixer software is at exactly this point!