It just works... or does it? The dark side of Macs in HEP

If you attend a particle physics meeting these days (and most of us do, several times a day... this is not a good thing) it looks rather different to how it did 10+ years ago. Not that everyone paid attention then, but the type of laptop everyone's focusing on rather than the speaker has shifted, from the olden times array of various clunky black boxes to the situation now where 2/3 of the room seem to be wielding shiny silver Macbooks.

It seems like a no-brainer: Windows is pretty much 100% dysfunctional for computing-heavy science (unless you are either in a fully management role and never touch data, or for some reason love doing all your work though a virtual machine), but Linux is unfamiliar territory for most starting PhD students. Sure, it's a lot more user friendly than it used to be, with more helpful GUI ways to manage the system and the wifi even works out of the box most of the time. But Macs are perfect: beautifully designed, friendly, but with Unix underneath ... and they only cost an extra 50%! Ideal for HEP users who need Unix computing but want it to just work out of the box... and who doesn't? As the Apple advertising used to say "It just works". But does it?

From my perspective as an author and contact point for an awful lot of particle physics software packages, I can put my hand on my heart and swear that about 90% of user problems are from people with Macs. And even if we adjust for the base rate effect that there are an awful lot of Macs out there, it's still the case that a majority of issues relate to the fact that it's a Mac that they are using. This trend toward everyone having a Mac naturally brings an assumption that, just like the mail client and web browser and office suite, HEP software should "just work" on these machines like it does on the offically supported Linux platforms. But that's often not true, for various reasons, and the demand that everything also work on a rather dysfunctional platform which is not part of the LHC's data processing plan puts rather a heavy load on us developers and maintainers. If you want to be able to run HEP code on your Mac, rather than just use it to log into Linux servers, then Macs are not the "it just works" path of least resistance that you might expect. I've called Macs "dysfunctional" there: accordingly legions of fans will be grinding their teeth and preparing to send me hate mail. Well, first-off I'd like to come clean and admit that I used to be a Mac user... and that I liked it a lot. My Macbook was sleek, light, cool to the touch, bonged endearingly and played Epel when I first switched it on, the screen was vibrant and sharp, the desktop apps were all superbly integrated with the hardware: wonderful! And then I needed to do some work.

This was back in 2002 or so, and Macs didn't have multiple desktops: I had to install a hack. There was also no way to override some keybindings, e.g. turning my Caps-lock into a backspace: another hack. The terminal application was junk... a toy to tick the "we're Unix and good for power users" box, a la the Windows command prompt. And compilers... I could get an old GCC through XCode, but no Fortran compiler: I installed Fink and had to work out how to pick up consistent build tools from /sw. And then the X-based apps from Fink didn't integrate nicely with the OS X display server, didn't know about the existence of that big "Apple" command key, expected a right-hand mouse button, and in short half the programs I needed to use just didn't behave in that beautifully integrated Mac way: I felt like a second- class citizen on my own machine. And since half these hacks weren't package-managed, I needed to do all that by hand, too. Much though I loved my sleek silver machine, after 6 months in my first postdoc job, where there was less time for this sort of frippery, I pulled the ripcord and bought a Lenovo. They have their own issues and my next jump (several Thinkpads later) may be to Dell, but on Linuxes like my current Xubuntu installation a consistent and self-updating set of developer tools is at my fingertips via a quick sudo aptitude search|install command. By comparison when it came to development tools and command-line operation, my Mac did not just work... in fact it was much harder.

Things have moved on quite a bit in Mac-land since then: there are multiple desktops built in, the terminal is better and apparently X-server integration now works, at least to some extent. But it still does not "just work" when it comes to developer tools, and accordingly everyone needs to patch their system if they are to get any Real Work done. In XCode itself, Apple long ago stopped updating GCC, leaving it at version 4.2 while they migrated to clang, and hence forcing many HEP users to install their own copies via Fink, MacPorts, HomeBrew, or manually. And if you needed a Fortran compiler, as you probably did, you definitely needed to get that from somewhere else. (I once had a wonderful exchange with a Mac user who said that LHAPDF was broken and wouldn't compile... ten emails later it turned out that they didn't have a Fortran compiler and couldn't read the "confusing" error message that essentially said "compiler not found". Ten mails after that, it turned out that they had found one online... and dragged it to their desktop as they would a Mac application. I'm not sure we ever managed to solve all their computing woes.)

This array of ways to install the extra tools that particle physicists need, but which Apple doesn't support you at all in doing, and the broken and inconsistent state of Apple's own compiler suite, means that there are a huge number of HEP Macs out there which are essentially borked. They have multiple compilers, in multiple locations on their system, which generate incompatible binary code objects and libraries. They have set different environment variables to try and random-walk their way around the generally screwed-up states of their machines. And every one is different, and most of them (as far as I can tell) still expect that our code will "just work". It doesn't and it can't.

To be clear here: I have no issue with bug reports about bugs in our software, those are always welcome even though we wish we hadn't messed up in the first place! And I get annoyed but can't blame our users for the fact that Apple uses a version of sed that behaves differently from the Linux one, that it uses DYLD_LIBRARY_PATH rather than LD_LIBRARY_PATH, and other such glorious inconsistencies. No, the big problem is that because of their shiny computers' shortcomings as development (or Real Work ;-) ) machines, people have applied hacks upon hacks, from all sorts of different sources, to try and make their computers functional again, without really knowing what's going on. Given this, I'm not sure why I feel like the bad guy when (after trying to help -- I really do) I also gently suggest that maybe it's their responsibility as computer users to make sure that their system is, y'know, functional.

Often I've had responses implying that I'm some sort of tech elitist for this -- and sure, I'm pretty good with computers and "the Unix way", I damn well should be after all these years -- and that this is unreasonable, that everything should just work, that it's their job to do science not computing, etc. ... but is it really? I certainly wasn't born knowing how to script in bash or run a compiler, I worked it out. I also didn't know how to install or administrate a Linux machine: I worked it out (at a time when that was a lot harder than it is now, both because distros were less user friendly and because Q&A resources like StackExchange didn't exist). So I do expect that others, especially particle physicists who are pretty much synonymous with "mad smarts" in the public perception and who need to use this stuff every day whether or not they love computers, can work it out too.

(Hint, if you dislike computers or programming, then experimental particle physics in particular is not going to be a good fit for you. And, worse, you're going to do many key things badly for a lack of interest in the hard-earned craft of doing them well. As a combination of being an inveterate cynic and caring that the science of particle physics is done well, this disturbs me. We undoubtedly already have a malign "standard path" treadmill which involves getting out of the dirty, hard, hands-on, technical business of making and doing cool things with data and code as soon as possible, and in to the higher-valued ranks of the "coordinators" and "convenors" which for some reason are sought by departments appointing new tenured academic staff. Encouraging that trend is not going to end well when there is no-one left to actually do anything anymore... a situation which feels painfully close already!)

To their credit, despite software developers not being a big target market for Apple -- at least not compared to graphic designers and hipsters -- in their 10.9 Mavericks release they have finally sorted out their system compilers. Still no Fortran as far as I know, and there is unlikely to ever be one, but if you can live with just C and C++ then a suite of LLVM-based compatible compilers centered around clang(++) works perfectly. If you buy into that, get rid of all the gcc and g++ that you can get your hands on, and maybe export CC=clang; CXX=clang++ variables in your environment then all should be well. Except one thing: demonstrating how much attention they pay to this area of their activity, Apple managed to issue a clang upgrade which failed to upgrade Python and Ruby at the same time. The significance of this is that Python in particular is an important interpreted language which can build and use compiled "extension modules" that interface to C++ libraries: we do exactly this for HEP projects like LHAPDF, Rivet, and YODA. And that in this clang(++) update, Apple removed support for several command line flags which were still built into how Python and Ruby call clang: cue several users with mysterious bug reports that looked like their systems were borked. Which is just business as usual, but this time it was Apple that did the borking. For the record, until Apple issue another update you should add -Qunused-arguments to your CFLAGS and CPPFLAGS (and maybe CXXFLAGS) variables. I don't know how long that update will take, but I suspect this is not a big blip on Apple's radar.

Finally, as another indicator of just how much Apple aren't into supporting code development, the Rivet team noticed recently that OS X does not have an explictly named python2 alias, as all other systems do these days. For about 5 years, having such an alias has been recommended as a major part of the strategy to allow staged upgrading from Python 2.x to 3.x. Linux installations like Arch have already moved their main python executable to Python 3, so it's important that we be able to ensure that our Python 2.x scripts get run with the correct interpreter until we can update our code. But Macs are once again the sticking point.

There is no way back from the Apple invasion, and I truly sympathise with the users who like their sleek and functional Mac machines and can't understand why they're getting these confusing error messages when they try to build standard, stable bits of HEP software. But they have wasted time for both users and package developers (and "user" in HEP really means "personal developer" -- everyone codes), as well being part of a more subtle and more destructive message that everything should just work and if it doesn't then it's the job of one of the "techie guys" to sort it out for you. Pretty they may be, but the most generous I can be is to say that the jury is still out on whether they have brought a net benefit to the particle physics world.


Comments powered by Disqus