Insectnation

The ROOT of all evil, 20 years later

Andy Buckley — Thu, 02 Apr 2026 10:35:31 GMT

I was contacted today by an undergrad student who has just completed a research project in particle physics, using the ROOT analysis framework, and had a … mixed experience. It sounds like they did a great job, really prepared to try new things to speed up and otherwise improve the processing, but something about the experience made them also happen upon my now rather ancient article about ROOT and to get in touch about it.

That article was written just after my PhD, now almost exactly 20 years ago — it’s only on writing this that I realise my first postdoc contract started in April 2005, and my belated graduation in March or April 2006, with the article written somewhere in-between. In it, I was reflecting on my PhD experiences with ROOT, as contrasted with what I had learned and was still learning about good software design and development. Every now and again I skim through it once more, and find that for a relative neophyte my judgements on good practice were surprisingly spot-on.

Maybe a little preachy, certainly purist/naive about workload… but then, I have come to think that the strongest distinguishing feature between a mediocre and an excellent (scientific) software developer, particularly when it comes to frameworks that others will use programmatically, is readiness to put in the effort to re-engineer something that technically works, because it could be better.

ROOT has, since 1991 and certainly up to the end of the v5 series, largely been defined by aggregation of more half-assed, kitchen-sink features (and assimilation of other libraries that had been more useful standalone), and unwillingness to revisit suboptimal designs in its core, most heavily used parts. ROOT6 started to overturn that, with the many-year Cling project to replace the Cint C++ parser (this was billed as a quick collaboration with LLVM in 2008, finally released a first version in 2014), and since then has added features like RDataFrame in response to the popularity of such structures as introduced by Pandas and other Python tools. From version 7 — I’m sure I’ve been seeing talks since at least 2019 — there’s finally a revisiting of the histogram API, the plotting backends and rendering quality, etc. So maybe the lesson of iterating and improving has arrived with the new — I think third — generation of developers. But is it in time?

Back to the student enquiry, which asked me for my feelings about the tool now, many years later. Honestly, I minimise my contact with ROOT, and have ever since those first encounters. There have always been other ways, and lots of what its core features do are so uncomplicated that anyone can spin them up. That was true in the C++ days, and it’s super-true in the current, more Python-oriented world. I’m also a great believer in the importance of “write it yourself, once” as the best way to truly understand a concept — maybe I’m a bit slow, but I would never have truly grokked sampling, or likelihood profiling, or myriad other things, if I hadn’t made myself write code snippets to implement them, until the penny dropped. It’s a world apart from learning to write the config file for someone else’s implementation, and the sort of thing you can only take the time to do during PhD or postdoc “self-learning years”. Every now and again I reconnect, to e.g. write the YODA-to-ROOT bidirectional histogram converters, and depart vowing not to go there again for some time.

So I use it for file I/O only, in fact preferring third-party tools like uproot, and would never touch it for plotting these days. I think one of ROOT’s greatest damages done to the physics community is to conflate data-processing with presentation: in any sane workflow these are separate things. And in general, staging and filtering data to ease reprocessing — especially the very many iterations needed for final-plot and uncertainty studies — is an important skill that we don’t train or emphasise enough.

My general feeling about ROOT7 is that they’ve missed the boat. In the era when not-invented-here syndrome ruled high-energy physics, researchers and students largely weren’t aware of alternatives, and tolerated / Stockholm-Syndrome’d ROOT as the one way to do analysis. Some even took this out into industry, inflicting ROOT on insurance and banking quants as the guru-level tools by super-scientists in Geneva; to be fair, their sector tools were probably even worse. Everyone was also expected to write C++ then, for which ROOT’s tutorials and API encouraged myriad very poor software-engineering practices. But by virtue of inertia and navel-gazing, it remained the centre of most HEP experimentalists’ lives.

What I see now is an explosion of better-quality plotting and more performant, industry-standard analysis tools from outside particle physics, and this time researchers and students are aware of them and migrating. ROOT is going to remain the main storage and I/O format for HEP raw data, but in LHC collaborations I see it increasingly being ditched in the analysis phase for non-HEP Python-based toolchains. The rise of ML outside the ROOT/TMVA monolith — it is a monolith, despite old claims of modularity — has further encouraged that migration.

So the ROOT 6 and 7 developments of dataframes, new histogramming etc. are nice. Indeed, they look a lot like things I wrote in that article fresh out of my PhD! But this ain’t 2006, and people have a lot more alternatives now. At some point, CERN and the HEP community at large may decide that specialisation is better: ROOT as a focus for our peta/exabyte-scale I/O and filtering, but environments composed of interoperating, cohesively designed small tools as the better way to make a modern ecosystem. In the end, researchers vote with their feet for the tools that work best.

UKRI/STFC funding crisis: a letter

Andy Buckley — Sun, 15 Mar 2026 23:49:03 GMT

The UKRI/STFC crisis rumbles on, through an impressive Select Committee showing by research-community reps and a fairly dismal performance by the STFC Executive Chair at the same. As there has again been no clear sign of appropriate urgency among the council leaderships, who appear keen to kick the consequences of their actions into the long grass, time for a bit of awareness-raising to representatives – in Scotland meaning MSPs and Scottish Parliament committees as well as MPs. For interest, here’s what I’ve sent:

Dear XXX …

You may already be aware that there is something of a crisis in UK science funding, under new leadership of the UK Research and Innovation (UKRI) body which distributes national research funding on behalf of DSIT. In particular, the inspiring “blue skies” research programmes under the Science & Technology Facilities Council (STFC) are facing existential cuts. The science minister and UKRI chief executive have stated this effect is unintentional but show little evidence of acting to remedy it. With this letter I want to highlight the extraordinary damage that this situation is doing to science in Scotland, where STFC research has a prominent history and leadership across particle physics (e.g. Higgs-boson Nobel Prize), astrophysics (e.g. gravitational waves Nobel Prize), and nuclear physics.

The background appears to be a government directive to encourage more research focus on interdisciplinary and “impactful” science – in particular ringfencing money for AI and quantum technology. “Appears”, because the true situation remains unknown among conflicting, often disproven, and ever-changing explanations from the UKRI and STFC leadership. While claiming to have protected “curiosity-based” research, we have been advised to expect large cuts to long-term science projects in which the UK has established international leadership. This has included summary cancellation of UK pledges to international projects. Chief among these are the LHCb antimatter-experiment upgrade, where the UK was the leading player and the Glasgow and Edinburgh research groups were central technology leaders; and the US Electron-Ion Collider (EIC), where the Glasgow Nuclear & Hadron Physics group has key leadership and R&D roles, and research-staff contracts were predicated on work we now cannot do. Our reputation has already been severely dented by these shock withdrawals, announced over the Christmas break with zero community consultation or evidenced consideration of impact, and drawing ire from international research leaders. That the LHCb U-turn happened precisely as the UK took up the leadership of the international CERN lab for the first time in decades highlights an incredible lack of joined-up thinking.

This first chaotic misstep (technically both cancellations were separate from STFC, though its Chief Executive’s recusal from the meeting due to one of several conflicts of interest cannot have helped) has been followed by botched announcements of further reorganisations to research council funding before they are even defined. This self-inflicted uncertainty has led multiple research councils to slam the brakes on their grant programmes. The vaunted new investments in AI and quantum might be able to fill the other gaps – fundamental science is big on use and development of that tech – but no-one knows if they can alleviate the headline damage, as their funding structure has not yet been decided. I find it impossible to view deployment of the destructive part of a reorganisation before the constructive part is decided as anything other than peak leadership failure.

Theoretical physics, up against delays in announcement of its annual funding, has missed the internationally agreed research recruitment deadline for 2026 and hence the chance to recruit the best new graduates. STFC in particular, hamstrung by the large fixed costs of its national and international research facilities, has told its research community to plan for up to 60% cuts to project and university-group funding. This is ludicrous: no tightly-managed project can survive 30%, let alone 60%, cuts and deliver any more than a small fraction of their original plan. Community review of these projects for UKRI is already a huge administrative overhead, to which the response has been years of “flat cash” cuts: there is no wastage to be “efficiency-saved”. So STFC and UKRI, their new leadership having immediately haemorrhaged all community support, are on course to rapidly eviscerate the UK’s international reputation in fundamental science, and to ruin the careers of an entire generation of young UK-based researchers, all while still paying hundreds of millions of pounds per year for membership of research organisations we cannot afford to properly use.

This mess is made worse by the leadership’s reticence to discuss openly with the community and to spray around changing explanations and deflections, defence of the institution seeming higher-priority than fixing the problem. The current claim is that they need to wait for the (recently submitted) community impact statements; it is perfectly obvious what this paper-chase will say, so the feeling is very strongly of complacence or hoping the problem will spontaneously dissipate. Meanwhile real damage accumulates daily, not to mention the very substantial opportunity costs of scientific communities diverting their efforts from productive research into administrative and political response-mode. This was called out in impressively straight-talking terms by the Commons Science & Technology Select Committee, following hearings with both the UKRI & STFC leadership, and with representatives of the STFC research community. The most recent hearing even emphasised the mismatch of the scale of impact to the size of budget-reduction involved, which was described as a “Treasury rounding error”. But yet the botched plan, insofar as it even exists, remains unamended more than two months since the alarm was sounded.

Our understanding is that government, and the public and MPs in general, are proud of the UK taking a leading role in this science, and understand that while abstract, its spin-offs are not: a pipeline of highly trained researchers moving into UK industry, inspiration for school students to apply to university science & technology courses, and direct technology such as medical scanning and radiation treatments, and of course (from the previous generation of experiments) the World Wide Web. This science is a key part of the growth agenda that the Government promised, albeit one that works on longer timescales than an election cycle. Cutting off the long-term roots, perhaps to chase the coat-tails of the AI boom/bubble, is an excellent way to kill the tree of sustainable economic growth. Your support and help in drawing parliamentary attention to this issue, and its impacts locally in Scotland, would be greatly appreciated.

Thank you.

A grant don't come for free

Andy Buckley — Mon, 02 Mar 2026 14:12:41 GMT

In my last post on the ongoing self-made funding crisis at UKRI and its subsidiary science-funding agencies, I made a final reference to the administrative overheads in modern science, which would obviously have been too much burden for an already overlong piece to bear. So here is a follow-up on that issue, one not specific to the transient effects of the latest disruption, but generally how we have made science cumbersome and inefficient in the name of accountability.

If you run into any UK research (and teaching) academic and ask them how things are going, chances are – in danger of breaching the British politeness convention of an upbeat non-answer – there’s a good chance the answer will contain some variant on “good, except for all the paperwork”. I don’t know what most of the public imagine that professors do day-to-day, but a never-ending round of bureaucracy is probably not foremost in mind. Funders and our own institutions demand applications, reports, surveys, etc. plus the reviewing that constitutes the other side of that. Not to mention the departmental and teaching administration, and the more engaged management of our research teams, if we’re lucky enough to have them.

It’s a glorious day when I carve out enough “science time” to personally make some progress on a research issue. This is why sabbatical time has become so prized – not a “year off” or a chance to write a book or similar, but to recapture some of the life-affirming personal creativity and exploration of our PhD and postdoc years that drew us into the system in the first place. How did we get here?

The usual justification is “accountability”. This was Ian Chapman’s primary defence for his upturning of UKRI’s funding structures: the public needs to know what their money is being spent on. And as a principle that is fine, even good. But it has limits: how much detail does the public really need or want to know?

Take the 175 glossy pages of the UKRI 2024-25 report… how much of this has ever been read? The main use I have every found is how some tables in the back showcase the lie about increasing international subscriptions. Every funded project also submits tens of pages of reporting every year, even though the research outputs can be discovered through existing publication/output metrics. The REF exercise forces us to spend years once again presenting these outputs in the best possible light, and academic panels reviewing thousands of the already peer-reviewed publications. Grant applications again spread over tens of pages, these days with restrictions on institutional applications that mean in particular junior staff spend significant time on proposals of which the majority never leave their own institution; note that this requires other academics to perform both the internal and then later the external review processes. At least the STFC Consolidated Grant system only requires whole-group applications every 4 years, unlike the bulk of research-council grants which are individual to each project and typically for half that time. I am not aware of the detailed information from these ever being used; the reporting from a previous grant is not fed to the reviewers of the next one to identify when funds have been inefficiently used.

And then we have the regular crises like the current one: sitting in a 3-hour online response meeting with 400 other UK academics and researchers I couldn’t help but tot up the numbers and note that that single meeting had used up research resources approaching a working year! When you note that the attendance was (by dint of greater responsibility/enthusiasm for such strategic issues) skewed toward the more senior and hence better paid, it’s even worse. Think of how hard we fight for research assistant posts, and there go 1-2 years of FTE via a meeting that only existed because of UKRI mismanagement. (Yes, yes, we inflict plenty of science meetings that don’t need to exist, too. Smart researchers tend to figure out which ones are worth their while.) Talking to some of the most senior particle-physics academics in the country, responding to the UKRI crisis has been their virtually sole activity so far this year, so you can multiply up a few more FTEs that way, too.

Bear in mind that this is all in the name of good governance! I am reminded of Parkinson’s Law – one of few “laws” not to fall foul of Stigler’s Law – and in particular “The number of workers within public administration tends to grow, regardless of the amount of work to be done.” I once attended an “academic entrepreneurship” course with a speaker who was a lot more frank than the organisers intended, who dismissed the role of university technology-transfer offices as “if you fill an office with patent attorneys, what do you think they’ll do? They’ll file patents, it’s all they know how to do. Of which nearly all are useless and are never taken to completion”.

We have grown a culture of the same form in research administration, certainly at UKRI and other levels both national and international, but also within universities, into teaching administration and oversight, and beyond. Academics, who are not generally hired for their interest in or aptitude at bureaucracy, are being flattened by the onslaught, from an army of self-perpetuating administrators who do not intrinsically value the researchers’ time.

Getting out of this hole requires leadership. Partly the growth in oversight burden is because the people employed by agencies value that work and see everywhere opportunities to increase it. Good leadership requires the CEOs to push back and demand the lightest overhead needed to assure proportionate accountability. We repeatedly see in application review that the majority of shortlisted applications are “fundable but not funded”, i.e. good proposals but not enough money left: this is not a position that requires heavy-handed oversight, it’s a buyer’s market for quality research. The agency CEOs and eventually their political masters need a level of confidence in their departments’ work that their inputs and outputs substantiate, but which they seem to be lacking: a confident executive does not need to hide behind reams of ass-covering compliance screed.

Layering bureaucracy in the name of efficiency is a bad joke, which needs to be called out. I would love to see grant reporting have to honestly estimate the fractions of staff time occupied not with the research they pledged to do, but with managing the well-intentioned oversight demands on that research. This is never built into costing, but is nominally collected in university time-allocation reporting (if you think that can be trusted). What would the public think of the responsibility of the funders and universities, for squandering expensive expert-researcher time on so much busywork?

Funding crisis? Not another one...

Andy Buckley — Mon, 23 Feb 2026 18:00:41 GMT

What goes around comes around, and the start of 2026 has, for particle physicists of my vintage, brought strong feelings of deja vu. These harken back to 2007, when the UK government self-inflicted a funding crisis with the creation of STFC, the UK agency that has since then funded our fundamental science research. For those new to the issues old and/or new, sit back, this is going to take a bit of explaining…

Born to be bad

Back then, the problem was that politicians had decided that it made sense to tie funding of large experimental facilities to the funding of science that would use them. This sounds pretty reasonable. Previously there had been a split, with the UK-based facilities and national laboratories funded and operated by CCLRC (no, you don’t care what the acronym stands for) and the research itself, plus international particle and astro physics facilities – most obviously, but not only, CERN – funded through PPARC (ok, fine: Particle Physics and Astronomy Research Council).

I was only a young’un back then and out of the loop on what particular genius cooked up this rearrangement, but the observant among you may note that from a particle and astro perspective, the facilities and scientific “exploitation” were already gathered together in a coherent way within PPARC. This was a very large deckchair-rearranging exercise… which for some reason also gathered up nuclear physics along with the particle and astro deckchairs.

The real consequence of the merger was to acquire an expensive set of tangentially related UK facilities, most notably the overspent Diamond light-source and ISIS neutron/muon source – both of which, despite being fairly big particle-accelerating rings, are of virtually no interest to particle physicists: other than prototyping of muon cooling, they are much more of interest to applied materials science and similar.

Their overspends, however, immediately became a budget problem for the particle and astro bits of the new STFC. This led to community action – including well-intentioned but irrelevant contributions from yours truly – and attempted engagement with the tail-end of the “New” Labour administration, who were dismissive and played somewhat on our naivety by asking us to hold back and let them sort it out – of course they did not. Ultimately, the campaign was crushed to invisibility when the rather larger and more immediate concerns of the 2007-8 “Credit Crunch”, now labelled the “Great Recession”, overtook the national bandwidth.

SNAFU: situation normal, all f’d up

Why this historical tour? Well, the last few months have seen the start of Ian Chapman as new chair of UKRI, the new (relative to 2007) umbrella funding organisation that distributes money to STFC and other research councils. And between the Christmas/New Year break and this month he has unleashed chaos upon the sector with what seem to be a set of unilateral reorganisations toward modish “government priorities”.

First, any confirmed project in the “infrastructure grant” channel unlucky enough not to have already received payments was abruptly cancelled – though, in Whitehall fashion this was bowdlerised to first “deprioritised” and now “paused” – regardless of the extent to which international partners were locked in on the understanding of UK contributions.

Not content with the dog’s breakfast of communication surrounding this and his new (and seemingly arbitrarily resourced) funding “buckets” system, Chapman compounded his science-community popularity by forcing screeching halts to established funding schemes across all of UKRI. This involved horrifying suggestions of 30% cuts (sorry, that word wasn’t used; try “efficiencies” or similar bowdlerisation) as standard, and potentially up to 60% cuts on some projects. While this applies across all UKRI research, the headline cuts to “curiosity based research” fall heaviest on STFC – not only because STFC’s science is predominantly “curiosity based”, but particularly because of its structural combination of expensive, fixed-cost facilities and the rather squishier researchers who use them.

It would be obvious to a moderately intelligent child that if half their pocket money is inflexibly ringfenced to pay for bus fares, then a 30% reduction in the total would translate to a 60% reduction in their spare cash for sweets or Robucks, but it doesn’t seem to have occurred to the extraordinarily rapidly-elevated Chapman that a similar logic applies to STFC. Cuts on a budget with large fixed commitments are multiplied on to the remaining, more flexible parts.

In his defence, Chapman was made chief executive of the UK Atomic Energy Authority within 9 years of his PhD, having already joined the senior management team 2 years previously, and is UKRI CEO in just over 18 – he was still a PhD student when the 2007 STFC crisis hit, and like most politicians behind this policy lacks institutional memory of the structural fixed-cost multiplier issue. However, he and they should be aware that just a few years ago we were in precisely the position of paying handsomely to join international facilities that we then didn’t fund researchers for; this was patched up at that time, and yet here we go again.

Chapman then proceeded to compound this offence to the researchers operating – and looking for post-PhD jobs – in STFC science by claiming the problem has come from currency fluctuations driving up the cost of international subscriptions: we know from our 2007 dealings that in fact the government hedges against such increases, and a fairly basic analysis shows that in fact the GDP-indexed CERN subscription bill has slightly reduced over the last 5 years. What I hear is that a more honest assessment of cost overruns are based in (human and energy) operating-cost increases at STFC’s UK labs. Welcome back to 2007.

My overall impression is of a personal prejudice and cavalier failure of planning diligence on the part of a hyper-ambitious science-administrator keen to do whatever will impress his new masters – the increasingly desperate, “growth-focused” Starmer government. While the exact meaning of his “buckets” is yet to be seen, the mood music seems clear that rather than investing in the sort of abstract but technically challenging fundamental science that created the Web, medical PET scanners, early machine-learning advances, and other such economic goodness, “industrial priorities” now means the government micromanaging its research toward established industrial R&D areas like “AI”. Those, in other words, that are already well along their hype cycle and where we are already late to the party for the subset that do have lasting substance. The lack of vision is … disturbing.

The rebellion

There have already been several excellent public push-backs, from Brian Cox, Jon Butterworth, Paul Nurse (doyen of UK science, ex-head of the Crick Institute, former and current head of the Royal Society, and parent to an ex-particle physicist), Vincenzo Vagnoni and Tim Gershon (spokespersons current and elect of the LHCb experiment, whose UK-led Phase-2 upgrade project was abruptly… ahem, “deprioritised”) and many others. The STFC science community, and more generally the wider UKRI one are undivided and determined not to be played this time.

Discussions with, and more recently public statements from, Chapman and science minister Patrick Vallance have suggested that the consequences on STFC science were unintended. And I found it heartening that in the recent Science & Technology Select Committee interrogation of Ian Chapman the committee several times raised UK fundamental science as an unqualified good thing. Less inspiring has been the effective abdication of leadership and community representation by STFC’s own relatively fresh leader, Michelle Docherty, who in a separate indication of basic failures of judgement ~~was forced~~ chose to stand down from their parallel and contradictory role as president of the Institute of Physics. (As noted here, a conflict of interest / judgement gap remains in her role as Astronomer Royal.)

Break the cycle

So there we go. Different decade, same shit – although this time I find myself living through the mess not as a fresh-faced junior postdoc, but a relatively grey-bearded prof. There are (very faint) hints of light at the end of this self-inflicted tunnel, if we read substance into the softening of rhetoric and admission of unintended consequences, but the onus is on government to do something. Reconsidering the position of the ambitious new administrator who’s managed to cause two sector-wide panics and broad funding hiatus in as many months of being in post would be a start.

And above that, the politicians who came in preaching growth and infrastructure investment should understand that basic research (and separately, the UK university sector as a whole) are national assets to be resourced and leveraged not over a few-year election cycle but over decades.

This post is quite long enough already, so I’ll stop here. Thanks for reading. But in particular I also noted the cost of this uncertainty as researchers junior and senior shift more or less of their active time to following and countering these disruptions: the integral of this opportunity cost is extremely high. And it’s not just crises: administrative overheads in science in general are a generally unaccounted opportunity cost acting as a drag-anchor on research. If the public that apparently demand this accountability were aware of the cost of providing it, government/funder attitudes would likely change. More in a follow-up, when I’m able.

PS. I see Ken Rice and Peter Coles have also covered this ground] nicely. Good to also have an astro perspective – we really are in this together, and need to stay that way.

Function plotting and the inverse cdf

Andy Buckley — Wed, 06 Mar 2024 23:19:41 GMT

I previously wrote about using the inverse-cdf transform not just for its classic application of efficient random sampling, but as an optimal histogramming-binning generator mapping exactly to expected population quantiles.

(If you think of the random process that populates the histogram as itself being done via uniform random sampling of the unit cdf interval, mapped back into the coordinate value using inverse-cdf, it's even clearer why a uniform binning of the cdf interval would be optimal.)

In this post I'd like to show that we can usefully bend the inverse cdf at least a little further, into applications where there isn't even traditionally a positive-definite pdf. My motivation is efficient function plotting.

In the YODA new plotting system, I'd like us to be able to overlay analytic -- or at least Pythony -- functional forms on top of data- or MC-populated histograms. But just creating a linspace() of lots of points in $x$, and hoping that'll be enough to smoothly render the most rapidly varying bits of a plot -- the usual approach -- is unsatisfying.

If that plot has a mixture of curvy bits and straight sections, we could distribute a given number of points more efficiently, as straight parts need only one point at either end to draw a (Postscript or PDF) line between, and the points budget would be better spent in the curvy parts where straight lines are nowhere a good approximation. If we're stuck with using straight-line segments -- which we more or less are with matplotlib -- then distributing their endpoints more efficiently means we can get a good visual effect everywhere on the plot with fewer points, and hence a smaller file-size.

Let's think a little: as intimated above, the key issue is not the gradient $df/dx$ of the function $f(x)$, as I can draw a steep-gradient straight line just as effectively with two points as I can a flat one, but the second derivative $d^2f/dx^2$, aka the "curvature". So let's make a guess that what our brains want to see proportionally mapped out is point density in proportion to the curvature. So we can use the inverse-cdf method again, right: the cdf is $$F(x) = \int \frac{d^2 f}{dx^2} dx = \frac{df}{dx} ,$$ right?

Ah, but curvature can be both positive and negative: that messes things up. We really need $$F(x) = \int \left| \frac{d^2 f}{dx^2} \right| dx ,$$ but that's a more awkward object: we'd need to decompose into the set of positive-curvature and negative curvature intervals, and add up the integrals from each, e.g. $$F(x) = \sum_i \pm \int_{\omega_i} \frac{df}{dx} \Big|_i ,$$ with appropriately identified $\pm$ signs.

We could maybe do this for arbitrary functions with a symbolic algebra library like sympy but it's awkward. And doesn't solve the elephant-in-the-room issue that most monotonic functions, and hence most cdfs, don't have an analytic inverse. (This problem is also the major limit of the inverse-cdf method in general, of course.)

When in doubt, sample to victory! Let's just throw lots of evenly spaced points at our function, numerically compute an array of second derivatives and take their absolute values, then compute a numerical cdf by taking the cumulative sum of the array:

	`xs = np.linspace(XMIN, XMAX, 1000)`
	`ys = f(xs)`
	`ypps = np.diff(ys, 2)`
	`ayps = np.cumsum(abs(ypps))`
	`ayps /= ayps[-1]`

where the final line normalises to make sure the cdf really adds up to 1 as intended. Then we can throw a more modest number of points linearly into the cdf interval, and numerically invert via straight-line interpolations:

	`def G(q):`
	`return np.interp(q, ayps, xs[1:-1])`
	`qs = np.linspace(0, 1, N)`
	`xs_opt = G(qs)`
	`ys_opt = f(xs_opt)`

Let's see the result on a pretty nasty function, $$f(x) = (2.5 - x) + e^{-2x} \cos(20 x^{0.7}) ,$$ using just 50 points:

Not bad, huh? The red line is this optimal sampling approach, the green (with visibly bad-approximation straight lines in the wibbly left-hand part) is the uniform straight-line approximation, and the true function is hidden underneath in blue. The grey lines in the background show the "optimal" sampling points, uniform in the $y$-axis of the cdf, overlaid in orange.

Now, I've cheated a little here, because to my eyes that assumption that the (magnitude of the) second derivative is proportional to the optimal sampling density isn't quite correct. Unsurprisingly, our visual cortexes are a bit better than that; in particular we seem to be fairly good at compensating for scale differences, so we see deviations from low-amplitude bits of smooth curve not so differently from on high-amplitude versions of the same shape. In the full code, reproduced below, I added a micture fraction $f$, so you can fade smoothly between purely "optimal polling" and "uniform polling" strategies. There are probably smarter ways.

I hope this was interesting. It feels to me like there's more potential to play and improve here, but hopefully we'll deploy this feature into YODA's plotting before too long!

curvplot.py (Source)

#! /usr/bin/env python

import numpy as np
import matplotlib.pyplot as plt

N = 50
XMIN, XMAX = 0.0, 2.5
F = 0.6

def f(x):
    "Function to plot"
    #return np.exp(-x) * np.cos(10*x)
    return (2.5 - x) + np.exp(-2*x) * np.cos(20*x**0.7)

## Naive version
xs_lin = np.linspace(XMIN, XMAX, N)
ys_lin = f(xs_lin)

## Densely sample to build an cdf of |f''|
xs = np.linspace(XMIN, XMAX, 1000)
ys = f(xs)
ypps = np.diff(ys, 2)
#ypps /= (ys[1:-1] + np.mean(ys))
ayps = np.cumsum(abs(ypps))
ayps /= ayps[-1]

## Mix with linear in F:1-F ratio
byps = np.cumsum([1. for x in xs[1:-1]])
byps /= byps[-1]
ayps = F*ayps + (1-F)*byps

## Invert cdf
# TODO: make more efficient!
def G(q):
    return np.interp(q, ayps, xs[1:-1])
qs = np.linspace(0, 1, N)
xs_opt = G(qs)
ys_opt = f(xs_opt)

## Plot
for x in xs_opt:
    plt.axvline(x, color="lightgray", linewidth=0.8)
plt.plot(xs, ys, label="$f(x)$")
plt.plot(xs[1:-1], ayps, "--", label="CDF, $F(x)$")
#plt.plot(xs[1:-1], ayps, "*")
plt.plot(xs_lin, ys_lin, label="Linear")
plt.plot(xs_opt, ys_opt, label="Optimal")
plt.legend()
for f in [".png", ".pdf"]:
    plt.savefig("curvplot2"+f, dpi=150)
#plt.show()

Optimal binning and the inverse cdf

Andy Buckley — Tue, 27 Feb 2024 23:25:13 GMT

One of my favourite tricks in numerical methods is [inverse transform sampling](https://en.wikipedia.org/wiki/Inverse_transform_sampling), which is a supremely elegant way to efficiently sample from a probability distribution.

If you know a distribution's cumulative density function (cdf) -- or can approximate it; see the follow-up post -- and the distribution is positive everywhere, as probability densities should be, then uniform sampling from the unit interval and applying the inverse function of the cdf to the samples is exactly the transform required. It's elegant and intuitive -- once you've seen it -- because of course large fractions of the [0..1] cdf interval are taken by regions of high density in which the cdf grows quickly.

The problem is of course that most interesting functions -- even the Gaussian, dammit -- don't have analytic cdfs. But nevertheless, use of the inverse transform is the analytic endpoint for lots of other strategies such as importance sampling, where an analytic distribution close to the desired one is a key ingredient.

What I see remarked upon much less is the equivalence of sampling and binning of histograms. In fact, a culture of making histograms assuming uniform widths for all bins is so engrained that you'll find [eight different strategies](https://numpy.org/doc/stable/reference/generated/numpy.histogram_bin_edges.html) "to calculate the optimal bin width and consequently the number of bins". This remarkably ignores that there is a well-defined ideal strategy for achieving equal relative statistical errors across a histogram, and that is to bin with variable widths in proportion to the reciprocal of the expected density function!

When following this recipe, by construction the product of density and width then gives equal bin populations and hence equal statistical stability. You can then choose the number of bins by dividing the sample size by the desired statistically stable population of each bin. Extending to fix a minimum bin width to respect non-statistical limits on binnable resolution is a fairly simple task.

Let's see this in action for a classic function with a huge dynamic range of densities: the [Lorentzian or Breit-Wigner distribution](https://en.wikipedia.org/wiki/Cauchy_distribution), describing physical resonance effects, with pdf $$f(x) = \frac{1}{ \pi \gamma \left[1 + \left(\frac{x-x_0}{\gamma}\right)^2 \right] }$$ and cdf $$F(x) = \frac{1}{\pi} \arctan\left(\frac{x-x_0}{\gamma}\right) + \frac12 .$$

With uniform binning, a BW distribution is doomed either to be so coarsely grained that all detail will be missing from the statistically robust peak, or (with finer binning) to have wild instabilities in the low-population tails. Like this:

But we can use variable binning, with uniformly spaced samples in $F(x)$ corresponding to distribution quantiles, mapped back into the expected distribution as bin-edge positions using the inverse cdf: $$x = \gamma \tan((F - \frac12) \pi) + x_0 . $$

Given that these bin edges correspond to distribution quantiles, I guess we could call this approach "expected quantile" binning, or similar, if it needs a name. Here's a bit of Python/numpy code implementing this binning strategy for the Breit-Wigner of the Z-boson mass peak around 91.2 GeV:

	`M, Gamma = 91.2, 5.5`
	`gamma, x0 = MGamma, M*2`
	`qmin, qmax = 0.05, 0.95 #< quantile range to map`
	`qs = np.linspace(qmin, qmax, NBINS)`
	`xs = gamma * np.tan( (qs-0.5) * np.pi ) + x0`
	`binedges = np.sqrt(xs[xs > 0]) #< eliminate any negative E^2s`

And giving the following edge distribution (this version actually engineered to place the qmin, qmax quantiles at 70 and 120 GeV):

And finally the dynamically binned distribution:

Nice, huh? The full code listing follows. I'll follow this up with a post on how to use a variation of the same idea to optimally sample a function for visual smoothness, based on sampling density proportional to curvature...

bwedges.py (Source)

#! /usr/bin/env python3

import argparse
ap = argparse.ArgumentParser()
ap.add_argument("DATFILE", help="unbinned data file to read in")
ap.add_argument("OUTNAME", nargs="?", default="mee",
                help="hist name to write out as .dat and .pdf")
ap.add_argument("--dyn", dest="DYNBIN", action="store_true",
                help="hist name to write out as .dat and .pdf")
args = ap.parse_args()

import numpy as np
vals = np.loadtxt(args.DATFILE)


## Binning
NBINS = 50
RANGE = [70,120]
binedges = np.linspace(*RANGE, NBINS)
if args.DYNBIN:
    ## Dynamic binning, by inversion of the Breit-Wigner CDF:
    ##  https://en.wikipedia.org/wiki/Cauchy_distribution
    ## PDF = 1 / [pi gamma (1 + (x-x0)^2/gamma^2)]
    ##   with x = E2,  x0 = M2, gamma = M Gamma
    ## CDF = arctan( (x - x0) / gamma) / pi + 1/2  
    ##   -> x_samp = gamma tan((rand - 0.5) pi) + x0
    M, Gamma = 91.2, 5.5
    gamma, x0 = M*Gamma, M**2
    qmin, qmax = 0.05, 0.95 #< quantile range to map
    qs = np.linspace(qmin, qmax, NBINS)
    xs = gamma * np.tan( (qs-0.5) * np.pi ) + x0
    binedges = np.sqrt(xs[xs > 0]) #< eliminate any negative E^2s

    
## Plot and save
import matplotlib.pyplot as plt
fig = plt.figure(figsize=(10,7))
fig.patch.set_alpha(0.0)
counts, edges, _ = plt.hist(vals, bins=binedges,
                            density=True, histtype="step")
plt.xlim(RANGE)
plt.xlabel("$e^+ e^-$ pair invariant mass, $m_{ee}$ [GeV]")
plt.ylabel("$\mathrm{d}N / \mathrm{d}m_{ee}$ [count/GeV]")
plt.yscale("log")
for ext in [".pdf", ".png"]:
    plt.savefig(args.OUTNAME+ext, dpi=100) # transparent=True
np.savetxt(args.OUTNAME+"-hist.dat",
           np.stack((edges[:-1], edges[1:], counts)).T)

Showing you care

Andy Buckley — Tue, 27 Feb 2024 22:52:11 GMT

I recently did an interview with the David Hume Institute – through a friend – about how (or if) personal financial issues affect my work. I won’t forensically document how that aspect went, except to say it was a fun chat and they are doing good things. But the thing that’s stayed with me is how I involuntarily laughed at the suggestion that my employer would provide any sort of benefits in addition to pay.

The thought had never even occurred to me! I’ve used government-organised schemes through work, like Cycle Plus and Childcare Vouchers, but the employer is a pretty passive partner in those. It turns out that a bunch of companies do actually offer staff discounts – as a loss-leader, of course, and not entirely enticing: I have yet to make use of my small discounts at Kilt Warehouse or Beauty Boutique – and again this is zero-effort from the uni. Virtually nothing involving the uni actively working to provide better quality of life for its employees. Should it be?

This brought into relief the contrast I see via engineers I play with in a band: their employers shower them with subsidised meals, massages, music tuition, rewards for patents, you name it. It’d be enough to make jealous, except they also sound like brutal workplaces in other ways, and despite the overwork and underappreciation, academia’s relative freedom and autonomy is something to be cherished. But the very idea of a work culture that tries to make employees feel valued is alien if you’ve always worked in universities.

This doesn’t surprise me: the UK university sector is in a terrible bind. Outside Scotland, the switch to funding through student fees at just over £9000/year led to a pullback of previously index-linked government funding, and there has not been political courage to significantly increase the charges in line with inflation over the last 12 years. In Scotland, the situation is even worse as the government payment per student has not even kept pace with the rest-of-UK figure. The sector is chronically underfunded, full of people working several job’s worth of tasks because they believe in the mission, at a time when increased expectations from students (on the teaching side) and for project-management diligence (on the research side) mean we need substantially more staff across the board. I could also add the intrinsic inefficiencies and perverse incentives of a system that’s evolved to recruit staff primarily for their research expertise, then drowns them in teaching and departmental admin, while their research colleagues get used to them “not being useful anymore”. And that genius governmental ideas like systematic 80% FTE funding of research projects (i.e. systematically making research financially unattractive) make it impossible to balance the books without rinsing international students for fee money.

There is no capacity in the university system to make meaningful change, and were I a VC I’d probably do the same: try to keep the locomotive on the tracks for as long as possible despite it being chronically overburdened, lobby government in the background, and hope not to be in charge when the crash comes. But that fatalism seems even to have propagated to cost-free indications of caring about the frontline staff. In 20+ years I have never once seen a VC or dean of college visit a Physics department or take staff Q&As. How hard would that be, once a year? I also happened to chat with a previous vice-chancellor last year during strikes, who enthused “sensible folk, physicists” at the news there weren’t many Physics staff refusing to mark exams; he, of course, was a perfectly nice chap but never taught or researched in a university in his life. How would he know what it’s like? Well, he might have asked…

In the big picture, senior management giving staff their time and attention would say a lot more than trinkets. Or the occasional lick of paint on our decaying buildings – I should say that the culture in departments is generally good, with the infuriating obliqueness of central management a source of solidarity! But in UK academia, for now, the old mood music remains the same: keep heads down, be grateful, and for god’s sake keep the paying students coming…

More heat than light

Andy Buckley — Tue, 27 Sep 2022 22:20:30 GMT

Another week, another row about particle-physics methodology involving the field’s latest engagingly controversialist internal critic – older readers may feel a pang of deja vu from the “Not Even Wrong” years. But this time, the maelstrom has somehow escaped Twitter and been platformed in Guardian Science.

I feel a pang of guilt about criticising this article. After all, as scientists we are meant to question ourselves constantly – the Royal Society, with a decent claim to being the leading grouping of natural philosophers as scientific method established itself in the mid 1600s, after all adopted a Latinised “Take nobody’s word for it” as a motto. And within the field, I’d be lying if I claimed never to have felt frustration at perceived timidity and herd instinct. There’s also a good practical reason not to comment, since that’s probably what is hoped for, all publicity being good publicity when you have wares to promote.

But this really is a terrible piece, and on the whole I think better to engage than let such things slide and enter public consciousness unopposed. It starts with quirkily hypothesised portmanteau animals and the cunning plan of an invented group of zoologists to travel the world in search for them – then asserts that this is what particle physicists, or at least beyond-Standard-Model (BSM) theoretical physicists do with their days. Experimentalists don’t get let off easy: we are apparently slack-jawed rubes, so uneducated or uncritical about physics that we hang on every theorist’s word. I get the feeling Sabine has not tried selling any theories to a CERN experimentalist audience recently.

This is deeply disingenuous stuff. First off, it’s a gross mischaracterisation of the model-building process. Even as a non-expert, I know that the majority of models are proposed not just willy-nilly, but to solve a perceived problem – or ideally, more than one. Where most of us differ from Sabine’s value system is in what we consider an above-threshold modelling problem. She has asserted many times that the Standard Model can accommodate everything that has been observed, which is not true: neutrino masses require a mechanism not established in the SM, cosmological matter-antimatter asymmetry requires a mechanism of CP violation far stronger than achievable in the SM quark sector, and so-on. These seem fairly unambiguous areas where new mechanics are needed, and I’ve not even mentioned her preferred touchstone of dark-matter particle vs. MOND.

But most of us also take seriously, though perhaps not as seriously, vaguer questions of model stability (the hierarchy problems) and of why our model contains the components it does in the form it does. If we should take nobody’s word for it, we should also be sceptical of fringe calls to just give up and accept the world as it seems to be. It is an entirely reasonable scientific endeavour to try and understand why things are the way they are. To deny that this is rational requires either a particularly naive take on philosophy of science, or bad faith. Just because the likes of the anthropic principle (things are the way they are because we’re here to see it) have some intellectual merit doesn’t mean that fundamental scientists must Eeyorishly resign ourselves to not even trying.

Most “organising” theories that might solve big conundrums of this sort – ranging from more technical data-model discrepancies to the borderline-philosophical – have consequences that could potentially be measured, and so we should search for them and cut away the models that fail to appear. And, to give us some credit, some such organising principles have borne fruit before, in the forms of the W, Z, and Higgs bosons, and various exotic hadrons. This is a long way from hypothesising acontextual flying cave-worms: it’s more like – to extend an analogy in a field I know as little of as Sabine does – observing several separate evolutionary responses to selection pressures, hypothesising that they could interact interestingly, and proposing to look for them in places with the appropriate conditions. Maybe that’s the sort of thing zoologists should be funding, maybe it’s not, but it’s not a category error to consider it.

This brings me to the final, and I think most offensive, aspect of the article, which is the argument that we either pursue these hypothetical hints of organising principles through clueless herd instinct or through rampant careerism. And the reason this annoys me so much is that there is undoubtedly a kernel of truth here. I think everyone in the field has at some point encountered a physicist who can’t explain why they’re interested in what they’re doing, but it’s what the group or their PI is interested in, or because they just like the process, or because it’s an area publishing lots of papers and they’d like to ride that bandwagon (cf. the absolutely correct criticism of LHC 3sigma-anomaly chasing). Pin the blame for that on our intrumentalised version of research-performance measurement, a superheated academic job market (guess what, folks want a job in a stimulating area they spent their intellectually formative years mastering), and the raging bin-fire that is the rentierist academic publication business. By overextending this reasonable criticism to the sort of gasp-inspiring cartoon that gets one a Guardian splash, the whole argument jumps the shark and we learn nothing.

But, by-and-by, most of us know about this problem. Most research-active academics are trying to find areas where they can do something impactful, not just be a cog in the machinery… and actually, proposing or searching for unmotivated exotic new particles is not a rational bet. I’ve seen properly cynical, unmotivated models, and no-one outside the proposer’s group works on them or pays the blindest bit of attention. Blunderbuss criticism in a very public forum also risks destabilising institutional support for the whole field. Funding agencies generally recognise particle physics as mostly worthwhile and balance their involvement across its facets, but this could become harder to do if populist tales of careerist physicists cynically living it up on taxpayer funds find purchase in the wrong ears.

So, not everything said is wrong. But it is dressed up in such a pantomime-dame version of the critique that it can’t be taken seriously. And that’s a shame: there are conversations here which could perhaps usefully be made more open and explicit. There are horrifying degrees of rentierism and perverse incentive in academic careers, publishing, and conferences – let’s talk about them, too. But straw-man arguments about modelling whimsy and bad faith distract from these real problems and more nuanced questions of scientific value; as quintessentially rational people, we need to reject them and platform the valuable discussions instead.

Leaving Labour

Andy Buckley — Sun, 22 Jan 2017 16:27:02 GMT

Just sent:

Dear <CLP Secretary>,

It's with sadness that I write to tell you I have decided to leave the Labour Party.

Despite having initially welcomed the arrival of a nominally more socialist national party leader, the Corbyn team have backtracked, vacillated, and failed to deliver any coherent progressive economic message. In their hands, Labour has become an incoherent policy vacuum with constantly bungled media management. I don't believe that leftist or revolutionary policies make a party unelectable -- look at the two big electoral shocks of the last year -- but an incoherent party that occasionally strikes radical poses (before immediately backtracking) is going nowhere.

This has been going on for many months, but I have remained a member in the hope that the party would rediscover its purpose -- currently to hold the government to account, at a time that that is needed more than ever before in my lifetime. I continue to respect the commitment and competence many of the Party's MPs and MSPs. But Labour's failure over the last 6 months to provide an alternative to the Government's partisan & scorched-earth attitude to Brexit, crowned by last week's (yes, again bungled) declaration that Labour MPs will be either encouraged or whipped to vote for the Government's A50 bill, without demanding answers to the detail, has been the final straw for me.

I appreciate that many of these issues are not primarily the remit of the Scottish Labour Party, especially with its current paltry Westminster representation, and that by cancelling my membership I am unfairly tarring you with the same brush. But this is the only way I can make my displeasure known to the Party as a whole, and with the rhetoric coming from Theresa May and other representatives of "Little England" I cannot even say that my commitment to the Union -- the main distinction between Labour and SNP -- is as unwavering as it used to be.

Yours, Andy Buckley

Science and the English language

Andy Buckley — Tue, 22 Nov 2016 20:55:42 GMT

Having just completed the tortuous process of publishing an ATLAS data analysis, in particular 6 months of back-and-forward text-iteration, I find myself thinking of the excellent guidance on writing in English provided by George Orwell in his essay Politics and the English Language.

This is largely concerned with how banal, obfuscated and characterless English language can act as a cloak for vapid thought, and a smokescreen for vile political acts. (We've certainly seen plenty of that in the UK, USA, and parts of Europe in the last few years.) It is crucial short reading for anyone who aspires to good communication of facts and ideas, and I point all my students at it -- especially those who seem to believe that to be convincingly sciencey, a report has to be a Jackson Pollock composed of obfuscation and undefined technical jargon.

The pithy take-home message from the essay is a list of 6 excellent rules on language, emphasising clarity above all. The final rule is the one to rule them all: "Break any of these rules sooner than say anything outright barbarous." This invaluable instruction is expressed within a few short pages, to which I can't help but compare the ATLAS experiment's 51-page style guide, which says nothing remotely as interesting or profound.

But, as I was recently and painfully reminded, this tedious document is held up as the touchstone of English language by several individuals within the experiment. Frankly, the sort of people who will volunteer themselves to comment on hyphenation, the existence or not of possessive 's'es on words ending in s, or detailed footnote-marker placement are precisely the sort of busybodies who should be kept a good pole's length from any sort of editorial review. Rather than act as critical maintainers of ATLAS paper readability, they have made themselves the priests and guardians of an arcane grammar style guide, far more concerned with the trees than the wood. The joke is that the target journals (themselves hardly paragons of quality English) will anyway revert half these decisions, whether by design or accident.

As scientists, and particularly as scientists in a discipline with quite mathematical foundations, pedantry can come naturally. But we should -- must -- primarily concern ourselves with clear and correct description of our methods and results. After that comes readability, a nebulous concept involving not just clarity but also character. Formal grammatical and punctuational correctness honestly do not get a look-in: have a look through any of the best books on your shelf, either fiction or non-fiction, and you'll find "errors". Except they are not errors, they are character, voice, and awareness of what is important and what is trivial noise.

English language, like any other, is not subject to rules of algebraic correctness. It lives, breathes, and evolves. And its worth is tied into its variety and willingness to be interesting -- a style guide that thinks it appropriate to ban the use of the present perfect tense everywhere is a problem rather than a solution, by flattening the resulting language into a bland and repetitive drone.

Uniformly applied present tense, a limited and repetitive set of overly-adjectived nouns and ... this is the textual equivalent of speaking with someone on strong personality-suppressing drugs. Why would we want to inflict that on our readers in the name of "correctness"?

You should be concerned when a bunch of physicists or mathematicians appoint themselves as guardians of language. Unlike those subjects, language -- in general, and in its written representation -- does not have well-defined rights and wrongs. There are some robust don'ts for technical writing rather than creative, but far fewer of them than many seem to think. What matters is the readability of whole phrases or sections, not an algebraic absolutism on the level of grammatical atoms.

The experience genuinely made me question my interest in doing physics with ATLAS. That was several months ago, and fortunately it passed, but look at it this way: we already waive a lot of individuality by working in a big collaboration -- it involves following a lot of internal rules, of not having your name clearly identified with work which you led, or not having the right to be primary choice of conference speaker on your own work. So we have to make do with small things, and in corollary the small things matter... such as the small freedom to "voice" your own paper. It's hard enough to have busybodies with poorly calibrated comment filters make dubious change requests on work that they played next to no role in. And harder still to be asked to revert those comments in the next round of review, and so-on. But then to find that the collaboration places so little trust in its members' collective ability to produce a high-quality scientific paper, that it explicitly employs someone to make a bunch of acontextual (and frequently wrong) grammar complaints... blimey.

This is not to say that the feedback on language has been unhelpful. There have been places where phrasing or clarity has been improved. But it's a question of threshold: not a single hyphenation, nitpick on precise choice of sub-tense, or anal retentive replacement of "systematic" with "systematic uncertainty", or change from "observable" to "observable's value" has influenced scientific readability. And in some cases being technically correct really misses the point about communication: it has to be approachable and engaging, otherwise already dry scientific papers easily become about as captivating as the phone book.

So sure, give me feedback on language: but please keep to the stuff that makes a difference. Without this nonsense we'd have published nearly 6 months earlier, with a more readable document, and with my nerves significantly less shredded! But on the plus side, it's certainly revitalised my empathy for the authors of papers where I'm the reviewer. Proportionality and knowing when to stop... not natural physicist traits, but we need to learn -- especially when there's so bloody many of us.