Making infeasible runs feasible

The full-annulus simulations the group needed would not run. The reason was not the one everybody assumed.

The scaling study

Resolving what happens to particles through a multi-stage compressor properly means a full-annulus simulation: every blade passage, not one sector with periodic boundaries. That is a 100M+ cell problem, and the group could not run it.

The instinct in that situation is to ask for more cores. I ran a systematic scaling and performance study instead: the lab workstation against three Slurm partitions, across core counts, node counts and memory configurations.

More cores on one machine is not the answer. On the workstation the wall clock bottoms out and then rises: 965 s at 68 cores, 1002 s at 110. Adding cores made it slower. The cluster scaled 3.7× going from 8 to 64 cores at about 45% efficiency, against 1.8× at 25% on the workstation, and at 96 cores the same case ran roughly 30% faster, a one-month simulation becoming about three weeks.

My working hypothesis was that the constraint was memory, since a job had already been killed outright on a partition with 2 GB per core. A computational scientist at the university's HPC centre corrected the mechanism: memory has little to do with speed, and the large-memory nodes I was aiming at actually clock slower, 2.45 GHz against 2.75 GHz on the normal queue.

Both of those are true, about different questions. Memory capacity per rank decides whether the job runs at all: below the threshold it is killed, not slowed. Clock speed decides single-node throughput. Neither predicts what happened next.

Standard practice for this solver is to minimise node count. Domain decomposition makes the partitions interdependent, every iteration exchanges boundary data between ranks, and across a network that costs 10–100× what it costs inside a node. More nodes should mean a slower run.

It did the opposite. Holding 64 cores per node and adding nodes: 838 s → 711 s → 530 s across one, two and three. At 14 cores per node, 2269 s on one node fell to 576 s on sixteen, a 3.9× speed-up where the textbook predicts a penalty.

So the original memory instinct was right about the mechanism and wrong about the remedy. The answer was not one machine with more memory; it was more machines, each with its own memory bus. A large-memory node would have made it worse, since those clock slower.

Extrapolated to 16 nodes × 64 cores the month-long case lands near 8.75 days, bounded not by hardware but by the 400 ANSYS licences the group can hold at once. The timings behind all of this, and the script that turns them into the plots, are in cfx-cluster-scaling.

Two smaller results from the same study: disabling hyperthreading measurably improved throughput, and multi-node jobs failed with solver error 255 at certain rank counts: the workaround was core counts that are not powers of two, 36 rather than 32, 68 rather than 64.

That changed what we asked for: normal-queue nodes in quantity rather than a single large-memory machine, and licence capacity as a first-class constraint rather than an afterthought. 100M+ cell full-annulus simulations became feasible for the group for the first time, and the group moved off one workstation and onto the cluster.

None of this was my dissertation. It was a side investigation that the whole group ended up running on. A master's student completed an LES study in three months on the strength of it, and a colleague took the method further, benchmarking Owl against Tinkercliffs on NASA Stage 35 and measuring wall-clock-per-iteration improvements of up to 89% from multi-node arrangements. His report credits the original benchmark and records the finding that started it: past 64 cores the cluster beat my workstation by only 20–30%, and the real gain was multi-node.

One methodological note from that follow-up worth keeping: benchmark numbers fluctuated until jobs were submitted with --exclusive. Sharing a node with another tenant makes timings unrepeatable, and a scaling study built on unrepeatable timings measures nothing.

The useful output of a scaling study is rarely a speedup number. It is finding out which resource you are actually short of, which is often not the one being requested, and sometimes not the one you were sure of yourself.

When my own model broke the solver

The breakage model creates particles. A fragmenting parcel becomes several, those fragment again, and the tracked population grows through the run. The aerothermal solve stayed stable; the job died anyway: dynamic memory exhaustion inside the solver's own particle-track bookkeeping, the integer stack and track-info structures, once counts climbed far enough.

The multiplication factor is stable at roughly 6–7×; what changes is the absolute scale, and with it the bookkeeping. The solver's static memory flags do not touch this; they size the general solver pools, not particle multiplication, fragment lifetime, or how much history is kept per particle.

There was an easy way out. Inject fewer particles, write less trajectory output, and the problem disappears, along with the statistics the study exists to produce. I did not want to pay that before knowing whether the infrastructure could be made to carry the real workload.

So I wrote it up as an engineering question rather than a complaint and took it to the university's HPC staff: scaling evidence on particle-count growth and track-file size, the Slurm script and solver flags in full, and specific questions on MPI layout, memory per rank, load imbalance and filesystem choice.

The diagnosis was not what I expected. The cost was not bandwidth; it was many small reads and writes to remote storage. On a cluster, /scratch, /home and /projects all live on a storage cluster reached over the network, so a stream of small sequential I/O operations is dominated by per-operation latency rather than throughput. It is the failure mode that does not exist on a workstation with a local SSD, which is exactly why nobody arriving from one expects it.

Multi-node makes that harder rather than easier: every node has its own $TMPDIR, so input has to be broadcast to each one and output gathered back before the allocation ends, or it is gone with the node. Which is the recurring shape of this work: the fix is available, and it costs you something in job-script complexity that you have to decide is worth paying.

The campaigns

On top of that: structured parameter sweeps and DoE-style studies across operating conditions, mesh resolution, timestep and particle parameters, on large-memory nodes under MPI. Enough runs that managing them by hand stops being possible, which is its own engineering problem.

Turning output into something you can model

A campaign of this size produces up to 450 GB per run. The solver's own post-processing tools will not answer questions at that scale, and several of the questions I needed answered are not ones they can express at all.

So the analysis is a pipeline: automated, reproducible, and written to be run again. One 254 GB particle-track file is parsed down to 10,797 mass-verified fracture events: every one checked for mass conservation on the way through, because a fracture event that does not conserve mass is a bug in my model and I would rather find it than average over it.

That dataset is what makes the reduced-order modelling possible: a breakage model collapsed to three substrate-independent parameters (RMSE 0.051 across 48 conditions), with leave-one-surface-out cross-validation transferring between alloys at ΔRMSE < 0.001: a held-out generalisation test, not a fit.

Recovering what the solver does not export

CFX writes one row per particle-wall impact and does not write which particle it belongs to. So you can count impacts, and you cannot ask the question that decides whether an erosion map means anything: how many particles is it made of?

I recovered the identity by matching every impact against a 14 GB trajectory file on its position and velocity signature, streamed in chunks with a spatial index rather than held in memory: 586,764 of 586,764 rotor-blade impacts attributed, to two micrometres and a millimetre per second.

The answer was that 4.6% of the particles produced 92% of the impacts. The physical map is a narrow band at the leading edge, at 28° median incidence. The rest is a small trapped population grazing the surface at 0.08°, writing hundreds of rows each. No per-surface quantity can show you that, because a contour plot has no notion of a particle.

The first version of that pipeline took 2 h 15 min and then ran out of memory at 4.5 GB. Parallelising it bought nothing and rewriting it in C++ bought nothing, because the cost was algorithmic rather than a shortage of machine. Changing the question from compare everything to everything into look up the nearest few took it to four minutes on three times the data. Knowing which of those two situations you are in is most of the skill.

Infrastructure as an output

Solver extensions, tracking methodology, job workflow, analysis pipeline: this is now the infrastructure the group's other doctoral projects and its sponsor deliverables run on, and I train the junior students on it.

I mention it because it is the part that does not appear in a publication list. Building the thing several other people's work depends on is a different contribution from writing a paper, and I would rather be measured on it.

The code and the data

• cfx-cluster-scaling

  Every number on this page, with the measurements underneath it. Wall-clock timings as CSV, the figures regenerated from those CSVs by a script rather than drawn by hand, the sanitised Slurm job scripts, and the full write-up of the dynamic-memory and I/O work, the failure signature, the per-rank evidence, and the questions I took to the HPC centre.

• cfx-particle-id-recovery

  The attribution above: the streaming matcher, the per-particle fate pass, the record layouts the vendor does not document, and the frame problem that still blocks the stator surfaces and their 3.23 million impact rows. It opens with the profiling table and the two things that did not work, because those took longer than the fix.

It also records what to distrust: each point is a single run rather than an average, the 8.75-day figure is an extrapolation, and the error-255 workaround is documented as something that worked without my ever establishing why.