Moving a 55-country HIV portfolio from remote desktop to parallel batch

The bottleneck

The engagement modeled combinations of next-generation HIV prevention and treatment products: oral and injectable PrEP, rings, implants, bNAbs, a future vaccine and cure, VMMC, and ART variants. Each scenario translated portfolio assumptions and market-dynamics rules into Spectrum inputs, then extracted infections, deaths, and coverage. Running country files manually, one at a time through remote desktop, made a full portfolio pass impractical. Scenario and sensitivity branches multiplied that bottleneck.

Four-stage pipeline

A fully automated R pipeline in four stages:

• Templating: reads an Excel assumptions database, enumerates every product-combination branch, and applies market-dynamics rules (synergy, amplify, cannibalize). For example, long-acting PrEP can cannibalize oral PrEP.
• PJNZ editing: programmatically edits Spectrum country files, scoped to priority countries, in parallel.
• Extraction: runs Spectrum in batch over every branch, with a validation-and-auto-retry loop for corrupted or incomplete jobs.
• Post-processing: consolidates every branch into an analysis-ready dataset of infections, deaths, and coverage by country and scenario.

The architecture

The pipeline detects who is running it and on what OS, and picks a matching config. On Windows it runs Spectrum locally. On macOS it opens an SSH connection to the Windows box and dispatches the batch there. It multiplexes the connection for speed and pulls results back over SSH. Later, Dropbox acted as a shared filesystem so files propagated on their own. The portfolio owner set up their side with a short OpenSSH and key walkthrough I wrote into the docs. From their seat it was one command. The Windows box ran the parallel batch and the outputs appeared on their Mac.

My role

• The full pipeline (templating, PJNZ editing, extraction, and post-processing) and the market-dynamics rule engine.
• The cross-machine architecture: OS-aware config, SSH dispatch with connection multiplexing, and the Dropbox shared-filesystem fallback.
• The compute box itself, built, tuned, and parallelized for multicore batch.
• The onboarding: an OpenSSH / key-auth / remote-access guide so the portfolio owner could launch and monitor parallel batches from their Mac.
• The owner-facing documentation site the whole workflow ships with.

Throughput

The owner still controlled the assumptions and launched runs from their Mac. One command dispatched country and scenario jobs across the Windows box’s cores, checked outputs, retried incomplete jobs, and returned the results. Full-portfolio runs, scenario branches, and sensitivity analysis became operationally feasible.