A new story by Jon Cohen - a longtime follower of the HIV vaccine field and author of an excellent book on the subject - has just appeared on the Science Insider blog at Science Magazine, entitled "Unrevealed Analysis Weakens Claim of AIDS Vaccine 'Success.'" It suggests that, as some observers (including TAG) had feared, the very limited data from the Thai HIV vaccine trial that was announced on September 24th does not represent the whole story. Specifically, it appears that the statistically significant result that was announced was derived from what's called an "intent-to-treat" (ITT) analysis; this type of analysis includes data from all participants, whether or not they completed the full protocol-mandated series of immunizations. ITT is a standard and important way of analyzing data that typically avoids over-estimating the efficacy of an intervention. The less conservative type of data analysis is usually called a "per protocol" (PP) analysis; in this case only study participants who have completed the protocol as specified, including all immunizations, are included. Based on Cohen's report, it appears that the PP analysis of the Thai HIV vaccine trial also showed fewer HIV infections in the vaccine group, but the difference with placebo was not statistically significant. This seems odd, as PP analyses involving a successful intervention typically show a greater, not lesser, effect. Given the potential importance of the trial results, it is unconscionable that this discrepancy was not discussed when the initial announcement was made.
Cohen closes his piece by citing an excellent and prescient "anticipating results" document about the Thai trial that was released by AVAC just a couple of weeks ago. Under the heading "Advocates' Guide to Statistical Terms" it says:
"Other terms that might be used in discussion of initial results are per-protocol (PP) and intent-to-treat (ITT) analyses of results. PP results only include infections that occurred in those participants who received the full course of vaccinations . ITT results count every infection after trial enrollment, regardless of whether a participant completed the full regimen. ITT is considered a gold standard because it considers every randomized participant. Both approaches provide important information. The safest route is to report both PP and ITT and to analyze any difference. Advocates’ take-home message: no matter what the headlines say, a single number is not the full result."
Comments