Discussion

Chapter 15: Methodology of the RPRP

Obtaining Research Proposal Abstracts to Identify Studies of Interest

The RPRP involved the collection and review of documents related to recently funded, federally supported human radiation research. This included research supported or performed by the DOD, DOE, DHHS, NASA, and VA. Each agency funds intramural research conducted by agency staff members at various field sites and extramural research conducted outside the agency by contractors or grantees. The Advisory Committee requested and received abstracts or similar descriptions from these agencies for all intramural and extramural studies newly approved and funded between fiscal years 1990 and 1993 (that is, "new starts" in those fiscal years) that fell within two general categories: (1) studies involving the exposure of human subjects to research applications of ionizing radiation (or follow-up studies of such exposures); and (2) nonradiation research involving human subjects. These abstracts represented the "universe" of federally funded contemporary human research from which studies were then selected for review.

Selection of Studies Involving Ionizing Radiation

For purposes of the RPRP, a radiation experiment was defined as any federally funded or performed investigation where the exposure of human subjects to ionizing radiation is an element of the research design. In addition, we included follow-up or epidemiological studies of exposures of humans to ionizing radiation.[2] Any procedures involving radiation incidental to a subject's enrollment in a study (for example, a diagnostic x ray in research involving chemotherapy) were not considered experimental for purposes of the review.

To select studies to review from the many abstracts we received, nuclear medicine experts on the Advisory Committee staff first reviewed and stratified the

Definitions of Biomedical Categories

Tracer/biodistribution studies: Studies involving the measurement of administered radioactive chemicals within the body (in vivo) using radiation detectors directed at the body from the outside, or in body fluids such as blood and urine in the test tube (in vitro).

Biodistribution studies are distinct from tracer studies in that their object of study is radioactive contaminants themselves, in order to understand their distribution and metabolism within the body. By contrast, tracer studies employ radio-labeled variants of ordinary biological chemicals to provide information on natural metabolic processes involving those chemicals. Tracer/biodistribution studies differ from research involving external sources of radiation (such as x rays), because tracer/biodistribution studies involve the administration of radioactive chemicals into a subject's body.[a]

Studies involving potential therapeutics: Studies that involve novel or nonvalidated uses of radiation for therapeutic purposes on sick individuals.

Studies involving potential diagnostics: Studies that involve experimental uses of radiological or nuclear medicine diagnostics (for imaging) that are experimental in that their efficacy has not been established. This includes research involving different types of radiation exposure as well as applications of established radiation imaging techniques (such as diagnostic x-rays or CAT scans), for new diagnostic purposes.

Epidemiological/observational: Studies of health effects in people who have experienced exposures to ionizing radiation. This research does not employ radiation, but attempts to understand health effects on humans exposed to ionizing radiation using follow-up studies, medical monitoring, and retrospective records reviews.

^{a .} Henry N. Wagner, Jr. and Linda E. Ketchum, Living with Radiation--The Risk, The Promise (Baltimore: The Johns Hopkins University Press, 1989), 77-78.

study abstracts obtained according to the biomedical categories that the Advisory Committee established for radiation research: tracer/biodistribution studies, studies involving potential therapeutics, studies involving potential diagnostics, and epidemiological/observational studies. These categories were intended to parallel roughly the various types of past radiation experiments identified by the Advisory Committee. We recognized that placing radiation experiments into discrete categories was a difficult task. The purpose of the categorization, however, was to sample proposals across the range of radiation research conducted on human subjects rather than to identify specific research as falling into strict categories.[3] Definitions of the biomedical categories used in the Research Proposal Review Project are listed in the accompanying box.

We then selected studies[4] to ensure that each funding agency and each biomedical category of human radiation research (tracer/biodistribution, therapeutic, diagnostic, and epidemiological/observational) was adequately represented in the random sample[5] of studies to be reviewed. Eighty-four radiation studies were selected from proposal abstracts provided by the agencies. These included 31 extramural proposals representing nonfederal research institutions,[6] primarily universities, and 53 intramural proposals[7] from the DHHS, DOE, DOD, NASA, and VA.

Selection of a Comparison Group of Nonradiation Studies

For purposes of selecting a comparison sample of nonradiation studies, the 84 radiation studies were reclassified according to the following categories: (1) federal funding agency, (2) extramural/intramural, and (3) cardiology/cancer/neither cardiology nor cancer.[8] Approximately half as many studies (41) were selected for the comparison sample and distributed in each of the three categories in comparable proportion to the distribution of radiation studies. We drew our sample of nonradiation studies from the same grantee institutions that were included in the radiation sample.

Data Sources

In total, the Advisory Committee identified for review 125 research proposals involving human subjects (84 involving ionizing radiation, and 41 not involving radiation) that were approved and funded by DHHS, DOE, DOD, NASA, or VA between fiscal year 1990 and fiscal year 1993.[9]modified sampling scheme that was conceived in order to pare down the number of studies reviewed to 125. Long-term epidemiological studies that were initiated before fiscal year 1990 and continued through this period were included in the review in cases where the methodology and/or consent procedures for such studies were found to have been updated in recent proposal renewals.

The Advisory Committee requested the following documents[10] for each of the 125 studies it identified for review:

1. Grant proposal submitted by investigator to federal agency;[11]
2. Institutional review board (IRB) application;[12]
3. Original consent form submitted to the IRB;
4. Consent form, as approved by the IRB;[13]
5. The IRB's final disposition letter;[14]
6. Documentation concerning any changes to the research design, methods, or consent form approved by the IRB after the IRB's initial approval of the study;[15]
7. If relevant, the application submitted to and the official letter of approval from the radioactive drug research committee (RDRC);[16]
8. If relevant, the application submitted to and the official letter of approval from any institutional human use committee other than the IRB or RDRC.

All of the relevant federal agencies and the 47 extramural grantee institutions to which the Advisory Committee submitted a request complied with this request. The willingness of institutions to voluntarily make available documents for review indicates their commitment to research ethics, which the Committee very much appreciates. The openness shown by the biomedical research community is important evidence of improvement in the ethics of human subject research over the fifty-year history reviewed by the Committee.

Review Process

Three basic elements were considered in developing a system to review the research materials supplied to the Advisory Committee: the procedures for obtaining informed consent, the balance of risks to potential benefits for the subject, and the selection and recruitment of subjects. An evaluation form was developed by a subcommittee of Committee members and staff to assist reviewers in organizing their assessments of the research documents (grant proposal, IRB application, RDRC application) and the consent form(s).

The documents for each proposal were reviewed by a team of two individuals, with at least one member of the Advisory Committee placed on each team, so that documents from every proposal were reviewed by at least one member of the Committee. Review teams consisted of either two Advisory Committee members or one Committee member and one staff member. One member of each team had expertise in research ethics, while the other had expertise in radiation science, radiation medicine, another branch of medicine, or epidemiology. Reviewers were never assigned documents from their own institution; they were also required to recuse themselves if they were well acquainted with the principal investigator of a proposal.

Documents were first reviewed independently by each reviewer and then by the reviewers together as a team. At the end of this process, each team completed a single evaluation form representing a joint assessment.

Limitations

The Research Proposal Review Project was designed to provide insight on an exploratory basis into the current practice of human subjects research conducted or supported by the U.S. government. The project was not undertaken with the expectation that our results would be generalizable to all research involving human subjects or to research sponsored by nongovernmental agencies. Of necessity, we reviewed documents from only a small sample of proposals for human subjects research funded in fiscal years 1990 through 1993. In a given year, DHHS supported 16,972 projects and subprojects involving human subjects research.[17] At the same time, however, our sample includes examples of both radiation and nonradiation research funded or sponsored by five different federal agencies across a variety of biomedical categories and medical specialties. Moreover, the proposals whose documents we received and reviewed were selected at random; there was no attempt to identify proposals that appeared from the outset to pose human subjects problems or high levels of risk and therefore no reason to suspect that the sample chosen was biased in favor of more problematic or higher-risk studies.

Within the Committee, reviewers rarely disagreed in their reviews. Although these reviews are based on interpretation and opinion in the context of Committee deliberation, it should be noted that so, too, are the evaluations of IRBs, on which the protection of human subjects now rests.

Perhaps the most significant limitation of this project is that the evaluation of each proposal was based only on the documents that were provided by the federal funding agency and grantee institution. The documentation we received was not always complete. Moreover, IRBs may have had access to sources of information not available to the Committee. Some IRBs invite principal investigators to make presentations at IRB meetings; others encourage reviewers to discuss proposals with principal investigators before IRB meetings. Thus, in some cases, IRBs may have reviewed the proposals evaluated by the Committee with a fuller and more accurate understanding of the project than was available to the Committee. It is therefore possible that some of the research projects that raised concerns for us based on the documents we reviewed, would, with the provision of additional information, be deemed unproblematic from a human subjects perspective. Conversely, it is possible that some of the research projects whose documents raised no concerns may nevertheless have inadequacies affecting the rights and interests of human subjects that we could not detect.

From the outset, the Committee neither desired nor thought it possible (because of our limited tenure and resources) to make judgments about the extent to which these 125 research projects were in fact being conducted in an ethically acceptable manner. This would have required a careful evaluation of far more than the documents that we received.

Neither IRB interactions with principal investigators nor documents speak to what actually happens between investigators, their assistants, and potential subjects. What investigators in fact say to potential subjects, the tone with which they say it, and the conditions under which the interaction takes place are pieces of information that were unavailable to the Advisory Committee in its review of the documents from contemporary human research proposals, just as they are generally unavailable to IRBs.

The Advisory Committee's review of research proposal documents thus was not intended to evaluate the performance of particular IRBs or the ethics of the conduct of particular investigators or specific insitutions. Rather, by examining documents from a wide variety of research projects funded by many agencies of the federal government, we hoped to offer insight into the general state of the protection of the rights and interests of human subjects.