| |
Evaluation of Three Rapid Screening Assays for Detection of Antibodies to Hepatitis C Virus - false positives for HIV+ persons
|
| |
| |
Download the PDF here
The Journal of Infectious Diseases Sept 15 2011
Bryce D. Smith,1 Jan Drobeniuc,1 Amy Jewett,2 Bernard M. Branson,3 Richard S. Garfein,4 Eyasu Teshale,1 Saleem Kamili,1 and Cindy M. Weinbaum1
1Division of Viral Hepatitis; 2Oak Ridge Institute for Science and Education, Clinton, Tennessee; 3Division of HIV/AIDS Prevention, Centers for Disease Control and Prevention, Atlanta, Georgia; and 4Department of Medicine, Division of Global Public Health, University of California, San Diego, California
"The 3 anti-HCV rapid assays evaluated in this study were found to have varied performance characteristics. When compared with the CDC approach, the OraSure and Chembio assays had the highest sensitivity (99.3% and 97.8%, respectively) and MedMira had the lowest sensitivity (88.3%). All 3 assays had very high specificity (99.5%-99.8%). Although the comparison with the SA approach appeared to provide slightly lower sensitivity results for the rapid assays, these differences were not statistically significant.
Controlling for all other variables, specimens from HIV-positive persons were 11 times more likely to have a false result than those from HIV-negative persons using the Chembio test, and 4 times more likely using the MedMira test. The proportion of false negative results (8.5%) among HIV-positive persons across all 3 assays compares favorably to a European study that found a false negative proportion of 22.4% among HIV-positive persons when using a rapid tests
The sample of HIV-positive specimens (both anti-HCV positive and negative) was small (n = 43), and the associated estimates are imprecise.
False results (positive and negative) varied by assay and reference approaches (Table 1). Across the 2 reference approaches, Chembio had 0.2% false positives and false negatives ranged from 2.2% to 3.9%, MedMira had 0.2% false positives and the range of false negatives was from 11.7% to 13.4%, and OraSure had a false positive range from 0.4% to 0.6% and a false negative range from 0.7% to 2.2%. Demographic information regarding race and age were not predictive of false results for any of the rapid assays. Of the 43 HIV-positive specimens, 26 were anti-HCV positive by both reference approaches. False anti-HCV results were associated with HIV positivity for both approaches for the Chembio (aOR = 8.2-11.0, P < .01) and MedMira (aOR = 3.7-4.0, P < .01) assays (Table 2). However, false rapid anti-HCV results were not associated with HIV positivity for the OraSure assay (aOR = 4.7-7.2, P > .10).
The sensitivity and specificity of anti-HCV rapid assays need to be evaluated in field settings where they are most likely to be implemented. These evaluations should include an assessment of the readability of the assays, including operator agreement. Given that anti-HCV rapid assays will likely be implemented in HIV testing settings and ~3%-6% of HIV-positive persons have an impaired HCV antibody response [31, 32], anti-HCV rapid assays need to be evaluated using an HIV-positive specimen panel to establish an expected proportion of false-negative results. Also, demonstration projects should be conducted to see how these assays would be implemented in HIV testing settings, including the development and testing of HCV-specific prevention messages. Lastly, effective methods to link newly identified anti-HCV positive persons to medical evaluations and subsequent care and treatment should be evaluated.
During the course of drafting this manuscript, FDA approved OraQuick HCV Rapid Antibody Test for clinical use with venous and finger stick blood specimens. Given the likelihood that anti-HCV point of care assays will be available in the US market soon, public health and other direct service providers need to be prepared to make appropriate use of these disease prevention tools. CDC is beginning the process of developing guidance for persons who could be using these tests in the field when they become available and will also be updating testing algorithms and guidance related to their use for laboratory, epidemiologic, and surveillance purposes."
Abstract
Background. The Centers for Disease Control and Prevention (CDC) estimates that 3.2 million Americans are living with chronic hepatitis C virus (HCV) infection and 50%-70% are unaware of their status. Although therapies are available that can suppress or eliminate infection, identifying persons infected with HCV is challenging. Rapid tests could help identify many of these persons more expeditiously.
Methods. Three manufacturers, Chembio, OraSure, and MedMira, submitted HCV antibody (anti-HCV) rapid screening assays to the CDC for evaluation and comparison with established anti-HCV screening assays. The panel consisted of 1100 specimens drawn during 1997-1999 from persons reporting injection drug use. Sensitivity and specificity were assessed using 2 reference approaches, one based on the reactivity of samples in an anti-HCV screening assay and the other based on CDC HCV testing algorithm.
Results. The sensitivities of the Chembio, MedMira, and OraSure assays across the 2 approaches were 96.2%-98.0%, 86.8%-88.3%, and 97.8%-99.3%, respectively. The 3 assays had specificity of 99.5% or higher with no differences between assays. False rapid assay results were associated with human immunodeficiency virus positivity for both approaches for Chembio and MedMira.
Conclusions. Rapid anti-HCV tests can provide sensitive and specific anti-HCV results for high-risk patients.
BACKGROUND
The Centers for Disease Control and Prevention (CDC) estimates that 3.2 million Americans are living with chronic hepatitis C virus (HCV) infection [1]. CDC recommends routine HCV antibody (anti-HCV) testing for persons at risk of infection [2], yet estimates of the proportion of infected persons who are unaware range from 50% to 70% [3, 4]. Identifying persons infected with HCV is challenging for a variety of reasons, including difficulty in reaching high-risk populations (eg, persons who inject drugs), stigma of the disease and risk behaviors, provider reluctance to assess risk factors, and patient reticence to admit risk behaviors [5, 6].
The current standard of care for HCV treatment (pegylated interferon and ribavirin) has been found to have rates of sustained virologic response (SVR; undetectable virus >6 months after completion of treatment) ranging from 45% to 75%, depending on HCV genotype [7-10]. However, new therapies that may significantly increase these rates could be on the US market within the next year. SVR has been found to contribute to improvement in quality of life and survival among infected individuals [7, 11] and to potentially reduce further virus transmission [12]. Knowledge that one is infected with HCV may prompt entrance into medical care and treatment.
Immunoglobulin G antibody to hepatitis C virus (anti-HCV) is used to screen for HCV exposure, and 70.4%-86.6% of persons in whom anti-HCV is detected are subsequently determined to be chronically infected [1]. However, antibody tests cannot establish chronic infection; nucleic acid testing must be performed to make that determination [13]. Persons identified as anti-HCV positive can be counseled to alter behaviors that may lead to transmission of virus and exposure to other infections, decrease alcohol intake, and avoid certain medications that could damage the liver [2, 14, 15]. Persons who inject drugs and know that they are anti-HCV negative are less likely to share needles with an anti-HCV positive partner than are persons unaware of their infection status [16]. Burt and colleagues [17] found that persons who injected drugs and were aware of their HCV status were more likely to selectively share needles with partners of like HCV status. Knowledge of HCV status has also been associated with a short-term reduction in alcohol and noninjection drug use [18].
The current CDC-recommended testing algorithm relies on testing of blood specimens for anti-HCV by screening immunoassays followed by supplemental confirmatory assays if the screening assay (SA) is reactive but below a signal-to-cutoff (S/CO) ratio threshold [13]. This process can require long wait times for patients to receive their results. Rapid test technologies have been implemented successfully in human immunodeficiency virus (HIV) testing for almost 10 years [19]. Studies show that persons receive HIV test results more often when rapid tests are used (83%-100%) compared with conventional testing (43%-84%) [20-23]. Studies conducted in a sexually transmitted disease clinics and in high-risk settings found that 97% and 82%, respectively, of newly identified HIV-positive persons kept their first medical appointment after learning of their status from the results of HIV rapid tests [20, 24]. At the time this study took place, no rapid tests for anti-HCV detection were available in the United States or approved by the US Food and Drug Administration (FDA) for clinical use. The current paper reports on the laboratory evaluation of 3 anti-HCV rapid screening assays.
MATERIALS AND METHODS
Anti-HCV Rapid Assays
In spring of 2009, 3 anti-HCV rapid assay manufacturers responded to a Federal Register Notice announcing an opportunity to collaborate in the evaluation of rapid diagnostic assays for HIV and anti-HCV. These were OraQuick HCV Rapid Antibody Test (OraSure Technologies, Inc), Chembio DPP HCV test (Chembio Diagnostic Systems, Inc), and Multiplo Rapid HIV/HCV Antibody Test (MedMira Laboratories, Inc). Investigational use only kits for the 3 selected assays, all in vitro qualitative immune-chromatographic assays to detect anti-HCV, were provided by the manufacturers to the CDC for the evaluation. All rapid assays are single use, disposable chamber assays that provide visual results in <40 minutes. The assays were performed per the manufacturers' instruction manuals in the Assay Development and Diagnostic Reference Laboratory (ADDRL) of the Division of Viral Hepatitis at CDC using serum specimens.
The Chembio DPP HCV Test is a screening assay for the detection of anti-HCV in whole blood, serum, plasma, or oral fluid specimens. The Dual Path Platform chromatographic immunoassay uses recombinant multiepitope chimeric HCV antigen containing structural (Core) and nonstructural (NS3, NS4, and NS5) HCV proteins. The time required to perform the assay is between 15 and 30 minutes. Detailed information can be found at http://www.chembio.com/newtechnologies.html for the Chembio assay.
The Multiplo Rapid HIV/HCV (MedMira) Antibody Test is a flow-through device with a nitrocellulose membrane for qualitative detection of antibodies to HIV type 1 and 2 and HCV in serum, plasma, or whole blood specimens. Antigens derived from conserved regions of surface glycoproteins of the HIV virus and the NS3 and core proteins of HCV are printed in distinct areas of the nitrocellulose membrane. Captured anti-HIV and anti-HCV antibodies are visualized by subsequent reaction with protein A labeled colloidal gold. The time required to perform the assay is approximately 3 minutes; a reaction of the assay can be read immediately.
Detailed information can be found at http://www.medmira.com/default.asp?mn=1.25.49 for the MedMira assay.
The OraQuick HCV Rapid (OraSure) Antibody Test uses an indirect immunoassay method in a lateral flow device to detect antibodies to HCV in whole blood via finger stick, serum, or plasma via venipuncture, or oral fluid via swab. In this device, antigens from the core, NS3, and NS4 regions of the HCV genome are immobilized on a single test line on a nitrocellulose membrane; antibodies reactive with these antigens are visualized by protein A labeled colloidal gold. The time required to perform the assay is between 20 and 40 minutes. Detailed information can be found at http://www.orasure.com/products-infectious/products-infectious-oraquick-hcv.asp for the OraSure assay.
Specimen Panel
The evaluations reported in this study were conducted using a specimen panel that was originally collected as part of the Collaborative Injection Drug User Study (CIDUS) II from 1997 to 1999 [25]. All participants in the CIDUS II study were 18-30 years old and reported injection drug use in the previous 12 months. The panel contained 1100 serum specimens, which were frozen at the time of the study and remained in storage at -70° C until they were thawed for the current study.
Assay Performance
The manufacturers of the kits provided all the necessary documentation and instructions for performing the assays but were not asked to demonstrate for the laboratory staff. Interpretation of the results was similar for all 3 assays. An assay was interpreted as negative if a control line was present (regardless of intensity) with no corresponding test line. The appearance of a control line and a test line indicated a positive result. A missing or broken control line indicated an invalid result, regardless of test line presence. Each specimen was tested and read by 4 independent laboratory staff, henceforth designated as operators. A rapid assay result was classified as positive if at least 3 of the 4 operators independently interpreted the assay as positive.
Reference Assays
All specimens in the evaluation panel were tested for anti-HCV with the Ortho VITROS anti-HCV Chemiluminiscent Immunometric assay (VITROS ECi Immunodiagnostic System, Ortho-Clinical Diagnostics, Inc) in accordance with the manufacturer's instructions. A specimen was considered reactive when the S/CO ratio was ≥1.00, nonreactive when the S/CO was ≤0.90, and in a gray zone when the S/CO ratio was >0.90 but <1.00; such specimens were retested to define reactivity.
A third generation recombinant immunoblot assay (Chiron RIBA HCV 3.0 SIA; Novartis Vaccines & Diagnostics, Inc) was used to confirm antibody positivity for those specimens that were reactive but had a S/CO <8.0. RIBA-positive specimens are reported as confirmed anti-HCV positive, RIBA-negative specimens are reported as anti-HCV negative, and indeterminate results are reported as indeterminate.
Data Analysis
Performance of each anti-HCV rapid assay was evaluated in comparison to the anti-HCV screening assay (SA) and the CDC anti-HCV testing algorithm approaches [13]. First, we compared the results of the rapid assays to the anti-HCV screening assay results based on whether a sample was reactive or nonreactive. Second, we compared the results of the rapid assays to the results of the anti-HCV screening assay and RIBA based on CDC's testing algorithm approach, which requires supplemental confirmation of all screening reactive samples with a S/CO ratio <8.0 established for Ortho VITROS anti-HCV assay. A reactive specimen with a S/CO ≥ 8.0 was considered anti-HCV positive. If a reactive specimen with a S/CO <8.0 was positive by RIBA, it was considered anti-HCV positive. A nonreactive specimen with a S/CO <1.0 was considered anti-HCV negative. If a reactive specimen with a S/CO <8.0 was negative by RIBA, the specimen was considered anti-HCV negative.
Sensitivity, Specificity, and Interoperator/Intra-Assay Agreement.
Sensitivity and specificity were assessed by comparing results of each rapid assay to results of each of the reference approaches. Sensitivity was defined as the number of reference test positive specimens detected by the rapid assay divided by the reference approach positive results. Specificity was defined as the number of reference approach negative specimens identified by the rapid assay divided by the reference approach negative results. Confidence intervals (CIs) for sensitivity and specificity were calculated using the Wilson Score method [26]. Logistic regression was used to analyze whether false results (false positive or false negative) were predicted by demographic variables. All data analyses were performed using SPSS 17.0 (SPSS, Chicago). Interoperator and intra-assay kappa agreement was calculated using cross tabulations between each of the 4 operators within each of the rapid assays [27, 28]. Results of individual specimens across the 4 operators were used to assess the overall sensitivity and specificity for each assay.
The use of trade names and commercial sources is for identification only and does not imply endorsement by the CDC.
RESULTS
From the original 1100 specimens, 17 (1.5%) and 19 (1.7%) were excluded from the anti-HCV SA and CDC algorithm approaches, respectively, for the following reasons. For both approaches, 13 (1.2%) were excluded due to insufficient specimen quantity and 4 (0.4%) were excluded due to missing data, resulting in a sample size of 1083. Of these, 546 (50.4%) were anti-HCV positive and 537 (49.6%) were anti-HCV negative for the SA approach.
Of the 10 anti-HCV screening reactive specimens that had S/CO ratios between 1.0 and 8.0, 1 was positive by RIBA, 7 were negative, and 2 were indeterminate. These 2 (0.2%) indeterminate samples were excluded, resulting in a sample size of 1081 for comparison with the CDC testing algorithm approach. Of these, 537 (49.7%) were anti-HCV positive and 544 (50.3%) were anti-HCV negative.
From the original 1100 specimens, 1 specimen was in the gray zone (S/CO = 0.92). This specimen was retested per protocol with the same result. Therefore, it was included in all analyses as anti-HCV negative.
The sensitivity and specificity of the 3 rapid assays compared with the anti-HCV SA approach ranged from 86.8% to 97.8% and from 99.6% to 99.8%, respectively (Table 1). Sensitivity for OraSure was higher than Chembio (P = .012) and MedMira (P < .001). Sensitivity was also higher for Chembio than MedMira (P < .001). There were no significant differences between the specificity results of the 3 rapid assays (Figure 1).
Using the CDC approach, the sensitivity ranged from 88.3% to 99.3% and the specificity ranged from 99.5% to 99.8% (Table 1). Sensitivity for OraSure was higher than Chembio (P = .021) and MedMira (P < .001). Sensitivity was also higher for Chembio than MedMira (P < .001). There were no significant differences between the specificity results of the 3 rapid assays (Figure 1).
False results (positive and negative) varied by assay and reference approaches (Table 1). Across the 2 reference approaches, Chembio had 0.2% false positives and false negatives ranged from 2.2% to 3.9%, MedMira had 0.2% false positives and the range of false negatives was from 11.7% to 13.4%, and OraSure had a false positive range from 0.4% to 0.6% and a false negative range from 0.7% to 2.2%. Demographic information regarding race and age were not predictive of false results for any of the rapid assays. Of the 43 HIV-positive specimens, 26 were anti-HCV positive by both reference approaches. False anti-HCV results were associated with HIV positivity for both approaches for the Chembio (aOR = 8.2-11.0, P < .01) and MedMira (aOR = 3.7-4.0, P < .01) assays (Table 2).
However, false rapid anti-HCV results were not associated with HIV positivity for the OraSure assay (aOR = 4.7-7.2, P > .10). False results were negatively associated with female gender (aOR = 0.10; 95 CI, 0.01-0.39; P = .03) for the OraSure assay when compared with the anti-HCV screening approach. However, gender was not associated with false results using the CDC algorithm approach for OraSure, nor was it associated with Chembio or MedMira for either approach.
Operator agreement within each assay was calculated using kappa scores (Table 3). Chembio's assay Kappa scores ranged from 0.96 to 0.99, MedMira's ranged from 0.89 to 0.92, and OraSure's ranged from 0.98 to 0.99
DISCUSSION
The 3 anti-HCV rapid assays evaluated in this study were found to have varied performance characteristics. When compared with the CDC approach, the OraSure and Chembio assays had the highest sensitivity (99.3% and 97.8%, respectively) and MedMira had the lowest sensitivity (88.3%). All 3 assays had very high specificity (99.5%-99.8%). Although the comparison with the SA approach appeared to provide slightly lower sensitivity results for the rapid assays, these differences were not statistically significant.
False-positive and false-negative results were not associated with race or age variables in any of the assays. Controlling for all other variables, specimens from HIV-positive persons were 11 times more likely to have a false result than those from HIV-negative persons using the Chembio test, and 4 times more likely using the MedMira test. The proportion of false negative results (8.5%) among HIV-positive persons across all 3 assays compares favorably to a European study that found a false negative proportion of 22.4% among HIV-positive persons when using a rapid tests [21]. In previous studies using conventional anti-HCV screening assays, false-negative results among HIV-positive individuals have ranged from 3.8% to 5.5%, somewhat lower than the 8.5% proportion found across the 3 rapid tests evaluated in the current study [20-22]. However, the sample of HIV/HCV coinfected persons was so small (n = 26) that the associations had wide CIs indicating instability in the estimates. Female gender was negatively associated with false results when the OraSure rapid test was compared with the anti-HCV screening approach. Race, age, gender, and HIV status were the only demographic information collected, and we were unable to analyze the association between gender and false results any further.
The readability of rapid assays used in field settings is critical. One way to assess the readability of an assay is through the comparison of interoperator agreement with the use of Cohen's Kappa test [28]. A level of operator agreement above 0.75 is considered to be excellent [28, 29]; in this study operator agreement was over 0.85 for all 3 assays, which falls within the acceptable range [27]. The agreement scores for OraSure and Chembio were very high, implying that the same interpretation can be obtained regardless of who reads the assay. The lower level of agreement between operators performing the MedMira assay suggests more difficulty interpreting results from this assay. Field settings generally use persons who are less experienced than laboratory medical technologists, so readability of a rapid assay with a lower agreement score in those settings may be more challenging.
We used reactivity in conventional anti-HCV screening assays and the CDC algorithm as reference approaches to evaluate the performance characteristics of the assays. In general, the sensitivities of 3 rapid assays appeared to increase when the CDC algorithm was used in comparison to the anti-HCV SA-only reference approach, but these increases were not statistically significant (Figure 1).
Also, our study found that the addition of the CDC algorithm to the anti-HCV SA-only reference approach reduced the proportion of false-negative results by approximately 1.5%, which would be a significant practical reduction in a testing setting. Because only 10 specimens fell in the S/CO range requiring RIBA confirmation, we do not have sufficient data to comment on the practical value of the use of RIBA in rapid anti-HCV screening algorithms.
Anti-HCV rapid assays could be implemented in many settings. Hard-to-reach, high-risk populations, such as persons who inject drugs, are unlikely to be screened for anti-HCV using conventional testing, and when they are, they are less likely to return for their results [21, 24]. Anti-HCV rapid assays could be administered in syringe exchange programs (both through store fronts and mobile units), methadone maintenance treatment programs, and other programs that provide direct services to persons who inject drugs. Rapid assays are also often used in health fairs that are frequented by persons who may not have access to medical screenings due to being uninsured or underinsured. Rapid assays are regularly used by military personnel in the field [30], and anti-HCV rapid assays could be used as part of the ambulatory blood bank system. Anti-HCV rapid assays could be used in numerous medical settings from emergency rooms to primary care. Epidemiologically, given the number of anti-HCV outbreaks over the past few years that required testing large numbers of potentially exposed patients quickly, an anti-HCV rapid test would have been especially useful. For areas with limited laboratory resources that require high volume testing to be cost effective, a single use rapid anti-HCV test could be more cost effective than current conventional anti-HCV screening assays. Lastly, an anti-HCV rapid assay could be integrated into established HIV testing services provided in medical office settings and by organizations that are effective in reaching hard-to-reach risk populations (eg, persons who inject drugs) who have not been tested for anti-HCV.
Our study has some limitations. All rapid assay testing in this study was conducted by professional laboratorians in a reference laboratory; therefore, the results are not generalizable to field settings where the testing personnel may not have the same laboratory expertise. Because the participants who provided the specimens reported recent injection drug use, it is possible that some specimens represent acute rather than chronic infections. We did not document the number of invalid or indeterminate results of the rapid assays and therefore cannot compare the assays in those terms. The only covariates available for this analysis were race, age, gender, and HIV status. Therefore, limitations exist in analyzing other potential confounding variables. The sample of HIV-positive specimens (both anti-HCV positive and negative) was small (n = 43), and the associated estimates are imprecise.
There are several issues to be considered in future research. The sensitivity and specificity of anti-HCV rapid assays need to be evaluated in field settings where they are most likely to be implemented. These evaluations should include an assessment of the readability of the assays, including operator agreement.
Given that anti-HCV rapid assays will likely be implemented in HIV testing settings and ~3%-6% of HIV-positive persons have an impaired HCV antibody response [31, 32], anti-HCV rapid assays need to be evaluated using an HIV-positive specimen panel to establish an expected proportion of false-negative results. Also, demonstration projects should be conducted to see how these assays would be implemented in HIV testing settings, including the development and testing of HCV-specific prevention messages. Lastly, effective methods to link newly identified anti-HCV positive persons to medical evaluations and subsequent care and treatment should be evaluated.
During the course of drafting this manuscript, FDA approved OraQuick HCV Rapid Antibody Test for clinical use with venous and finger stick blood specimens. Given the likelihood that anti-HCV point of care assays will be available in the US market soon, public health and other direct service providers need to be prepared to make appropriate use of these disease prevention tools. CDC is beginning the process of developing guidance for persons who could be using these tests in the field when they become available and will also be updating testing algorithms and guidance related to their use for laboratory, epidemiologic, and surveillance purposes.
|
|
| |
| |
|
|
|
