Abstract
Free full text
Reliability of Nucleic Acid Amplification Methods for Detection of Chlamydia trachomatis in Urine: Results of the First International Collaborative Quality Control Study among 96 Laboratories
Abstract
The first European Quality Control Concerted Action study was organized to assess the ability of laboratories to detect Chlamydia trachomatis in a panel of urine samples by nucleic acid amplification tests (NATs). The panel consisted of lyophilized urine samples, including three negative, two strongly positive, and five weakly positive samples. Ninety-six laboratories in 22 countries participated with a total of 102 data sets. Of 204 strongly positive samples 199 (97.5%) were correctly reported, and of 506 weakly positive samples 466 (92.1%) were correctly reported. In 74 (72.5%) data sets correct results were reported on all samples, and 17 data sets (16.7%) showed either one false-negative or one false-positive result. In another 11 data sets, two or more incorrect results were reported, and two data sets reported a false-positive result on one negative sample. The Roche COBAS Amplicor test was performed in 44 (43%) data sets, the Abbott LCx assay was performed in 31 (30%) data sets, the Roche Amplicor manual assay was performed in 9 (9%) data sets, an in-house PCR was performed in 9 (9%) data sets, the Becton Dickinson ProbeTec ET assay was performed in 5 (4.9%) data sets, and the GenProbe TMA assay was performed in 4 (3.9%) data sets. The results of the Roche Amplicor manual (95.6% correct), COBAS Amplicor (97.0%), and Abbott LCx (94.8%) tests were comparable (P = 0.48). The results with the in-house PCR, BD ProbeTec ET, and GenProbe TMA tests were reported correctly in 88.6, 98, and 92.5% of the tests, respectively. Freeze-drying of clinical urine specimens proved to be a successful method for generating standardized, stable, and easy-to-transport samples for the detection of C. trachomatis by using NATs. Although the results, especially the specificity, for this proficiency panel were better than most quality control studies, sensitivity problems occurred frequently, underlining the need for good laboratory practice and reference reagents to monitor the performance of these assays.
A multiplicity of nucleic acid amplification test (NAT) procedures are now available for the diagnosis of Chlamydia trachomatis infections. C. trachomatis amplification tests can be performed on endocervical or urethral swab specimens and urines (6). In addition to in-house PCR tests, commercial NAT assays are available, such as Abbott LCx, Becton Dickinson ProbeTec ET, GenProbe TMA, and Roche Amplicor (COBAS and manual) PCRs. All tests aim to be rapid, sensitive, specific, and easy to perform. However, previous studies have shown that application of NATs may be unreliable because of cross-contamination, inappropriate treatment of the clinical samples leading to the loss of target DNA, or the presence of inhibitors in the sample (1, 3, 8, 11).
Members of the European Union Concerted Action on Quality Control (EU-QCCA) of Nucleic Acid Amplification in Diagnostic Virology and members of the Study Group on Molecular Diagnostics of the European Society for Clinical Microbiology and Infectious Diseases formed a working party to establish a quality assessment program for the evaluation of currently used NATs for the detection of C. trachomatis. The aim of the study was to assess the quality performance of laboratories for detection of C. trachomatis in a panel of samples that could be tested with all existing commercial and in-house test systems and that would resemble clinical samples. Since commercial systems require their specific transport medium for swab specimens, urine samples were chosen as the most suitable specimen for the EU-QCCA panel. All commercial systems include a protocol for testing C. trachomatis in urine samples.
In order to avoid the high costs associated with shipment of infectious goods on dry ice, lyophilized urine specimens were prepared.
MATERIALS AND METHODS
Collection of urine samples.
Negative urine samples were obtained by collecting up to 50 ml of early morning first-catch urine samples over a period of 3 months from two healthy females (urine samples B and C) and one healthy male volunteer (urine sample D). Urine samples from each volunteer were pooled. The volunteers had no history of infection with C. trachomatis or N. gonorrhoeae, and the pools were examined for these microorganisms by culture and NAT. These tests were all negative. A mixture of urine types was formed by mixing equal amounts of the pooled urine samples (mix A).
First-catch urine samples from patients visiting the clinics for sexually transmitted diseases of the Erasmus Medical Center in Rotterdam, The Netherlands, and of the National Institute of Public Health in Oslo, Norway, were collected and stored at 4°C. Samples were tested for the presence C. trachomatis DNA and Neisseria gonorrhoeae DNA with the Roche COBAS Amplicor PCR and the Abbott LCx tests according to the manufacturer's procedures. C. trachomatis-positive urine samples were diluted 10−1 and 10−2 and retested the same day. Urine samples positive at the 10−2 dilution were considered strongly positive. After estimation of the volumes, all urine samples were frozen at −70°C, with two extra aliquots of 1 ml for further pilot testing. It appeared that some of the urine samples from Rotterdam, but not those from Oslo, contained N. gonorrhoeae DNA in addition to C. trachomatis DNA.
All positive urine samples were thawed, and two different pools were prepared. The 28 urine samples collected in Rotterdam were used to prepare a C. trachomatis-positive-N. gonorrhoeae-positive urine pool (pool I). The 22 urine samples collected in Oslo were used to prepare a C. trachomatis-positive-N. gonorrhoeae-negative urine pool (pool II). Tenfold dilutions of pools I and II were prepared in the C. trachomatis-N. gonorrhoeae-negative urine mix, and 1.7-ml aliquots were freeze-dried by using a Leybold-Hereaus GT 2 freeze-drier and housed in a temperature-controlled environment (17°C).
Quality control assurance and production of the panel.
The lowest dose 50% endpoint (LD50) was defined as the lowest concentration of target that gave a 50% chance of a positive test result. The 10-fold dilutions of urine pools I and II were tested six times by each of four different commercial test systems: Abbott LCx, BD ProbeTec ET, COBAS Amplicor PCR, and GenProbe TMA. Three ampoule contents were pooled to provide the 4-ml volume required by the BD ProbeTec ET test. The LD50 value of a positive response in each test was calculated by the method of Kärber (7). The weakly positive samples in the final panel contained 10 times the concentration of the LD50.
The final panel consisted of 10 1.7-ml portions of lyophilized urine samples: 3 negative samples, 2 strongly positive samples, and 5 weakly positive samples. Three different urine samples from healthy volunteers were used for dilution of the positive pool I and II to generate a heterogeneous panel (Table (Table1).1). The samples were coded and randomized.
TABLE 1.
Composition of the panel and overview of correct results per sample
| Sample no. | Composition of the sample
| Expected result | No. of samples tested | No. of correct results (%) | |
|---|---|---|---|---|---|
| Background | Positive urine addedb | ||||
| 2 | Urine B | Negative | 101 | 100 (99.0) | |
| 5 | Urine C | Negative | 102 | 102 (100) | |
| 9 | Urine D | Negative | 101 | 100 (99.0) | |
| 1 | Mix Aa | I | Strongly positive | 102 | 100 (98.0) |
| 4 | Mix A | II | Strongly positive | 102 | 99 (97.1) |
| 3 | Urine B | I | Weakly positive | 101 | 95 (94.1) |
| 6 | Urine C | I | Weakly positive | 102 | 92 (90.2) |
| 7 | Urine C | I | Weakly positive | 101 | 93 (92.1) |
| 8 | Mix A | I | Weakly positive | 102 | 97 (95.1) |
| 10 | Mix A | II | Weakly positive | 100 | 89 (89.0) |
Quality assurance for the quality control panel was performed by four reference laboratories, and tests were performed with Abbott LCx, BD ProbeTec ET, COBAS Amplicor PCR, and GenProbe TMA. Laboratories performing the BD ProbeTec assay received three vials of each specimen; these had to be pooled after restoring the volume to 1.7 ml to obtain the requested specimen volume of 4 ml.
Organization of the study.
The proficiency panel was distributed from the production laboratory in Oslo, Norway, by surface mail at ambient temperature to the participants. The packet also included an information sheet with instructions as to how to open the vials and restore the volume to 1.7 ml. Reporting of arrival, results, questionnaire, coding, and confidentiality was organized as described previously (15). The Neutral Office in Manchester, United Kingdom, received all results. Data were analyzed anonymously in Leeuwarden and Rotterdam, The Netherlands, by using the Statistical Package for the Social Science (SPSS version 10). Statistical evaluation of the collected data was performed by using the Fisher exact test; probability tests were two-tailed.
RESULTS
Validation of freeze-drying of C. trachomatis-infected urine specimens.
A set of 1-ml portions of 10-fold dilutions of a pool of strongly positive urine samples was lyophilized, randomized, and sent to other laboratories to test for the presence of C. trachomatis DNA or RNA with COBAS Amplicor PCR (five laboratories), Abbott LCx (four laboratories), GeneProbe TMA (two laboratories), and BD ProbeTec ET (one laboratory); one set of samples was tested with the Vidas Probe assay (Biomerieux, Boston, Mass.). Reproducible results were received from duplicate series tested with the same method at the same or at different laboratories. The sensitivity of the COBAS Amplicor, LCx, Vidas Probe, and BD ProbeTec ET assays were similar: dilutions 10−3 to 10−4 tested positive. The laboratories performing the GenProbe TMA assay reported a lower sensitivity (10−2 to 10−3); this probably resulted from the use of a 1- ml specimen, whereas the provider prescribes 1.5-ml of urine to be tested. Samples from the same batch were stored at room temperature and retested 5 months and 1 year later with COBAS Amplicor and LCx. Comparable results were obtained that confirmed that the lyophilized samples were stable for at least 12 months.
Distribution and response.
The panel was sent to 105 laboratories in 22 countries. The median transport time was 3 days, with a range of 1 to 36 days. Results were received from 96 participants (91.4%). Ninety laboratories returned one data set, and six participants performed two tests and returned two data sets, which yielded a total of 102 evaluable data sets.
Performance of the laboratories.
In total, 304 negative samples were tested. The composition of the panel and the results on the individual samples are presented in Table Table1.1. Only two laboratories reported one false-positive result; these were both experienced laboratories examining more than 2,000 samples per year. Of the 204 strongly positive samples, 199 (97.5%) were reported correctly; of 506 examinations of weakly positive samples, 466 (92.1%) were reported correctly.
In 74 (72.5%) data sets all samples were reported correctly. In 17 (16.7%) data sets one incorrect result was reported, in 16 of these 1 weakly positive sample was reported as a false-negative result. In six (5.9%) data sets two weakly positive samples were reported incorrectly; one (1%) data set missed three weakly positive samples. In three (2.9%) data sets two or three weakly positive samples plus one or two strongly positive samples were reported incorrectly, and one (1%) data set incorrectly reported one weakly positive sample, one strongly positive sample, and one negative sample.
Relation between score, type of assay, and procedures.
Table Table22 presents an overview of the scores classified by the type of assay used. The BD ProbeTec ET has the highest rate (98.0%) of correct results; however, only five participants used this new assay. Equal percentages of correct results were reported with the two Roche tests—Amplicor manual (95.6%) and COBAS Amplicor (97.0%)—and with the Abbott LCx test (94.8%) (P = 0.48). The GenProbe TMA test scored lower (92.5%) because only 85% of the results of the weakly positive samples were correctly reported. This result was in accordance with the expected performance of the test as predicted by the random chance calculation of the results from pilot tests and LD50 calculations (data not shown). The in-house PCR tests had the lowest score (88.6% correct results).
TABLE 2.
Comparison of results and type of assay
| Method | No. of data sets | No. of samples with correct results/no. of samples tested (%)
| |||
|---|---|---|---|---|---|
| Negative samples | Strongly positive samples | Weakly positive samples | All samples | ||
| COBAS Amplicor | 44 | 131/132 (99.2) | 87/88 (98.9) | 207/218 (95.0) | 425/438 (97.0) |
| Abbott LCx | 31 | 92/93 (98.9) | 59/62 (95.2) | 142/154 (92.2) | 293/309 (94.8) |
| Amplicor manual | 9 | 27/27 (100.0) | 18/18 (100.0) | 41/45 (91.1) | 86/90 (95.6) |
| PCR (in house) | 9 | 25/25 (100.0) | 17/18 (94.4) | 36/45 (80.0) | 78/88 (88.6) |
| BD Probe Tec ET | 5 | 15/15 (100.0) | 10/10 (100.0) | 23/24 (95.8) | 48/49 (98.0) |
| GenProbe TMA | 4 | 12/12 (100.0) | 8/8 (100.0) | 17/20 (85.0) | 37/40 (92.5) |
| Total | 102 | 302/304 (99.3) | 199/204 (97.5) | 466/506 (92.1) | 967/1014 (95.4) |
Application of an internal control to monitor inhibition of the reaction in each sample was reported by 42 data sets. Three participants, two using COBAS-Amplicor and one using in-house PCR, reported one or more samples with inhibition. When these samples were tested again, the inhibition problem was solved in the COBAS Amplicor assay but not in the in-house PCR assay.
The majority of participants (91.7%) were laboratories that carry out NAT for clinical diagnostics, 76 (79.2%) were hospital or public health laboratories, and 12 (12.5%) were commercial (private) laboratories. Five (5.2%) laboratories mentioned that they were manufacturers of NAT assays. Three participants did not answer this question. Commercial (private) laboratories tended to score lower than diagnostic laboratories and manufacturers (92.8, 95.7, and 100%, respectively; P = 0.08).
The number of NATs performed for detection of C. trachomatis on an annual basis varied from one laboratory performing 10 to 100 tests up to 21 laboratories examining more than 10,000 samples. More than 80% of the respondents were experienced users of NATs (>1,000 tests per year). However, there was no correlation between the number of tests performed annually and the quality of the results.
Four participants performed NAT in a single laboratory room; data sets from these labs showed a higher number of incorrect results, i.e., 6 of 39 (15.4%) versus 41 of 945 (4.3%) for those using more than one room (P = 0.008).
DISCUSSION
This is the first international quality control study for the detection of C. trachomatis based on molecular diagnostic methods and lyophilized urine samples. The pilot studies and the results from the final panel show that freeze-drying of clinical urine samples for NATs is a successful method for obtaining stable, standardized samples that are easy to transport.
The panel was composed to resemble clinical samples, with strongly and weakly positive urine samples from women and men. The weakly positive samples contained approximately the same number of C. trachomatis organisms believed to be present in asymptomatic infected humans (R. P. Verkooyen, unpublished data). Since more than 60% of all female C. trachomatis infections are asymptomatic, the working party thought it important that participants should be able to detect these low numbers of bacteria. Also, population-based screening for C. trachomatis is expected to detect many asymptomatic infections (2, 5, 13).
Since double infection with C. trachomatis and N. gonorrhoeae has been observed frequently (9, 14), we investigated whether the tests were able to cope with samples containing both C. trachomatis and N. gonorrhoeae. The results showed that the presence of these dual targets did not adversely affect test sensitivity for C. trachomatis (Table (Table11).
The scores described here were very good; 89.2% of data sets scored 90% or higher. However, five (5%) data sets scored ≤70%. The primary purpose of the proficiency testing was to estimate whether a laboratory is capable of providing reliable results and not to test the performance of different commercial assays. However, as the number of participants applying certain commercial assays was high, the results also yielded useful information on the assays used. It showed that there was no significant difference between the Roche PCR and the Abbott LCx tests, although the mean score from the LCx test was 2.2% lower. This difference was mainly due to the errors with the weakly positive samples. Results from the pilot tests suggest that COBAS Amplicor and LCx tests have the same chance of correctly reporting weakly positive samples. The fact that the results from participants in study performing the LCx assay were slightly less successful than with the COBAS assay may be due to inhibition of the amplification reaction sample (1, 3, 8, 11). From previous investigations we learned that the accuracy of molecular tests with urine samples may be compromised if urine residues remain after centrifugation; such a problem would not be detected in the LCx system since this system lacks an internal inhibition control. On the other hand, a weakly positive sample may also be missed when the pellet is lost after centrifugation. In that case, there is no inhibition and a false-negative test result is reported.
Previous quality control studies revealed that the number of manually performed pretreatment steps and the difficulty of processing samples before samples enter an automatic system are the major causes of false-positive and false-negative results (4, 10, 12, 17). The procedures used for the detection of C. trachomatis in urine samples have relatively simple pretreatment protocols, which may explain the differences in results between this quality control study and the other studies. The present study shows a very low number of false-positive results, and only two negative samples were reported as positive in two different data sets. These low rates were also reported in recent quality control studies and may reflect the greater expertise of the participating laboratories compared to several years ago and the increasing use of commercial kits in general (15, 16).
In the present study, weakly positive samples were missed approximately three times more frequently than the strongly positive samples, indicating a problem in test sensitivity. The question always arises as to whether in-house PCR amplification is suitable for routine clinical microbiological diagnostics. The group using in-house PCR tests had the lowest mean score (88.6% correct results), which shows that results obtained by in-house amplification procedures should be interpreted with care.
For each of the commercial methods the manufacturer prescribes the volume of sample to be used. Participants were asked to calculate the sample volume equivalent used in the amplification reaction and to report whether they adhered to the manufacturer's protocol. These answers were not always in concordance; some participants calculated a different volume than the one prescribed but answered that they used the manufacturer's protocol. In some cases, only 50% of the prescribed volumes for reagents and sample were used. Use of a smaller or a larger volume may compromise the sensitivity of the method or increase the inhibition rate. Ten participants apparently used lower equivalents of urine in the test than are prescribed by the manufacturer, resulting in a significant decrease in sensitivity, compared to the participants who used the prescribed sample equivalents (P = 0.036). Only three participants violated the test protocol by increasing the equivalent sample volume used in the test. No significant difference was observed in these cases.
Participation from different European countries was low in comparison to the number of laboratories performing NATs for the detection of C. trachomatis. Some laboratories may underestimate the importance of external quality assessment as part of quality management systems. At the time of the present study, only 31.2% of participant laboratories were accredited. We expect that the need for QC panels will increase if in more countries accreditation becomes obligatory. The outcome of the present study underlines the need for reference reagents and standardized operating procedures, which would enable experienced technicians to perform reliable quality control assessment of nucleic acid amplification methods. The first steps are taken toward a standardized procedure for quality control of nucleic acid amplification assays for the detection of C. trachomatis, which should be available for every laboratory performing NATs in routine clinical practice.
Acknowledgments
We thank Abbott Diagnostics, Roche Diagnostics, Biomerieux, and Becton Dickinson for support. The EU-QCCA Program was supported by a grant from the EU Biomed 2 Program.
We are grateful to J. Mouton (Canisius Hospital, Nijmegen, The Netherlands), P. Oostvogel (Westeinde Hospital, The Hague, The Netherlands), A. van den Brule (Free University of Amsterdam, Amsterdam, The Netherlands), J. Thjie (Academic Hospital Maastricht, Maastricht, The Netherlands), C. Roemer (Medical Microbiological Laboratory, Utrecht, The Netherlands), L. Sabbe (Public Health Laboratory Zeeland, Goes, The Netherlands), and M. de Vries (LCPL, Leiden, The Netherlands) for performance of pilot experiments and reference testing. We also thank Margaret Barringer for all administrative work and all participating laboratories.
REFERENCES
Articles from Journal of Clinical Microbiology are provided here courtesy of American Society for Microbiology (ASM)
Full text links
Read article at publisher's site: https://doi.org/10.1128/jcm.41.7.3013-3016.2003
Read article for free, from open access legal sources, via Unpaywall:
https://europepmc.org/articles/pmc165323?pdf=render
Citations & impact
Impact metrics
Citations of article over time
Article citations
Current and Future Perspectives on Isothermal Nucleic Acid Amplification Technologies for Diagnosing Infections.
Infect Drug Resist, 13:455-483, 12 Feb 2020
Cited by: 68 articles | PMID: 32104017 | PMCID: PMC7024801
Review
Free full text in Europe PMCAdvances in Directly Amplifying Nucleic Acids from Complex Samples.
Biosensors (Basel), 9(4):E117, 30 Sep 2019
Cited by: 21 articles | PMID: 31574959 | PMCID: PMC6955841
Review
Free full text in Europe PMCEvaluation of Copan FLOQSwab for the molecular detection of Chlamydia trachomatis by Abbott RealTime CT PCR.
Acta Clin Belg, 70(6):398-402, 24 Jun 2015
Cited by: 3 articles | PMID: 26107497
Application of HTB-SiHa cells transfected with a recombinant plasmid for external quality assessment of Chlamydia trachomatis PCR.
Ann Lab Med, 34(5):360-366, 21 Aug 2014
Cited by: 0 articles | PMID: 25187888 | PMCID: PMC4151004
Advances in sampling and screening for chlamydia.
Future Microbiol, 8(3):367-386, 01 Mar 2013
Cited by: 14 articles | PMID: 23464373
Review
Go to all (25) article citations
Similar Articles
To arrive at the top five similar articles we use a word-weighted algorithm to compare words from the Title and Abstract of each citation.
External quality assessment program for Chlamydia trachomatis diagnostic testing by nucleic acid amplification assays.
J Clin Microbiol, 40(8):2893-2896, 01 Aug 2002
Cited by: 6 articles | PMID: 12149347 | PMCID: PMC120684
Comparison between the LCx Probe system and the COBAS AMPLICOR system for detection of Chlamydia trachomatis and Neisseria gonorrhoeae infections in patients attending a clinic for treatment of sexually transmitted diseases in Amsterdam, The Netherlands.
J Clin Microbiol, 39(3):829-835, 01 Mar 2001
Cited by: 48 articles | PMID: 11230391 | PMCID: PMC87837
Comparison of manual Amplicor PCR, Cobas Amplicor PCR, and LCx assays for detection of Chlamydia trachomatis infection in women by using urine specimens.
J Clin Microbiol, 35(2):402-405, 01 Feb 1997
Cited by: 51 articles | PMID: 9003605 | PMCID: PMC229589
Detection of Chlamydia trachomatis by nucleic acid amplification testing: our evaluation suggests that CDC-recommended approaches for confirmatory testing are ill-advised.
J Clin Microbiol, 44(7):2512-2517, 01 Jul 2006
Cited by: 39 articles | PMID: 16825373 | PMCID: PMC1489520
