This article requires a subscription to view the full text. If you have a subscription you may use the login form below to view the article. Access to this article can also be purchased.
- Address for Correspondence: Michael Rock
, University Wisconsin-Madison, mjrock{at}wisc.edu
ABSTRACT
The cystic fibrosis (CF) Quantum test (CFQT) showed promise in a previous pilot study; however, there was greater imprecision in one patch lot. Following the pilot study, the manufacturer changed their fabricating procedures. Participants with previously diagnosed CF or participants who required a sweat test for clinical reasons were invited to undergo the CFQT research test and a conventional sweat test (Macroduct collection and chloride analysis via the ChloroChek chloridometer). Previously diagnosed CF (n = 41) and CF transmembrane regulator–related metabolic syndrome/CF screen positive inconclusive diagnosis (n = 3) patients and patients who required a sweat test for clinical indications (n = 22) were recruited to have bilateral CFQT along with the Macroduct test performed on the same day. Pairs of data from each test were plotted as a correlation graph, bias plot, and Bland Altman plot. Coefficient of variation (CV) between extremities and quantity-not-sufficient (QNS) rates for both tests were calculated.
The CV between left and right extremities was greater in the CFQT (9.5%) compared with the Macroduct (4.8%). The QNS rates of the two tests were comparable (CFQT, 6.8%; Macroduct, 6.0%). There was greater imprecision with the CFQT results. The diagnostic agreement between the two tests was 100% positive percent agreement (95% confidence interval [CI], 90%–100%), 100% negative percent agreement (95% CI, 80%–100%), 67% intermediate percent agreement (95% CI, 30%–80%), and 92% overall percent agreement (95% CI, 80%–100%).
This follow-up study demonstrated that the CFQT is not analytically nor diagnostically reliable (Clinicaltrials.gov identifier NCT01345617).
- CF - cystic fibrosis
- CFF - Cystic Fibrosis Foundation
- CFSPID - cystic fibrosis screen positive inconclusive diagnosis
- CFTR - cystic fibrosis transmembrane regulator
- CFQT - cystic fibrosis Quantum test
- CLSI - Clinical and Laboratory Standards Institute
- CRMS - cystic fibrosis–related metabolic syndrome
- CV - coefficient of variation
- POCT - point-of-care test
- QNS - quantity not sufficient
INTRODUCTION
Cystic fibrosis (CF) is the most common, life-shortening autosomal recessive disease in White people, occurring with a frequency of approximately 1 case in every 3300 live births. The basic defect in CF is dysfunction of the CF transmembrane regulator (CFTR) protein.1 CFTR is a chloride channel that is expressed in sweat ducts, respiratory epithelial cells, pancreatic ductules, and exocrine cells in the reproductive system (the vas deferens and cervix). Common symptoms of CF are recurrent pulmonary infections leading to progressive loss of lung function, infertility in the majority of males due to congenital bilateral absence of the vas deferens, and exocrine pancreatic insufficiency resulting in malabsorption of protein and fat with subsequent failure to thrive in infants. The pulmonary infections account for the majority of morbidity and mortality in CF.
Although the discovery of the CFTR gene in 19891 opened the door to a genetic diagnosis, there are patients with CF in which 2 variants cannot be identified. Thus, sweat chloride testing2,3 will always be necessary for diagnostic purposes and to assess the effect of protein modifier drugs. The quantitative pilocarpine iontophoresis sweat test was first described by Gibson and Cooke in 1959.4 Pilocarpine, a cholinergic agonist, is delivered to the sweat glands by iontophoresis: A small electrical charge delivered for 5 minutes drives pilocarpine into the skin. This is followed by a 30-minute collection period in which sweat is collected into gauze or filter paper. After sweat collection, there are steps to elute sweat out of the gauze or filter paper. Lastly, sweat chloride analysis occurs via quantitative analysis using a coulometric/amperometric chloridometer. There are many steps in this process in which errors can and do occur. An alternative collection method that is approved by the Cystic Fibrosis Foundation (CFF) is the Macroduct method.5 Similar to the Gibson-Cooke method, there is a 5-minute pilocarpine iontophoresis step. In the Macroduct method, sweat is collected in microbore tubing for up to 30 minutes. The pure sweat sample can be placed directly into a chloridometer for sweat chloride analysis. Although there are fewer steps in this method compared with the Gibson-Cooke method, technicians must be meticulous in all aspects of the procedure: iontophoresis delivery of pilocarpine, sweat collection, and sweat chloride analysis.5 Potential errors can also occur with the Macroduct procedure.
A novel point-of-care test (POCT), the CF Quantum test (CFQT) (PolyChrome Medical, LLC; Eden Prairie, MN), was developed in an effort to simplify the determination of sweat chloride.6 This utilizes an electrode and controller set that can be worn on the arm for the delivery of pilocarpine. This differs from the iontophoresis device for the Gibson-Cooke and Macroduct devices in which the patient is tethered by wires to a box. After iontophoresis, sweat is collected on a patch containing silver nitrate. An ion exchange reaction occurs between the chloride in the sweat and the nitrate, resulting in silver chloride, which is an insoluble white precipitate in the center of the patch. In theory, the surface area of the white precipitate is proportional to the sweat chloride value. After collection of sweat, the patch is placed into an analyzer that consists of a camera and computer software. The sweat volume and chloride level are derived by computer software in the analyzer. The CFQT does not involve the handling of any liquids (including sweat) and is a simpler procedure compared with Gibson-Cooke and Macroduct. All methods for sweat collection and analysis require a minimum amount of sweat, not because of the instrumentation involved, but because a valid sweat chloride result depends on adequately stimulated sweat glands. Inadequately stimulated sweat glands will result in a decreased volume of sweat collected and can lead to false negative results. These inadequate sweat samples are referred to as “quantity not sufficient” (QNS).7 A national survey of CFF-accredited care centers demonstrated that QNS rates could range as high as 40%.8 Thus, there is a critical need for decreasing QNS rates with currently approved tests or development of a new test that has lower QNS results.
Although the CFQT was feasible in the previous 3-site, multicenter study, there was a patch lot in 1 center that showed greater imprecision than the patch lots tested in the other 2 centers.6 Following the results of the study, the manufacturer identified areas in the processing of the patches that could account for the lot-to-lot variability in the results and made changes to the patch manufacturing. After these modifications, this second multicenter study was conducted. The aim of the study was to determine the analytic and diagnostic validity of the CFQT and to compare the QNS rates of CFQT with collection of sweat in the Macroduct.
MATERIALS AND METHODS
A multicenter study (clinicaltrials.gov identifier NCT01345617) from February 6, 2017 to September 27, 2017 enrolled 44 participants with previously diagnosed CF or CF-related metabolic syndrome (CRMS)9/CF screen positive inconclusive diagnosis (CFSPID)10 and 22 participants who required a sweat test on clinical grounds (either as follow-up of an abnormal CF newborn screening test, or their provider ordered a sweat test). Participants were invited to undergo a CFQT and sweat test via Macroduct collection and analysis with the ELITech ChloroChek chloridometer Model 3400 (Logan, Utah). To assure that the reference method for collection and analysis (Macroduct/ChloroChek) was correctly performed according to CFF, Clinical and Laboratory Standards Institute (CLSI), and manufacturer’s guidelines, the sweat-testing laboratories were visited by the principal author of the CLSI guideline document on sweat testing, and a written evaluation was provided. It was mandatory that the suggestions for improvement were implemented prior to starting the study at each site. The Institutional Review Board approved the study at each site, and written informed consent was obtained from parents/patients (and assent, if applicable) prior to commencing with any study procedures.
Macroduct sweat stimulation and collection were performed bilaterally (ie, left and right arm) per the CLSI guidelines.7 Pilocarpine iontophoresis occurred for 5 minutes, and collection of sweat into Macroduct occurred for 30 minutes. If 15 μL of sweat was not collected within 30 minutes, then the test was deemed QNS. After collection and quantitation of sweat volume, the sweat was titrated using the EliTech ChloroChek chloridometer according to the manufacturer’s protocol. The chloridometer contains a silver electrode that releases silver into an acid solution containing the sweat sample, and a timer is started. Chloride in the sweat sample combines with the silver, forming insoluble silver chloride. When all of the chloride has been precipitated as silver chloride, the measuring electrode detects the appearance again of free silver ions, and the timer is stopped. The amount of time that silver is generated is proportional to the chloride concentration and is compared with an internal calibrator to convert the time to millimoles per liter. Three levels of commercial control solutions (low, intermediate, and elevated chloride concentration) were assayed every day of sample testing and had to be within the accepted range established by the manufacturer before study samples could be analyzed.
The CFQT was performed according to the manufacturer’s protocol. Sweat stimulation was performed bilaterally by pilocarpine iontophoresis for 8 minutes followed by the application of a collection patch (Figure 1). The maximum allowed time of sweat collection was 20 minutes. The collection of sweat on the patch occurred until the sweat front (a red circle on the patch) reached a stop test ring of 15 mm in diameter. The test was deemed QNS if the sweat front did not reach the stop ring by 20 minutes. After an adequate quantity of sweat was obtained, the patch was removed from the skin and allowed to dry for 15 minutes. The patch was then placed in the analyzer (Figure 2), and a sweat volume and chloride value were derived. The CFQT patch detection is based on an ion exchange reaction between the chloride in the sweat and silver nitrate in the test patch. When chloride ions in the sweat sample come into contact with the silver chromate in the patch, silver chloride, an insoluble white precipitate, forms in the center of the patch. An outer red ring, the sweat front, indicates the amount of sweat collected in the patch. The surface area of the white precipitate in the middle of the patch compared with the total surface area within the red ring is directly proportional to the sweat chloride value. The camera and computer software in the analyzer were assessed every day of sample testing using scanned photographs of 3 levels of results representing low, intermediate, and elevated chloride concentrations, and the values needed to be within the preset range.
CFQT collection patch.
CFQT analyzer.
If a participant was over 6 months of age, the Macroduct collection and CFQT both occurred on the forearm. For participants under 6 months of age (n = 10), there was inadequate space to perform both tests on the forearm. Thus, the Macroduct collection occurred on the forearm, and the CFQT occurred on the thigh. Areas of the skin only underwent sweat stimulation and collection once. Pilocarpine iontophoresis for the Macroduct and sweat collection occurred first, and the CFQT was performed second. It was possible to perform the CFQT during the 30-minute Macroduct collection time.
Agreement of sweat chloride values between left and right extremities were per the CLSI guidelines7: For sweat chloride values ≤60 mmol/L, the extremities must be within 10 mmol/L, and for sweat chloride values >60 mmol/L, the extremities must be within 15 mmol/L. Exceeding these thresholds resulted in an invalid test.
The interpretation of sweat chloride values was per the updated guidelines from the United States CFF.2 For all ages of participants, a sweat chloride value of ≤29 mmol/L was normal, 30–59 mmol/L was intermediate, and ≥60 mmol/L was abnormal and consistent with CF. With bilateral sweat testing being performed, the interpretation of the results used the higher of the 2 sweat chloride values.
Sample Size
A sample size of 300 participants, including n =150 participants with previously diagnosed CF or CRMS and n =150 participants referred to the sweat test laboratory for clinical reasons, was proposed for this study. It was estimated that of the 150 participants referred to the sweat test laboratory on clinical grounds, at least 120 would have sweat chloride values in the normal range. Hence, with a sample size of at least 150 participants with sweat chloride values in the nonnormal range and 120 participants with sweat chloride values in the normal range and assuming a true sensitivity/specificity of 0.95, the sensitivity and specificity of the CFQT would be estimated with a standard error of less than 2% and the lower bound of the two-sided 95% confidence interval (CI) of the sensitivity and specificity would exceed 0.9.
Statistical Analysis
Sweat chloride measurements obtained by Macroduct/ChloroChek and CFQT were considered 2 variables and were summarized in terms of number of observations, means, SDs, and ranges. The coefficient of variation (CV) between paired extremities by test (Macroduct/ChloroChek and CFQT) were calculated. Bias assessment was conducted according to CLSI guidelines for method comparison and bias estimation.11 A visual check for the relationship between the 2 variables was performed by evaluating (1) scatterplot of CFQT values versus Macroduct values, (2) bias plot of CFQT minus Macroduct/ChloroChek vs Macroduct/ChloroChek values, and (3) bias plot of individual results deltas vs the mean differences between the 2 tests (Bland Altman plot).12 The proportion of QNS sweat tests were compared between Macroduct collection and CFQT using Fisher’s exact test. All P values were two-sided, and P values <0.05 were considered statistically significant.
Categorization of diagnosis by test were summarized as follows:
| Macroduct/ChloroChek | ||||
|---|---|---|---|---|
| >60 mmol/L | 30–59 mmol/L | ≤29 mmol/L | ||
| CFQT | >60 mmol/L | A | B | C |
| 30–59 mmol/L | D | E | F | |
| ≤29 mmol/L | G | H | I | |
Overall, positive, negative, and intermediate percent agreements were calculated as follows:
Overall Percent Agreement = 100 × (A + E + I) ÷ (A + B + C + D + E + F + G + H + I)
Positive Percent Agreement = 100 × A ÷ (A + D + G)
Negative Percent Agreement = 100 × I ÷ (C + F + I)
Intermediate Percent Agreement = 100 × E ÷ (B + E + H)
Positive, intermediate, and negative percent agreements were reported along with the corresponding 95% CIs.
RESULTS
There were 66 participants at 4 CF centers who completed the study. There were 22 participants at the University of Wisconsin (center 1), 27 participants at the University of Minnesota (center 2), 13 participants at the University of Michigan (center 3), and 4 participants at the University of Alabama-Birmingham (center 4). The characteristics of the participants are in Table 1.
Characteristics of study participants
For Macroduct/ChloroChek, of the potential 132 test results (66 participants who had bilateral tests performed), there was one technical problem with the chloridometer. The stirring bar in the ChloroChek instrument stopped. The instrument was turned off and on, and although a final result was obtained, this must be considered an invalid test. For the CFQT, there were 13 technical problems in which no results were available: the analyzer generated an error code on 9 patches, and there was a stimulator error bilaterally in 2 participants. One CFQT result was invalid because of a sweat chloride of >160 mmol/L. Means, SDs, ranges, CV between extremities and QNS rates for the 2 methods are in Table 2. Of the 66 participants, 10 were infants with a positive newborn screen for CF. The Macroduct QNS rate in these infants was 20%, and the CFQT QNS rate in these infants was 15%. For sweat chloride values of <10 mmol/L (the lower limit of detection for both methods), the value was rounded up to 10 mmol/L. Although there was no significant difference in the mean sweat chloride value between the CFQT and Macroduct/ChloroChek, the mean sweat chloride value per CFQT reflects the positive bias (discussed below for Figure 4). Additionally, the higher CV between extremities for CFQT reflects greater imprecision.
Means, SD, ranges, CV and QNS rates
There were 9 infants who had the CFQT performed on the thigh. One study site performed Macroduct collection and CFQT both on the forearms in 1 infant. Although this yields a potential 18 comparisons with bilateral testing, because of QNS tests with both methods and stimulator errors with the CFQT, there were only 7 tests available for comparison of Macroduct/ChloroChek vs CFQT. None of these infants had CF; the mean value for Macroduct/ChloroChek was 15 mmol/L compared with a mean value of 19 mmol/L for CFQT (again reflecting a positive bias of the CFQT, but the number of tests is too small for statistical comparison).
The results of all bilateral Macroduct/ChloroChek were within the prestated agreement of each other. For the CFQT, there were 6 participants in which the results were invalid because they exceeded the bilateral level of agreement.
The average sweat collection time for the CFQT was 10 minutes vs 30 minutes for the Macroduct (P < 0.0001).
The method comparison graph of CFQT results vs Macroduct/ChloroChek is in Figure 3. The Pearson correlation coefficient was 0.97, y-intercept was –0.84, slope was 1.10, and SD x/y was 0.88. In a method comparison graph, the values obtained by the reference method (Macroduct/ChloroChek) are plotted on the x-axis, and values obtained by the new method (CFQT) are plotted on the y-axis. If identical values were obtained with both methods, the strength of the correlation would yield a correlation coefficient of 1.00, the y-intercept would be 0.00, the slope would be 1.00, and the SD x/y would be 0.00. In general, these values are interpreted such that the slope indicates proportional error, the y-intercept indicates constant error, and the SD x/y indicates the imprecision of the values around the correlation line.13
Method comparison graph of CFQT results vs Macroduct/ChloroChek; y-axis, CFQT (mmol/L); x-axis, Macroduct/ChloroChek (mmol/L). Center 1
Bias plot of CFQT minus Macroduct/ChloroChek plotted against Macroduct/ChloroChek. Circled symbols are from the second patch lot. y-axis, difference (CFQT, Macroduct/ChloroChek) (mmol/L); x-axis, Macroduct/ChloroChek (mmol/L). Center 1
Figure 4 is the bias plot of CFQT minus Macroduct/ChloroChek plotted against Macroduct/ChloroChek. The circled symbols are results from the second patch lot in this study. Only center 2 had progressed in the study to the point of using the second patch lot. The majority of the symbols on the bias plot are above zero, thus signifying a positive bias of the CFQT results compared with Macroduct/ChloroChek.
Figure 5 is the Bland Altman plot of the data. The solid line is the mean of the differences (–5.8 mmol/L), and the dotted lines are ±1.96 SD of the differences (13.5 and –24.8 mmol/L). Similar to the bias plot, the Bland Altman plot shows significant scatter between the 2 techniques and that there are 4 paired tests in which there are extreme outliers (more than 2 SD of the differences).
Bland Altman plot. Solid line denotes the mean of differences. Dotted lines denotes SD = ±1.96 of the differences; y-axis, sweat chloride difference (Macroduct/ChloroChek, CFQT) (mmol/L); x-axis, sweat chloride average (mmol/L). Center 1
In assigning diagnostic categories, the positive percent agreement was 100% (95% CI, 90%–100%), the negative percent agreement was 100% (95% CI, 80%–100%), the intermediate percent agreement was 67% (95% CI, 30%–80%), and the overall percent agreement was 92% (95% CI, 80%–100%) (Table 3).
Categorization of diagnosis
DISCUSSION
This study examined 66 participants undergoing comparative sweat chloride tests to evaluate a redesigned POCT device, the CFQT, and the reference method of Macroduct/ChloroChek. One of the potential advantages of such a POCT device could be decreased testing time, providing results faster to the physician with less operator intervention. In addition, a method with a low QNS rate would be highly desirable. A previous pilot study with 170 participants showed promise but noted concern about greater imprecision with differing lot numbers of patches, thus prompting this study, which was designed for 300 participants using a reformulated patch design. However, early results in this study demonstrated unacceptable positive bias with the CFQT, and the project was terminated after 66 participants.
A visual review of the comparison plot in Figure 3 shows a reasonable range of data and establishes a relationship between the pairs of sweat chloride values with slight proportional error when compared with the perfect correlation line. This proportional error is supported by an examination of both the bias plot (Figure 4) and the Bland Altman plot (Figure 5), which shows that the results of the CFQT may be anywhere from 14 mmol/L lower to 25 mmol/L greater than the reference method.
In evaluating the implications of the bias upon clinical decisions, the positive and negative percent agreement were 100%, but the intermediate percent agreement was only 67%. Thus, if one were to rely on the CFQT as a diagnostic tool, the categorization of patients with the CRMS/CFSPID diagnosis would be incorrect 33% of the time. CRMS/CFSPID is a consequence of newborn screening. These patients do not fit the full diagnostic criteria for CF, and their sweat chloride values are either normal or intermediate. The cause of the lack of agreement in the intermediate range and the overall proportional bias are unknown but may be related to the manufacturing process and composition of the test patches. In the manufacturing process of the patches, silver chromate is added to the patch, and excess reagent is removed by a roller apparatus. Prior to this study, the manufacturer of the CFQT obtained a new roller apparatus in an attempt to eliminate the variability of the results. Unfortunately, the new roller apparatus did not solve this issue. Post hoc analysis by the manufacturer utilizing scanning electron microscopy revealed that the silver chromate was variably impregnated into the chromatography paper due to the fibrous structure of the paper, random stacking and orientation of fibers, and variability in the thickness of the paper, which may account for the observed imprecision.
The bilateral agreement between the left and right extremities on the same participant was greater for the CFQT (CV = 9.5%) vs the Macroduct/Chlorochek (CV = 4.9%), with 6 participants having invalid results with the CFQT due to greater differences between the extremities exceeding accepted concentrations, suggesting greater imprecision with the CFQT. Unfortunately, it was not possible to further evaluate precision with the CFQT given the design of the device. Traditionally, as with the ChloroChek and other clinical laboratory-based analyzers, precision is determined by repeated measurements of the same level of control and calculation of SD and CV for within-run and across-run precision, which assesses the entire testing process. The CLSI has suggested that the CV for low controls be less than 7% and for high controls less than 5%.7 The CFQT controls were limited to assessing only the camera scanning and software portions of the test and did not assess the total testing process to include variation in patch manufacturing and design. Because of the potential variation in single-use devices, the College of American Pathologists requires that if a laboratory limits quality control to an internal device, such as electronic check, instead of also running external (liquid) controls, then the laboratory must develop an individualized quality control plan to evaluate risk and assess the effectiveness of the internal quality control and quality-assurance processes.14
A limitation of this study is that there may have been differences in analytical variation among the 4 participating sites. However, such differences for the Macroduct/Chlorochek procedures should have been minimal because each site needed to demonstrate that they could perform the procedure exactly as recommended by the CLSI. For the CFQT procedure, each site received an in-person instruction of how to perform this test at site initiation.
In conclusion, the CFQT device using chromatography paper–based patches did not yield results that were comparable to sweat collection with the Macroduct and chloride analysis with the ELITech ChloroChek chloridometer. In order for a new method of sweat chloride analysis to be accepted by the clinical laboratory and CF communities, the method must yield results as accurate as the established method of coulometric titration determination of chloride concentration.
ACKNOWLEDGMENTS
The authors acknowledge contributions of site investigators and research coordinators at the University of Minnesota (Terri Laguna and Cynthia Williams), the University of Michigan (Samya Nasr and Dawn Kruse), and the University of Alabama (Hector Gutierrez and Katie Brand). Our thanks to the clinical laboratories at the sites, families, and patients who participated in the study, and special thanks to Linda Makholm and Anita Laxova at the University of Wisconsin. None of the authors have any personal or any financial relationships that would constitute a conflict of interest.
Footnotes
Presented in part at the 31st Annual North American Cystic Fibrosis Conference, 2–4 November 2017, Indianapolis, Indiana
- Received September 9, 2019.
- Revision received February 20, 2020.
- Accepted March 2, 2020.
American Society for Clinical Laboratory Science
Jump to section
Related Articles
Cited By...
- No citing articles found.
More in this TOC Section
Similar Articles
Keywords
- CF - cystic fibrosis
- CFF - Cystic Fibrosis Foundation
- CFSPID - cystic fibrosis screen positive inconclusive diagnosis
- CFTR - cystic fibrosis transmembrane regulator
- CFQT - cystic fibrosis Quantum test
- CLSI - Clinical and Laboratory Standards Institute
- CRMS - cystic fibrosis–related metabolic syndrome
- CV - coefficient of variation
- POCT - point-of-care test
- QNS - quantity not sufficient
- sweat glands
- cystic fibrosis
- diagnostic tests
- routine
- bias
- statistical