QUAREP-LiMi: A community-driven initiative to establish guidelines for quality assessment and reproducibility for instruments and images in light microscopy.
Free full text in Europe PMC
Abstract
Free full text
QUAREP-LiMi: A community-driven initiative to establish guidelines for quality assessment and reproducibility for instruments and images in light microscopy
Abstract
A modern day light microscope has evolved from a tool devoted to making primarily empirical observations to what is now a sophisticated, quantitative device that is an integral part of both physical and life science research. Nowadays, microscopes are found in nearly every experimental laboratory. However, despite their prevalent use in capturing and quantifying scientific phenomena, neither a thorough understanding of the principles underlying quantitative imaging techniques nor appropriate knowledge of how to calibrate, operate and maintain microscopes can be taken for granted. This is clearly demonstrated by the well-documented and widespread difficulties that are routinely encountered in evaluating acquired data and reproducing scientific experiments. Indeed, studies have shown that more than 70% of researchers have tried and failed to repeat another scientist’s experiments, while more than half have even failed to reproduce their own experiments. One factor behind the reproducibility crisis of experiments published in scientific journals is the frequent underreporting of imaging methods caused by a lack of awareness and/or a lack of knowledge of the applied technique. Whereas quality control procedures for some methods used in biomedical research, such as genomics (e.g. DNA sequencing, RNA-seq) or cytometry, have been introduced (e.g. ENCODE), this issue has not been tackled for optical microscopy instrumentation and images. Although many calibration standards and protocols have been published, there is a lack of awareness and agreement on common standards and guidelines for quality assessment and reproducibility. In April 2020, the QUality Assessment and REProducibility for instruments and images in Light Microscopy (QUAREP-LiMi) initiative was formed. This initiative comprises imaging scientists from academia and industry who share a common interest in achieving a better understanding of the performance and limitations of microscopes and improved quality control (QC) in light microscopy. The ultimate goal of the QUAREP-LiMi initiative is to establish a set of common QC standards, guidelines, metadata models and tools, including detailed protocols, with the ultimate aim of improving reproducible advances in scientific research. This White Paper (1) summarizes the major obstacles identified in the field that motivated the launch of the QUAREP-LiMi initiative; (2) identifies the urgent need to address these obstacles in a grassroots manner, through a community of stakeholders including, researchers, imaging scientists, bioimage analysts, bioimage informatics developers, corporate partners, funding agencies, standards organizations, scientific publishers and observers of such; (3) outlines the current actions of the QUAREP-LiMi initiative and (4) proposes future steps that can be taken to improve the dissemination and acceptance of the proposed guidelines to manage QC. To summarize, the principal goal of the QUAREP-LiMi initiative is to improve the overall quality and reproducibility of light microscope image data by introducing broadly accepted standard practices and accurately captured image data metrics.
1 |. PREFACE
The QUality Assessment and REProducibility for instruments and images in Light Microscopy (QUAREP-LiMi) initiative (https://quarep.org/) aims at convening the light microscopy community with the explicit purpose of reaching a broad consensus concerning Quality Control and Quality Assessment guidelines for optical microscopy to be adopted worldwide. For the purposes of this discussion, by ‘light microscopy community’, we refer to everyone working directly or indirectly with light microscopes and image data, independent of the specific microscope design or configuration. Although we aim to satisfy the entire community’s requirements and views, we cannot claim sufficient diversity or coverage of the community for complete representation. Rather, this White Paper is the first of a series that will report our ongoing progress towards achieving the goals of QUAREP-LiMi as outlined in this document. While the work of QUAREP-LiMi aims at developing recommendations and guidelines that can be easily extended across disciplines (both physical and life sciences), for the sake of simplicity, the discussion is currently restricted to applications and examples drawn mainly from biology. Although our current efforts focus on establishing guidelines for widefield and confocal optical microscopes, we are keen to extend the breadth of our work subsequently to cover other light-microscopy-based imaging modalities.
2 |. BACKGROUND
A modern day light microscope has evolved from a tool devoted to making primarily empirical observations to what is now a sophisticated, quantitative device that is an integral part of both physical and life science research. Nowadays, microscopes are found in nearly every experimental laboratory. However, despite their prevalent use in capturing and quantifying scientific phenomena, neither a thorough understanding of the principles underlying quantitative imaging techniques nor appropriate knowledge of how to calibrate, operate and maintain microscopes can be taken for granted. This is clearly demonstrated by the well-documented and widespread difficulties that are routinely encountered in evaluating acquired data and reproducing scientific experiments. Indeed, studies have shown that more than 70% of researchers have tried and failed to repeat another scientist’s experiments, while more than half have even failed to reproduce their own experiments.1 One factor behind the reproducibility crisis of experiments published in scientific journals is the frequent underreporting of imaging methods caused by a lack of awareness and/or a lack of knowledge of the applied technique.2,3 Whereas quality control procedures for some methods used in biomedical research, such as genomics (e.g., DNA sequencing, RNA-seq) or cytometry, have been introduced (e.g. ENCODE),4 this issue has not been tackled for optical microscopy instrumentation and images. Although many calibration standards and protocols have been published, there is a lack of awareness and agreement on common standards and guidelines for quality assessment and reproducibility.5
In April 2020, the QUality Assessment and REProducibility for instruments and images in Light Microscopy (QUAREP-LiMi) initiative6 was formed. This initiative comprises imaging scientists from academia and industry who share a common interest in achieving a better understanding of the performance and limitations of microscopes and improved quality control (QC) in light microscopy. The ultimate goal of the QUAREP-LiMi initiative is to establish a set of common QC standards, guidelines, metadata models,7,8 and tools,9,10 including detailed protocols, with the ultimate aim of improving reproducible advances in scientific research and helping microscope custodians.11
2.1 |. Current situation
Since their introduction in the early 17th Century, microscopes have transitioned from basic, qualitative image-collecting tools to sophisticated instruments capable of automatically acquiring information-rich images that are further processed via advanced image processing and analysis steps to extract quantitative information about the underlying science. The robustness of the conclusions that we make from these observations will depend upon the reproducibility of the samples and the microscope system used to image them. Importantly, each instrument’s technical characteristics pertaining to the experimental conditions used need to be fully understood and documented to permit valid interpretation of imaging data. To enable the reliable and reproducible extraction of quantitative information, microscopes, including advanced widefield and confocal instruments, must therefore be well described, maintained, calibrated and in essence ‘quality controlled’ (Figure 1).
Acquiring imaging data that is both quantifiable and reproducible involves a myriad of factors, few of which are acknowledged or accurately recorded
Note: Intimate knowledge of the composition and performance of a system is essential for reproducibility. However, performance measurements may be tricky, and require specific protocols, tools, samples, training and data analysis methods. In order to help microscope users to assess and judge the performance of their systems properly, the community must agree on and publish guidelines and benchmarks.
Unfortunately, the requirement for robust microscope and image quality (IQ) assessment is not commonly recognized among the scientific community, leading to the infrequent application of appropriate quality control (QC) procedures. This is due to several barriers:
A lack of awareness of the importance of QC among many microscope users;
Insufficient training of microscope users on the complexity of performing quantitative imaging and on guidelines and standards for quality assessment and reproducibility;
The lack of widely adopted community-wide, universal guidelines and standards for light microscopy documentation and QC;
The limited willingness of the community’s stakeholders (researchers, funders and scientific publishers, e.g. Ref. 12) to enforce existing guidelines and standards.
As a result, rigor and reproducibility are limited, the reliability of quantitative analysis is severely impacted and the confidence in published data becomes eroded.
Encouragingly, the awareness of the importance of QC and reproducibility in light microscopy has gained traction in recent years, both within the scientific community and among funders (e.g. National Institutes of Health [NIH], European Research Council [ERC]) as well as within newly launched bioimaging networks (e.g. Max Planck Bioimaging Core Unit Network, https://www.bioimagingnet.mpg.de/aims). More active steps have been taken within several microscopy initiatives such as Global Bioimaging (GBI, https://www.globalbioimaging.org/), Euro-BioImaging ERIC (European Research Infrastructure Consortium) (https://www.eurobioimaging.eu/), the Royal Microscopical Society (RMS, https://www.rms.org.uk/), German BioImaging-Gesellschaft fur Mikroskopie und Bildanalyse (GerBI-GMB, https://www.gerbi-gmb.de/),13 BioImaging North America (BINA, https://www.bioimagingna.org/) and the RT-MFM technological network (Microscopie photonique de Fluorescence Multidimensionnelle, http://rtmfm.cnrs.fr/). GBI has published an overview of the current landscape for quality assurance and data management in imaging facilities,14,15 including recommendations for QC. This document also highlights multiple aspects concerning image data standardization, management and publication, such as the definition of image and microscopy metadata guidelines and data models7–9,14,16–18 and the need to provide open access to all raw data for accepted manuscripts (e.g. Image Data Resource and BioImage Archive),19–22 which are being addressed both within QUAREP-LiMi (see Working Group 7—Metadata) and by others7–9,14,15,23 within the imaging community. The Euro-Bioimaging ERIC includes the independent assessment of QC measures and implementation in the ongoing evaluation of existing Nodes and during the application process for new imaging Node candidates. Finally, BINA, the RMS, GerBI-GMB and the RT-MFM are all actively engaged in tackling QC and reproducibility issues via dedicated working groups (‘QC and Data Management [QC-DM, https://www.bioimagingna.org/qc-dm-wg]’; ‘QC Focussed Interest Group’, https://www.rms.org.uk/network-collaborate/focussed-interest-groups/quality-control.html; ‘Quality Assessment for Instruments & Facilities’, https://www.gerbi-gmb.de/WG1 and ‘Metrological Measurements on Microscopy’, respectively). All of the groups described are represented in QUAREP-LiMi and actively participate in the relevant Working Group sections (see below), giving us the chance to provide a single cohesive, internationally accepted approach to microscope QC.
2.2 |. Current approaches
Despite the importance of individual local efforts, they prove insufficient to overcome the global challenges associated with QC in light microscopy. In the following section, we highlight a few of these approaches and discuss why they are unable to completely overcome existing hindrances individually.
2.2.1 |. Quality Control procedures adopted by individual core facilities and laboratories
To tackle common QC issues, many core facilities and laboratories regularly perform maintenance and various QC tests of their instruments. However, the nature and frequency of the performed tests vary greatly, depending on the priorities set by researchers, imaging facility staf, and their institution. A survey initiated by the European Light Microscopy Initiative (ELMI) in 2019, https://lic-machform.vm.uni-freiburg.de/view.php?id=59721 highlighted that numerous core facilities and labs already perform QC, but a considerable percentage does not at all (Figure 2A). Likewise, there was wide variation in the respondents’ choice of tools, making any comparison and reproducibility of QC results between equipment difficult (Figure 2B).
Frequency and type of tests performed in core facilities
Note: Prior to a session on QC organized during the Microscopy Facility Day at the 2019 ELMI meeting, light microscopy core facility representatives around the globe were asked to complete a survey about the type and the frequency of tests performed in their facility. The link to the survey was opened in June 2019, 2 weeks before the meeting, and sent to all registered participants. It was also advertized multiple times on various international microscopy forums. Reminders were also sent to participants after the meeting and responses were subsequently collected until February 2020. The histograms in panels (A) and (B) summarize the responses from almost 200 facilities in a simplified manner. Panel (A) displays how often different quality checks are performed; the x-axis represents in percentage the respective frequency categories, namely, regularly, on demand and never. Panel (B) highlights which tools are used for performance evaluation of light microscopes; the x-axis represents the percentage of respondents using the indicated QC tool.
2.2.2 |. Guidelines by the International Organization for Standardization (ISO)
The ISO (https://www.iso.org/) has created standards for brightfield microscopy24,25 and, more recently, for confocal microscopy.26 These ISO standards provide researchers with directions as to what should be measured and tested. Nevertheless, there is little information describing how key measurements should be made within these documents, using which samples and tools, and with what frequency.
2.2.3 |. Tools and protocols by the community for the community
Several individual groups have published methods and software tools to streamline and automate microscope QC procedures (e.g.27–32 and recently reviewed in Refs. 33,34). In addition, several open-source software tools that provide different degrees of automation for different microscopy calibration tasks have been developed and made available both as ImageJ-based macros and plug-ins (e.g. NoiSee,35 MetroloJ,36 ConfocalCheck,37 AutoQC,38 PSFj39 and MIPs for PSFs,40 SIMcheck41) and stand-alone web applications (e.g. PyCalibrate, https://www.psfcheck.com/). Some National groups (RT-MFM and BINA) have made available QC suitcases,http://rtmfm.cnrs.fr/ providing groups with access to the hardware required for performing QC of their microscopes. Finally, international endeavors involving the global community were carried out and published by the Association of Biomolecular Resource Facilities (ABRF).42,43 These efforts provide both valuable results and metrics that can be saved locally and archived individually. However, these undertakings were affected by significant variations between individual groups. Moreover, they are neither comprehensive nor address standardization of metadata capture,9,10,18,23,44 and they are not fully aligned with the more recent recommendations put forth by standards organizations such as the ISO.
2.2.4 |. Regulations and guidelines imposed by third parties
Besides the ISO, funding agencies, scientific publishers and community organizations (e.g. GerBI-GMB, RMS, BINA, RT-MFM) often furnish QC guidelines for light microscopy. However, these guidelines are not exhaustive, are often issued in isolation, and are not accepted by the principal constituents of the imaging community (including commercial microscope manufacturers and the broader scientific community).
All of these approaches share a similar set of limitations: (1) they are currently adopted voluntarily and are therefore unenforceable; (2) they are often targeted at highly trained imaging facility staff and are often not accessible to less expert, non-facility microscope users and custodians in individual laboratories; (3) they are limited in scope and therefore do not guarantee proper QC and reproducibility and may not cover all of the variables required for the experimental setup and (4) they are not standardized and therefore show significant variability. The reasons for this are several-fold. First, there is a lack of agreement regarding the recommended standard samples, tools, protocols and metrics to be measured. While this is partially represented by the success of many smaller groups to create their own methods, there is a pressing need for the community to agree on what should be measured for each hardware component (e.g. laser, camera or objective lens) and calibration procedure (i.e. optical, intensity and mechanical calibration), which tools and samples should be used and the frequency of QC measurements for different metrics. Secondly, a commonly cited reason for minimizing or avoiding microscope QC is the lack of appropriate time and resources (i.e. personnel, machine time and required hardware) afforded to microscope custodians to perform the appropriate tests, the downstream analysis and the compilation of the results across time accurately and systematically. Finally, most protocols/methods currently being performed are marred by high variability of the measured values, almost entirely due to the lack of automation.
3 |. PROPOSED COMMUNITY-DRIVEN APPROACH
Following a discussion at the 2019 conference of the European Light Microscopy Initiative (ELMI 2019), members of the GerBI-GMB and RT-MFM networks launched a shared strategy to build a community consensus on QC measurements. This initial initiative rapidly integrated with similar efforts being conducted by the BINA QC-DM working group (https://www.bioimagingna.org/qc-dm-wg) and the RMS QC focussed interest group. (https://www.rms.org.uk/network-collaborate/focussed-interest-groups/quality-control.html) Shortly after, the publication of the Confocal ISO 2107326 provided the scientific community with an agreed minimal set of parameters that should be tested to assess the performance of confocal microscopes. Since ISO only provides parameters that should be measured and not in detail how this should be performed, a methodology manual was drafted as a first attempt to describe how the ISO-recommended QC metrics could be obtained in practice45 and presented to participants representing academia, industry and governmental standardization bodies at a meeting held on 28 April 2020. This led to the formal establishment of QUAREP-LiMi (https://quarep.org/)6 (coordinated by R. Nitschke, Gerbi-GMB).
QUAREP-LiMi is a grassroot global community that is open to individuals from academia, industry, government, funding agencies and scientific journals from around the world with interest in improving QC in light microscopy. As a testament to the timeliness of this strategy, QUAREP-LiMi quickly grew from 49 initial participants to 272 individuals (at the time of writing – 13 May 2021) from 24 countries (Figure 3). Compared to earlier approaches (see Section 2.2), the QUAREP-LiMi initiative is specifically designed to work in a completely transparent and open manner to foster ground-up participation from around the globe and ownership by all members of the community. By taking into account existing approaches and recommendations, the specific goal is to produce a consensus around shared, binding QC guidelines and specifications for the scientific community and corporate partners, releasing these as freely accessible, affordable and as open as possible. Additional output from QUAREP-LiMi will include outreach and review papers to educate users in the need for and employment of QC methods to satisfy the minimal metrics to support their experimental data sets. Furthermore, QUAREP-LiMi will work with both journals and funders to encourage stakeholders to adopt and enforce these standards.
Summary of current QUAREP-LiMi participants according to their origin and affiliation with the QUAREP-LiMi logo (updated: 13 May 2021)
3.1 |. Key beneficiaries
The deliverables from QUAREP-LiMi will benefit several groups related to light microscopy, from image data acquisition all the way to image data processing, presentation, sharing and reuse. All groups will benefit from heightened awareness of the need for microscope QC to support imaging data.
Research scientists and imaging scientists will take advantage of the harmonization and simplification of the QC procedures, facilitation of QC capture and storage and clear interpretation of QC results to better understand how the performance of their microscope impacts the interpretation of scientific results.
Scientific publishers and the general public will profit from an overall enhanced trust in the value and reproducibility of scientific publications, resulting from the publication of full descriptions (i.e. Material and Methods sections and microscopy metadata2,7,9,10,44) of the technical make-up of microscopes and of performance metrics to accompany raw image data.
Funding bodies will benefit from the planned improvement of QC practices that will undoubtedly improve image data reliability, reproducibility and openness, increasing overall the value and quality of scientific output and opening the way to truly FAIR data.46 Besides, the improved likelihood of data reuse towards novel discoveries will significantly impact taxpayer funds’ efficient use.
Core imaging facilities and their users, as well as non-facility microscope custodians and users, will profit from heightened confidence in the accuracy of their image data and the suitability of performed QC measurements to answer their scientific questions. Improving, prioritizing and streamlining the various required QC procedures will ease the time burden, facilitate the comparison of experimental data47 and improve the communication between imaging facility staff and commercial manufacturers based on common vocabulary, tools and protocols. As an added advantage, community standard procedures will allow users to submit standardized performance metrics with their published imaging data, which will greatly facilitate the interpretation, reproducibility and reuse of the results.
Commercial microscope and system component manufacturers will be able to take advantage of the availability of time-stamped and standardized microscope metrics to identify common microscope performance and quality issues, such as identifying faulty parts. They will also be able to utilize the metrics for future developments and the continued improvement of hard- and software products.
3.2 |. Initiated steps and dissemination
Although initiated around the Confocal Microscope ISO 21073 standard,25 the scope of QUAREP-LiMi has since increased. It is now devoted to establishing a comprehensive set of shared QC guidelines, tools for their capture and microscopy metadata specifications for their storage and automated reporting.
Direct and open engagement of the global imaging community, and public and effective dissemination of QUAREP-LiMi advances, are essential to foster consensusbuilding and global acceptance of QUAREP-LiMi proposals. Consistent with this goal and its foundational principles, QUAREP-LiMi actively seeks participation from any interested party to join the group and work towards establishing a global QC consensus. To this aim, regular updates on the progress achieved by individual working groups will be disseminated using various social platforms and the QUAREP-LiMi website for all interested parties to provide input. We have actively engaged with the current groups (see Section 2.1) to prevent duplication of work and to bring along the proponents of these groups as active participants in QUAREP-LiMi, and have organised outreach and education to present this year at the ELMI conference.
3.3 |. Organizational structure and immediate goals
Having established the general goals and founding principles of the group, members of QUAREP-LiMi agreed upon a set of essential topics to address and established the following organizational structure on 9 July 9 2020:
The group aims to achieve specific deliverables and established that work will be conducted by individual working groups, each led by an elected Chair and Vice Chair.
All interested parties are welcome to participate, either as observing members or as active participants within one or more working groups of their choice.
All participants, regardless of whether they are observing or active participants, will be allowed to provide feedback on the deliverables produced by individual working groups.
The number of working groups will be extended as needed to ensure coverage of all aspects of microscopy QC and satisfy all key beneficiaries’ desires.
More specifically, the following working groups were established, and when appropriate, will produce robust and straightforward protocols and training material based upon standardized samples or tools:
3.3.1 |. WG1 illumination power
Comparison of fluorescence intensities between images requires measurements of the illumination power and stability of the excitation light source. WG1 aims at establishing a recommended protocol for measuring the stability of a light source during both short- and long-term image acquisition sessions using calibrated external power sensors. This initial aim will be extended later to measure the absolute flux of light through the illumination path and irradiance of the sample. The initial protocol will be designed around lasers on confocal microscopy platforms (raster scanning and spinning disks). It will be later modified towards other microscopy techniques (widefield, TIRF, light-sheet, super-resolution and their respective light sources).
3.3.2 |. WG2 detection system performance
WG2 focuses on the detection system, comprising the detection path and its detector(s), and how it measures the signal from the sample. Members of WG2 aim to standardize the characterization of the detection system performance and create standard procedures for monitoring it over time, thereby revealing performance issues that could affect data reproducibility. Therefore, WG2 will define universal, externally measurable parameters applicable to any type of detector (e.g. photons, linearity and noise), together with measurement tools and protocols for measuring these parameters from common detector types. These universal parameters will be specified according to each distinct type of detector’s internal parameters, which have already been defined by the community.48,49 They will enable the evaluation and comparison of different detection systems, thus pinpointing the most suitable technology for given applications.
3.3.3 |. WG3 uniformity of illumination field – flatness
Illumination field uniformity is critical for quantitative imaging when comparing fluorescence intensities across a field-of-view (FOV) or a large tile of images capturing an entire sample. If the illumination is not constant over a large area, the fluorescence intensities will not represent the inherent fluorescence but rather the location within the image. Thus, WG3 aims at defining a set of universal protocols to assess the uniformity of illumination (i.e. ‘field flatness’) over the FOV of any photon-based imaging system and allow for correction of any non-uniformity. These protocols will identify the necessary tools, the procedures required to perform the measurements and the analysis methods required for their interpretation. WG3 will also define criteria regarding the cut-off for acceptability and the need for correction. A database will be created with ideal images of uniform FOVs from different microscope modalities and settings to be used as a reference by the community and validate the protocol and criteria.
3.3.4 |. WG4 system chromatic aberration and co-registration
Chromatic aberration refers to possible artefacts caused by the wavelength dependency of an imaging system’s optical properties, with the result that two colours arising from the same physical location within the sample appear separated in the image. Such artefacts result from the optical design of the system (e.g. well-corrected vs. poorly corrected objective lenses), the manufacturing tolerances of the system components and the alignment of the optical components. Co-registration accuracy more generally refers to the system’s ability to co-localize dyes of different wavelengths emitting from the same object within a particular experimental setup. This can be affected by both the experimental setup and the system architecture. Working within the assumption that microscope users are ultimately interested in co-registration accuracy, WG4 aims to use sub-resolution and larger multi-coloured bead preparations to measure co-registration accuracy. Alternative tools for performing these measurements will also be evaluated. WG4 will compare reproducibility across different laboratories to determine the best protocol.
3.3.5 |. WG5 lateral and axial resolution
This WG focuses on the microscope lateral and axial resolution, which is essential for reporting size measurements of near-resolution limit objects or distances between them. Resolution is highly related to the objective quality but depends strongly upon other parameters ranging from the sample preparation to the signal detection.
The WG aims to define sample preparation, image acquisition and data analysis protocols for testing resolution, first using sub-resolution fluorescent bead preparations and second employing alternative pattern-based methods. Monitoring the resolution (Point Spread Function in the case of beads) over time will identify possible new aberrations in the system. Pooling the data from multiple laboratories within the WG will allow them to compare reproducibility for sample preparation, data acquisition and data analysis tools, thereby determining a robust, easy-to-use protocol to propose to the community.
3.3.6 |. WG6 stage and focus—precision and other
The mission of WG6 is to ensure the performance and QC of stage platforms and sample holders and the optomechanical focus of the optical system as it relates to X, Y, Z movement, stability, reproducibility and repeatability. The goals are defining the terms typically used to address QC, providing standardization of the measurements and testing protocols and establishing performance benchmarking levels. Though initially applying these towards confocal light microscopy, the WG will endeavor to include details for standard incident light fluorescence microscopy and more advanced techniques such as super-resolution and light-sheet microscopy.
3.3.7 |. WG7 microscopy data provenance and QC metadata
For proper interpretation, microscopy images must be accompanied by both human-readable (i.e. Materials and Methods sections) and machine-readable (i.e. metadata) descriptions of all steps leading to image formation (i.e. ‘data provenance’ metadata) as well as by QC metrics detailing the illumination, detection, chromatic, optical and mechanical performance of the microscope.7 Nevertheless, no universally accepted community guidelines exist defining what ‘data provenance’ and QC metadata should be reported for distinct types of imaging data. Therefore, the metadata automatically recorded by different commercial microscopes can vary widely, posing a substantial challenge for microscope users to create a bona fide record of their work. To meet these challenges, the 4D Nucleome (4DN)50 Imaging Working Group and the BINA QC-DM WG (https://www.bioimagingna.org/qc-dm-wg) have developed a tiered framework of Microscopy Metadata specifications that provide a suite of extension of the OME Data Model16 that scale with experimental complexity and requirements, and are specifically tailored at enhancing comparability and reproducibility in light microscopy. WG7 aims to systematically evaluate the structure and semantics of the initial 4DN-BINA-OME (NBO) extension proposal7–9 and to launch a coordinated outreach and training strategy towards reaching a community-wide consensus around the proposed metadata specifications and the adoption of these specifications by public image data repositories.14
3.3.8 |. WG8 White Paper
The remit of WG8 is to relay both short- and long-term goals of QUAREP-LiMi by the publication of a set of White Papers to communicate and seek cooperation from the community. The principal aim of these White Papers is to promote QUAREP-LiMi to (1) Prospective new members: to actively engage with the work of QUAREP-LiMi; (2) Imaging scientists and bioimage analysts: to raise awareness of QC issues; (3) Group Leaders/Principal Investigators: to engage a critical mass of academic researchers (top–down); (4) Research scientists (graduate students and post-doctoral researchers) with expertise in the specialized WG topics and imaging scientists: to influence the research group leaders (bottom–up); (5) Scientific publishers: to raise the quality of methods reporting and rigor and reproducibility in publications; (6) Leads (CEO/directors) of companies and commercial application specialists: to work alongside QUAREP-LiMi to facilitate ease of measurements and reporting and (7) Prospective funders (funding agencies, private sponsors): to support the work of this initiative.
3.3.9 |. WG9 overall planning and funding
The principal aim of WG9 is to coordinate and promote the activities of QUAREP-LiMi. Within this WG, there is representation from all other WGs in addition to key global, regional, and national microscopy communities. WG9 will also liaise directly with corporate partners, scientific publishers and funding bodies.
WG9 will focus on the following activities: (1) Ensure that the output of QUAREP-LiMi achieves maximum impact within the imaging community by raising awareness of the need for QC across all stakeholders in light microscopy (via white paper, website and publications); (2) Seek to obtain support from our corporate partners (microscope manufacturers/technology companies); (3) Obtain funding and support from national bodies, scientific publishers and learned societies to help cover the activities of QUAREP-LiMi (allow us to stage physical meetings, cover publication costs, help with organization and add impact); (4) Keep stakeholders informed and share information through a regularly updated website and tools database (towards internal and external communication and impact) and (5) Coordinate QUAREP-LiMi WGs and future QUAREP-LiMi meetings (virtual and physical).
3.3.10 |. WG10 IQ
Good IQ is essential for any subsequent image processing, analysis and presentation steps. However, the notion of IQ is very broad and encompasses concepts that might differ between various microscope types. The aims of WG10 are (1) to define a set of basic IQ parameters (quantitative criteria, metadata and QC metrics8) for light microscopy; (2) to weight the significance of the individual parameters for different experimental techniques and microscope types and (3) to facilitate the assignment of a microscope- and experiment-specific QC rating to individual images. Ultimately, WG10 will work to summarize the upshot of these steps in the form of easy-to-use workflows. The integration of IQ ratings as part of the metadata8 associated with every imaging data set is a long-term goal of this WG.
3.3.11 |. WG11 microscopy publication standards
WG11 will work together with scientific publishers to promote the adoption of best practices in the reporting of metadata (for both image acquisition and analysis) throughout scientific journals and books. Only by ensuring all relevant constituents (researchers and imaging scientists submitting publications and designing research; editors, scientific publishers and reviewers monitoring and preparing publications and funders, researchers and educators evaluating and disseminating publications) are working in concert can we raise the bar to ensure reproducibility in imaging experiments. WG11 will focus on the following activities: (1) inform scientific publishers of the standards and metadata put forward by the other QUAREP Working Groups; (2) liaise with and encourage individual journals to modify their imaging guidelines to align with these recommendations; (3) work together with the scientific publishers to enforce high standards of imaging metadata reporting in all research works accepted for publication; (4) facilitate the involvement of technical reviewers with significant microscopy expertise during the review of papers that rely heavily on imaging techniques; (5) work together with publishers to promote and increase the appropriate acknowledgement and co-authorship of imaging scientists and core imaging facilities in publications; (6) encourage publishers to compel authors to make raw imaging data available if, and when, required for validation of published research and to make reasonable suggestions regarding duration of storage of raw imaging data relevant to published results; (7) propose minimum standards for figure quality, figure colour selection, scale bars, inserts, annotations and labelling, in order to render all microscopy figures easily interpretable by experts and non-experts alike.
3.4 |. Future steps and perspectives
The ultimate goal of QUAREP-LiMi is to benefit everybody in the light microscopy community. Our future strategy can be subdivided into medium-term goals to be achieved within the next few months and long-term goals to be realized within the next 1–2 years.
3.4.1 |. Medium-term goals
Growth and diversification of QUAREP-LiMi member body:
The vast majority of current QUAREP-LiMi members are imaging scientists representing academic labs, core imaging facilities and standardization organizations. An overview of QUAREP-LiMi’s members is listed on our webpage. (https://quarep.org/members/) For the mission of QUAREP-LiMi to be successful, it is imperative to achieve greater engagement with industry (currently 17 companies with 25% of the total members), scientific publishers, funding agencies, commercial microscope manufacturers and observers of such. Moreover, a high priority is to achieve a better worldwide representation of the imaging community by including more members outside North America and Europe.
Establishing a consensus of accepted guidelines within the WGs:
Each WG working towards a defined QC method and overall microscopy metadata specification will finalize a proposed solution and methodology for their topic. They will present this to the entire QUAREP-LiMi community (see Section 3.3) for evaluation, including some beta-testing. This will result in revised versions of individual WGs’ proposals submitted for final approval by the imaging community. The final approval of guidelines and methods will be a community decision and result in documented and openly accessible, free QC protocols and where possible, software. This kind of workflow is adopted and slightly modified from the proven workflow of the ISO.
3.4.2 |. Long-term goals
Dissemination of new guidelines to the scientific community and its stakeholders:
The QUAREP-LiMi guidelines will be published like those for RNAseq, proteomics, microarrays, and so forth,4,51–53 and highlighted at national and international scientific meetings. Furthermore, the inclusion of the guidelines in teaching materials and training courses (both for microscope users and for imaging facility staff,54 as well as for commercial developers and corporate partners) will ensure their wide-spread adoption among microscope users and developers. The QUAREP-LiMi guidelines, initially developed for widefield and confocal microscopy, are intended to be adopted and extended to other imaging modalities, such as light-sheet and super-resolution microscopy.
Implementation of the new guidelines within the community:
By engaging the entire imaging community throughout the development of new guidelines and specifications, we strive to implement a standard procedure for end-users and to promote the integration of the guidelines into commercial microscope hardware and software as well as freely available microscope and image analysis software. As this process becomes easier and more streamlined, the long-term aim is to enable the automatic measurement of these metrics. Thus, early engagement with commercial manufacturers and other developers is critical to ensure simple approaches towards acquiring QC data.
Working with stakeholders to promote the implementation of new guidelines:
A straightforward solution to encourage the uptake of minimal QC metrics would be for scientific publishers to adopt these standards as part of their standard requirements to accept material for publication. Ideally, access to raw data should also be provided. Such initiatives are gaining ground and being backed at the national level by funding bodies. Databases to store the QC data for each microscope should enable creating simple reports to accompany published experimental results, demonstrating the system’s real-time performance across the data collection period. In other fields (genomics, transcriptomics, proteomics, etc.), public data repositories have played a key role in implementing community-proposed standards and accelerating their adoption. Data repositories and journals worked together as de facto enforcers of a standard. Journals required data related to peer-reviewed manuscripts to be submitted to repositories. In contrast, repositories themselves enforce standards either at the time of submission or by converting submitted data to a standardized format for publication and download. With the establishment of several bioimage data publication systems,20,21,55,56 there is now an opportunity within the field of light microscopy to use a similar approach based on these successes.
Modification of the existing ISO and establishing of new ISO standards based on guidelines developed by QUAREP-LiMi:
The final formalization of the QUAREP-LiMi guidelines will be achieved by their inclusion in new editions of the respective ISOs.24–26,57 While we expect that microscope QC will be a constantly evolving area, as new technologies become mainstream, the establishment of fixed versioning of the current guidelines for widefield and confocal systems will provide the community with a strong baseline for further developments of ISOs. It will cover the vast majority of current microscopy-based research.
4 |. CONCLUSION
The international nature, size and breadth of QUAREP-LiMi is critical for its mission, which will only succeed with sufficient buy-in from all stakeholders. The first step will be to reach a consensus between microscope and system component manufacturers, users and microscope custodians regarding precisely what needs to be measured, how and at what frequency, taking into account the experiment being performed and the downstream image analysis strategy. Next, a set of common, practical tools to accomplish these measurements must be developed and provided to the entire community. The microscope manufacturers can provide some of these as internal QC tools that align with the QUAREP-LiMi guidelines, thereby facilitating rapid, simple measurements by all microscope users and custodians. Such tools would help the companies ensure their instrumentation’s consistent performance, facilitate the more rapid diagnosis of problems and permit imaging scientists to perform necessary checks and alignments that must currently be performed by dedicated service engineers. Commercial microscope manufacturers can also support this paradigm shift by raising awareness of the importance of imaging QC with their direct customers; thus, their involvement in the QUAREP-LiMi initiative and working groups is crucial.
An equally critical factor in the agreed-upon guidelines’ global adoption will involve education and raising awareness through publications, workshops and meeting presentations explicitly targeted at research and imaging scientists. Since a significant part of many imaging scientists’ responsibilities already lies in maintaining instrumentation at optimal performance, they are typically more familiar with the problems and challenges outlined above. Hence, they have tremendous potential to educate researchers on the importance of imaging QC, the tools available and recommended and to disseminate the QUAREP-LiMi guidelines to their facility users and researchers who have microscopes in their laboratories. Funding bodies and scientific publishers could also encourage adopting these guidelines by requiring their implementation in all imaging-focused research. Scientific publishers can further educate their reviewers in imaging QC or bring in expert technical assessors to interrogate data quality and reliability. Publishers and reviewers should also encourage the sharing of imaging data in public repositories. Finally, repository hosts could help enforcement by automating data quality validation and ensuring that the data are made widely available to the broader scientific community.
QC is costly and requires significant time and effort, but its lack undermines trust in the quality of data, equipment, scientific rigor, reproducibility and data exchange. By providing a clear community-driven way forward and working closely with all stakeholders, QUAREP-LiMi has the potential to drive a culture change. This will benefit the entire community by fundamentally transforming image data quality and reproducibility.
ACKNOWLEDGEMENTS
We thank https://www.somersault1824.com/somersault18:24 BV (Leuven, Belgium) for help with Figure 1. E. C.-S. was supported by the project PPBI-POCI-01-0145-FEDER-022122, in the scope of Fundação para a Ciência e Tecnologia, Portugal (FCT) National Roadmap of Research Infrastructures. R.N. was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) Grant number Ni 451/9-1 - MIAP-Freiburg.
Funding information
The Deutsche Forschungsgemeinschaft (DFG, German Research Foundation), Grant/Award Number: 528 Ni 451/9-1
APPENDIX A
A.1 |. Glossary
| Benchmarking | The comparison of device performance to an accepted standard for a given task. |
| Corporate Partners | Umbrella term including vendors and companies. We chose to use the word ‘partners’ to encourage them to work with QUAREP-LiMi. |
| Data Management | Data Management comprises all disciplines related to managing data as a valuable resource. In particular, image data management can be defined as a process that includes all phases of the image data life cycle, from its ’pre-life’ to its ’after-life’, including experimental procedures, sample preparation, image acquisition, quality control, data-storage, -protections, -finding, -validation, -processing, -analysis, -interpretation, -sharing, -integration, presentation, re-use and archiving. As such, the ultimate goal of Data Management is to ensure the accessibility, reliability, reproducibility and timeliness of image data for its users. In this context, it is important to note that the ‘FAIR Guiding Principles for scientific data management and stewardship’ have been established to assess the quality of data management for scientific data. FAIR data are data that meet principles of findability, accessibility, interoperability and reusability.46 |
| Device Monitoring | The process of overseeing all activities and tasks that must be taken into account to adhere to a given level of standard performance. |
| Facility Microscope User | People using microscopes housed in core facilities (Note: a student or post-doc could be BOTH a facility and non-facility user—the term used would depend on the given context). |
| Good Laboratory Practice | The Principles of Good Laboratory Practice are a managerial quality control system covering the organizational process and the conditions under which non-clinical health and environmental studies are planned, performed, monitored, recorded, reported and retained (or archived). |
| Image Metadata | Metadata are data that provide information about other data. In particular, Image Metadata consist of any and all information that allows imaging results to be evaluated, interpreted, reproduced, found, cited and re-used as established by measurable data quality criteria (i.e. FAIR principles46). As such, Image Metadata includes metadata that document all phases of a typical bioimaging experiment including experimental treatment, sample preparation and labelling, image acquisition, instrument performance and image quality (i.e. Microscopy Metadata), and, last but not least, image processing and analysis (NOTE: As defined Image Metadata includes Microscopy Metadata). |
| Imaging Facility Staff | An umbrella term including directors, managers, staff scientists and technicians working in a recognized facility. |
| Imaging Scientists | An umbrella term for scientists whose job it is to make discoveries with imaging possible. They are typically trained across multiple disciplines and utilize a wide breadth of imaging technologies, (i.e. image data acquisition, analysis and management) for the furtherance of science. An important role of imaging scientists is to interface with others using and developing this technology. For more information: http://www.imagingscientist.com/ http://www.imagingscientist.com/ |
| International Organization for Standardization (ISO) and ISO Standards | The ISO is an independent, non-governmental organization made up of members from the national standards bodies of 165 countries. ISO issues proprietary, industrial and commercial standards to ensure the quality, safety and efficiency of products, services and systems. Relevant standards for the light microscopy community are, for example, ISO 21073:2019 Microscopes - Confocal microscopes - Optical data of fluorescence confocal microscopes for biological imaging.26 |
| Microscopy Metadata | Metadata are data that provide information about other data. In particular, Microscopy Metadata are metadata that document the process of Image Acquisition using a Microscope and the Quality of the resulting Image Data. As such, Microscope Metadata can be subdivided into two sub-categories: ’data provenance’ metadata include microscope hardware specifications and image acquisition settings; while quality control metadata include calibration metrics that quantitatively assess the performance of the microscope at the time of acquisition and therefore allow the evaluation of image quality (NOTE: As defined, Microscopy Metadata are a sub-categoty of Image Metadata). |
| Non-Facility Microscope User | A researcher who uses their own microscope(s) in their own labs—or ‘Individual research groups operating microscopes’ as a collective term. |
| Quality Assessment (QA) | Action performed to ensure the quality of a specific factor involved in image acquisition and analysis. |
| Quality Audit | Quality audit is the process of systematic examination of the quality system used to ensure image acquisition and analysis. It might be performed by internal or external experts, and ideally is comparing system quality over time to identify any issues. |
| Quality Control (QC) | Procedure performed to ensure the quality of all factors involved in image acquisition and analysis. |
| Quality Standards | Accepted level of device performance for a given set of tasks as defined and published by a recognized organization such as the ISO or DIN. |
| Standard Operating Procedures (SOPs) | Established methods to be followed routinely for the performance of specific operations. |
| Tools | Instruments, samples or software used to determine the quality of all factors involved in image acquisition and analysis. |
Footnotes
COMPETING INTERESTS
R.R.B., R.H., D.J.M., C.D.W., S.B., J.B., K.A.B., J.B., R.D., F.E., A.F., W.I.G., G.A.H., C.B.J., S.C.J., J.L., E.R., A.R., V. S., S.S. and D.S. have a financial interest as employees of companies producing, selling or distributing light microscopes or products related or used with light microscopy equipment. A.D.C. has a financial interest in PSFcheck and PyCalibrate.
REFERENCES
Full text links
Read article at publisher's site: https://doi.org/10.1111/jmi.13041
Read article for free, from open access legal sources, via Unpaywall:
https://repository.escholarship.umassmed.edu/bitstream/20.500.14038/44422/2/jmi.13041.pdf
Citations & impact
Impact metrics
Citations of article over time
Alternative metrics
Smart citations by scite.ai
Explore citation contexts and check if this article has been
supported or disputed.
https://scite.ai/reports/10.1111/jmi.13041
Article citations
High-throughput image processing software for the study of nuclear architecture and gene expression.
Sci Rep, 14(1):18426, 08 Aug 2024
Cited by: 1 article | PMID: 39117696 | PMCID: PMC11310328
Successful 3D imaging of cleared biological samples with light sheet fluorescence microscopy.
J Cell Biol, 222(11):e202307143, 17 Oct 2023
Cited by: 2 articles | PMID: 37847528 | PMCID: PMC10583220
Review
Free full text in Europe PMCFully automated point spread function analysis using PyCalibrate.
Biol Open, 12(11):bio059758, 10 Nov 2023
Cited by: 2 articles | PMID: 37815435 | PMCID: PMC10651089
Artificial intelligence in imaging flow cytometry.
Front Bioinform, 3:1229052, 09 Oct 2023
Cited by: 1 article | PMID: 37877042 | PMCID: PMC10593470
Community-developed checklists for publishing images and image analyses.
Nat Methods, 21(2):170-181, 14 Sep 2023
Cited by: 19 articles | PMID: 37710020
Review
Go to all (25) article citations
Similar Articles
To arrive at the top five similar articles we use a word-weighted algorithm to compare words from the Title and Abstract of each citation.
QUAREP-LiMi: a community endeavor to advance quality assessment and reproducibility in light microscopy.
Nat Methods, 18(12):1423-1426, 01 Dec 2021
Cited by: 34 articles | PMID: 34021279 | PMCID: PMC9443067
The future of Cochrane Neonatal.
Early Hum Dev, 150:105191, 12 Sep 2020
Cited by: 5 articles | PMID: 33036834
Scientific basis of the OCRA method for risk assessment of biomechanical overload of upper limb, as preferred method in ISO standards on biomechanical risk factors.
Scand J Work Environ Health, 44(4):436-438, 01 Jul 2018
Cited by: 6 articles | PMID: 29961081
An Experimental Approach to Genome Annotation: This report is based on a colloquium sponsored by the American Academy of Microbiology held July 19-20, 2004, in Washington, DC
American Society for Microbiology, Washington (DC), 02 Oct 2020
Cited by: 0 articles | PMID: 33001599
ReviewBooks & documents
Free full text in Europe PMC
Funding
Funders who supported this work.
Cancer Research UK (1)
Grant ID: 22308
NCI NIH HHS (1)
Grant ID: U01 CA200059
NIBIB NIH HHS (1)
Grant ID: U01 EB021238
The Francis Crick Institute (1)
Grant ID: 10010
