Consensus Builder

Version 1.1

Introduction

The NIST Consensus Builder (NICOB) serves to combine measurement results obtained by different laboratories or by application of different measurement methods, into a consensus estimate of the value of a scalar measurand. The NICOB qualifies the consensus estimate with an evaluation of measurement uncertainty that captures not only the stated uncertainties associated with the individual measured values, but also any additional component of uncertainty that manifests itself only when these measured values are inter-compared.
In addition, the NICOB can also report the differences between the individual measured values and the consensus value, and the differences between pairs of values measured by different laboratories or methods, in both cases qualifying these differences with evaluations of associated uncertainty. In the context of Key Comparisons, these differences and associated uncertainties are called (unilateral, and bilateral, respectively) Degrees of Equivalence (DoE) (Comité International des Poids et Mesures (CIPM), 1999).
When the reported measurement uncertainties associated with the individual measured values are qualified with the numbers of degrees of freedom that they are based on, these numbers are taken into account as well. In general, the numbers of degrees of freedom convey the reliability of the evaluations of measurement uncertainty, expressing the extent of the underlying evidentiary basis, be it the size of the experimental data or the strength of other information used when producing the evaluations.
For more information see the User's Manual.

Quick Start

Enter Data

  • Labels designating the n participating laboratories (required — character strings comprised of letters or numbers, separated from one another by commas)
  • Measured values x1,..., xn produced by n different laboratories or measurement methods, independently of one another (required — numbers separated by commas)
  • Measurement units to qualify the numerical values of the measured values which are used to label axes of plots (optional — character string)
  • Standard uncertainties u1,..., un associated with the measured values (required — positive numbers separated by commas)
  • Numbers of degrees of freedom ν1,..., νn that the standard uncertainties are based on (optional — positive numbers separated by commas)
  • Coverage probability (positive number between 0 and 1) desired for the coverage intervals (required, default: 0.95).
  • Indication, by means of a checkbox, of whether degrees of equivalence should be computed (default: Not computed).
  • If this box is checked, the following additional inputs appear:
    • Indication, by means of a radio button, of whether degrees of equivalence should be computed as defined in the MRA or based on leave-one-out estimates.
    • Number of bootstrap replicates for degrees of equivalence uncertainty calculation. This is only used for the DerSimonian-Laird procedure (default: 10000); the Hierarchical Bayes and Linear Pool procedures use for this number the sample sizes of their method specific inputs.
Buttons at the bottom of the Enter data page allow the user to load and save configuration files with inputs for the NICOB. Clicking the button labeled Save Configuration File downloads a plain text file named consensus.ncb to the local machine, which specifies the current inputs for the NICOB. To use a previously saved configuration file, search for and select the file using the Browse button.
Alternatively, the NICOB also accepts configuration files with inputs specified as comma separated values and extensions .ncb, .csv, or .txt. Each row of the file designates data from a different laboratory or measurement method, and the file can have two, three, or four comma separated columns. For each row, data should by entered in the order: name (if available), measured value, standard uncertainty, and number of degrees of freedom (if available; missing or infinite degrees of freedom should be entered as Inf).

Choose a method for analysis

The three methods of data reduction implemented in the NICOB are not meant to be interchangeable, and the user should consider their characteristics, including advantages and disadvantages, to determine which may be best fit for the purpose that the results of the data reduction are intended to serve.
For more information see the User's Manual.
  • For the DerSimonian-Laird procedure:
    – If the Knapp-Hartung adjustment is desired, check the corresponding box;
    – To apply the parametric bootstrap for uncertainty evaluation, check the corresponding box. This reveals an input field for the desired number of bootstrap replicates (Suggested value: 10000).
  • For the hierarchical Bayesian procedure:
    – Positive numbers in the corresponding boxes which are used as the medians of the prior distributions for the between-laboratory and for the within-laboratory (or, between-method and within-method) variance components (default: robust indications of spread of the measured values for the between-laboratory variability, and for the laboratory-specific uncertainty)
    – Total number of iterations for the Markov Chain Monte Carlo sampler (default: 250000).
    – Length of burn-in for the Markov chain, which is the number of values discarded from the beginning of the realization of the chain (default: 50000).
    – Thinning rate for the Markov chain (default value is 25, meaning that only every 25th value generated in the chain should be kept).
  • For the Linear Pool:
    – Weights (non-negative numbers separated by commas) to be associated with the different measurement results (default: 1 for all).
    – Size of sample drawn from the mixture distribution of the measurand (default: 100000).

Output

  • Consensus estimate, associated standard uncertainty, and coverage interval for the true value of the measurand;
  • If degrees of equivalence were requested, then estimates, standard uncertainties, and expanded uncertainties for differences between measured values and the consensus value, and between pairs of measured values are reported and depicted graphically.

List laboratory labels, measured values, standard uncertainties, and (if available) numbers of degrees of freedom, separated by commas.

* Required field

Degrees of equivalence





Save configuration file

Fit laboratory effects model using DerSimonian-Laird procedure



DerSimonian-Laird

The consensus estimate is:

The standard uncertainty is:

The dark uncertainty (tau) is:

Download plot

Parametric bootstrap for uncertainty evaluation

The standard uncertainty is:

The coverage interval ranges from:



Download bootstrap output

Unilateral degrees of equivalence

Download unilateral DoE plot

Bilateral degrees of equivalence

Yellow squares (with black asterisks in the center) indicate results that differ significantly from 0 at 95% coverage. Light blue squares indicate results that do not differ significantly at 95% coverage. Dark squares are space-fillers for results when compared to themselves.
Download bilateral DoE plot

Fit using Bayesian method with weakly informative priors

Default is the median of the absolute values of the differences between the measured values and their median
Default is the median of the lab-specific standard uncertainties


Bayesian procedure

Assuming weakly informative prior distributions and allowing for uncertainty in standard uncertainties

The consensus estimate is:

The standard uncertainty is:

The credible interval ranges from:

The dark uncertainty (tau) is:






Download MCMC output Download plot

Unilateral degrees of equivalence

Download unilateral DoE plot

Bilateral degrees of equivalence

Yellow squares (with black asterisks in the center) indicate results that differ significantly from 0 at 95% coverage. Light blue squares indicate results that do not differ significantly at 95% coverage. Dark squares are space-fillers for results when compared to themselves.
Download bilateral DoE plot

Linear opinion pooling

If no weights are entered, default weights will all be equal to 1

Linear opinion pooling

The consensus estimate is:

The standard uncertainty is:

The coverage interval ranges from:



Download linear pool output Download plot

Unilateral degrees of equivalence

Download unilateral DoE plot

Bilateral degrees of equivalence

Yellow squares (with black asterisks in the center) indicate results that differ significantly from 0 at 95% coverage. Light blue squares indicate results that do not differ significantly at 95% coverage. Dark squares are space-fillers for results when compared to themselves.
Download bilateral DoE plot