Reliability and Science

Reliability is something that every scientist, especially in social sciences and biology, must be aware of.

In science, the definition is the same, but needs a much narrower and unequivocal definition.

Another way of looking at this is as maximizing the inherent repeatability or consistency in an experiment. For maintaining reliability internally, a researcher will use as many repeat sample groups as possible, to reduce the chance of an abnormal sample group skewing the results.

If you use three replicate samples for each manipulation, and one generates completely different results from the others, then there may be something wrong with theexperiment.

1. For many experiments, results follow a ‘normal distribution' and there is always a chance that your sample group produces results lying at one of the extremes. Using multiple sample groups will smooth out these extremes and generate a more accurate spread of results.

2. If your results continue to be wildly different, then there is likely to be something very wrong with your design; it is unreliable.

Reliability and Cold Fusion

Reliability is also extremely important externally, and another researcher should be able to perform exactly the same experiment, with similar equipment, under similar conditions, and achieve exactly the same results. If they cannot, then the design is unreliable.

A good example of a failure to apply the definition of reliability correctly is provided by the cold fusion case, of 1989

Fleischmann and Pons announced to the world that they had managed to generate heat at normal temperatures, instead of the huge and expensive tori used in most research into nuclear fusion.

This announcement shook the world, but researchers in many other institutions across the world attempted to replicate the experiment, with no success. Whether the researchers lied, or genuinely made a mistake is unclear, but their results were clearly unreliable.

Reliability and Statistics

Physical scientists expect to obtain exactly the same results every single time, due to the relative predictability of the physical realms. If you are a nuclear physicist or an inorganic chemist, repeat experiments should give exactly the same results, time after time.

Ecologists and social scientists, on the other hand, understand fully that achieving exactly the same results is an exercise in futility. Research in these disciplines incorporates random factors and natural fluctuations and, whilst any experimental design must attempt to eliminate confounding variables and natural variations, there will always be some disparities.

The key to performing a good experiment is to make sure that your results are as reliable as is possible; if anybody repeats the experiment, powerful statistical tests will be able to compare the results and the scientist can make a solid estimate of statistical reliability.

The Definition of Reliability Vs. Validity

Reliability and validity are often confused, but the terms actually describe two completely different concepts, although they are often closely inter-related. This distinct difference is best summed up with an example:

A researcher devises a new test that measures IQ more quickly than the standard IQ test:

• If the new test delivers scores for a candidate of 87, 65, 143 and 102, then the test is not reliable or valid, and it is fatally flawed.

• If the test consistently delivers a score of 100 when checked, but the candidates real IQ is 120, then the test is reliable, but not valid.

• If the researcher's test delivers a consistent score of 118, then that is pretty close, and the test can be considered both valid and reliable.

Reliability is an essential component of validity but, on its own, is not a sufficient measure of validity. A test can be reliable but not valid, whereas a test cannot be valid yet unreliable.

Reliability, in simple terms, describes the repeatability and consistencyof a test. Validity defines the strength of the final results and whether they can be regarded as accurately describing the real world.