The Definition of Reliability - An Example

Imagine that a researcher discovers a new drug that she believes helps people to become more intelligent, a process measured by a series of mental exercises. After analyzing the results, she finds that the group given the drug performed the mental tests much better than the control group.

For her results to be reliable, another researcher must be able to perform exactly the same experiment on another group of people and generate results with the same statistical significance. If repeat experiments fail, then there may be something wrong with the original research.

Testing Reliability for Social Sciences and Education

In the social sciences, testing reliability is a matter of comparing two different versions of the instrument and ensuring that they are similar. When we talk about instruments, it does not necessarily mean a physical instrument, such as a mass-spectrometer or a pH-testing strip.

An educational test, questionnaire, or assigning quantitative scores to behavior are also instruments, of a non-physical sort. Measuring the reliability of instruments occurs in different ways.

Test - Retest Method

The Test-Retest Method is the simplest method for testing reliability, and involves testing the same subjects at a later date, ensuring that there is a correlation between the results. An educational test retaken after a month should yield the same results as the original.

The difficulty with this method is that it assumes that nothing has changed in that time period. Staying with education, if you administer exactly the same test, the student may perform much better because they remember the questions and have thought about the questions.

How many times have you left an exam and, after a couple of hours, thought; “How could I have been so stupid - I knew the answer to that one!” Of course, next time, you will get that question right, meaning that the test is unreliable.

For this reason, if you have to retake an exam, you will be faced with different questions and may be marked a little more strictly to take into account that you had extra time to revise. This is not the complete picture, because the two exams will need to be compared, to ensure that they produce the same results. This shows the importance of reliability in our lives and also highlights the fact that there is no easy way to test it.

Internal Consistency

The internal consistency test compares two different versions of the same instrument, to ensure that there is a correlation and that they measure the same thing.

For example, sticking with exams, imagine that an examining board wants to test that its new mathematics exam is reliable, and selects a group of test students. For each section of the exam, such as calculus, geometry, algebra and trigonometry, they actually ask two questions, designed to measure the aptitude of the student in that particular area.

If there is a high internal consistency, and the results for the two sets of questions are similar, then the new test is likely to be reliable. The test - retest method involves two separate administrations of the same instrument, internal consistency measures two different versions at the same time.

A horribly complicated statistical formula, called Cronbach's Alpha tests the reliability and compares the various pairs of questions but, luckily, computer programs take care of that and spit out a single number, telling you exactly how reliable the test is!