Published article

Threats to the validity of eye-movement research in psychology
Behavior Research Methods
Jacob Lund Orquin & Kenneth Holmqvist
 

In this paper the authors present nine common mistakes in eye-movement research. These mistakes were mostly identified while reviewing or editing own or other researchers’ manuscripts.

Comparing apples to oranges. Many researchers wish to compare eye movements between two or more objects in a visual scene, for instance, do people look more at other’s eyes or mouths during conversation. Unfortunately, many such comparisons suffer from confounding from object position, size, salience etc.

Analyzing multiple metrics. Some eye-trackers (read Tobii) provides a wealth of possible metrics and many researchers uncritically analyze several of these highly correlated metrics and inflate their type one error rate. 

Total fixation duration. This metric, also called total dwell time, is best avoided. Our analyses show that it is in the best case undiagnostic and in the worst case actually misleading.

Too small objects. All eye-trackers are noisy and report slightly offset eye positions. This noise sets a lower boundary for how small your object of interest can be. Even for the high resolution Eyelink 1000 we suggest a minimum object size 3.2 degrees of visual angle.

Hidden defaults. There is a tendency for eye-tracking researchers to propagate others’ mistakes by default, such as excluding fixations shorter than 300 msec. bad idea.

Free vs. fixed exposure time. Unless you have a good reason, it is best to avoid fixed exposure time since it can bias cognitive processes.

Assuming eye-mind relationship. Eye-trackers do not measure attention unfortunately, although many researchers seem to believe so. Be careful before claiming such relationships.

Undersampling of stimuli. In some disciplines, it is common to use a single critical trial. Unless the stimuli is carefully controlled, this easily confounds the results. 

Generalization of eye movements. It is disappointingly difficult to generalize eye movements from the lab to the real world. One solution is to focus on causal mechanisms behind eye movements instead.