The Implicit Association Test (IAT) is one of the most commonly used measures for the implicit assessment of preferences and attitudes. Nevertheless, the meaning of the IAT effect remains unclear. Since the IAT is based on the speed and accuracy with which different stimuli are categorized in the category to which they belong, the sense of the IAT measure is strongly related to the functioning of the stimuli used to represent the categories. Thus, a model providing a fine-grained analysis at both respondents and stimuli levels would help in clarifying and interpreting the IAT measure. By applying Linear Mixed Effects Models to IAT accuracy and time responses, this study aims to present a modeling framework for IAT data resulting in such a detailed analysis at both respondents and stimuli levels. The proposed models provide a detailed picture of the contribution of each stimulus to the IAT effect, allowing for the identification of malfunctioning stimuli that can be eliminated or substituted to obtain better performing IATs. Additionally, the detailed information on respondents’ performance yields a better understanding of the classic measure of the IAT effect (i.e., the D score). Implications of the results and future research directions, also implying different implicit procedures, are discussed