Engagement as Attitude: A Bifactor Measure of Employee Engagement

Psychometrics
R
Co-authored manuscript developing and validating a new 18-item engagement survey, with CFA and item-order analyses in R.
Published

June 10, 2023

Full title: Engagement as attitude: Development and initial validation of an intentional bi-factor measure of employee engagement Authors: John Kulas, Renata Garcia Prieto Palacios Roji, Casey Osorio-Duffoo, Morgan Russell, & Mike DeFabiis Code: jtkulas/EngagementPaper

About the paper

Engagement is usually defined by three sub-dimensions: vigor, dedication, and absorption. Attitudes are usually split into cognitive, affective, and behavioral components. This paper develops and validates an 18-item measure that intentionally crosses both models, so every item captures one dimension from each.

Data & methods

  • Snowball sample: 282 working adults, with items shown in four counterbalanced orders
  • Qualtrics panel: 343 adults saw items grouped by attitude; 404 saw items grouped by substantive dimension
  • Reliabilities ranged from α = 0.78 to 0.91 across the six subscales
  • Confirmatory factor analyses across 1,025 respondents showed fair fit for both the substantive and attitudinal structures (RMSEA = 0.09)

Lollipop chart of reliability (Cronbach's alpha) for six subscales, from 0.78 (Vigor, Affect) to 0.91 (Dedication).

Key findings

  • Grouping items by dimension gave respondents only slight cues about the structure; the item-order effect was small
  • Measurement invariance held across the four snowball-sample conditions, but not across all six conditions once the panel samples were added
  • The main source of misfit was the scale’s single reverse-coded item (“Thinking about work saps my energy”)

My contributions

  • Automated cleaning of messy free-text survey responses (for example, “hours worked”) so the code works on new data as it arrives
  • Recoded reverse-scored items, split and merged data across survey conditions, and ran item analysis
  • Built interactive plots of results
  • Ran the confirmatory factor analyses and fit the bifactor model, including troubleshooting models that wouldn’t converge
  • In 2023, cleaned up the code base, documented each data source, and got the APA-formatted manuscript knitting to PDF

Tools

R, R Markdown, papaja (APA-formatted manuscripts), lavaan, tidyverse, LaTeX, Git/GitHub