Introduction

The roots of employee (aka work; e.g., Schaufeli & Bakker, 2010) engagement research likely started with theoretical expansions of forms of employee participation and job involvement (e.g., Elloy, Everett, & Flynn, 1991). Like many other constructs within the I-O psychology field, our knowledge of engagement has been at least partially informed by its measurement and documented association with other work attitudes and behaviors. There are currently many engagement scales used for either academic and/or applied purposes (for example, the UWES, ISA, or Q12).

The focal survey discussed in this paper is described in greater detail at a nearby poster, but is primarily informed by Schaufeli et el.’s (2002) structure, with engagement being defined as: a mental state wherein employees: a) feel energized (Vigor), b) are enthusiastic about the content of their work and the things they do (Dedication), and c) are so immersed in their work activities that time seems compressed (Absorption). A differentiating feature of our survey, however, is that these three components are each measured at three attitudinal component strata: d) feeling (e.g., affect), e) thought (e.g., cognition), and f) action (e.g., behavior).

This study examines the impact of item ordering on factor structure using this measure that has been intentionally specified within two different dimensional frameworks: 1) attitudinal, and 2) substantive.

Methods

There were two populations sampled for this study. The first was a snowball sample initiated by I-O Psychology faculty and graduate students which resulted in a total of 282 participants. There were four counterbalanced orderings of item presentations within this administration, as well as an additional 16 candidate items. In the second data collection initiative, Qualtrics panels were solicited. These US workforce representative adults responded to two counterbalanced orderings of the focal 18 items along with 2 additional candidate items. The respondents of the second sample included 343 who responded to attitudinally clustered items and 404 who responded to substantively clustered items.

Materials

Our 20-item engagement measure was crafted to be intentionally complex (each item is intended to load on two constructs). This complexity stems from a crossing of the attitudinal components of affect, cognition, and behavior with the substantive engagement components of vigor, dedication, and absorption. The 6-point response scale is: Strongly Disagree, Disagree, Somewhat Disagree, Somewhat Agree, Agree, Strongly Agree. The item stems as well as their intended scale associations are presented in a concurrent SIOP poster session.

Results

Two omnibus confirmatory factor analyses (CFAs) were imposed on the data conforming to the two scale definitions. Regardless of item ordering, across a collapsed sample (n = 1,029) both models exhibited fair fit (\(χ^2_{substantive}\) = 995.34, df = 132, RMSEA = 0.09; \(χ^2_{attitudinal}\) = 1,103.47, df = 132, RMSEA = 0.09). Additional fit indices for the two models (as well as models run separately within each condition) are presented in Table 1. Figures 1 and 2 are visual representations of the omnibus CFA models. Omnibus data α’s (across all conditions) were 0.81 (Absorption), 0.91 (Dedication), 0.78 (Vigor), 0.78 (Affect), 0.89 (Cognition), and 0.83 (Behavior).

The anticipated order effects emerged as predicted from all six clustered item ordering conditions, but were quite moderate (for example, \(Δχ^2_{Cond1}\) = 9.55, \(ΔAIC_{Cond1}\) = 10.53). Measurement invariance was evaluated across a series of progressively greater restrictive conditions. Because our six conditions were realized across two qualitatively different sampling procedures, we applied our analyses of measurement invariance twice. The initial snowball sample across four counterbalanced conditions can be reasonably characterized as exhibiting strong/scalar invariance for the attitudinal structure and marginally attaining scalar invariance for the substantive CFA. Secondly, the follow-up Qualtrics panel respondents there was evidence of weak/metric invariance for the substantive CFA across the two Qualtrics item orderings (\(ΔΧ^2\) = 13.84, df = 15, p = .54), but a lack of invariance for the attitudinal model.

Table 1: Summary fit indices across item ordering conditions

Condition	Model	Chisq	df	RMSEA	SRMR	CFI	TLI	AIC
Condition 1	3-factor substantive	300.86	132	0.14	0.11	0.68	0.63	3,282.88
	3-factor attitudinal	290.33	132	0.14	0.11	0.70	0.65	3,272.35
Condition 2	3-factor substantive	310.01	132	0.15	0.10	0.71	0.66	3,257.45
	3-factor attitudinal	322.52	132	0.15	0.10	0.69	0.64	3,269.96
Condition 3	3-factor substantive	252.07	132	0.12	0.10	0.78	0.74	3,510.32
	3-factor attitudinal	275.74	132	0.13	0.10	0.73	0.69	3,534.00
Condition 4	3-factor substantive	224.96	132	0.10	0.09	0.82	0.79	3,421.64
	3-factor attitudinal	228.99	132	0.10	0.09	0.81	0.78	3,425.66
Condition 5	3-factor substantive	549.8	132	0.10	0.05	0.90	0.89	14,932.57
	3-factor attitudinal	497.9	132	0.10	0.04	0.92	0.90	14,880.67
Condition 6	3-factor substantive	468.02	132	0.09	0.05	0.91	0.90	17,953.02
	3-factor attitudinal	610.6	132	0.10	0.06	0.88	0.86	18,095.61
Overall	3-factor substantive	995.34	132	0.09	0.05	0.91	0.90	46,915.05
	3-factor attitudinal	1,103.47	132	0.09	0.05	0.90	0.88	47,023.18

Discussion

Through counterbalancing the order of item presentation, we provide response cues regarding structural priority. Analyses of measurement invariance reveal generally positive support for invariance across item orderings. When items are presented “together” in terms of conceptual cohesion (either grouped by attitudinal dimension or substantive dimension), the CFAs yielded slightly better fit indicators than when items were not presented in conceptually cohesive groupings. This was true across different levels of cohesive emphasis. There is evidence of at least partial invariance within both administrations. However, the idea of invariance has greater support within the initial four administration conditions than with the second sampling method.

It is certainly feasible that the content cue becomes more powerful as respondents become more fatigued (e.g., with truly longer assessments). In a similar consideration, it is possible that our scales were SO short that the respondents did not pick up on the cue (“hey – these are all asking about how I feel [or think, or act]”). Longer scales would represent a stronger manipulation in the sense that there is greater opportunity for the cue to be perceived. This poster was created via posterdown (Thorne, 2022).