Are there differences in PathwayU assessment scores by gender or race?

 

Impact of Gender and Race on Scores

With heightened awareness among educational institutions and employers of social justice concerns, a common question is whether gender or race adversely affects the scores people receive on their PathwayU assessments, and whether any such differences negatively impact their outcomes. Here we clarify whether PathwayU scores show group differences and explain what score differences mean and when they matter.

Are there differences in PathwayU assessment scores by gender or race? Yes and No.

Differences on interest and value dimensions.

• PathwayU’s interests and values measures were originally developed by the U.S. Dept. of Labor. In terms of absolute differences between demographic groups across score domains, PathwayU’s interests measure shows the same small differences between women and men that emerge on other reliable and valid measures of interests—specifically, women score slightly higher on average than men on Social and Conventional interests, and men score slightly higher on average than women on Realistic and Investigative interests. In terms of work values, PathwayU’s measure shows a small gender difference only for Achievement values, with women scoring very slightly higher than men. In terms of race differences, Black respondents score slightly higher than White respondents on Social, Enterprising, and Conventional interests, and on Recognition values. These observed differences are very small in magnitude—generally about one-fourth the size of the differences found between racial groups on certain aptitudes.

Differences in criterion-related validity.

• A key question is whether an assessment used to inform career decision-making predicts outcomes (such as satisfied occupational membership) equally well across groups. The PathwayU research team monitors score patterns from its assessments on an ongoing basis. We recently examined a sample of nearly 1,000 employed adults comparing their interest fit to their current job, a key measure of criterion-related validity of scale scores. We did not find any statistically or practically significant differences between women and men in how well the assessments predict. We also examined scores across race categories, and again found no differences. The complete absence of systematic differences suggests that PathwayU assessments are equally predictive of satisfied occupational membership across gender and race.

Differences in efficacy of PathwayU-driven counseling interventions.

• In a large-scale experiment (with more than 600 students across four campuses) testing the efficacy of a brief career decision-making workshop featuring PathwayU, we found that workshop participants (compared to those in a control group) reported greater career decision confidence, fewer career decision difficulties, and greater confidence that attaining their career goals will result in valued outcomes. Results suggested that size of the effects did not differ across gender, race/ethnicity, or disability status. Results did reveal a difference across self-reported socioeconomic status (SES), with the most positive effects reported by those low in SES. Other research conducted with our educational partners reveals that PathwayU has a significant positive impact on students’ career attitudes and decision-making; this study suggests that these differences are equivalent across key demographic categories, with the exception of being particularly helpful for low-SES students.

Analysis.

• PathwayU’s instruments are not designed to predict how “good” a person will be in a particular job; rather, they predict the extent to which people will find certain types of work interesting, meaningful, and satisfying. Small group differences are found on some PathwayU interests and values dimensions. These effects conform to what is known within the broader body of research on vocational interests and values, and are very small in magnitude—much smaller than what is found when examining group differences in aptitudes. Large systematic differences in scores across demographic groups would present a concern to educational institutions when they correspond to adverse impact. This can happen when institutions use criteria such as high school GPA or placement tests (e.g., the ACT or SAT) to select students for admissions. However, schools typically do not use measures of career interest and values (like those offered by PathwayU), which are intended to guide students to better career choices, for this purpose.

Caution:

• When interpreting scores on a career assessment, it is crucial to recognize that differences between people within a group are much larger—usually five to ten times larger—than differences between groups; generalizations from a group to an individual can never be made responsibly without further information. Furthermore, unlike many other career assessment instruments, PathwayU uses criterion- based rather than norm-based scoring. Scores from a normed-based system tell students how they compare to “people in general.” In contrast, PathwayU’s criterion-based approach shows how a person’s scores correspond to the item response anchors, and also how a person's scores compare to scores on other scales for that person (e.g., "Your score on Social is moderate-to-high indicating that you enjoy Social tasks, and it's your highest interest type"). Since students are essentially compared to themselves in this method, group differences are much less relevant. A criterion-based scoring system also has the important advantage of facilitating more inclusive interpretation strategies than do norm-based systems that use separate-gender norms, which are difficult to interpret with clients who identify as nonbinary.

Recommended Use:

• Questions about differences across groups must move beyond mean differences on particular dimensions and address the question of whether scores on assessments are more effective at predicting outcomes for some groups than others. PathwayU’s assessments have been demonstrated to have no differences in this type of validity across the demographic categories examined to date.
• Finally, it is important to consider how assessments like those within PathwayU are deployed within an Institution or organization. Usually they are embedded within a career intervention such as individual or group counseling, a course, or a workshop. Do some students receive more benefit than others from these interventions, in a way that varies systematically across demographic categories? Our research demonstrates that for PathwayU, the answer to this question is no, with the exception that such interventions are more helpful for students lower in self-reported SES.

Conclusion:

• PathwayU’s assessments show small differences across gender and race on some interest and value dimensions. Systematic score differences across gender and race are a concern when scores are used for admission decisions. Most higher education institutions do not use PathwayU assessments this way. There are no differences across gender or race in the validity of PathwayU’s assessments for predicting outcomes, nor differences in benefits from PathwayU-focused career interventions.