This is something we are often asked. In answering it, the critical issue to bear in mind is that validity has to do with the conclusions, inferences, or decisions we want to make based upon the assessment, not the assessment itself. Consider a typing test.  If it is reasonably constructed, it is pretty likely that it measures typing speed and accuracy. However, is it valid? Without knowing what the test is going to be used for, we don't know. If we are using it to select administrative assistants, we are likely on the right track if we assume keyboarding is important for job success in this role. However, by the same token, it is likely to be invalid for selecting groundskeepers.

Commercially available assessments have technical documentation that details how the test measures what it is designed to measure.  And this is certainly something that enters into the validity equation. However, the key question about validity has to do with the intended use of the assessment. The underlying question really isn't whether a test itself is valid but rather Valid for What?

Reid Klion

The Super Bowl, coming up in about a week and a half, will be a re-match of the 2008 game with the New England Patriots facing the New York Giants. An interesting fact is  how different the rosters of the two teams are today from four years ago. While the quarterbacks and coaches are the same, the Patriots only have 5 starters (of 24, counting the punter and kicker) from the 2008 team while the Giants have 9 remaining starters. (Going a bit further, there are only 7 Patriots and 16 Giants on the team roster in total today who were there 4 years ago.). While this is in part testament to the relatively short tenure of most players in the NFL (around 3.5 years on average) and, admittedly, two very good quarterbacks, it is even stronger evidence about the importance of selecting and training new talent. So while many of the individual players are different, the strength of the talent management skills of both these organizations is a constant thread.

Reid Klion

When we help a client to implement an assessment program, we are sometimes asked to use a “profiling” approach which involves giving the assessment to handful of the best performing employees, look at their scores,  and then use this to create a benchmark for hiring new employees. While this model is intuitive, quick, and easy to execute, it has a number of drawbacks. From a measurement perspective, the first challenge is that it is virtually impossible to create norms that are stable and meaningful when they are only based on a handful of cases. Secondly, since most assessments employ a large number of scales and sub-scales, it is rather difficult to decide which scales are actually predictive of performance without additional analyses of the job at hand and what it requires. (Alternately, it is possible that the selected assessment many not measure what actually differentiates high and low performing employees unless when have a sense of what the job requires when we initially select the assessment.)  Another factor to consider is that an employee’s skills may change and improve as a result of on the job experience. That is, successful employees may look significantly different after several months on the job than they did when they were hired. Finally, unless we also make a systematic job task analysis, we may lose sight of the likelihood that there are several ways for someone to be successful in the job.

The take-away lesson is that while “profiling” may give us a sense of the competencies of some of our most successful employees, it is not a sufficient foundation upon which to base an assessment program. Unless we take additional steps to fully understand the requirements of a position, we are likely to implement a solution that is less than fully effective. In the final analysis, our task is to match individuals to a job, not individuals to each other.

Reid Klion

While there have been some slight improvements in the national employment rate (now down to 8.5% from a high of 10% in October 2009), millions of people are still seeking work. A study was recently released which examines the relationship between college major and unemployment. Recent grads with majors in healthcare, education, and agriculture/natural resources had unemployment rates at or below 7%. The major with the worst prospects was architecture with nearly 14% of its recent grads still looking for work. Also interesting to note is that individuals with graduate degrees tend to have very low levels of unemployment (though these folks almost by definition are older and more experienced). Finally, the study looked at long term prospects for income. Here, individuals who majored in the sciences, engineering, or business tended to do much better from a wage perspective than those with a background in the humanities, psychology, and education. 

Reid Klion

I just saw another round of discussions in the press as to whether student scores on standardized tests should be used to evaluate teachers. The debates often devolve into all-or-nothing propositions such as “yes, we need to fire teachers whose kids do poorly on tests” vs.  “only a teacher can evaluate another teacher and test scores don’t mean much of anything.” This also takes place in a context where some schools devote very substantial amounts of time “teaching the test” and searching for ways to improve test scores (which in a few unfortunate cases involve cheating). While some of this reflects the irrationality that often seems to emerge whenever public education policy is discussed, it also highlights the larger issue of what is known as the “criterion” problem in I/O psychology.

A topic I have written about previously, we face a major challenge (which is often ignored) in measuring work performance in a meaningful way, especially for higher level positions. We often lack good performance metrics and even these can be contaminated by a number of external factors. For example, sales revenue often reflects a number of things such as territory, market served, product mix, and tenure in addition to sales ability. (Also, consider the complexity of using student test scores when it is not uncommon in some urban districts for up to a quarter of children to change schools at least once during the course of a year.) As a result, we often need to use performance ratings which are prone to “halo effects” (e.g., if someone is liked, they are rated high on all attributes) and a lack of differentiation amongst employees (that is, other than a few exceptions, pretty much everyone tends to be highly rated).
 
In the case of education, it certainly seems reasonable that student test scores be considered as one factor in teacher evaluation but also indicators such as peer ratings, student input, and repeated observation by supervisors also taken into account. Just as we often advise that a single test score should not be used to ultimately determine whether someone should be hired or not, similar logic should apply the evaluation of job performance.

Reid Klion

The Association of Test Publishers (ATP) will hold its annual Innovations in Assessment Conference on February 26 to 29 in Palm Springs, CA. The program is now posted and registration information available.

Reid Klion

In an informal discussion letter issued late last year, the EEOC outlined how requiring a high school diploma may lead to issues with the Americans with Disabilities Act. The EEOC points out that if an employer requires a high school diploma and that requirement removes from consideration someone

who is unable to graduate because of a learning disability that meets the ADA’s definition of "disability," the employer may not apply the standard unless it can demonstrate that the diploma requirement is job related and consistent with business necessity. The employer will not be able to make this showing, for example, if the functions in question can easily be performed by someone who does not have a diploma.

This clearly follows the long held principle that selection criteria must be based upon a “business necessity.”

Additional, since the ADA is involved, employers also need to consider the principle of “reasonable accommodation.”  Here, the EEOC goes on to say:

Even if the diploma requirement is job related and consistent with business necessity, the employer may still have to determine whether a particular applicant whose learning disability prevents him from meeting it can perform the essential functions of the job, with or without a reasonable accommodation. It may do so, for example, by considering relevant work history and/or by allowing the applicant to demonstrate an ability to do the job’s essential functions during the application process. If the individual can perform the job’s essential functions, with or without a reasonable accommodation, despite the inability to meet the standard, the employer may not use the high school diploma requirement to exclude the applicant.

None of these are new principles or concepts. However, it is important remember that whenever screen out criteria are used for job selection purposes, it is critical that they be job-related and care must be taken that they do not  take actions that unduly discriminate against individuals with disabilities without consideration of reasonable accommodations.

Reid Klion

Best wishes to everyone for the New Year. December 31 is a common deadline for college applications. While the process of applying to college has gone on for many decades, it has also evolved. A much reviled part of the process is completing the typical standardized tests, primarily the SAT and ACT. There have been lengthy debates about the benefits and drawbacks of standardized tests to guide admissions decisions. However, virtually all colleges and universities use them because they provide an objective and common ground for evaluating potential students whereas letters of recommendation, GPA, and access to advanced classes can vary quite dramatically from school to school and community to community.

Before the advent of standardized testing, universities wrote their own admissions tests. A colleague recently sent me a link to Harvard’s 1869 admissions exam which included heavy doses of Greek and Latin as well as history, geography, and math. Questions included specifying the source of the Danube, Volga, Ganges, and Amazon rivers as well as proving that “the perpendicular from the center of a circle upon a chord bisects the chord and the arc subtended by the chord.” While the current regimen of ACTs and SATs is admittedly less than pleasant, it is very likely a major improvement from the very content specific exams that were used before their advent.

Reid Klion

The IPAC Call for Proposals has been posted and will be open until the end of January 2012 for its annual conference to be held July 22-25, 2012 in Las Vegas.

IPAC is an organization of applied personnel selection and assessment professionals to which a number of us at pan belong. IPAC focuses it efforts on increasing the awareness and application of empirically-based methods to HR practice. In addition to an annual conference, IPAC offers webinars as well as white papers on topics related to assessment and selection.

Reid Klion

There was a recent study published in Personnel Psychology looking at the development of executive leaders. The research question examined the relative roles of cognitive ability, personality factors, and work experience in predicting strategic thinking (which was measured by a series of five assessment center exercises). Based upon a sample of over 700 executives at an international consulting firm, it was found that cognitive ability (incidentally assessed by the Watson- Glaser Critical Thinking Appraisal and the Wesman Personnel Classification Tests, both published by Pearson and available on pan) was by far the best predictor of strategic thinking. Second most important was work experience. More specifically, it was found the broader the experience, both in terms of range of work activities and level of leadership, the more beneficial it was. The personality factors, openness to experience and extraversion, also made statistically significant contributions but to a much lesser degree. The takeaway lesson from this study is that if you are looking to develop individuals who can become strategic leaders, it is most important to find the brightest people and then provide them with a broad array of diverse work experiences.

Reid Klion