Evaluating Evaluations: A Meta-Evaluation Checklist by Michael Scriven
Abstract
Michael Scriven provides descriptive criteria for evaluation based on merit. The method is termed by the author as meta-evaluation checklist. The objective of the checklist is to provide a comprehensive and easy to use tool for professionals who may not necessarily be evaluators because of the use of easy-to-understand terms (Scriven, 2011, p.1). The checklist has six components or criteria as is referred to in the paper.
It includes validity, credibility, clarity, propriety, cost-utility, and generalizability. The first five criteria have “non-zero bars” i.e. pass-mark levels whose corresponding inadequate performance cannot be offset by good performance on other components. Generalizability has no level of achievement set for it and the tool does not depend on it for determination of merit but earns bonus to the work is being (evaluand).
Review of MEC’s Criteria of Merit
Validity is a major evaluation component of the tool (Ibid, p.2). It has four levels of achievement (bars) of which two are dominant. They include: contextual constraint (determination of the kind of evaluation required); coverage and correctness (probability of truth in the conclusions); reliability; and robustness. The tool assesses the validity of different types of work using a comprehensive composition of sub-criteria or bars. The tool, therefore, provides more accurate results on the basis of this consideration.
The second item is credibility (Ibid, p.2). This measures the relevance of the work or process being evaluated to stakeholders, staff and intended clients (audiences). By defining the credibility of the process or project, the tool reflects the possibility of bias (e.g. conflict of interest) and the level of expertise utilized. The third component is clarity. This measures the combination of concision and comprehensibility to stakeholders, staff clients, and audiences. It determines the level of acceptance and implementation. This component enables MEC to outperform other tools such as the Yellow book, KEC and PES.
The fourth component is propriety which determination of the ethical, cultural, and legal appropriateness of a process (Ibid, p.5). This covers issues of consent, privacy, timeliness, and the lack of consideration of gender, religion, age, ethnicity, race, and sexual orientation. The fifth component is cost-utility which determines the economic nature of process or work. It measures cost-benefit analysis. Generalizability is the sixth component which, in the tool, is not considered a major determinant factor but a source of bonus (Ibid, p.6). This measures re-usability, exportability and sustainability of the process or work. The sum of achievements in the 6 components provides an estimate of the merit and value of the work or process being evaluated. Component six determined is considered a bonus-dimension because the determination of merit or value can attain the highest grade without the inclusion it inclusion.
Conclusion
In general Meta-Evaluation Checklist (MEC) is a valuable process because it relies on a tool that is less dependent on small errors or variations of measurement of components involved. MEC provides a good lesson to evaluators that accurate evaluation depends on a specific and comprehensive measurement of the various dimensions of the work. However, the issue of generalizability is an important issue. For instance, the issue of sustainability is an important issue in all kinds of work and projects. Work is intended to benefit both the current and future situations. There, a recommendation for change should be that the sixth component should have enough weight to influence the total outcome of an evaluation exercise.
Reference
Scriven, M. (2011). “Evaluating Evaluations: A Meta-Evaluation Checklist (9th ed.). Claremont,
CA: Claremont Graduate University
Is this your assignment or some part of it?
We can do it for you! Click to Order!
