Outcomes Assessment: Conceptual Problems

Abstract: Despite their plausibility and their becoming common practice, programs of outcomes assessment are fraught with problems. The disclosure of those problems requires critical thought and a skeptical attitude which are at odds with most of the relevant literature. The practice lacks a basis in research, though circular and question-begging faux research attempts to fill the gap. Its real basis is discernible on investigation and is even more suspect. Further, the practice has several ill effects: it displaces teaching and learning; it divides faculty; it contributes to a misrepresentation of higher education as more narrow and simple than it in fact is; and it distracts us from more important and more urgent actions.

Outcomes Assessment: Conceptual Problems

            The basic argument for doing outcomes assessment in higher ed is almost Amish in its plainness. That plainness is a main source for its ability to garner assent even from those who do not understand outcomes assessment or like it. The argument also determines much of what counts as being outcomes assessment. The argument is so plausible that it displaces other arguments as superfluous, for example those grounded in controlled-studies-based research–or, indeed, any kind of research.

            Here is the argument. Higher ed uses other people’s money, people who are neither students nor staff nor administrators nor teachers. Those people who pony up the money deserve evidence they can understand that their money is being used wisely. Therefore higher ed should include structures which track, measure, and report measurable differences they produce in students who get degrees.

            This argument has been bought by legislators, boards of trustees, U.S. Secretaries of Education, and accrediting agencies. It has been accepted by university presidents. Programs of outcomes assessments are a result. In those programs, each university, each college within those universities, each department, each course and each professor teaching that course are required to articulate measurable outcomes by which the success of those courses, departments, colleges, etc., and the changes produced in students may be supported by evidence–and then that evidence is required to be gathered and reported. Enormous amounts of work are going into efforts to implement and follow up on these programs at universities across the country.

            I have offered this argument for students to review some basics of critical thinking of clarifying issues and describing arguments. We’ve never really worked it through–it would take too much time--and I tell my students that if they get it completely wrong that’s okay because everybody’s doing it. But the argument’s problematic assumptions, once exposed, eat it alive, and issues which keep leading to calls for increased accountability in higher ed are themselves problematic in ways revealed only by archaeological analysis into their sources.

            There is a positive aspect to the current situation, in that we might achieve more clarity now. To the extent that advocates have been successful at implementing outcomes assessment programs, it is easier to step back and think about appraising it and its effects in its turn.

            I pass by measurability problems, mostly because I see many faculty shut down in those discussions. The wars over operationalizing goals into measurable outcomes are strongly reminiscent of fifty-year-old controversies surrounding behaviorism, in which many on each side demonized (possibly still do) the other and then shut down with fingers in their ears. One side apparently thinks those insisting on outcomes assessment would turn all art into calculi and all wisdom into idiot savant checklists, while the other thinks their opponents would insist on reverence toward anti-scientific will-o-the-wisps and mass delusions while insisting on contempt toward evidence and arguments. These problems could indeed use fresh appraisal, but it will have to be smooth, mediating, and persuasive in ways for which I have no patience, no patience for either side.

             Other issues though are logically prior, more urgent, and less sclerotic. They have received less press and may offer hope for making progress. These other issues offer relevant considerations to policymakers deciding whether to implement outcomes assessments. The following are considerations which are easier to think through than the issue of whether OA is a result of fetishizing measurement. Advocates of outcomes assessment and people suspicious of it alike can work toward sorting pros and cons.

            Where Is the Research?

            What are the measurable effects of implementing programs of outcomes assessment?

            Outcomes assessment is undeniably succeeding on its own terms, in that it is becoming widely implemented. Also undeniably, outcomes-assessment-based education lacks evidence that it is an improvement over traditional education based on any other terms of evaluation. That is, when outcomes-assessment-based programs are compared with other programs lacking OA, one would expect to see some kind of results. Instead, the main results one can find are those based on question-begging measures of whether it has indeed been implemented. One wonders whether we are seeing another educational fad go nova, and what structures will remain among the clouds of gases. Advocates have succeeded in their efforts without the benefits of controlled studies. That is, the strongest advocates, including U.S. Secretaries of Education, accreditation agencies, and agencies which could have funded scientific research on these issues, make their case without appeal to hypothesis testing, without research projects, without publications of successes and failures, without comparison studies of any kind except those comparing before and after implementations, where indicators of success at implementation are assumed to be evidence for OA. No one has assessed the outcomes of implementing programs of outcomes assessment.

            Some to whom I have made this claim find it implausible. Perhaps we can count on rebuttals after this is published. Part of the puzzlement about how advocates of outcomes assessment could have succeeded without research can be resolved by realizing that many advocates think doing Outcomes Assessment just IS doing research because it results in data. But searches of the Internet, then, in frustration, the archives of The Chronicle of Higher Education and recent books on Outcomes Assessment and their bibliographies, yield a clear picture of the state of research, which is made up of substitutes for research. One is directed to manifestos claiming outcomes assessment makes sense, is needed, is succeeding, and is the wave of the future. Judging from the number of accounts of universities, colleges, school systems, and accreditation agencies in the throes of implementing outcomes assessment, one might be excused for thinking all these are true. Among some of these, the job of implementing a program of stipulating outcomes and assessing them is taken up by entities with names like Office of Research and Analytic studies, again suggesting that implementing outcomes assessment is as it were somehow doing research. A review of literature also turns up journalistic commentary on the manifestos, usually written as though simply chronicling waves of the future. There are a few dragging their feet and worrying about implications of implementing outcomes assessment, among whom I would place myself with this essay. Somewhere lower on the list are inquiries of the form, “Where the hell is the research?” Occasionally there are reports of educational organizations, such as the school system in New South Wales, Australia, withdrawing from outcomes assessment initiatives.

            It is striking how the question about lack of research bewilders advocates of OA–the question and the advocates seem ships passing in the night. By now we would expect reports comparing results of doing outcomes assessment with programs in which there is no outcomes assessment. But outcomes assessment is seldom mentioned except in contexts in which the speakers push to implement it. Advocates, convinced that the results of implementation will produce clear data, present before and after snapshots as evidence that implementation in fact increases an institution’s ability to meet its declared outcomes. The easy availability of that data is evidence that those advocates do not yet understand this question. Declaring outcomes and success at implementing outcomes is, after all, already a part of programs of outcomes assessment. What is needed is testing whether doing all that is educating better. And articulating outcomes is, after all, dependent on those who do the declarations being wise about what they do. Where is the research showing that this step leads to better education of students? If you are convinced going in that better education is education which is meeting measurable objectives, then we’ve a no-brainer: of course OA-shaped education will, provided it produces evidence that it really is OA-shaped, be better. On any robust conception of evidence, however, in which we insist on evidence which is not part of circular reasoning, the evidence is alarmingly lacking.

            A review of those alarmingly large sections of library bookshelves dealing with outcomes assessment also reveals overlaps with corporate calls for increased productivity in education. I have written elsewhere that calls for increased productivity should be informed by knowledge of what is being produced–what, that is, education is for. I return to this below under the headings of what gets left out and the final section on needed actions.

            If Not Research, What Basis Is There?

            The surprising lack of research raises the issue, what explains advocates’ faith in the efficacy of outcomes assessment? The issue now becomes informed by a lesson I often attribute to Freud as a part of the argument for doing critical thinking: often we believe not on the basis of arguments but on the basis of desire. This may require some digging. Are there clues in the history of the idea or in relationships of the arguments for outcomes assessment to the research-based arguments we would expect but do not find?

            This question about the history of an idea may look innocuous or threaten to take us off on a tangent. But when movements hold their block parties, intellectual historians don’t get invited. And if intellectual historians do show up at the party, everyone starts observing what nuns call chastity of the eyes, looking at the ground. For example, Gerald W. Bracey tells of such in Final Exam, p. 239: “At a Brookings Institution luncheon symposium for the release of Hanushek’s book [this was Making Schools Work, Washington DC: Brookings Institution, 1994] he reiterated the claim that in the last 20 years expenditures for schools had doubled and test scores had been flat. I held up a chart that showed test scores declining in the 1960's and 1970's, then reversing and rising to record levels. ‘They aren’t flat,’ I said. Hanushek and symposium chair, Chester E. Finn, Jr., cast me a glance, then continued as if the chart did not exist. Shortly thereafter, I penned a piece called, ‘The Right’s Data-Proof Ideologues’ that was published in the Commentary section of the January 25, 1995, issue of Education Week.)” There’s a reason for hostility to historians. Historians are powerful corrosives of the work of True Believers (as Eric Hoffer’s philosophical and historical work of that title is corrosive of the general concept). Further, reading histories of higher ed tends to cause cynicism and despair regarding reform. The history of outcomes assessment is rootbound in the same pot with American positivism and with behaviorism, and it is closely related to histories of management theories, which theories mostly make progress by renaming themselves. Thorsten Veblen’s 1918 The Higher Learning in

America: A Memorandum on the Conduct of Universities by Business Men contains analyses of how the secularization of universities led to imposition, at many levels, of values and of administrative structures which are extraneous to the academy’s work. From Richard Hofstadter’s 1962 Anti-Intellectualism in American Life (New York: Alfred A. Knopf) we can learn, among other things, how insistence on accountability can be based on hostility and lack of understanding of education. Gerald W. Bracey’s Final Exam presents five powerful historically-based essays anatomizing contemporary movements in education, including the move to broaden access to include all but insisting that that same all will achieve at a minimum level (which has since led to “No Child Left Behind”); the move to reform based on the claim, recurrent since Adam and Eve’s kid flunked the unit on brotherhood, that schools are going to hell in a handbasket, always have, and that the fix will start with better testing and assessment; and the move to fix education by articulating and imposing new standards. Jacques Barzun’s 1959 House of Intellect and 1968 The American University are attempts to distinguish core purposes of higher ed from administrative tasks, money management, and oversimplified models of education which distract us from that core. Learning, guarding learning, and teaching are not the same as managing budgets, getting students, publishing, offering to name classrooms for people with money, or testing whether learning has happened. Learning, even, must be distinguished from the excitement which is an effect of learning and from other effects. Barzun’s recommendations for bringing us back to what we are about are bracingly radical and specifically counter to historical trends calling for external accountability (see 1968, Chapter 8, “The Choice Ahead,” especially pp. 247-259). And of course we can remind ourselves of how some terms, such as strategic, evidence (and culture of evidence), accountability, carry a kind of authority which helps put critical faculties to sleep both in those who hear them and in those who speak them.

            One of the main lessons of history, if we are to avoid repeating it, is that we have to keep our critical faculties. The history of educational reform, as Bracey points out, proceeds like the Bill Murray film Groundhog Day–we do the same day over and over and over as long as the lessons do not get learned. If we step back and regard not the history of educational reform but the literature, there is a striking division between the great number of those who have remedies to sell and those who think. For those who are selling something, the world is a simple place. Critical faculties, as Veblen teaches us, are out of order when the agenda is advertising. But history offers powerful therapy for oversimplification. (J.L. Austin says of philosophers, and we could include most educational theorists, “Oversimplification is the occupational disease of philosophers–unless it is the occupation.”) Historians tend not to make good salespersons. –Though it’s not necessarily being a historian which makes a person thoughtful and critical, or we would not have to choose our histories carefully. Perhaps some of the critical attitude comes from historians finding their best work comes with a certain distance. Outsiders who are not historians, like bell hooks, James Herndon, Socrates, Paula Gunn Allen but who provide powerful critiques of their and our cultures provide some confirmation of this idea.

            The arguments for assessing outcomes are expressions of models of corporate business management. These models are startlingly temporary, in the sense that there is always a new one that gets attention and aspires to capture the hearts and minds of managers and administrators. (Though many of them fade from prominence only asymtotically: like behaviorists and Jehovah’s Witnesses, a Remnant remain.) Joel Best’s Flavor of the Month: Why Smart People Fall for Fads gives examples for higher education, and Robert Birnbaum’s Management Fads in Higher Education anatomizes several of these. One example is Total Quality Management or TQM, whose blood quantum shows in Outcomes Assessment. TQM presents itself as a new, humane, compassionate but still hard-headed approach to doing business by bringing to each activity not a stopwatch but the lens of quality. Some impetus for that model may have come when Robert Pirsig made the word “quality” into a mantra, beginning with Zen and the Art of Motorcycle Maintenance in 1974.

            We too might be well-advised to meditate on the word. One of the most difficult parts of implementing Outcomes Assessment has to do with appraising quality. This issue of appraising quality is going systemic in higher ed. The current rash of program prioritization efforts on campuses is least regimented and least tied to criteria when it comes to providing evidence for a program’s quality. Robert C. Dickeson’s Prioritizing Academic Programs and Services, which my administration has handed out like necklaces at Mardi Gras, includes a guarded note about the move to assess outcomes rather than inputs or processes, in its section on appraising program quality. A few suggestions are made for possible outcomes to observe, but this section is much less prescriptive (which gets translated on campus as “less helpful”) than most others. The last paragraph owns up to this lesser aspect by acknowledging that “Assessing quality outcomes is generally regarded as more difficult and less precise than assessing quality inputs,” pp. 66-67.] Assessing quality is difficult partly because, in the history of the term, qualitative has usually been distinguished from quantitative, where objective evidence and measurability helped guarantee that the appraisers were not subject to delusion. The subjective-objective distinction, then, with its polluted sources in Cartesian dualism, still wells up to influence our work.

            These sources, then are not new. Indeed, one characterization of them is present in Thorsten Veblen’s 1917 The Higher Learning in America. He anatomizes, provides examples, and then predicts pathologies which result from putting captains of industry and co-captains of the pecuniary system in charge of higher education. One of his prophecies we can see fulfilled in reports of surveys of college graduates’ ignorance. Over the last hundred years the alleged steady rise in the ignoramushood index among college graduates could easily be an effect of increases in power and control by non-academics over universities. External calls for increased accountability erode the quality of education. This hypothesis might help explain the dreariness of the history of educational reform. Calls for increased accountability may be especially corrosive (and particularly consistent with Veblen’s account) when they come from those whose interests are not deeply informed by education: narrowly educated theoreticians, managers, government bureaucrats, legislators and trustees. We need to produce inhabitants of those categories who are more broadly educated.

            Neglected (Because Unknown) Outcomes: What Gets Left Out

             Applying outcomes assessment to academic programs requires that those who assess know and understand what the programs do. This turns out to be problematic. That is, undergraduate degree-granting institutions need to know what it is they produce. I have become convinced that very few academics, much less trustees and administrators, have a clear picture of the contemporary bachelor’s degree and what it means. If they did, general education would not be such an ineffective mess. What are the causes of this tunnel vision? We can round up all the usual suspects: the withering of interdisciplinarity and emphasis on specialization; hardening of turf boundaries in the face of declining budgets; economic hardships in the larger society leading to emphasis on preparation to serve the labor market despite that market’s regularly being beaten bare by winds of change; general education programs being shoved off the end of the bench by majors; a decline in the number and quality of scholars who focus on the largest issues facing the academy.

            In Humboldt State University’s preparation for our accreditation renewal, a group of smart people with surprising good will and astounding lack of agendas worked hard to come up with a brief list of the university’s goals for graduates with a baccalaureate. We are now engaged in implementing outcomes assessment plans for these goals. I think ours are as good as it gets, though the Web is rife with these lists. Boiling all our work down to a list of seven goals may be–no, it is–problematic. We can see this, if it is not obvious, by looking again at the process we followed and asking what gets left out.

            We began the work of articulating these goals by asking groups of people to reflect on their own undergraduate degrees, on what had been the most important results of those years, and on particular outstanding students we have now, on what makes them outstanding. What struck me then was how varied the responses were, how insightful but also how idiosyncratic and difficult (at least for me) to regiment the answers into categories. Much of what we said back then does not show in final lists of goals and objectives.

            Here are some of the missing factors, including some mentioned in those beginning conversations. College provides a safe vestibule for fledglings being kicked out of the nest, and its library provides an entrance to the world. College is still a large part of many adults’ coming of age. It provides a substitute for apprenticeships and community rituals. It is instrumental in helping people find a calling, a vocation. Jobs may come and go, but college often still helps shape interests and directions which influence the rest of a person’s life, and provides abilities and encouragement to pursue them. Small classes and conversations with faculty expose our mistakes and give us practice in arguing (and joking) with professional people in which we are treated seriously, perhaps even treated as peers. This kind of education prepares students not just for taking a place in an existing power structure or class system, but instead prepares them for analyzing, influencing, and changing power structures. Reading Homer, Aeschylus, Sappho, Chuang Xu, Plato, Lesley Marmon Silko, Maimonides forcibly decenters particular and provincial points of view. Faculty who share their own academic lives with us, who welcome us into doing research, provide us models of scholarly involvement and activism. Faculty who share their own personal lives with us provide models of interests and enthusiasms which help us take our own seriously. Faculty who are broadly educated, broadly activist, and broadly interested help us keep perspective on the regimented disciplinary boundaries which still exert powerful influence over the degree. Faculty who manage to teach and write and serve on committees and still raise children and drink and coach Little League, provided we can get to know them, help us integrate academics and community, gown and town. Faculty who pay attention to analysis of social structures and raise consciousness of the roles of gender, race, class, culture, money, and provincialism prepare students to cross boundaries imposed by those factors. Faculty who took our faculty under their wings–mentors–were mentioned repeatedly in those first conversations.

            Other important omissions in the thinking behind outcomes assessment are demonstrated in research on attempts to close gaps in educational achievement between groups who are successful and those groups who have been less successful Closing those gaps requires attention to lives outside the classroom, to informal social processes, to relationships with parents and peers–none of these easily measurable in the classroom and none of them amenable to outcomes assessment in most courses (Ferguson, Kim-Hall). (“Outcome two for History of Ancient Philosophy: Students will have hung out with the Plato wonks. Assessment measure: # hours students spend in informal settings with the Plato wonks.”)

            None of this is new–we could regard the forgoing list as an application of John Dewey’s recommendations that a school be a part of a community and part of the world. Of course that means a lot of different things to different people, but one main meaning is a repudiation of the factory model emphasizing measurements focused on inputs, productivity, and outcomes. Assessing the items on this list via outcomes is a parody of education. R. G. Collingwood, the philosopher of history (among other things), teaches us that a primary determiner of the answers we get is the questions we ask, and that another determiner, then, is the questions we don’t ask. Not asking what education is for may doom us to a stunted list of alternatives. Not asking what assumptions shape programs of outcomes assessment results in a parody of what we want, and distracts us from our own goals.

            OA Pulls Resources from Teaching and Learning

            As implemented, programs of outcomes assessment shift resources and emphases within programs away from those results which have been largely unarticulated and toward those which are being measured. Departments and faculty teach to the tests. In order that course content not be eroded, this requires the tests be fair and complete. But in fact, completeness of testing interferes with completeness of courses at least to the extent that assessment becomes a larger part of the courses. As a result, completeness of both are routinely sacrificed for sampling efforts, and teaching to the test becomes a more dangerous possibility because the evidence of good teaching comes from a smaller portion of the course than before. After circulating drafts of these remarks to faculty and administrators, vehement agreement regarding this point was the most common response I got.

            OA Divides Faculty

            One problem, with effects on morale, is that many who are required to implement OA understand the motives behind it and feel rebuked. Boards of trustees (which are as Veblen predicted a hundred years ago almost entirely captains of industry), the chancellors and presidents they hire (and deans, if trustees can pull it off down that far on the chart), and the legislators and governors and governors’ staff for higher ed still often look at four-year colleges and universities with envy and uncomprehending distrust. Work weeks measured by contact hours are short; faculty salaries are higher than factory workers who do real work; faculty get a quarter of the year off; faculty sit staring off into space and claim it’s work; and new graduates keep demonstrating breathtakingly new levels of ignoramushood. Those who foot the bills and look over the budgets don’t trust the faculty. Outcomes assessment gets taken up by such players as a Missouri “Show Me” project. It is a result of distrust of the university on the part of those who pony up the money, or rather administer for those who pony up the money. It also reinforces those faculty who enjoy rebuking each other in underhanded ways (at times I am in this number as well.) The fact that we are in the habit of preaching skepticism as part of critical thinking helps lower our guard to this kind of attitude.

            Standard practice from civil service and corporations illuminates this source for OA. If an employee is not performing, is dragging feet or otherwise not productive, then performance reviewers may put the employee on a plan to become better. These plans are called by various names: Personal Development Plan, Performance Improvement Statement. That plan has to move down the ladder of abstraction from the job description under which the employee was hired. It will contain more specific goals, called for this purpose objectives, and clear criteria for whether the employee at the next performance review can be judged as improving. Putting employees on such development plans serves notice that their work is not good enough and is a forcible reminder that employees are underlings and that their continued employment is at stake along with respect for them in the workplace. Such plans provide outcomes and outcomes assessment measures as ways to make the subsequent decision, up or out, into a public, verifiable process based on measurable evidence. Those measurements are meant to forestall grievances and answer charges of unfairness. In higher ed they nudge up against Retention, Promotion, and Tenure processes.

            Some faculty, then, with such practices as background, feel misunderstood, threatened, unrecognized, and defensive in ways which do not promote cooperation. They often point out that they give grades, they read papers, they give tests, students take away transcripts. These are assessments. Further, they do assess, rather than inputs, outcomes. Ignoramuses can be predicted and in fact are predicted from their transcripts, their GPA’s, and their faculty letters of recommendation. Faculty feel universities are being pressured to become more like On Job Training programs. This diminishes higher ed to the extent that what higher ed really is about is preparing students to handle a world in which their jobs will keep changing and their roles as citizens and thinkers will require more information, more investigative, creative and critical skills than just one generation ago. Further, the supervisory functions are already performed within the academic world. Faculty are not very good at trusting each other, and so can point to policies, procedures, committees, and the work they already have to do to show each other they are putting out a “quality product.” Those policies and committees can be offered for consideration as effective tools of supervision and quality assurance. For example, policies are in place which stipulate procedures, forms, and required approvals any time a faculty member wants to claim that a course should be a part of a university’s General Education Program.

            Against oversimplifications, faculty insist on neglected aspirations in higher ed. Higher ed is not just about memorization or skills but is also about evaluating what has been memorized. Higher ed is about liberation, about questioning and arguing values and authorities, about criticizing and refining citizenship roles, about acquaintance with the best ideas humans have had and adding to those, and about developing better peripheral vision to get past the blinders we don with our cultures and dogmas. Some may think we exaggerate the importance of these goals. Certainly for many students the goal of economic independence, preparation for a job, is the crucial first goal and should not be discounted. These other goals, though, are important to a larger picture, and faculty are right to remind us.

            Faculty do disagree. Some faculty regard it as reasonable that those who pony up the money be given evidence the money is put to good use, and that the evidence be easily understandable by those who do the ponying. For some this view is not only accepted but is endorsed with enthusiasm. Other faculty roll their eyes but regard the push for outcomes assessment as inevitable and not outrageously unreasonable, and so work with good will to put together assessment programs which will pass muster among the campus assessment coordinators and committees. But others, a great many among those with whom I talk, regard outcomes assessment as evidence that the town does not understand the gown. They regard the demand for evidence that the academy is doing its job well as unfair and as based on wilful refusal to investigate how in fact faculty do track and provide evidence of success appropriate to what they do. The strength of these negative reactions varies, including those who are merely depressed and those who are ready to hoist flags over the ramparts. These divisions among faculty are of course not all bad; like committee work and strategic planning, they keep faculty off the streets and out of the halls of the legislature. Most faculty would agree though that OA and other productivity, efficiency, and accountability initiatives have real impacts on teachers and classrooms. They tend to dilute education and do divide faculty. The stronger versions of these views turn into indictments: outcomes assessment initiatives add to workload, increase class sizes, reduce conversations between teachers and students and between students and other students, reduce then the actual accountability of the students, reduce the amount of writing involved in a degree, push the students out the door more quickly, and hide from students the degree to which they are being cheated.

            Nobody Reads Them

            First, some comments which might seem to contradict the claim that no one reads outcomes assessments. I now serve on several campuswide committees by virtue of having been elected General Faculty President, and have served on more. The University’s Curriculum Committee, the Provost’s Council, and an accreditation task force routinely look at program goals and outcomes assessment plans, and have been involved in implementing requirements to include statements of goals and objectives in syllabi and in program review documentation. All curriculum changes now are accompanied by statements of how the changes will accomplish goals and how those goals generate objectives and how those objectives will be assessed. I read a lot of syllabi, and the syllabi are getting longer. I help write objectives in order that the Western Association of Schools and Colleges can see how we at my university accumulate evidence that we are accomplishing our goals. I read program reviews in General Education as well as for departments and majors. The California State University is once again engaged in a systemwide strategic planning effort, “Access to Excellence,” in which the language of goals, objectives, and assessment are prominent. All over the 400,000-student California State University the talk is of what our objectives are and how we will assess them, and faculty and staff are writing statements about how we will do these things. I keep hearing proud talk, coming from the accrediting agencies and from the Secretary of Education, about “nurturing a culture of evidence.” In many departments evidence of how those objectives have been assessed are accumulating in drawers and in large computer files. Departments do exit interviews and tests, students accumulate portfolios, teachers collect data from each course and forward that data to department chairs and to committees. Each committee rows through tides of paperwork, higher tides in winter and bad weather, writing reports. The University Curriculum Committee has a two-and-a-half-inch binder each year for each member and then for each meeting we receive a packet including more outcomes. As I write this two-thirds through the year my packets no longer fit into four folders. Outcomes and assessments structure about a fourth of those folders. But reports of assessment of outcomes assessment are a much smaller part of the whole than are the stacks of promissory notes. In each department, there are one or two faculty who have to review assessments and forward them. On each campus are a few administrators who have to review what has been forwarded. These reviews are done for someone else who is far away. These efforts began with people who are off campus–systemwide administrators, the Board of Trustees, influential businessmen, legislators, staff in the state governor’s office, national education organizations operating at a level of abstraction to which one must bring one’s own oxygen, accrediting agencies, the U.S. Secretary of Education. Not a single one of those is going to look at a single document my department generates, nor will my department’s results even find their way into a summary of all those documents written by my committees or my college. The culture of evidence has turned into a culture of reams of paper, a culture of file cabinets and large .pdf files. Our students’ transcripts get looked at seldom enough–at job interviews, by graduate admissions committees–but they do get looked at. They are blunter and quicker than the results of my department’s exit interviews or exit exams or the course outcomes assessment data or portfolio reviews. While transcripts do get read because they are useful, no one will ever read OA reports unless someone pays for it and requires doing so as part of the position description.

            The Argument Disguises Its Speakers

            Because of the talk in the argument about educators using other people’s moneys (governments, taxpayers), it might seem as though the argument is being put forward by those other people. Strikingly, it is not. One has to watch out for the genetic fallacy here–the argument’s source need not be relevant to its force. But when the argument is partly about the existence of a need–the people need the academy to render an accounting–then the question of who is speaking for the people becomes germane. Do the general run of taxpayers have problems with traditional means of assessing students, namely grades, majors and transcripts, letters of reference? They clearly do not. Does this issue arise as a result of taxpayers or employers finding fault with college records? Clearly not. Perhaps one might think these concerns arise with claims of rampant ignoramushood among graduates. Those who do pursue this argument are certainly happy to cite surveys emphasizing ignorance, which remain consistent since the 1920's (Bracey 1995). But these calls for reform and for increased accountability do not have their origins with the public. Instead they show up among administrators under the influence of schools of thought among management theorists in business. These schools of thought are discussed in books by the historian Gerald Bracey, by Robert Birnbaum, and by Joel Best. All three provide bracing criticisms. Birnbaum and Best explicitly regard them as fads. All three are dubious about their final value and point to serious problems in the processes by which they are justified and by which they get implemented. History provides evidence that these remedies will likely yield to other inflated promises, other oversimplified cures, other fads. Fads give their devotees the appearance of being involved with a Cure, a movement, a Help, and administrators are under pressure, as Veblen points out, to produce something visible which will mark their campus or agency as belonging to those particular administrators and will mark it as different from others in ways which increase prestige. One cannot help but wonder how many millions of dollars and hours of faculty and staff time could be saved if, instead of letting new presidents and provosts initiate strategic plans and new initiatives, we could just march them around the boundaries of the campus and have them pee on the corners.

            There Are Appropriate Actions for Us to Take Now

            If the forgoing is at all correct, then there are clear implications for actions to improve our situation and for our work regarding outcomes assessment. Those actions will not appease those calling for more or faster implementation. They will involve analysis at a more thoughtful level rather than accelerating the training of faculty. They involve, wait for it, philosophical re-examination. Ten years ago I wrote in this journal on what education is for; my main claim was that when you are wrong about what education is for then it is mad to insist on increased productivity, and that even though it looks a lot like doing nothing, doing some philosophy might save us from that madness. Same here.

            Scientific analysis would also help. A controlled study of whether degree programs which implement outcomes assessment do in fact produce better educated graduates than programs which do not has not been done. One reason such a study cannot quite be done yet is that it will take some prior reflective thought. Such a study will need a non-circular, non-question-begging description of what it is to be “better educated.” The study itself would be a major undertaking and would require careful design work and money. The design would include the possibility of disconfirming the claim that outcomes assessment programs help educational effectiveness. This means such a study would be unlike those studies in which measures of implementation are taken as measures of success–which concept of research makes research like a dog incapable of barking in the night-time or the day, much less having bite. One would need to control carefully for other variables. One variable William James points out which can confound results at a subliminal level is paying attention–if an institution can be improved simply by paying attention to what it is doing, then that is not evidence that a particular form of paying attention is the main factor in any resulting improvement.

            One might save work, even, by doing a thought experiment: take an institution of unquestionably high quality, such as, perhaps, Harvard or Stanford or Columbia or St. Johns; select some programs at that institution for implementation of outcomes assessment and select others as controls for a several-years study. We might have to begin by overcoming doubts as to whether such an effort would be profitable, but perhaps we could threaten their continued accreditation.

            We also need an analysis which compares calls for accountability urged from outside higher ed and those which emerge within faculty. We need such an analysis because of the possibility that such an analysis might reveal that calls for accountability from outside the academic world are based on misunderstandings and oversimplifications of education (for instance, thinking that education is about learning facts and job skills.) Historians of education, especially educational reform, could help. A history of higher ed which focuses on separating changes which have been most beneficial from enthusiastic but ultimately futile reform efforts might illuminate our current situation. The history of reform in higher ed can be matched for dreariness only by history of housing reform, and articulating the relations of meaningful and positive reforms to theories provides mostly a cautionary lesson about theory.

            Finally, there is an opportunity now for real reform of a quite different kind, not driven by theory and much more meaningful. Right now, there is a relevant, interesting, perhaps pregnant pause in discussions of general education, that core part of the degree not presently included in majors. One gets the impression, (something similar may be happening in linguistics) that discussants are waiting for a crucial innovation. Part of this is because doing outcomes assessment in general ed is particularly difficult, raising more recalcitrant issues than the issues within most majors (Katz, CHE May 23 2008). Part of it is captured in the report from the UC’s Center for the Study of Higher Ed’s report on General Education for the 21^st Century, which includes useful surveys of the state of general education in American universities and accounts of many attempts at reform. The report offers recommendations for what is needed, but those recommendations do not address the recurrent problem that the reforms they document turn out to be piddling. Careful work by a strong task force–or perhaps just implementation by a strong institution-- to articulate what is needed and what a coherent general education program would look like might presently find a receptive audience and might even influence discussions of outcomes assessments.

            For instance: it may be that general education, after decades of being nearly pushed off the bench by expanding majors and by trustees pushing for faster throughputs, and in the face of increased and pressing world complexity, should be expanded by about a year, that each student should be required to take a minor, at least the equivalent of a semester’s work completing an organized cluster of courses, in humanities, another in social sciences, and another in sciences, separate from the major, and that each student should take a two-year sequence of courses in the intellectual history of the world.

            Outcomes assessment is an odd business. It is not to the credit of higher education that we have tolerated this external assault on our work. Its origins are suspect, its justifications abjure the science we would ordinarily require, it demands enormous efforts for very little payoff, it abjures wisdom, it requires yielding to misunderstandings, and it displaces and distracts us from more urgent tasks, like the teaching and learning it would allegedly help.

             References

J.V. Baldridge et al. Baldridge National Quality Program: Educational Criteria for Performance Excellence (Washington, D.C.: National Institute of Standards and Technology, 1999).

Joel Best, “From Fad to Worse” in Chronicle of Higher Education, Volume 52, Issue 32, Page B6, April 14 2007, revised from his book, Flavor of the Month: Why Smart People Fall for Fads (Univ. California Press, 2007). Best offers explanations for the prevalence of fads, but in his focus on why people choose to endorse fads he neglects the powerful role of true believers among administrators, who require implementation from those who do not endorse the fad in question. This is conspicuously the case with outcomes assessment, where accrediting agencies, U.S. secretaries of education, trustees, presidents, and provosts have decided in favor and so drag universities into compliance. Though Veblen found grim hope, in the face of business taking over higher education, in the fact that businessmen don’t really understand higher ed and so are generally impotent to work changes, that hope may not apply here.

Robert Birnbaum, Management Fads in Higher Education, (Jossey-Bass: San Francisco, 2000). Prof. Birnbaum, who says, “I was once myself a votary,” mostly focuses on the structures and life cycles of fads, but nevertheless offers a veritable cold shower of examples. With many of those examples we get a brief history and genealogy, enough to get a sense of a taxonomy. The vocabulary in his analysis shows bad seed also visible in Outcomes Assessment. Birnbaum also comments in his preface on the narrow perspective involved in asking, “Why can’t a college be more like a business?” (one hears Rex Harrison as Professor Higgins, about to burst out in song), and contrasts that with a better question, “Why can’t a business be more like a college?”

Gerald W. Bracey, Final Exam: A study of the Perpetual Scrutiny of American Education (TECHNOS Press: Bloomington IN, 1995). This work is a model of critical thought based on wide-ranging historical foundations.

Ken Buckman, “What Counts as Assessment in the 21^st Century?” Thought and Action, Fall 2007, pp. 29-38.

California State University Institute for Teaching and Learning. Student Outcomes Assessment: What Makes It Work? (CSU, Long Beach, CA: 1992). For considering what counts as research, see especially “Evaluation of Student Outcomes Assessment Pilot Projects in the California State University,” by M. Riggs and J. Worthley, Pp. 1-22, and the same authors’ “Lessons from Pilot Projects,” Pp. 23-30. Of particular concern is the degree to which faculty and administrator participants’ support and acquiescence are identified as predictors of success. For true believers confirmation is easy to find.

Chickering, A., and Gamson, Z. “Seven Principles for Good Practice in Undergraduate Education.” The Wingspread Journal, 1987, 9 (2), 1. This is online at

http://www.johnsonfdn.org/Publications/ConferenceReports/SevenPrinciples/SevenPrinciples_pdf.pdf This very widely cited reference is interestingly different from how it is characterized by the outcomes assessment movement, which sees the seven principles as amenable to treatment as outcomes. For instance, see also the citation of the above and its characterization in Darrell W. Krueger. “Total Quality Management.” Making a Difference: Outcomes of a Decade of Assessment in Higher Education (Jossey-Bass, San Francisco: 1993) Pp. 269-278.

R. G. Collingwood, Autobiography (London: Oxford Univ. Press, 1939).

John Conant, The American High School Today: A First Report to Interested Citizens (NYC: McGraw-Hill, 1959).

Robert C. Dickeson, Prioritizing Academic Programs and Services (Jossey-Bass: San Francisco, 1999).

Peter F. Drucker, The Practice of Management (Harper/Collins: NYC, 1954).

E.E. Evans-Pritchard, Witchcraft, Oracles and Magic among the Azande (Oxford, 1937), pp. 202-250, provides an account of the education of medicine men. A briefer and more generalized account is in John S. Mbiti, African Religions and Philosophy (Heinemann: Nairobi, 1969), pp. 166-171. There is, of course, a crying need for implementation of programs of outcomes assessment in these educations.

Ronald F. Ferguson, Toward Excellence with Equity: An Emerging Vision for Closing the Achievement Gap (Cambridge MA: Harvard Educational Press, 2006).

E.N. Friedland, Introduction to the Concept of Rationality in Political Science (Morristown, N.J.: General Learning Press, 1974).

James Herndon, The Way It S’pozed to Be, (Simon & Schuster: NYC, 1968), and How to Survive in Your Native Land (Simon & Schuster: NYC, 1971).

bell hooks, Teaching to Transgress (NYC: Routledge, 1994).

Soo Kim Abboud and Jane Y. Kim-Hall, Top of the Class: How Asian Parents Raise High Achievers–And You Can Too (NYC: Berkley Books/Penguin, 2006).

John Henry Newman, On the Idea of a University ed. C.F. Harrold (NYC: 1947, original 1852).

Neil Postman, The End of Education: Redefining the Value of School (NYC: Knopf, 1995)

Brent D. Ruben, ed. Pursuing Excellence in Higher Education (Jossey-Bass: San Francisco, 2004). This substantial volume gives a good snapshot of more or less current work written by major luminaries, informed by what Wittgenstein suggests is a recurrent source of pseudoproblems in theory, namely thinking that for every substantive (e.g. excellence) there must be a substance which it names. Perhaps because many of the companies turned out not to be so excellent after all, and because of a lack of interest in history, no mention is made of this book’s ancestry going back to T.J. Peters and R.H.J Waterman’s best-seller In Search of Excellence: Lessons from America’s Best-Run Companies (Harper-Collins: NYC, 1982).

D.T. Seymour, On Q: Causing Quality in Higher Education (Macmillan: NYC, 1992). See especially the summary of eleven different elements (one might say meanings) of “quality,” pp. 19-21, and the explicit relation to corporate models for education, p. 128.

Thorsten Veblen, "The Higher Learning as an Expression of the Pecuniary Culture" in The Theory of the Leisure Class (NYC: Macmillan, 1899).

Thorsten Veblen, The Higher Learning in America: A Memorandum on the Conduct of Universities by Businessmen (NYC: Macmillan, 1918).

Wingspread Group on Higher Education, An American Imperative: Higher Expectations for Higher Education (Johnson Foundation: Racine, WI, 1993).