Why We Haven’t Measured College-Level Learning (column)

Two weeks ago in this space, I made a case that this vulnerable moment for higher education—of diminishing public confidence, financial strain, political attack and more—demands that colleges and universities show their value in part by proving that they’re actually enabling learning on their campuses.

I heard from a wide range of people, some agreeing and others sharpening their pitchforks. This is contested terrain: Some people who work in higher education just don’t like to be asked to prove themselves. Others worry (understandably given the history) that trying to “measure” whether and how much students learn might be reductive and produce endless administrative busywork.

I’ll soon respond here to the good questions some of you have raised since my original post. But first I want to finish what I started.

In the next couple of weeks, I’ll explain why we’ve historically tried (and failed) to measure learning and the importance of doing so now. Provide my early, and undoubtedly imperfect, ideas about how we might do that. And, importantly, distinguish what I’m envisioning from the accreditation- and accountability-driven “assessment” that a lot of you (not wrongly) view as the devil.

Today’s column offers my best attempt to explain how we ended up here: lacking, as I argued previously, “tangible, direct and incontrovertible proof that colleges and universities actually help learners develop the capabilities, skills and habits of mind that the institutions purport to deliver.”

Colleges and universities have long taken steps to understand whether students were learning, though in differing ways to suit different needs and satisfy different audiences.

For their own internal purposes, institutions tracked students’ progress toward a degree, largely through a process involving faculty development of a curriculum, student exposure to that curriculum measured in credit hours or “seat time,” a set of assignments and assessments graded by faculty experts deemed best able to judge whether students were learning, and a diploma backed up by a transcript with little detail.

For decades, most employers and Americans seemed to take on faith that people with a degree had learned enough or otherwise derived enough value to be “worth it.” That’s if they even thought about it; for many it was just assumed.

Institutions used also assessments for other “internal” purposes: to evaluate professors’ performance for tenure and promotion purposes; to “improve” courses and programs, understanding what works and doesn’t, and where students go off track; and to help instructors and designers rethink teaching practices and revise the curriculum as a result.

Those efforts aimed at institutional improvement had quite a bit of internal buy-in (since people tend to support things they’ve had a hand in creating); the central theme of Scott Gelber’s 2020 book, Grading the College: A History of Evaluating Teaching and Learning (Johns Hopkins University Press), he says, is that “when people come up with their own methods of evaluation, they have more faith in the validity” than in approaches imposed on them from outside.

But the internal methods lacked any currency outside the college (or even academic department) in question. And at various points in time, when questions arose about higher education’s quality or effectiveness, researchers, policy makers or politicians pressed for different kinds of learning assessment that might offer more comparability across institutions.

The first big push occurred in the mid-1980s, in the form of the 1984 report of the Study Group on the Conditions of Excellence in American Higher Education, which followed closely on the heels of the “Nation at Risk” report that spurred decades of K–12 reform.

The study group, which was commissioned by a unit in the then-new U.S. Department of Education but made up of prominent academics, recommended that colleges define educational excellence in terms of measurable student “outcomes” (knowledge, capacities and skills) rather than “inputs” such as the academic credentials of incoming students.

It also called on accreditors to hold colleges accountable for defining expectations, assessing whether students were meeting those expectations and developing systems for improving teaching and curricula based on those assessments.

Peter T. Ewell, who’s been called the “dean of the outcomes assessment movement in higher education,” cites this moment as the birth of that movement, and notes that it was plagued from the start by the dual purposes laid out by the study group—for internal improvement and for external accountability. (One of my all-time favorite op-eds in Inside Higher Ed was titled “Assessment for Us and Assessment for Them.”) Proponents’ desire to use the data to improve teaching and learning was increasingly greeted with skepticism from professors who believed the information would be “ritualistic” and potentially used against them in the court of tenure.

That tension escalated two decades later when then–Education Secretary Margaret Spellings’s Commission on the Future of Higher Education built its often-harsh critique partly around the idea that colleges and universities should define and measure “student learning outcomes” and that accreditors should police their work to ensure quality.

While Sen. Lamar Alexander blocked Spellings from taking formal action to force accreditors to set minimum levels of performance for student learning, colleges and accreditors alike got the message that, as the former college president and policy maker Jamienne Studley recalls, “if we didn’t do it ourselves, they would do it to us.” Even a self-described “curmudgeonly old classicist” like W. Robert Connor, then head of the Teagle Foundation, believed the time had come to measure learning more vigorously.

The threat of government-imposed requirements spawned a slew of initiatives over the next decade.

Some, like the Collegiate Learning Assessment, which was championed by the Spellings Commission, were standardized exams that sought to measure students’ critical thinking, analytic reasoning and written communication skills through a series of “performance tasks” and “writing prompts.” The CLA was notable because it was designed to measure how much learners grew over time, by comparing how students performed at the beginning and end of college. (The CLA’s embrace by the Spellings panel may have helped doom it.)

Many other advocates for stronger assessment focused not on standardized exams but on assessing portfolios of actual student work to find evidence of learning growth. The American Association of Colleges and Universities’ Valid Assessment of Learning in Undergraduate Education (VALUE) designed a set of rubrics to define 16 broad learning outcomes, applied the rubrics to samples of student work produced in courses and programs, and scored it. The now-defunct Multi-State Collaborative to Advance Quality Student Learning tried to do this at scale; hundreds of campuses still use VALUE in some way.

Still others intent on assessing student learning embraced competency-based education, an effort to recognize mastery of skills and knowledge without regard to how much time a learner spends in a program.

But none of these efforts fully took hold. Richard Arum, whose 2011 book Academically Adrift used data from experiments with the Collegiate Learning Assessment to argue that shockingly little learning was happening at many colleges, says there has been “no systematic investment in the use of these tools, let alone further development of them.”

He’s unsurprised that most colleges didn’t show much interest in measuring whether their students are learning—“that’s just basic self-interest” in not wanting to be held accountable. But Arum is disappointed that funders and research organizations haven’t pushed harder.

(Important side note: The colleges and universities that typically “lead” in higher education—the most visible institutions that are also often the wealthiest and most selective—have the least incentive to measure learning. The public already assumes they’re the “best,” so any mechanism that showed us which colleges actually helped their students grow the most might well diminish the “elites’” status rather than improve it.)

Meanwhile, the fears of many faculty members—that institutions and accreditors would adopt a top-down assessment strategy that purports to gauge whether students are learning but mostly creates box-checking busywork—has come to pass.

Rank-and-file faculty members (who, let’s be honest, will complain about anything they don’t see as being primarily in their interests) aren’t the only ones who think so: At one accreditor’s conference I covered in 2019, a researcher who has dedicated her life’s work to assessing student learning called the state of the field a “hot mess.”

“We had a round of assessment that was really detrimental, incredibly measurement focused,” she said.

What has happened to learning in the meantime? The whole point of this series of columns is that we don’t really know, but most of the few indicators we have suggest that the quality and quantity of learning aren’t going in the right direction. The latest measures of adult literacy and numeracy in the United States show declines, and while the most prominent studies finding declining academic workloads for college students are 15 years old, anecdotal evidence aplenty suggests that professors are assigning less reading and writing and students themselves are doing less work.

All while college grades have risen unabated: The best available (read: imperfect) evidence we have, from the Education Department’s National Postsecondary Student Aid Study, suggests that the average college grade point average rose from about 2.7 in the late 1980s to roughly 3.15 today. That’s widely interpreted as evidence of grade inflation rather than actual improvement, because students’ performance on standardized measures like the ACT and national exams like the National Assessment of Educational Progress have declined at the same time.

You know what’s also increased during this time? Public doubt about the value of higher education and of degrees. Is that because people doubt that learning is happening? That’s not a reason they tend to cite. But they are questioning whether graduates are prepared for employment and whether the time they spend at our institutions will help them enough to be worth what it costs.

Is learning the only piece of that? Of course not, which is why I’ve argued previously for a “broadly framed but very specific menu of indicators that would present a fuller picture of whether colleges and universities are delivering on the promises they make to students and to society more broadly.”

Learning should be at the center of that, though. I’m not talking about measuring it for either of the two historical reasons for assessment—“improvement” or “accountability”—but for a third purpose: “assessment to prove value.”

To prove to students and families and to employers (and maybe to ourselves?) that our colleges are actually delivering to learners what our mission statements say. That we have an actual plan for what we want them to know and be able to do after undertaking a set of learning experiences at our institutions, that we and they understand what that is, and that we’re committed to showing them they’re actually getting what we’ve promised.

I’ll save my working vision of what that might look like for the third and last piece of this series later this month. But at a high level, we would:

1. Define much more clearly (and ideally coherently, at an institutional or even cross-institutional level) the main capabilities, habits of mind and behaviors) we think a learner ought to develop (for me, it would be some set of durable skills on which faculty members, employers and others can find common cause and language);
2. Commit to a goal of helping every learner who puts in the work develop competence (if not mastery) of those capabilities;
3. Figure out some way (or ways) to measure students’ development of them so we (and they) can know how we’re all doing; and
4. Use the resulting knowledge for some combination of continuous curricular improvement and persuading our publics that we’re doing what we promise.

Sounds simple, right?