The Way(s) Forward on Measuring College Learning (column)

This is the third and last piece in my miniseries on measuring learning. Thanks for sticking it out; you’re almost there. Buckle up: This one’s long.

Three weeks ago in this space I explained why I believe colleges and universities must prove they’re actually helping students develop the capabilities, skills and habits of mind they say they are.

Last week I described how we got here—to the point that the public increasingly questions the value of degrees and of going to college at all, in part because they doubt learning is occurring. Confidence in higher education’s internal evaluations based on faculty judgment and grades has eroded, and external accountability systems have done as much damage as good.

Now comes the really hard part: suggesting how we might certify that students are ready, to prove to the public that postsecondary education is delivering what it promises, at a time when the emergence of artificial intelligence is reshaping how we work and live.

As a journalist for 40 years, finding solutions wasn’t my responsibility. I mostly pointed out others’ problems and nagged them to fix them. Now I feel an obligation to be more constructive given my current role as … wait, what am I? Consultant? Talking head? Believer in higher education yet healthy skeptic of how it’s performing?

Still, I venture into this space humbly. I’ve never designed a curriculum. Or (formally) taught a group of students. Or tried to measure learning. I know what I don’t know.

What I do know is that higher education has a credibility problem. Too many people now doubt whether the time and money they expend pursuing a credential at American colleges and universities are worth it, and whether the credentials they receive will be valued by employers.

Measuring learning won’t address the credibility gap by itself; we also need to make postsecondary education more affordable, improve postcollege earnings and other outcomes, and, yes, tell our story better so Americans remember that higher education actually does a lot of important things well.

But documenting whether and how much learning is happening is essential to the “value” argument; as I wrote a few weeks ago, “it’s hard to persuade people that your primary ‘product’ is worth paying for when you can’t prove they’re actually getting it.”

So how might we reach the “holy grail” of truly measuring student learning?

Step 1: What Do We Want to Measure?

The question of “what is college for?” has many answers, and no single valid response. But many of the arguments we have—is it for career preparation or broad knowledge? Personal growth or earnings potential?—are binary to the point of banality.

Yes, colleges in different sectors have varying missions, and institutions that specialize in fields with licensure exams (think nursing, law and engineering) have external, structured targets for student knowledge and development.

But if you zero in specifically on what colleges promise to their undergraduate students (as opposed to their local communities through business development, or to society at large through research), they look way more alike than different.

Take the core educational competencies that, say, the Virginia Community College System wants its students to develop, and compare them to Michigan Technological University’s 12 “essential abilities”; Stanford University’s “ways of thinking, ways of doing”; and Carleton College’s “higher-order thinking skills.”

Given the differences in these institutions’ missions, it’s not surprising to see modest variation in the language they use to describe what they are striving for in educating learners. But all of them identify capacities such as critical (or analytical) thinking, communication, quantitative reasoning, ethical or civic capacity, breadth (or depth) of knowledge, and working with others (or across difference).

Widening the prism, we’ve seen several larger past efforts to define the competencies and skills of a college-educated person. The Liberal Education and America’s Promise initiative’s Essential Learning Outcomes Framework from the American Association of Colleges and Universities morphed into the VALUE rubrics, and Lumina Foundation’s 2011 Degree Qualifications Profile sought to gauge what learners should know and be able to do by the time they’ve attained an associate, bachelor’s or master’s degree, regardless of field.

These frameworks offer a useful starting point to show we could develop a common definition of the capabilities, knowledge, skills and habits of mind that learners should acquire with the help of an educational experience.

Both are at least a decade old, though, so they probably need revisiting to account for the emergence of AI, which is scrambling our understanding of work and what current and future workers (and human beings) need to know and be able to do.

Let’s throw into the mix, then, an emerging set of frameworks around what people outside higher education (and a few inside it) call durable skills: the capabilities and competencies employers believe will remain relevant in the face of whatever technological advancement or other massive societal changes are visited upon us.

These models come from organizations at the intersection of education and employment, like the National Association of Colleges and Employers, America Succeeds, the World Economic Forum, and the Education Design Lab. The Carnegie Foundation for the Advancement of Teaching is developing its own right now.

These frameworks overlap substantially with the ones developed by AAC&U and Lumina; they all include some version of critical thinking and reasoning, problem-solving, ethical judgment, communication and collaboration.

But they diverge in other ways: Those developed with higher education in mind prioritize deep knowledge of a discipline, which doesn’t appear in the work-focused systems. The durable-skills frameworks, meanwhile, emphasize the importance of personal traits like self-management, resilience and capacity for growth, which are largely absent from the Degree Qualifications Profile and VALUE frameworks.

Picture a process, then, in which we convened a group of people who approach post–high school learning from varying perspectives: faculty members from multiple fields, assessment experts, administrators from an array of colleges and universities, employers, researchers on the future of work, high school educators, and maybe a few actual learners.

Their goal: coalesce around a forward-looking, detailed framework for what a person “educated” after high school ought to know and be able to do to work and function effectively in society. (I’d picture it having 10 or fewer high-level capacities, cascading to a larger number of subskills. Broad enough to capture what’s important, targeted enough to be meaningful and to give us a snowball’s chance of being measurable.)

The diversity of the group would be essential because of a core tension underlying this whole conversation: Who judges what learning is and whether it’s happening? For centuries it was largely left to colleges’ faculty, but the employers who hire students and the learners themselves clearly have a stake, too.

That’s why we must strive for a consensus among them on any attempt to define goals for what colleges and universities should be shooting for in educating learners, and ultimately what we should be measuring against.

Step 2: A Question Before ‘How’?

Some wise souls have warned me that reaching any kind of collective agreement on what a college-educated person should know and be able to do is a fool’s errand. I take that seriously, as I don’t want to be Don Quixote at this stage of my career. Humor me for a moment and assume that the process I just described results in a blueprint that is palatable to professors of all stripes, corporate HR managers and parents and students alike. (A boy can dream.)

Before we get to the really tricky dilemma of how we might measure whether learners are reaching the goals we set, one key question demands consideration. (People like John Warner have raised others, not all of which I can get to in this column.)

• Proficiency or growth? An age-old debate in education research is whether it’s more important to assess whether learners know or can do something (proficiency or competency) or how much more they know or can do than they did before (growth).

The debate particularly rages in accountability circles (especially in K–12) because measuring proficiency sets a clear common standard but doesn’t account for differences in where students started, so it benefits schools (or colleges) dominated by well-prepared students (and hurts those that serve many disadvantaged learners).

For the purposes of this column, the question is whether we’d strive for some measure of competence (upon completion of a degree program or curriculum, does Jane know x and can Johnny do y, and how well?) or seek some kind of value-added measure that shows how much better (or worse) Alex and Pat are at knowing or doing things than they were when they appeared at our door.

Here’s why that matters. A competence measure would tell us most of what we need to know for the sake of learners themselves: Are they ready for what comes next in work and life? It could certainly tell us something about the performance of colleges and universities, too: Are Monumental State U or Cozy College generally delivering on the commitments they make to learners?

A value-added measure would answer those questions and provide something else, too: a sense of how much the college or university journey itself has helped develop its learners. By gauging what learners know and can do at the beginning and end of an educational experience (or at points along the way), we’d also get a sense of which institutions do the most with the students they enroll, regardless of their previous preparation and background.

The latter appeals to me personally, because one of my stubborn beliefs is that the public regularly assumes that the “best” colleges, where you learn the most, are the highly selective and wealthy ones that frequently filter out the students who need the most support. I’ve argued before that I’d bet that students at Fill-in-the-Blank Community College learn and grow more during their time there than do their peers at Highfalutin University in the fancy suburb next door.

A value-added learning tool would prove me right (or wrong), so I’m drawn to that. It would also be more useful from an accountability standpoint, since it would create a more level playing field for judging the contributions of the institutions themselves.

But creating a system that measured growth would also be much more complicated and expensive, because it would require evaluation at multiple points. And ultimately, what matters most to me (given my primary focus on restoring public confidence in higher ed’s value, rather than on accountability or internal improvement) is whether learners are prepared or not for what comes next. So I’d favor a measurement mechanism focused on competency, at least as a start.

Step 3: How Might We Measure?

I’m going to spend a lot less time on this final section than I originally planned, for two reasons: (1) As I worked on this set of columns, I’ve realized that the most important task by far is defining (and reaching consensus on) what we want today’s and tomorrow’s learners to know and be able to do. So many good things would flow from doing that that how we measure it might matter less. And (2) the how is really hard, and I honestly have few answers for that. But I’ll try anyway.

The short version is this: Once we’ve developed a common framework for what students know and can do, there are two likely options—a capstone test/exit exam, or a method of evaluating authentic student work against the framework.

• The Test: The prospect that some kind of examination could accurately measure whether students have the capabilities, skills, knowledge and habits of mind is enticing. It would satisfy those who favor a significant level of cross-institutional “objectivity” over the subjective judgment that plagues our historical approach: a diploma purportedly certifying learning based on faculty judgment (grades) on a set of completed assignments and assessments, plus time spent in the hallowed halls (credit hours).

I suspect that if we poured a ton of money and energy into the effort, we could improve on previous efforts like the GRE and the Collegiate Learning Assessment (see last column) and end up with something that might satisfy psychometricians and politicians. Concerns about the price tag and how you’d incentivize learners to take the test seriously might be hard to overcome.

Most faculty members would hate it for more reasons than I can list, and they’d be right about some of them.

Would it be worth trying? Heck yes. Am I optimistic that it would work, or that it would be the best option? No.

• Portfolios of student work: If the goal is to develop a mechanism for evaluating and certifying learning in a way that is persuasive to the public yet acceptable to faculty members, it’s much more likely to come from analysis of authentic student work than from some external examination. (There are more and less justifiable reasons that many professors dismiss assessment of learning, but faculty worries that students would focus on an externally derived tool at the expense of their actual course of study are legit.)

The use of electronic portfolios as an assessment tool in higher education emerged in part as a response to testing because they allowed for evaluation of actual student work in a low-stakes environment tied to material professors valued. They also build in students’ reflection on their own growth in a way that most exams don’t.

Picture an approach where the set of capacities developed by my learning supergroup would be turned into a rubric against which a collection of authentic student work was judged. (If this sounds like how AAC&U’s VALUE system works, yes, same ballpark. Some competency-based education approaches work this way, too.)

I’m drawn to the competency-based approach of gauging learners’ levels of proficiency (e.g., novice, developing, proficient, distinguished) rather than a grade or numerical rating for each competency. Whether that would be specific enough to satisfy all the audiences this whole process might need to serve is a question for another day.

Inertia and faculty distrust of common rubrics substituted for their own personal judgment are two major (and I’d say flawed) reasons why competency-based approaches haven’t taken hold in the way advocates hoped. The convoluted regulatory environment around federal financial aid has probably been the biggest impediment.

But another, very legitimate practical one is the cost and labor intensity of actually scoring student work. Creating an infrastructure to collect, tag and organize all the artifacts that learners produce—assessments, papers, videos, oral presentations, team projects, statistical models, musical scores—and then training and compensating faculty to judge and score the student work against a rubric is complicated and expensive. Nobody has had the incentive to invest in doing so at scale so far.

I’d argue that we arguably have the incentive now—you know, to maybe prove that going to college is worth it, to save the industry all those professors work in?

But this is also where technology, which may be our collective downfall, may also be an ally.

What if we could marshal the power of artificial intelligence to effectively (over time, with an enormous amount of training and human oversight) grade authentic student work conceived and assigned by real, live faculty members against the competency rubric we’ve developed?

I’m way outside my depth here, but smart people like Eddie Watson and Kate McConnell at AAC&U and Tim Knowles and Brooke Stafford-Brizard at Carnegie are already thinking about this, as I’m sure are others I haven’t spoken to. Technology companies are making lots of promises, too, and I take every single one of them with tablets, not grains, of salt.

The End, for Now

I’m running out of steam here (as are you, if you’ve made it this far, probably), so I’ll conclude with this.

If the grand vision I’ve just laid out came to pass, the next few steps would involve using the new information we’ve gleaned (about how well students are developing the capabilities we want them to have) to embrace more of the curricular and teaching approaches that research has shown to work best.

We’d make the information publicly available to help upend a student decision-making process that too often is based on colleges’ reputation, wealth, marketing budgets and the like.

We’d develop better navigation systems to match learners and potential employers based on the documented strengths and weaknesses they exhibit on the durable capacities we’ve agreed on.

I’m sure I’ve tried to do too much in this series of articles, and yet not enough, too.

Mainly what I want is to try to stimulate more and better thinking about what we want college-level learning to be in this important, possibly existential moment, and how we might reassure those who’ve lost faith in us that we can, and want to, be better.