Measuring Learning Means Answering a Lot of Questions First

Provoked (in a good way) by Inside Higher Ed co-founder and current fellow columnist Doug Lederman’s most recent “Tough Love” installment on how “failure to measure learning is hurting higher education,” I sat down to write a response on how we can indeed measure learning.

I felt confident in my response, because shifting the orientation of writing courses from what I call “schooling”—essentially following prescriptive processes in search of a grade—to “learning” (you know, actually developing the capacity to do something and then translate that doing to other things) has been my focus for nearly two decades.

I know the difference between when a student goes through a school-ish process and when a student learns something. Students know this, too, by the way. The challenge, though, is to “measure” this learning in a way that satisfies the criteria Doug—I am going to dispense with the conventions of referring to authors by their last name here because I have known Doug for like 15 years—lays out in order to “win back the public trust,” namely “whether and how much students learn.”

Lederman (for some reason it feels right to go with the last name in starting a sentence^[1]) is not of the group that thinks learning isn’t happening but also believes that asserting from authority (what I do in the paragraph just above this one) “is different from knowing … And knowing is different from proving.”

I agree that asserting from authority is different from knowing, but I will still submit that I know—almost without fail—when students are learning, and when I’m not sure, I can have a conversation with a student to provide clarity for both them and me. In this particular case, my authority is rooted in experiences, and I believe my track record is pretty good on this front.

But the more I thought about Doug’s argument (there I go switching again, should I just go with Lederman and be done with it?), the more I realized that there are many more questions we have to consider—if not answer—before we can even head in the proper direction toward our goal.

Failure to understand the whole mission risks an Odyssey-esque journey where it would be something of a miracle if we managed to keep from getting consumed by a cyclops, or turned into a pig, or seduced by Charlize Theron and her lotus flowers just when we’re ready to head home to Anne Hathaway. (Too cute? Eh, why not throw a bone to the Odyssey discourse? People who have seen the movie will get it and those who haven’t will likely at least understand the reference.)

I don’t think this list of additional questions is even exhaustive, but it’s a start on some of the unspoken factors that must be made explicit if this mission stands any chance of success. I have some thoughts on answers to these questions, but for now, I’ll confine myself to the questions themselves.

Why is it a necessary step to “win back the public trust” now, whereas (as Lederman notes) in the past we let it slide? What has shifted in the underlying society and systems that requires this change, and is it possible that addressing those shifts more directly would be a superior route to restoring the public trust?

Who is the measurement for? Who is the audience for the measurement? Students? Parents? Employers? Government? If it’s students, I think we can solve this problem pretty quickly, but each of those other groups would be a tougher nut to crack.
What is this measurement going to be used for? Will it help inform pedagogical practices? Will it determine funding levels for fields of study or even entire institutions? Is this measurement part of a consumer guide for students in a consumer marketplace, or is it the backbone of public policy decision-making?
Is the measurement meant to rank students (and/or institutions) against each other or a prescribed standard, or is it relative to the individual student—e.g., how much each student progressed over time?

When will the measurement be made, and how often will it be made? Do we do this annually? Course by course? At the start of the college journey and then again at the end? Or should it be continuous, like a health monitor? What about doing it five and then 10 years postgraduation? Wouldn’t that be kind of interesting?
Do we need to measure everyone or just a representative sample?
How precise must the measurement be? Do we need to establish a scale with gradients, or can it be more like that thing on the Thanksgiving turkey that just pops out when it’s done? Do we need to establish a detailed map of what Doug Lederman calls “durable skills,” or will it be enough to look at a student and say, “They’re soup!”
Wait, what is a durable skill?
What kind of evidence will be acceptable? Self-testimony? Subjective judgment? Objective test?
What sort of variance (if any) will be made for students in different fields of study? Can the same measurement serve for the musical performance major and the engineer?
Finally, and to my mind, most importantly, how do we avoid the trap of Campbell’s law, where the measurement becomes the goal and therefore ceases to be a good measure, a phenomenon that has repeatedly swallowed well-meaning attempts at quantifying learning across all educational levels?

Compiling these makes me feel sort of excited for the challenge of measuring learning, because it’s a big question, and I think we could learn a lot in the process, even if we fail. If nothing else, surfacing the fact that there likely isn’t a solid consensus about what constitutes “learning” would be progress over the status quo.

This is particularly exciting if learning becomes a focal point of the work of universities in ways it currently is not. Like Doug, I’m confident that learning is happening in colleges and universities, but my personal experience suggests that learning, per se, is not the main operational focus of our educational institutions.

I cannot help but return to the words of Carol Christ in 2016, then vice chancellor and now chancellor emerita of the University of California, Berkeley: “Colleges and universities are fundamentally in the business of enrolling students for tuition dollars.” This seems as true now as it was then, and no doubt is at least part of the reason for this declining trust. The notion that surfacing learning could fundamentally change how people relate to institutions would presume some fundamental shifts in priorities.

For example, just a couple weeks ago I wrote about how little institutions seem to care about teaching writing, based on the actual resources they put to the task. Would this initiative change these choices?

Would proving learning or showing that some institutions are better at inculcating learning change the choices students make? I have some doubts. For example, Harvard could be shown as quite average in terms of learning, at least as measured by growth, rather than relying on the dependent variable of admitting smart and (predominantly) wealthy kids to your institution, but I think it would continue to retain its prestige.

If we found out elite institutions are kind of average, learningwise, would we see a further erosion of trust of higher education, or would we see a renewed appreciation for more accessible institutions that may be doing better on our metrics?

So many questions. Hopefully, before the “mucky-mucks” Doug Lederman has been talking to about this problem get down to deciding the fates of the people who do the work of teaching, we get as many questions on the table as possible and commit to pursuing an answer that’s real and meaningful, rather than merely convenient.

(Do I want to end on this semi-bitter note? I think so, given that I smell yet another top-down initiative that will be imposed on the laborers of higher ed without their participation or soliciting their learned [and earned] expertise.)

^[¹^] Some of these parentheticals are an example of how I know students are developing their writing practices—essentially, that they’re able to internally narrate the choices they’re making as they’re writing. I would put this kind of metacognitive awareness at or near the top of what Doug calls “durable skills.”

Source by [author_name]