Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Problem with Assessment


Published on

ATI January 2019 Event - Friday, Jan. 18
Open Education & Student Agency: Empowering the Student Voice & Facilitating Access

Published in: Education
  • Be the first to comment

  • Be the first to like this

The Problem with Assessment

  1. 1. Curiosity, stream of consciousness, critical thinking, + communities of practice Jess Mitchell Sr. Mgr. Research + Design @jesshmitchell Open Education & Student Agency: Empowering the Student Voice & Facilitating Access When I was a child I spoke in what my parents considered a very frustrating manner — it was stream of consciousness (barely consciousness). They claimed it lacked a kind of conscientiousness for audience, a regard for those who had to listen to my wandering paths of thought and what seemed like a lot of randomness. I annoyed people with my chatter — my random revelations and my near constant eureka moments. I learned how to keep most of those moments inside and to figure out how to talk in a way that others might understand and tolerate me. Let’s hope I learned enough :) I was ready to burst with excitement. I couldn’t sit still — I wanted to learn, but I didn’t really want to sit and read! And I was told that’s how you learn. So, I tried, I’d dive into magazines (Skiing magazines, Sports Illustrated, Rolling Stone, Field and Stream, Archeology, National Geographic, Smithsonian — they all played a big role in my learning). Anyway, I liked the combination of the pictures and the text, the human-interest stories with the exciting discoveries. I was making sense of the social, the world, the wonder… I was learning. There’s a TED Talk about a game the speaker (Rives) came up with when the Encyclopaedia Britannica stopped being printed. He called it Chimborazo — it’s a sort of scavenging game on the internet called Chimborazo. The game consists of a kind of rabbit hole wandering from links within Wikipedia. The ‘player’ chooses a link that is something they didn’t know (as Rives describes it, “It's like ‘eureka’ and ‘bingo’ had a baby. I didn't know that; that's pretty cool. Chimborazo!” (01:24) From revelation to revelation the player moves until it all comes full-circle. Imagine my delight when I saw this TED Talk. Rives has made a game that describes exactly what my mind does-- what I delight in which I feel gives it some legitimacy. So, we’re going to do some streams today — I’m kind of giddy about that. Still, I feel somewhat apologetic about what I’m about to subject you to. How many of you have heard me speak before? How many of you saw or heard or read my OpenEd Keynote? Ok, I have some homework for you. I ask that you capture two barnacles from what I’m about to say. That way, we’ll have no shortage of things to chat about later.
  2. 2. No Sales Tax Live Free or Die So, some free association about NH — I think you should know what I, as a visitor, think about when I think about NH… Elizabeth Warren, her husband Bruce, and her golden retriever campaigning Swenson quarry — last one that was NH owned — now owned by a company in Quebec. The granite makes me think of…
  3. 3. This guy in mountains… you used to have this guy there in Franconia Notch and the mountains make me think of…
  4. 4. And those mountains — ahhhh. This one, Burke Mountain — in a region of big ski areas, I love the way it felt to go skiing at Burke Mountain. There were no bullhorns, no gimmicky loud speaker things happening. It was quiet, beautiful, and lovely.,g_custom/v1/properties/7076/l4pl9auukmxz4dkvo2is
  5. 5. Skiing is fun, and it can be functional too! Model-T with ski kit Virgil White of West Ossipee, New Hampshire was a successful garage owner that was the authorized Ford dealer in that part of the world during the 1920s. In 1913 he had come up with the snowmobile concept but it wasn’t until 1923 when he would actually try to make a go at marketing and selling them.
  6. 6. Speaking of Henry Ford and the Model T This guy has a cast-iron steam engine which once belonged to Henry Ford built into the multi-story center atrium of his house no far from here as I understand it. This is of course Dean Kaman, an inventor of many things (all very different from each other). By Airman 1st Class Keenan Berry -, Public Domain,
  7. 7. Kaman invented the iBot a mobility device (NOT A WHEELCHAIR) that could climb stairs. Here is Michael Graves, now gone famous designer using an iBot. Graves became paralyzed later in life and in this TED talk he reflects on how his design perspective changed when that happened. This underscores the importance of diverse voices, experiences, and ideas in our design teams and groups. There’s a wonderful video online about how Graves redesigned his home in Princeton NJ so he could access the rooms, the dishes in the kitchen, the shower and making it amazingly beautiful design: His Tuscan barn sanctuary home in Princeton. And anywhere in NJ reminds me of my family… though I grew up in the Midwest, my family is from the NY/NJ region — immigrants.
  8. 8. Live Free or Die And being an immigrant to this land is something I spend time thinking about… I think about how to meaningfully give a land acknowledgement when I speak. And it isn’t something I want to read from a copy/paste of text from a website with a database of unceded territories. I want to say a little bit about my connection with it. My commitment to the land… I was born in Concord, an immigrant of sorts. My family lived here in Manchester for two years. It’s the place where I got chicken pox. And the time I felt most in awe and connected to this land here in NH was when years later, as an adult, I took my parents on a kayak trip on the Androscoggin River. We paddled through the land of various Abenaki tribes, NH largely divided between the Androscoggin Abenaki and Pennacook nations. And I imagined the history that took place on this land — who paddled down that river first — where are their ancestors are now? Are they here today at this event? And if not, why? That is my acknowledgement of the land we are on today. And you’ve survived round one of chimborazo!
  9. 9. Open Education & Student Agency: Empowering the Student Voice & Facilitating Access why aren’t we empowering student voices? why aren’t we facilitating access? what do we want to change? why haven’t we?? let’s break this down a bit to understand…
  10. 10. I want to start here. Right here. This is the place. This is where we had the “gravel stop.” That’s what my mom called it. Back then there was a train track just ahead. This is a Google Maps Streetview — clearly the railroad track is gone. But the gravel is still there. My mother skid her tires in the gravel and stopped short. She pointed her index finger at me and executed a parent-level tirade. I would go to college. I would stop with this resistance to the structure of education — even if it made no sense — I would play along and that was final. That was the best deal she had to offer me. I felt fairly cheated and severely put-upon. I can’t remember which time this happened: was it when I refused to memorize the countries and capitals of Africa (I wanted to dive in and explore each place and I thought memorizing was a waste of time); was it the time I refused to memorize the periodic table of elements (show me, let me experience them — memorizing made me feel patronized and bored); was it the time I had to memorize all the prepositions in alphabetical order — about, above, across, after, against, along, among, around, at, before, by, down, during, for, from, in, into… nope… it wasn’t that time. I was in Grade 5 and I did that one and I’m still bitter about the uselessness of that exercise and how much space it takes up in my brain. It was early that I learned that education was a series of memorization responsibilities. There was no thinking, no learning, no exploring. Surely success wasn’t just measured as retention of data! I got more from wandering around in the woods than I did from the classroom. I got more from skipping school to watch my peers build a half pipe, and watching them work with local authorities to make skating a sport, not a crime. I got more from memorizing song lyrics that expanded my world and my imagination (and indulged my moody teenage self). Me, my thoughts are flower strewn Ocean storm, bayberry moon I have got to leave to find my way Watch the road and memorize
  11. 11. I always had this image when I was in my teens and twenties. There were two trains travelling parallel to each other (in the same direction). I was riding one train (the one on the right) and when I looked out the window, through the passing trees between the tracks, I could see to the other train. That train was where the party was. It was where people were free, happy, curious, full of wonder, discovery, experimentation. They were tromping around in Gombe National Park in Tanzania interacting with gorillas. They were in the Antarctic studying penguins. They were building a Stirling Engine that would convert dirty puddles to potable water. They were writing brilliant plays. They were the most gorgeous dancers you could imagine. They had great hair, they were cool. There was Glitter on the mattress Glitter on the highway Glitter on the front porch Glitter on the hallway — B52s, Love Shack And I was stuck on boring train memorizing the prepositions in alphabetical order. I was struggling through Advanced Algebra. I was asked to memorize the periodic table and the countries and capitals of Africa. I wanted to BE in Africa. I wanted to be on that train so badly. And I had no idea how to jump the tracks. I had no idea if/when/ how I’d get to that train. Somewhere (not a specific moment in time) my life jumped the tracks and I found myself more and more on that train. What happened? When?? I mean, I don’t hang out with penguins, but I got to come here today to talk to you! I had opportunities that were (in large part) given to me — and opportunities beget more opportunities… Someone took a chance, then someone else did, then someone else noticed and they too took a chance… that was my career path — that was my learning path — a
  12. 12. There really is no substitute for experience So, how do you get new experiences? You either take risks yourself or someone gives you space/permission to take risks. Experiences are often given as rewards for achievement (success) based on following the rules. To get opportunities, and this was my Mom’s point, you have to play the game and ACHIEVE. You got to know when to hold ‘em know when to fold ‘em know when to walk away know when to run… — kenny rogers, the gambler To get onto the cool train you either work really hard or you step into some wicked luck, or both. It’s all just a gamble! Which brings me to assessment and harm.
  13. 13. The Potential Harm We Cause with the Game Assessment Achieve Career/ Success Opportunities You have to achieve to have opportunities and we know opportunities are how you gain skills, experience, insight — the sorts of things you build a livelihood/career/ upon. Opportunities are the amino acids (building blocks) of advancement. And opportunities often come from achievement. What is success? How do we determine achievement? ASSESSMENT The building blocks of all of this pathway to success really do start with memorizing the periodic table?? I don’t know about you, but this doesn’t look like the sexy train with the people with good hair on it… This looks like a lot of drudgery. And I’m not sure how much learning it represents. Isn’t it just knowing? Learning to me smells more like thinking, exploring, epiphany, critique, questioning, burning and building. Where does this drudgery come from? The University determines the minimum requirements (often GPA and SAT scores). The department determines the requirements to graduate with a degree (core and subject-specific). The syllabus lays out the rules. The Grammar Book lays out the rules. RULES RULES Sign, sign, everywhere a sign Blockin' out the scenery, breakin' my mind Do this, don't do that, can't you read the sign?) — Five Man Electrical Band, Signs I met a man at the OpenEd 18 conference who came up to me and told me he’d been rethinking his visceral reaction to grammar errors in his students work. He teaches in a poor community in Southern Florida — the kids are often non-native English speakers). He and I both grew up in households full of English speakers with a firm respect for grammar. My mother was a grammarian. In addition to teaching me to lose the stream-of-consciousness chatter, they broke me of using the work ‘like’ as a
  14. 14. Form & Function You see, we in this room are the ones who design the game. And we know the deep relationship between form and function. I can design a pathway to encourage you to walk in a certain way — IKEA does it with the arrow on the floor. The way things begin matters. Each decision is a design decision — this and not anything else — and those have a deep impact on how we function in the spaces (literal, virtual, and conceptual). Put them in chairs all lined up and staring at you and you’ll get silence and focus entirely on you. And very little engagement. Give them a strict rubric, they will function as though that strictness is a barrier between them and success. Give them a strict path to success, they will focus on the path, not on the things all around them — the things just off the path. Give them a go/no-go, some will simply crumble under the pressure. Lead them through something exploratory — they might sink right back into the rut of what they’ve always done, not knowing how to do it differently. Doing it differently doesn’t come in a recipe. It isn’t some one way — it should be the outcome of creativity, context, thought, and engagement. One Size does not Fit All.
  15. 15. My sisters: a case study in birthday cakes An interesting thing happened during a co-design workshop I facilitated. We created in an open, exploratory, intersectional, messy, collaborative, creative way and then it got boiled down to…
  16. 16. a. Definitions (C. Blake / CCDI) i. Defining Diversity ii. Defining Inclusion iii. Defining Equity vs. Equality iv. … b. What is the Business Case for Diversity and Inclusion? (C. Blake / CCDI) c. Guiding Principles (C. Blake) i. How to Engage Collaboratively / Respectful Relationships ii. Accountability iii. Shared Authority iv. Making Time for Mindful Reflection d. Self-Assessments (C. Blake / R. Myers / CCDI) a. Core Questions (C. Blake) i. What are my lenses? ii. Am I just confirming my assumptions, or am I challenging them? iii. What details here are unfair? Unverified? Unused? iv. What’s here that I designed for me? What’s here that I designed for other people? v. Who might be impact by what I’m developing? vi. Is my audience open to change? vii. What am I challenging as I create this? viii. Whose voice is here? Whose voice is missing? b. Case Studies (Catalyst / Advocate Teams) – as museums use the guide, opportunities to add case studies / stories to site i. *see Toronto Ward Museum mock-up for example (R. Myers / T. Pinkofsky) a. General Resources b. Training c. Consultants d. Community Organizations e. Employer Tools a. Blog / Forum i. Ask Questions ii. Create space for reflection and discussion iii. Address failures and access support (R. Myers) a. Webinars We flatten them and formalize them and seek to validate them! This is what was presented as the outcome from the collaborative info architecture session. We dressed up that creativity, combed its hair, gave it a starched button up, and some serious shoes. And in doing that we lost all the great work we had done. The people who weren’t at the design session began questioning the taxonomy of what we’d done and got bogged down in form and function we’d created. We all do this…
  17. 17. Control We like structure. We often fall back on it when we’re most unsure. We like certainty, completeness, simplicity, we like knowing. We approach the world with a transactional check-book accounting expectation – if we document our inputs and outputs, it should all reconcile neatly in the end. If I eat a healthy diet, if I exercise, if I sleep well, if I do all the things the doctor recommended, then I should know my outcome will be good health. If I memorize all the things the professor said, then I should get a good score on the exam. As a result, our institutions have adapted to contend with this fundamentally flawed, oversimplified causal thinking. For example, medicine contends with a patient that thinks a visit to the doctor should result in a tangible something, a new prescription, a referral, a plan; and education contends with a student who thinks that if she does everything the syllabus says, then she deserves an A (she’s done her part of the contract, she has learned).
  18. 18. Conundrum Rubrics tell us: Adherence to the rule is success; deviation from the rule is failure. So we codify the transaction in structure! This is the syllabus — this is what you’ll be measured on Strict adherence with little to no spark will get you a B, full-borne deviation will get you a D — because an F is really brutal. Forget about glitter, forget about good hair — these are marching orders. And that will be how we measure success. We will measure you all by exactly the same criteria — because that makes us feel fair; no exceptions — and there will be no substitutions to your pizza order. But we know this does damage… this is NOT A NEUTRAL ACT. Listen, everybody knows the fight is fixed; the poor stay poor, the rich get rich. That’s how it goes. Everybody knows the deal is rotten; old black joe’s still picking cotton for your ribbons and bows and everybody knows - Leonard Cohen
  19. 19. The over-reliance on metrics and measurement has become a tool used to remove questions of responsibility, morality, and justice from the language and policies of education. [It’s] a kind of mind-numbing investment in a metric-based culture that kills the imagination and wages an assault on what it means to be critical, thoughtful, daring, and willing to take risks. The distinction between information and knowledge has become irrelevant in this model and anything that cannot be captured by numbers is treated with disdain. Henry Giroux This was our little xmas present last year — in an interview with Henry Giroux he said the following… These are the tools of accountants and have nothing to do with larger visions or questions about what matters as part of a university education. The overreliance on metrics and measurement has become a tool used to remove questions of responsibility, morality, and justice from the language and policies of education. [It’s] a kind of mind-numbing investment in a metric-based culture that kills the imagination and wages an assault on what it means to be critical, thoughtful, daring, and willing to take risks. Metrics in the service of an audit culture has become the new face of a culture of positivism, a kind of empirical-based panopticon that turns ideas into numbers and the creative impulse into ashes. … The distinction between information and knowledge has become irrelevant in this model and anything that cannot be captured by numbers is treated with disdain. In this new audit panopticon, the only knowledge that matters is that which can be measured. Henry Giroux
  20. 20. Conundrum Many syllabi are the rules by which we measure success or failure. Adherence to the rule is success; deviation from the rule is failure. When measurability is success, it becomes an end in itself. We begin asking questions that lead us to measurable answers. We begin measuring those things that are easily measured. And those are not neutral acts. We act on our measurements – data become the tea leaves for decision-making, the map for change, the path toward advancement (data-driven decision-making in Education). And we feel a sense of comfort having followed the directions given to us from the disembodied data. a pushmi pullyu
  21. 21. Data Will Not Save Us What do we measure now? We’re measuring admissions; we’re measuring retention; we’re measuring graduation; we’re measuring in a micro form the regurgitation of information… if it’s hard to measure we often ignore it; because without measures/data we can’t make claims about it.
  22. 22. Strong leaders teach the hard stuff.- Vala Afshar @ValaAfshar sense of urgency critical thinking creativity customer empathy team commitment humility unselfish giving judgement grace and dignity positivity optimism bias towards results active listening do you measure? what you measure is what you value: Strong leaders teach the hard stuff.- Vala Afshar @ValaAfshar Chief Digital Evangelist @Salesforce
  23. 23. Whose rules? Adherence to the rule is success; deviation from the rule is failure. We measure what we value. Who is deciding what to measure? Let’s go back to that social justice issue: equity, diversity, and inclusion. The horrifying truth is, we perpetuate and amplify existing cycles of exclusion and further exclusion for many. Who is making the rules? Are the rules revisited as culture, humans, the economy, health, the environment change? Do the rules flex for edge or special cases that they don’t fit? What about current events?
  24. 24. Adherence to the rule is success; deviation from the rule is failure. Achieve Career/ Success Opportunities Whose rules? Achievement is defined by adherence to the syllabus — because we need to measure to feel fair, validated, and objective. No bias here folks! Opportunities are based on achievement over time. Opportunities help build skills, gain experiences, which parlay nicely into a career. Success! If you’re disadvantaged or excluded or left out of the measures at the beginning of this game, you never catch up. a high school student in Detroit and a high school student at phillips exeter will not have the same chances at any part of this equation. The playing field is not level. And we’re all different. Adherence in many cases is merely conformance with the form of the game to give the intended output or (function). Draw a person — how would you evaluate Picasso, or Miro, or Kandinsky? Have you done so equitably?
  25. 25. • Pace, Path, Content, Delivery Method • text, visual, sonification, video… • individual, group, didactic, participatory • Motivation – external, internal, positive, negative • Social support – peer, instructor, other • Degree of structure What if Kandinsky took longer? What if Miro wanted to do it alone, rather than in a group? What if Picasso wanted to do it somewhere else all together — from another location? Pace, Path, Content, Delivery Method text, visual, sonification, video… individual, group, didactic, participatory Motivation – external, internal, positive, negative Social support – peer, instructor, other Degree of structure These are just some of the diverse needs or preferences for learning. How do these fit into your rubric? Does your syllabus flex to these? Advantage some? Disadvantage others? Is the preferred mode even available?
  26. 26. Who do the rules limit? Adherence to the rule is success; deviation from the rule is failure. No exceptions You must report on your department’s success using these measures Only English speakers can learn from this material Only graduate students can take this course Your thoughts must be organized in a 5 paragraph essay You cannot use Wikipedia as a reference And why do we create and tolerate these rules? How else can we get good numbers with which to make our data-driven decisions? And how worried are we that we’ll lose you through a false negative in our assessments?! False positives I worry less about. Perhaps you’ve been innovative in gaming the system — perhaps it will all catch up with you. False negatives I worry A LOT about. Not just the diamond in the rough, but the pearls, the sandstone, the sparkles that we squash out.
  27. 27. Failure Adherence to the rule is success; deviation from the rule is failure. With this: • We fail to go anywhere. • We often fall back on the way it has always been • comfortable ruts of thinking and doing This folks, is failure. But we say we want other outcomes. We don’t want stagnation. We don’t want to do it the way it’s always done without critique. Who likes ruts? Besides deer?
  28. 28. how we design matters; and every decision is a design decision For change to happen within any institution, the thing you want to change has to be a value; something you practice and continue to challenge and nurture. Inclusion is not a checklist item. It’s like breathing and bathing — you got to keep at it evermore. Nothing is neutral. every decision is a design decision that either contributes to or challenges existing structures of privilege, power, and presence. Each decision is an opportunity and a responsibility we have as leaders in these public spaces
  29. 29. & mroF noitcnuF What happens if we take a different approach to designing our rules?
  30. 30. What if adherence to the rules is failure and deviation is success?! What failure should be: • adherence to a rule • knowing without thinking (memorization) Ask of your students that they understand the rule and then go beyond it. That they undo it from the inside. That they express something in the following of the rule. This is a place for a critical mind. For discover, wonder, curiosity, creativity, love of learning. 3 cheers for the atypical let’s look at this model for a moment. Indigenous ways of knowing often characterize movement and progress in circles. The linear might just be a measure that isn’t universal, that isn’t fair.
  31. 31. What is Failure then? Adherence to too much structure
  32. 32. What is Success then? Just enough structure Personal control We should be creating only as much structure as is absolutely needed to move thinking forward and no more. We shouldn’t feel as though this means we’re unprepared – we should prepare ourselves with our instincts, our own experiences, and more than anything our ability to listen, hear, and change. 3 levels of structured structurelessness we need to focus on: 1. Syllabus 2. Pedagogy 3. Individual goals and definition of success We need to get better at balance (somewhere between structure and less structure). Are there disciplines for which this wouldn’t work? No. Architects need to build stable bridges though! Yes, and they need to make design decisions that impact all of us – the building shape us and we shape the buildings – Churchill Scientists need to adhere to the scientific process! Yes, and that process includes an awful lot of wonder, experimentation, failure – science gets wonky when it ignores the things it doesn’t know yet… All scientists should be natural experimenters. All of these things, you see, exist within our social and political structures. And each of these disciplines, teaching opportunities, and the individuals within them require a different and unique balance of structure and structurelessness. And so if we’re training people to follow the rules, then stick to structure. But if we want people to learn concepts, to understand history, to embrace diversity of thought --- then use barely the amount of structure that is absolutely necessary.
  33. 33. ReThink Syllabus Pedagogy Individual goals and definitions of success 1. Syllabus 2. Pedagogy 3. Individual goals and definition of success We need to design the syllabus so anyone can “walk through it” – like an architectural dreamland where the building adapts to your needs and preferences. We need to create teaching and learning environments that are focused on the goals and notions of success of the learners. We need to treat people as people, not as numbers that we’re afraid to touch.
  34. 34. what you measure is what you value When information is more important than knowledge, and certainty and measurability are more important than thoughtfulness, risk, exploration and discovery, what do we lose? What are we relinquishing? If to value something we have to be able to measure it and vice versa, what are we overlooking and missing? What is the byproduct or waste that is created by our need to have neat, simple, exact corners (in education)?
  35. 35. Activity • What do you measure? What do you measure? Who does it advantage? Who might it disadvantage? What are you measured against? What do you measure your students against?
  36. 36. Activity • Is that what you value?
  37. 37. Structure We start here — a computer-generated hopscotch field File:TiuFeiKei(Hopscotch)_pattern.JPG&psig=AOvVaw2AMcRvNb14d9RV7CcG7zAu&ust=1529104487597222
  38. 38. Regurgitated Structure Then we go here — a chalk-drawn and worn hopscotch field
  39. 39. Structured Structure-lessness But this is what we know is actually the context we live, work, grow, and learn in. Julie Mehretu — dimensionality; complexity; intersectionality; cyclical; global HOW THINGS ACTUALLY ARE: Unpredictable Dynamic Emerging Adapting When we let go of notion of clarity, we see differently. This is the sexy train. Now let’s talk about how to do this and answer these questions: why aren’t we empowering student voices? why aren’t we facilitating access? what do we want to change? why haven’t we??
  40. 40. beware the taxonomy beware the binary beware certainty beware of completion
  41. 41. @jesshmitchell