share interactive transcript request transcript/captions live captions
|
MyPlaylist
GRETCHEN RITTER: A couple of years ago, Larry Summers published an article in the New York Times titled "What You Really Need to Know." Summers began his essay by noting that leading universities often set the terms for the US educational system more generally. And yet, at our leading universities, undergraduate education has changed remarkably little over recent decades. Summers believes that this needs to change and that universities need to move beyond inertia to become more innovative in our approaches to education.
I would add that this inertia in our educational system is all the more striking given two recent developments-- namely the spread of information technology as well as the growth in cognitive science. These two great developments offer opportunities for us to strengthen the pathways to educational success, while building on the skills our students will need to compete in the job market once they're done with college.
But let me return Summers. Despite his concern about the prevalence of what he considered old school thinking regarding undergraduate education, his essay was mostly positive and forward-looking. He wrote about six emerging trends and opportunities in an undergraduate education.
I won't review all of those here. But I can tell you that much of what Summers laid out in terms of future approaches that will advance undergraduate education matches with the work that we're doing here with the ALI Initiative at Cornell. And that's what we're going to hear about today.
So what is the Active Learning Initiative? In brief, it is a program designed to improve student learning and academic success at Cornell. To achieve these goals, the faculty you will hear from today have been working with experts from the Center for Teaching Excellence to redesign core lower division courses by leveraging what we've learned from cognitive science and what we can do with educational technologies.
New approaches are being adopted in physics and biology, approaches that enliven classes, encourage collaborative learning, and allow students to practice and master core concepts. What this initiative represents is a shift to a more student-centered educational model. We are migrating from a model of education that is focused on the instructor-- think about the sort of brilliant instructor sharing their wisdom three times a week in 50-minute chunks-- to one where education is more focused on the student and having students take responsibility for their own classes, for setting their learning goals, for practicing core skills and concepts until they master them.
We're moving from a model of education that is about how much time the student is in the classroom each week to one that is about the competencies and orientations they develop by the end of a course and the skills they will carry with them into their next course. Part of this involves using educational technology, like online learning resources, like classroom response system, and like learning analytics.
Why does that make sense? Because we want to equip our students to be lifelong learners. And we want to provide them with the opportunities to develop the skills they're going to need to succeed, not just in the job market, but in life and in an interconnected world, where information technology has already revolutionized business practices, science, medicine, politics, and the news, among other things.
It is also fast reshaping how people connect socially and how they access and process information. This is where our students live. And it's a reality that we need to take account of in our role as educators.
If we do this right, then technology enhanced approaches education will make our students more engaged in the learning process. If they are more engaged, they are less likely to fail or quit a course. If they are more engaged, they are more likely to become more empowered and self-actualized learners.
Well, as you're about to hear, the preliminary results on this experiment are in. And I think you'll agree, we're having a lot of success. Finally, let me say before I turn this over to our presenters, that when I look at the work that's being done in this program, I'm inspired by the level of commitment and innovation we've seen from the faculty, the staff members, and the departments involved in this project. These dedicated teachers care deeply about our students and are willing to go above and beyond to provide opportunities for our students to get the most out of their time at Cornell. So thank you all for your great work.
[APPLAUSE]
PETER LEPAGE: I'm Peter Lepage. And I'm the director for educational innovation, which is an organization that has exactly one employee, and that's me, and I'm half-time. But I've been helping Gretchen with the ALI, which is the Active Learning Initiative.
The first iteration of the ALI began four years ago. What it is a an internal grant effort, where we make funding available and departments write proposals in a grant competition. And they write proposals for the resources to try and make major changes to their curriculum.
What you're seeing today are results from the first iteration of the ALI, which started four years ago. We're in the process of launching a new competition right now, inviting departments again to submit proposals for new efforts. Our first competition was focused on STEM teaching, because that's where there's a mountain of research coming from university STEM departments from cognitive psychologists and others that give you some ideas and moves that can be very, very effective.
In this round, we're opening it up to the whole college and inviting proposals from departments in any subject. These grants can be very large, like up to a million dollars. We also have opportunities that are much smaller than that. If people want to find out more about the ALI effort, you Google Active Learning Initiative Cornell, and you'll get several links to our web page just by doing that.
What we're going to hear today is presentations. We asked faculty members from the three departments that were involved, which are ecology and evolutionary biology, neurobiology and behavior, and physics. And we've asked each department to send over a faculty member who is involved in that effort to talk about their experiences in re-engineering their courses to build in active learning. And I think they'll tell you what we mean by act of learning just as part of talking about what they did.
Before we get to those speakers, though, we invited Julia Thom-Levy to come talk. Julia is the provost's fellow for pedagogical innovation. So she has university-wide responsibility for this kind of thing.
And I asked her to come and give us kind of a university perspective on it. But Julia also was involved in the ALI effort in physics. And so herself will share with us, I hope, a few words about what her experience was and what it was like.
We'll have each of the speakers talk, a few minutes for questions after each speaker. The idea is for you to ask anything you want to ask about what they did. This is an opportunity to find out what people have been doing in this initiative.
What's exciting here is, this is very much faculty-driven. It's what the departments make of it. And I agree with Gretchen, what's really impressive is what the departments have made of it and just the investment of time and energy and creativity that have come from the faculty involved. So let me turn it over to Julia first. Julia is also dean of Bethe House on West Campus, because she likes to have a life that's very full.
JULIA THOM-LEVY: Thank you for having me. It's very exciting to be here. I am a graduate of the ALI. Thank you, Peter, for the introduction.
And I just wanted to start out by saying that it had a huge impact on what I think where the learning outcomes, on me as a teacher. And it really sparked a whole interest in undergraduate education that I didn't know about before. So it was a huge, maybe the most important event in my teaching life for me.
So I was part of the ALI experiment in physics-- and Tomas will talk about it in detail-- but just maybe a few personal impressions. So we flipped the big intro classes, the freshmen intro classes, so there's are up to 300, 400 students in a big auditorium. And I was pulled into the initiative. I was a little leery of the effort that this would be.
And I had read the papers, was just a little hesitant, because that's not the way I had been taught. And it seemed strange to change everything when it had worked for me. And I'd been inspired by my lecturers, and so on.
Then as we were implementing the changes and preparing for a whole semester for rolling out these new courses, I was getting very worried. I had several identity crisis, where I thought, we cannot go out and they will not work with me. They will not respond, and I won't have a lecture.
I won't be able to talk to them. So what will we do? And Dean Ritter really was supportive during this time, I remember. that. That was very important for me.
And then the first lecture came along. And it just worked. I was just blown away. The lecture room came to life. People were talking to each other. They were talking to me. I was in there getting feedback, learning, even in the very first lecture, where people, I just have one activity, and I learned so much what they were stuck on, what they were thinking. I learned some names.
And it was just from then on, I was completely sold. I had not seen any outcomes, any exam scores, or anything. But it was just very clear that this is how we have to teach. It has to be student-centric. We have to take their input, we have to be coaches in the learning process and not a lecturer who's detached from the class and talking in front.
And from then on, it was just worry about how many years of damage I had done. Maybe I had just been a particularly miserable teacher before. So it just seemed the impact and seeing certain students come up to me who would never talk to me before was really incredible. So I consider it a big success.
And the question is, what made it successful? What were the ingredients? What was needed to do something like this in physics?
And there are a few things that come to mind. First of all, it was very important that it was done as a community, as a small community. So I had six colleagues to talk to throughout the process. When I was getting really scared, somebody else would say, no, let's just do this, it'll work.
And this community has even extended beyond the department and even across colleges. So somebody in engineering would talk to me about this. And I would meet people in biology or other departments. That was also very important for somebody only acting within physics at this point.
And then maybe even more importantly was the institutional support. So the department chair, Dean Ridder, Peter Lepage, having people who would express over and over that the institution was committed to this, this is what we want to do, it has been proven, and it work was incredibly important.
And then the third ingredient, I think, was the support through the central resources. So the Center for Teaching Excellence-- I see Amy-- who would help with assessment, who were there in class with me, taking stock, keeping notes, and then giving me feedback afterwards. And it was sort of an iterative process. Would help with Qualtrics feedback forms and just supportive all the way, this was also really, really important.
So on the latter point, in my position as provost fellow, I am fiercely committed to continuing in this tradition and doing everything I can to provide university-wide central resources for this effort and efforts like it. I really think this is important. Their experts know about learning analytics and assessment and can help us faculty implement them and really make great progress.
So looking forward, I'm excited to see interest and more people who are going to be part of this community. I will continue working on my teaching. And I'm just excited to be part of a vibrant community of teachers here at Cornell, working on these things. So handing over to the lecturers. Thank you.
[APPLAUSE]
KELLY ZAMUDIO: Ecology and evolutionary biology was one of the departments that jumped on board in the first initiative, in the first Active Learning Initiative round. And I'm Kelly Zamudio, one of the leaders of the large control bio class, called Evolutionary Biology and Biodiversity.
And we basically took a leap of faith. That's sort of how it feels four years later. And so what I'm going to do is tell you today a little bit about how we did that, what the outcome was. And then I'll give you some perspectives on how I felt four years ago when we started thinking about this and how I feel about it now.
Before I get started, I wanted to make sure I gave credit just to Cissy Ballen. Cissy Ballen was the Active Learning post-doc hired through the Active Learning Initiative to work with us. She worked very closely with me and the five other instructors for evolutionary biology and was critical in this process. So a lot of the work here is credit to her.
So our goal was to see if we could convert this large Intro Bio class into an active learning format. Some of you will recognize this-- all of you will recognize this, Call Auditorium. I've been teaching this large course since 2007 as part of a teaching team. And I remember, many, many years standing out here on this platform looking up at 300 students, every single one of them daring me to bore them.
And that's really what it felt like. We would put on shows. We would try to be entertaining. We would crack jokes.
But the bottom line is that they were doing everything here-- shopping for shoes, looking on Facebook, reading newspapers. It was an interesting time if I think back at it. So the goal was, the idea was, can we make something different that actually gets these students engaged to a different degree?
When Cissy came on board, she and I also discovered that we had some interests in common. We were interested in measuring what the effects of active learning would be on student performance in general, in terms of their engagement, in terms of their attitudes, and in terms of their learning. But both of us are very interested in issues of historically underserved populations in the sciences, such as Underrepresented Minorities, URM, or women.
And I don't have to tell this group, but basically, the demographic of our society is not reflected in any way in terms of diversity or gender in the STEM workforce. And there are many factors in the literature identified that contribute to this pipeline leak in terms of diversity in STEM. But one of them that has become more prominent recently in the literature has to do with issues of climate in the classroom.
So how is it that that climate in the classroom propagates a learning gap between URM students or women and other students of the majority demographic? And what does that mean in terms of their ability to continue in STEM professions? So we were interested specifically in addressing this question.
We had a number of challenges. Actually, I already referred to some of them. We were working with a large auditorium-style classroom, Call Auditorium. We had 280 students that were spread all over. It's a traditional lecture theater, so it's not set up in any kind of way to benefit small group interactions or dyad interactions or anything like that.
And I also teach a collaboratively taught course. And so that meant, I had to convince all the other faculty in that group. There was a particular dinner at somebody's house one day, where we made the decision and shook hands. And that's sort of how we moved forward.
I think everybody now in retrospect is very happy that we did so. But let me tell you a little bit about how we did this. So we set a relatively modest goal for ourselves for starters.
And we decided that when we made this transition that we were going to start by just increasing the amount of active material in each 50-minute lecture course by 20 minutes. So that's not a lot-- 20 out of 50. So at least 20 minutes of active material in each class was our goal.
We then made decisions in each lecture as to which material to actually remove from the lecture and have in what we call pre-lecture format-- so either a video, a vodcast that they watch before the class starts, some assigned reading, and so on. And these were usually associated with very low stakes-- assessment via a quiz. So by low stakes, I mean one or two little points-- an incentive, basically, for them to watch the videos.
We enforced that students sit in groups. We randomly assigned all 300 students to groups of five. They were forced to sit together during the entire semester and work together. We then used a number of different activities.
And by activity, I mean anything-- problem-solving, a series of iclicker questions that become increasingly complex, a concept map-- all sorts of activities, anything that basically had the students conversing among themselves, coming up with a solution, and then presenting that solution to the class. Again, we had incentives, low stakes, low number of points, but enough incentive to get everybody participating in these group dynamics.
We then did plenty of assessment with help from CTE. We got information about how much active learning was actually changing our teaching behavior. We also collected some other data, such as pre- post-tests on how students felt in terms of their confidence in the material. We also had a survey about the sense of inclusion and belonging that might, we thought, get a little bit at this idea of classroom climate.
So how did we do this? Let me just back up a little bit. On a daily basis, it sounds simple. It took time. I'm not going to fool you. But if it was not that complicated.
So I would look at a lecture. I would decide which concept or learning goal-- first, I would decide which learning goal I wanted to focus the activity on. I would then figure out the two or three most critical concepts necessary for that learning goal.
I would decide which part of the lecture to move before the lecture. And then I would just plop the activity right into the middle of the lecture. That's basically what happened.
We then equipped ourselves with the tools for active learning. Students were requested-- well, we requested that they bring clickers to class. We bought this wonderful Catchbox, which is a microphone that's embedded in a Nerf ball like material that you can throw around the classroom. And that really speeds up answering of questions, because the ball is thrown through the classroom from student to student.
We used heavily of this doc cam, because we do a lot of report backs. Students have to present their work to the class. We had a digital microscope here that basically takes tiny things and amplifies them on the screen.
And this semester, I'm playing around with these scratch cards. They're like lottery cards. They're immediate response cards.
So you have a series of clicker questions, and the group as a whole only gets a single scratch card. And this forces them, of course, to arrive at a consensus before they scratch it off and see that they got the star. Those should be pretty fun.
Here's some examples straight out of my lecture of the kind of activities that I would do. So in this case, this is a lecture on macroevolution. The concept that I'm trying to get across is that certain biological processes that have to do with species becoming different over time in the fossil record cause different patterns. So here, we have different rock layers. And over the course of time, this one species in this area has become two.
What is that going to look like in the fossil record? So I introduce them to the concept of time, morphology. Looking at variation in morphology, we discuss means. We discuss variance.
And after a series of about three or four questions, we end up basically inferring what the fossil record would look like given this process of speciation. The big learning goal here is understanding that patterns in the fossil record aren't necessarily always gradual-- sometimes they skip like this; sometimes they're punctuated-- but that that kind of variation can actually be explained by the underlying biological process. So that one worked pretty well.
They loved this one. This is a lovely gentleman called Philippe Padieu. He was the focus in the news media about six years ago because he knew he was infected with HIV and he purposely inseminated seven different women, I believe. It was in Texas. The women somehow learned of each other and got together and basically sued him.
And it's one of the first cases where phylogenetic reconstruction was used in a criminal court case. And so the objective here is to convince the students that, one, these data were actually useful in this kind of situation, but that there was a pattern in the phylogeny that could inform us of whether Mr. Padieu was guilty or not.
So we set up the situation. First, we brainstorm until we get these points. So the students are asked to discuss in groups what kind of evidence we would need from genes of the virus that is in Mr. Padieu's body and in his victim's body to come up with a verdict of guilty.
We talk a little bit about the transmission process-- what is it that happens when somebody gets infected with AIDS? We eventually land at this scenario, where we have Mr. Padieu's viral load here getting transmitted to, in this case, just two women. We arrive at the understanding through discussion and report back that this viral load of Mr. Padieu will then turn into its own population in the new host-- in this case, a woman-- and that that will be different from another woman.
Once we get to this point, the jump to understanding that the phylogenetic relationship of all these viral particles has a pattern that shows transmission-- ancestry descent-- that jump is pretty easy. So I ask them to come up to the doc cam. They have a phylogeny.
A number of misconceptions get cleared up. Students have a tendency to always want the ancestral point in the phylogeny to still be existent. And that isn't the case. So we clear up that concept.
And then we end up with this. Then we start actually asking questions of the actual tree. This is the actual tree that was submitted to court. The women and Mr. Padieu have been coded here, so that you don't know who's who. And then you can ask a bunch of questions about how many women were infected, who is Mr. Padieu on this tree, in what order were the women infected, and what is it about the shape of the tree that tells you that.
So this one worked really well. And they love it. Of course, they break out into applause when you tell them that Mr. Padieu got 45 years.
So CTE really helped us in terms of figuring out what we were doing. These are the results from our COPUS survey, an instrument that helps us see how the instructors are behaving and also how the students are responding. Our transition happened from the fall of 2014 to the fall of 2015.
In the fall of 2014, we were doing a traditional lecture. And much to my surprise, we were actually already doing a certain amount. We already had clickers. I can tell you from personal experience, we were not using these clickers very effectively, but we did have them. But we were still doing about 64% lecture.
After the transition, you can see that total amount of presenting straight lecture reduced to 45%. So it's exactly, the 20% that we were aiming for. The students showed a big jump in terms of the difference in their behavior. They went from 73% just sitting there listening or shopping for shoes to 57% with a big increase in this category here, which is working-- working together.
So what did we find? So we were interested, remember, in the effect that this would have on all students but also on URM students. And these data really blew me away. Let me orient you here.
In orange are the under-represented minority students. And in greenish colors are the non-URM students. But the darker colors are always the traditional lecture. And the lighter colors are the active semester. So it's a semester-to-semester comparison.
And what you can see is, whether you look at assessment learning gains, which was the instrument that we implemented pre and post in our semester, or whether you just look at the straight grades that the student got, the non-URM students maybe improve a little bit with active learning, but they're all up there. Look at the difference with URMs. This learning gap, which separated URM students from non-URM students is completely closed, such that in active semesters, URM students are statistically indistinguishable in their performance from everybody else in the class.
Now, we used these two metrics-- course grades and this instrument-- because we wanted to make sure that this effect was not just because of the way that the grade in the class was being assigned. We had to make room for these extra active learning points. This assessment learning gain here, it's a pre-, post-test. It has nothing to do with grade assignment. So these data to me are pretty convincing.
What about confidence and the sense of belonging? What about this classroom climate we keep thinking about? So this is-- again, the same color code applies here.
What we did is we asked students in a pre-, post-survey a number of questions about how they felt in terms of their confidence-- confidence to present hypotheses, confidence to pose questions, confidence to interpret tables and graphs. And for almost all the categories, you can see that the lighter colors, the active learning semester increased student confidence in all these activities that had to do with their performance.
This one down here is the saddest and perhaps the most interesting. This is confidence gains over the course of the semester. And you can see that, in the traditional semester, students lost confidence in their ability to understand course content.
So this has been noted before. This is in the literature already. Basically, this is what they call the weeder effect.
This is a student arriving first semester at Cornell, walking into a big classroom, getting hit by the fact that this is hard, this is not what I'm used to, and they lose confidence in their ability to perform. It's sad, but it's true. Both of those things are completely reversed with active learning-- both for URM and for non-URM students-- so again, big confidence boost.
We looked at confidence just a little bit further, because we were interested to see whether in-coming level of confidence, which is potentially correlated with in-coming level of preparation, had an affect. And here, we correlate that confidence with their course grade in the class.
And you can see that all the non-URM students improve in confidence-- I'm sorry-- All the non-URM students independent of their incoming confidence level improve their course grade. But that that increase is significantly larger for the URM students that come in with lower confidence. And in fact, these are the students that we are most likely to lose from the STEM pipeline, and especially these down here with the lowest confidence coming in.
The results for a student's sense of inclusion was also pretty interesting. This was a very simple survey. We added it at the end of the semester. And the question-- it was just four questions-- to what extent do you agree with these statements?
And two of these statements were significantly different across the fall semesters of the transition. One is, students in the class consider themselves as part of a community. The answer was significantly agree more in the active classroom.
And the second, which I think is a big deal here, I am comfortable making comments during class discussion. In a large 280-person lecture class, this is not the easiest thing to do. And you can see that there was a significant increase for all students, not just URM, in this case, in the active semester.
So that's what we found. It's been, I can say, incredibly satisfying to think about the mechanisms that underlie these changes in pedagogy and their effects on learning. We're continuing on now, trying to ask different questions about classroom climate, trying to ask questions, trying to perfect our activities, understand a little bit more about exactly what is working and what isn't.
So I wanted to leave you guys with a couple ideas about what you should do if you're thinking about doing this in your classroom. So I guess, if you can learn anything from the evolutionary biology transition is that it really doesn't need to be that complicated. So I'm not saying it's not work. It's work to transition this way.
You do have to create materials. You do have to develop activities. But they're relatively simple. You know more about evolutionary biology or about your topic than anybody else. So creating a 20-minute activity, that's like kindergarten stuff.
So that's how I began to think of it. It's like, oh, how can I mess with them today? Like, what am I going to challenge them to do today? And that sort of became part of the fun of the transition.
It's important that you stop being a control freak in your classroom. If I think back-- and the reason I say this is because you basically are ceding control of your lecture, and you're giving it to them. And if you are a person who likes the lecture to be perfect, that your intonation is always great, that you're a great lecturer, you say things exactly with the right cadence et cetera, et cetera, you're not going to like this, because it's chaos.
And it's chaos. And they get things wrong. And you need to adjust to that. You need to be constantly moving. You get good at that, too. It's surprising.
I like to think of this as an experiment. So we would always say, we're going to experiment with this or we're going to experiment with a different activity. Just as in any other experiment, those of you who are scientists, you manipulate variables. You try to figure out what works and what doesn't work. And then you start over.
Assessment, assessment, assessment, always-- we got a lot of it and a lot of help. And we even collected some of our own. And I'd like to reinforce this point here, which is, I think it's important in these cases to make use of young talent. We hired active learning post-docs in biology, and that was a great boon.
You might not want to believe this. But it turns out that you are 30-plus years older than your students when they're freshmen. And you may not know exactly what they think is funny or interesting. And Cissy was just the right bridge in terms of knowing what kinds of things we could push or where we could take the students.
And then finally, I would suggest that, if you're considering this, you shouldn't wait. The support for this experiment is on this campus, obviously, now, so go for it. So thanks. Any questions?
[APPLAUSE]
Andre.
SPEAKER 1: I'm always wondering-- and [INAUDIBLE] flipped [INAUDIBLE] also-- and I was always wondering, what is percentage of students that don't like that? And how do you deal with them?
KELLY ZAMUDIO: Yeah. So I've actually looked at-- the demographics of our classroom are pretty consistent year to year. And so most of our students, 60% are freshmen.
Another 20% or so are sophomores. And then we always have a few juniors or seniors that tend to be taking evolution late, for one reason or another. They should have taken it at the beginning of their STEM career if they were going to be bio majors.
A lot of the sort of informal interview questions that we've had have to do, I think, a little bit with some of those older students feeling like they're basically surrounded by a bunch of freshmen and sophomores and, on top of it, are being asked to do all these things that are kind of silly. So I think that's how they perceive it.
I think I get maybe two or three really negative reviews out of 280 every semester. And most of them are saying, I just want you to lecture, just shut up and lecture. And I guess we choose to ignore them.
These data convince me that we should be teaching to the class that you have with all the variance that's in there. And if those two students are already in those green bars and are already outperforming, and would perform well no matter what we did, then I guess I think that's the cost that they pay.
SPEAKER 2: So in terms of the kind of course it was, can you tell me what learning outcomes you were looking for? Were you looking for them to acquire skills, data, concepts?
Of course, it's going to be some kind of combination. But how is it weighted? And how did that affect your approach to things?
KELLY ZAMUDIO: So actually, the thing we did before we started anything, and after we found out that we were going to get this money, was that we got all the professors together, and we just set them into their learning outcomes, because that's what guided all the activity development. The learning outcomes are a combination of interpretation skills, understanding different concepts, understanding different facts about biological diversity, and applying those to different sorts of situations-- the court case being one example.
So yeah, it was a combination of them. I tried-- when I looked through each lecture and I decided-- so we have about six to seven learning outcomes in each lecture, learning goals, things we want them to absolutely walk away with from our course-- I would choose one. And then I would gather all the concepts that are associated with that goal. And that's what I would use as a starting point for our activity.
SPEAKER 3: Do you think they saw the infrastructure of the traditional classroom, the lecture halls that we have, do you foresee something where Cornell may, down the road, design some new lecture halls that support better the actual physical groups of students in the classroom?
KELLY ZAMUDIO: That would be nice. I know that a lot of universities are actually doing this. Cissy has moved on to the University of Minnesota for her second post-doc. And they have controlled classrooms there with tables like this, maybe a little bit smaller, with swivel chairs, so that each table has a group, and everybody is participating a different way.
It would be nice. What made me happy about this experiment was to show that it wasn't absolutely necessary. And I think that's a good thing, because it means we can move forward without actually having to wait for those buildings to be built, because we all know buildings don't pop out of nowhere. So yeah, it would be great. And maybe in the future, that's what classroom is going to look like all over the place.
SPEAKER 4: I'm curious if you felt that you needed to sacrifice any content to implement the active learning classroom.
KELLY ZAMUDIO: Yeah, so I would say there was probably a decrease in about 10% to 15% of the material that we taught. But what happened was, when forced to make decisions about what we were going to focus on, we very carefully analyzed whether our students really needed to know all of that.
We discussed a lot whether moving material to before lecture activities, both reading and video podcasts, whether we should hold them accountable for that material. And it turns out that, like some professors even in our group, said, the students are going to hate that, because you're not telling them. You're making them watch a video and do some reading and take a quiz. And then you're going to put it on the exam.
But they were fine with it. Like, they just understood that that was part of the content that they needed to know. And they dealt with it.
If they had questions, of course, they would ask in discussion section, or office hours, or whatever. So it's not that we ignored that material. It just was in a different order, in a different place.
So I think you lose some overall content. But the content that you do have, if it's closely aligned as it should be with your learning goals, should cover your main objectives. Another question here, I think.
SPEAKER 2: I'm guessing that a large course like this may have had discussions sections as well?
KELLY ZAMUDIO: We do have one-hour discussion sections per week, yeah.
SPEAKER 2: And how were those reframed as a result of what you were doing in the classroom?
KELLY ZAMUDIO: For this transition, we decided to not mess with that variable. So those stayed the same.
SPEAKER 2: Do you anticipate [INAUDIBLE]?
KELLY ZAMUDIO: Yeah. Yeah, I mean, that's one of the things that we need to think about and move in the future, is try to-- so the discussions sections already a bit-- they're smaller groups, 20 students. They're led by TAs. They have activities. They're already somewhat active learning. They reinforce the ideas that were giving in lecture.
It'll be fun. It'll be a good process, now that we've sort of established where we are with lecture, just sort of bring those discussion sections in. But we couldn't do it all at once. Carl?
SPEAKER 5: Did you add material assignments to the students that they would do at home? And how much time was that, if you did that? And did the students complain about it?
KELLY ZAMUDIO: Yeah. The assignments were the same thing. Before every lecture, you had to watch a vodcast, which included some material. And that's a Panopto video of me speaking over a PowerPoint or showing some images.
We also added a pre-lecture quiz, which was worth very little, only two points. But it was four questions that they have to answer. And we had assigned reading. And the quiz covered both the video and the assigned reading.
So what this meant is that, if they didn't do those two things, even if they guessed, they would not probably get-- it was an online quiz-- they would not get those points. Even though those were only two points before every lecture, we have 40-some lectures, that's a lot of points. And so they didn't complain. Thank you.
[APPLAUSE]
RON HARRIS-WARRICK: Thank you very much. And I'm glad to see you all here. So I want to talk to you about our experiences in flipping an introductory neurobiology course. And of course, the goals of active learning are to change this kind of class, which, unfortunately, many of you have perhaps seen before, into this kind of class, where the students, instead of passively absorbing or perhaps not absorbing material, become actively involved and really learn much more.
This work could not have been done without our teaching post-doc. We also used the money from the Active Learning Initiative to hire a teaching post-doc, who is Laura Manella. Laura is here, if you could stand up, Laura. And she did an enormous amount of work to help us do this.
And if you want to know a lot about nuts and bolts, I urge you to talk with her. We have a reception afterwards, right? And she's going to stick around for that and really has been wonderful for us. So this is work done by her and by us.
So the course is a sophomore-level course, BioNB2220, Introduction to Neuroscience. It's a general survey of how the brain works, basically from soup to nuts, from molecular mechanisms of neuron function up to cognitive activity-- emotions, learning, and so on. It's a gateway course for a major in neurobiology and behavior. All the majors have to take it.
But it's also quite popular as a distribution course for students looking for a biology course to take for their graduation. So I guess about half the students are not majors. They are just taking it as a distribution.
It's a smaller course. It had 175 students this last year, when we last taught it in the fall, or in the spring of this year. It requires a background in biology and chemistry. So the students already have to have prerequisites. So we have a smaller group.
Fortunately, some of the basic biology courses, like the one Kelly teaches, already are using active learning techniques. So by the time our students come, many of them have been exposed to this way of teaching.
Like Kelly's course, it's team taught. We have seven professors share in this course, each of whom gives half a dozen lectures. And this had its own set of problems, which you can imagine-- and how to flip a course and get seven people to agree on how to do this.
It has a discussion section for the students that are majoring in neurobiology and behavior. They get an extra credit for that. And there's also with three credit option for students who are taking a distribution requirement and don't have the discussion section.
We also organized with a series of undergraduate TAs about a dozen study sections, study groups, that the students could join to help them go through the material throughout the semester.
So what do we do for flipping the class? First of all, we've only worked on this for three years so far. The ALI, the Active Learning Initiative, is a five-year project.
And we started off quite slowly, because we had so many faculty who didn't know anything about active learning. So this is really a progress report. And we're not to the end-- nowhere near the end. We're still making major changes as we go.
The critical part of it is just what Kelly said. The students must prepare in advance for every lecture. So in most cases, this involved videos that they watch, Panopto videos, which is right embedded in Blackboard. So it's very easy for the faculty to set it up, very easy for the students to access it.
As Kelly said, the Panopto video has a little picture of you talking up in the corner. And then it shows your PowerPoint. And I put a pad on, so I can draw on it and circle things and write equations and so. So students watch that.
We have a handout for every lecture the students must read in advance. I've experimented with assigning specific figures from our textbook and telling the students, you must understand these particular figures beforehand because I'm going to show them and then we'll ask questions about them figures. And that also works.
So the idea is the students have done a lot of work before they come to the class. And this material is not taught in class. Instead, that time frees up the time for active learning exercises.
There was a question about how much do you lose in your content by doing this. And I agree with Kelly, it's in the ballpark of 10%. And it's the least important 10%. And I'm sure the students don't notice that they've lost anything at all.
Among the many things Laura did was do a bunch of analyzes of student attitudes and success, just like Kelly and Cissy. This is one of Laura's analyses-- how much work is this compared with other classes, because you're asking the students to do all this stuff before every class. And what the majority of them says, it's about the same.
They're not noticing that this is overwhelming. They're not coming in and screaming about how much extra work this is and how they're unhappy. A few of them are, but the majority are not.
So then since we've taken part of the lecture out and put it in video or in a handout, we then have time in class to do these exercises. And of course, these are designed to encourage active engagement with the core concepts and help develop creative and critical thinking skills.
So the idea here is not merely to learn stuff, but to learn to work with stuff, to solve problems, to come up with hypothesis, to be able to interpret things properly, things that involve kind of the top of the pyramid of Bloom's Taxonomy of learning, where recognizing and memorizing stuff is at the bottom. And you work up to evaluating and creating at the top. And that's the goal.
So it makes class more interesting. We don't see as many students sleeping or shopping or web surfing. And again, that's just what you found. Right now, we're spending about 40% of the class time on active learning stuff, again, with help from the Office of Undergraduate Biology on that. We continue with normal lecturing.
And it's my personal opinion that, with a course like this, we're never going to get to a completely flipped course. And it's not necessary to completely flip your course. That's not our goal.
And one of the reasons is that neurobiology is a very, very actively changing field. There's new stuff coming up all the time-- new imaging studies of human brains listening to Mozart or something. And by the time textbooks are published, they're out of date. Even the titles of the lectures, the topics we talk about from year to year are changing every single year.
So for that, we're going to have to teach some stuff that's not in the books. So we're going to have to continue lecturing. And I'm personally happy with this 40-60 split, maybe 50-50 in some cases. But I'd like to emphasize, it's not necessary to completely flip the class in order to get the benefits of active learning.
We use clicker questions to start class discussions. Most of you have probably seen a clicker, a personal responder, where we ask a multiple choice question. And they click in their answer. I've got a receiver that we plug into our computers so the faculty can see how the students are voting without showing it to the students. So students don't know how the other students are voting.
We don't use it simply to keep attendance or to test factoids, factual knowledge, but really ask questions-- detailed and difficult questions about simple concepts or interpretation of data. It's important to make them hard. They shouldn't be too easy. They should really challenge your understanding of the core concept, rather than just ask you to parrot something back.
In my personal opinion, we start with easier ones and progress to harder ones as the lecturer goes on. And we want to have a number of them be hard enough that only a minority of the students get the right answer. And then without showing them how they voted, we could just say, there's a lot of disagreement on this, why don't you talk with one another and discuss this. This is called think-pair-share in the teaching vocabulary.
And as both Julia and Kelly said, the room explodes. The students are all talking at once. They're gesticulating. With their hands, they're drawing graphs. And you can see they're thinking.
And I don't say a word, mostly because I'm hard of hearing and I should be walking around and listening. But I can't hear them. But then you have them revote. Without my having done any teaching, you'll go from 30% getting this very difficult question correct to 80% getting that very difficult question correct, because the students that have the right answer tend to be more persuasive in their arguments than students that don't have the right answer.
Then you can ask another question on the same difficult topic, and it'll go to 95%. All the students have got that concept. Then you can move on to another concept. And then afterwards, we discuss why the right answers are right and why the right answers are wrong.
So this is just what one looks like. Kelly showed you some examples. There's a question. This has to do with the strengthening of connections between neurons by serotonin.
And there's a bunch of possible mechanisms, all of which could affect it in one way or another. And we've never talked about some of these mechanisms, but they have the basic idea. And after working among themselves, they can get this right.
The other type of question that we've worked a lot with our discussion questions, which are open-ended, where there isn't just a list of possible answers. There's many possible answers. And we don't give them any. They have to come up with them themselves.
So we let the students think about it, then have them discuss it with their neighbors to kind of validate their ideas, whether what they're saying makes sense and is reasonable. And this then removes the disincentive to talk in class. And they're happy to raise their hands and talk.
So we then finally get better class participation to discuss the answers. So here's an example about schizophrenics who have poor working memory and how these new methods of looking into the brain of a schizophrenic versus a control person can help us understand how that works.
So Laura asked the students how effective these activities are in the lecture. And we got the same results that Kelly got, that the students find these very effective. So extremely effective is blue. And very effective is orange. And you can see, for most of these, it's a majority.
So for identifying core concepts, majority of the students say this is very effective or extremely effective; for identifying misconceptions, so that they walk out of class knowing what they didn't understand and having understood it further; for staying engaged, for keeping awake, for actually paying attention in class; and for being motivated to prepare for class, to actually do the pre-class materials.
We also have these discussions sections where we did change them to incorporate active learning techniques. And they actually got Amy from the Center for Teaching Excellence to come and train our TAs to use these techniques in the sections. We meet with the TAs in advance and figure out what the core concepts to cover from the previous week are.
The TAs, themselves, develop different types of active learning exercises. And they share them, so that, if each TA comes up with one, they end up with four, which covers most of the section. And so they can share their ideas.
And they use the typical active learning techniques of think-pair-share, something called gallery walk, where you have several different places to write down comments about different questions related to one. And each student walks by and adds a comment to the list of comments. And afterwards, you can see all the different comments students have made. Sometimes we use a debate format.
We also have a weekly quiz. We don't have quizzes before every class. But once a week, we have a challenging multiple choice quiz online, which is on core concepts from the previous week. And these, again, are not simple fact questions. But they're questions testing their understanding of the knowledge.
And this is worth a moderate amount. It's worth 10% of the grade, which is enough to keep the students taking the quizzes every week. And they say it helps themself calibrate their understanding and to seek help if needed. Because it spreads the grades out a little bit, they also say it reduces stress.
So here is some of Laura's data. They help me prepare for exams. The large majority of students think that the weekly quiz helps them a lot. And a large majority recommends that we continue this, even though it's extra work for them-- they got to do it every Sunday night. They have to do an online quiz before Monday class.
Laura checked to see how the outside tools-- which ones were most appreciated. And the ones that were most appreciative were the pre-lecture videos, which, again, is just me taking a piece of my lecture out and importing it over. It is not a lot of work for me to do. It's not a huge effort.
The weekly web quizzes were very much appreciated. The pre-lecture outlines, less so. And reading in the textbook, not so hot. Now, though still, a majority of people found that reading the textbook was good.
But these new things that we've added are the things that students liked the best. And it could be that this is a group that grew up with computers and the web. And doing these online things, they're familiar with them and willing to use them.
So how did we actually do it? Only two of us of the seven had had any prior experience in active learning. And many of the others were, quite frankly, suspicious of it. They were in the "ain't broke, don't fix it" category.
And so we had to have many, many meetings to discuss this, to talk about it. And to be very honest, the Active Learning Initiative helped buy-in their agreement to participate in it, although many of them were still very suspicious. So we had to start very, very slowly.
We got tremendous support for the Center of Teaching Excellence. And if you're considering doing this, CTE is an invaluable resource. Amy came over and led several workshops for our faculty to learn how to do it and to figure out what it even was, active learning. And she led workshops for our TAs and even the undergraduate TAs, who ran the study groups.
Laura was really invaluable. She is an expert in active learning methods. She met with every professor about every lecture to go over it. She helped actually the professors prepare the pre-lecture materials-- the videos, the readings, and so on. She suggested active learning exercises to use in class.
She wrote clicker questions for some of the faculty, for example, faculty who were not familiar with how this works and what kind of questions are useful. She met with the faculty afterwards and gave them feedback in an extremely polite way, such that senior professors were not offended by being criticized by a post-doc. She was very, very good at that. She also wrote the weekly quiz and did all the analysis that I've shown you. So she was really, really important.
So we've done this for three years. As I say, instead of making a switch in a single year, we've done it in a graded way, which I think is different than what the other groups did. The first year, each professor flipped one of their lectures. And the second year, they flipped two of their lectures.
And last year, I was in charge of the class. So then everyone flipped every one of their lectures. So this is the first year where the course is entirely flipped. And we've got two more years coming where we're fine-tuning and trying to improve that.
So does it work? This is the distribution of the exam scores that we had on average before flipping and the two years where we were doing more serious flipping. And what you can see is a shift towards higher scores and particularly a reduction of the lower scores.
The mean does change. But the reduction in the lower end, the poorer students seem to be doing better. And the better students are still doing extremely well. So that's good news.
Something that was very, very surprising was that introducing active learning in the classroom appears to do what the discussion sections were doing before. The students that would take the course for four credits would have a one-hour discussion section. And they would talk about the material. And this would help them to understand it better. So when we compared their scores on the exams, they always did better-- in fact, about eight points better on average.
And so we're still teaching the course with and without discussion sections. And this has started to fall. And this last year, when we fully flipped the course, you can see it basically dropped out to the point, where the difference between the students who had the discussion section and didn't have the discussion has disappeared.
It's almost as if the active learning exercises in the class have done the creative thinking that the discussion sections used to have. Now, we're hoping this trend continues. This is one year so far.
If it continues, what this does is liberate us to reconsider the discussion sections and to use them for some entirely different tool perhaps, bringing in additional creative things. Let them do additional experiments in their discussion sections, rather than going over the class material. So this is an unexpected result that really took us by surprise.
The other thing that's happening is the success of this is causing it to be spread out to other courses in our department. Our chair, after watching the success of the introductory neurobiology course, also assigned Laura, our teaching post-doc, to help with our other gateway course, Introduction to Behavior, which all majors take. And it's in its second year now and is really starting seriously to be flipped this year.
Several upper division courses are now using active learning techniques. I've actually been teaching an upper division neuropharmacology course called Drugs of the Brain. And I started flipping it in 2011.
And this allowed me over these years to make a strong emphasis on creative thinking and learning to think like a scientist, so that this class with all the active learning exercises in class allow the students to learn how to think about a hypothesis, to propose a hypothesis for data, to propose an experiment, to test a hypothesis, and so on-- things that I would lecture about before. But they learn it much better if they think of it themselves.
They're doing much better. I've been teaching this course for many years and have been grading it the same way every year. So for the five times I taught it before flipping, the final score was 81 plus or minus 2.5%. So I have a pretty tight grading.
I flipped it three times. And the mean is 86 with almost no variation. So it's gone up by more than two standard deviations.
Plus, the student comments are extraordinarily positive. Things like, now I know what it's like to be a scientist, rather than just learning science. The active learning techniques that we used in class and in homework really enhanced my knowledge of the core concepts. So this actually works.
So in conclusion, let me just say, active learning is more fun for the students. But, as Julia said, it's also a lot more fun for the faculty. It's really enjoyable for me, when the students are going crazy and they're all raising their hands.
And they come up with ideas that I hadn't thought of at all. And some of them are way out there. And some of them are way out there in a really interesting, novel, innovative way. And the discussion that we can have about that just makes going to class much, much more fun.
I'm in my 37th year, and I've never had so much teaching as I'm doing now. I'm really enjoying it more than I ever had. The students are learning more. They're definitely more sophisticated. And they're thinking about core concepts.
In my pharmacology class, the final project is to write a grant proposal to solve an unsolved problem of their own choice in neuropharmacology. And the sophistication of those NIH grant proposes has hugely increased since they started using active learning in the classroom. And finally, the classes, of course, offer many opportunities to engage in creative thinking, rather than just memorizing.
So let me just say that I and probably the others, if you're considering doing some flipping, I'd be happy to talk with you any time you want. I'll give you any help and advice that I can give, because I really do believe this is the future of college teaching. So thanks very much.
[APPLAUSE]
SPEAKER 1: Questions you have for Ron?
SPEAKER 2: Thanks, Ron. I do have actually a couple of questions if you don't mind. One is administrative-- seven professors all teaching the same course, how is that apportioned in terms of credit for teaching load?
RON HARRIS-WARRICK: Yeah, the teaching load in the sciences is somewhat different than the humanities for reasons that are historical. But part of it has to do with the fact that our running a laboratory and supervising students in the lab is considered part of our teaching load. And so historically, this team-taught course has counted as the teaching requirement for that semester. And that didn't change.
SPEAKER 2: The other thing that I'm, and you started to answer it at the end is, I think the attraction to something like this often is that it may be we're looking for ways to scale it up to larger courses. But I am also wondering how we scale it down to medium sized and small courses. And it sounds to me like you've begun that process. And I'm just wondering how you've done it and what the success of it is.
RON HARRIS-WARRICK: Right. So the neuropharmacology course I teach is a medium-sized course. It has between 60 and 90 students, depending on the year. And in that class, I started off-- it changes during the semester. At the beginning, I ask lots and lots and lots of clicker questions just to get the students into it.
And as I say, for each lecture, I try to make the clicker questions increasing in sophistication and difficulty as time goes on. And even in a particular core concept, I may give a couple of questions, then move to another core concept, and try to make them harder and harder and harder to test to allow students to see what they didn't understand.
As the course went on, I found myself writing more and more open-ended discussion questions, where I didn't have to come up with five answers. It might be 15, especially the questions based on things like, here's some data, propose a hypothesis to explain that there can be many, many possible answers. Here is a hypothesis, propose an experiment to test it. There can be many, many, many answers to that.
And the outcome is the students feel that they can do science much better than they could before, because in every single lecture, they're asked to do that. Their homework every week is an unsolved problem in neuropharmacology that nobody in the world knows the answer to. And again, they start off with the easy ones at the beginning of the semester.
I think, last time the first question was, should you borrow your ADHD friend's Adderall to study for tomorrow's exam? It turns out, nobody actually knows whether it works. What I read is probably it doesn't work. But it's not really clear. And so they could work on that.
Later on, the last question was epidemiologically, schizophrenia has a very, very strong genetic component that nobody can find the gene. So what's going on? Much, much, much more difficult question.
The students rose to it, and they found were challenged at every step and could deal with it. So they felt that they had succeeded. They felt really a strong sense of self-worth and accomplishment from having done that.
Now, in the humanities, of course, active learning is the game in the seminar course. In the seminar course, you do the work beforehand. You read the book or whenever. And you come into class prepared to actively engage in it and reading.
So in the seminar courses, this is what you guys do all the time. And what I think is interesting is, in fact, something that you're very familiar with in humanities try scaling it up to the big classes and see. You could do the thing. It works just as well.
RON HARRIS-WARRICK: I'll be discussing our activities in physics. Here, in physics, we called it the CUPID Project, Cornell University Physics Initiative in Deliberate Practice. And the real emphasis here is on practice-- practice, practice, practice. That's how we instill those skills.
And what the overall project represented was really a complete overall of our main line course sequence. It's a three-course sequence. And I had six wonderful faculty working with me. I don't have time to give all their names. But Julia was mentioned there as well. And I see a number of my friends here in the audience.
Now, so what I thought I'd do, really I feel my main role here is really to give a sense of what it is like as a faculty member down in the trenches and flipping one of these courses. So I'll just give a very quick overview of what we did overall in the CUPID Project and then really make it a testimonial of my own experiences in physics too.
So in big picture, we are talking about this three-course introductory physics sequence. The first two courses are required by every undergraduate in the College of Engineering. The third course is required of many of them. So we have a large footprint in the Engineering College. In fact, at the end of the day, each semester now, we impact 1,000 students with these new learning methods.
The basic idea, as we've heard in the previous two talks is really a shift in focus of what we do with those very valuable instructor contact hours, where now, the focus is not so much just on lecturing, but actually for us, it's doing physics, actually engaging the material, exhibiting the skills we're hoping to impart. And they're able to do this practice with expert feedback in the room. And we're just not simply presenting the material as a lecture.
We worked with some constraints that we imposed upon ourselves. We felt this was important to make sure we could spread the good news, so to speak, of these techniques and to make it easy to transfer them to other courses. We were very careful to maintain that the time commitment of both the staff involved in the courses and the students remained the same. We don't want this to balloon up and fill up all of our students' super-busy schedules.
Also we were very careful to be able to find creative ways to deliver this kind of active learning in the same physical plant, the same classrooms, and actually the same scheduling. So it's very easy to do this kind of a transformation internally to the courses. The bottom line outcome was, first of all, that it is just a very highly rewarding experience for the faculty. That's what I can really speak to you here today.
And why, really, I think it's so rewarding, in addition to some of the other issues that I'll mention in a moment, but personally, I became a physicist because I love doing physics, not necessarily that I love lecturing, although I don't mind lecturing too much. But that's what makes it so very rewarding. You're in there doing it with the students, mentoring them in that kind of engagement.
Another very rewarding aspect actually is to see the very significant learning gains. And I have data on these last three bullets that I'll share with you in depth. And we have very, very good measures of this, both from internal and external measures, as I will explain. So that's very rewarding too, to see the students learn more.
But what surprised us is that these learning techniques are actually so powerful and effective, they don't just learn the material better, they learn at a much deeper and more mature level. And I'll show you some data on that as well. And then finally, it never hurts in terms of reward to actually have the students appreciate your efforts. And we have very good reviews from the students as well that I will share with you.
So I want to focus then, drill down in the course that I was involved in nuts and bolts, which was our second course in the sequence, Electricity and Magnetism. My partner in crime was Kyle Shen. And just to keep the presentation short, I'll show you how we rearranged the course structure. It's a little similar but different from the previous two presentations. Then I'll drill into our data on how much more the students are learning and share with you a little bit on the student satisfaction.
So the best way to describe what we've done-- and you've heard already quite a lot, you've got the idea of the flip format-- but the best comparison I can draw is a before and after comparison. So here, I'm showing you the main activities in which the students were involved. There was a lab section. That we left alone. So I won't bother trying to describe that to you.
But when I taught this course in the traditional format in 2011 and 2012, we had two 50-minute lectures, where I would lecture the material. That was their first exposure. We then would have two 50-minute discussion sessions with usually a graduate TA, who would do a little more exposition and then solve problems for the students on the blackboard. And finally, they were sent home to bang their heads against the wall for about eight hours on a homework problem set assignment. That was our model.
Now, when I first came into this, I have to confess, I was perhaps the world's hugest skeptic on this whole flipping idea. So I worked very hard in how we redesigned this to make sure we're not losing anything from the old, but that we are actually delivering things in a much better way.
So my main message here is that the students in our flip here are engaging in the same activities for the same amount of time. The primary difference is the venue and social context in which they carry out those activities. And that's what makes all the difference for us.
So let's see then how we did this. We did use some technology to basically do some ground level thinking of how we could change the social venues. But the focus isn't the technology. It's what it's enabled us to do.
So here's how their week runs now. So prior to lecture, they watch a video lecture. Now, I went-- even though I was a skeptic, I went whole hog on this. The entire lecture is transferred before class.
So it's exactly the same lectures that I delivered in 2011 and 2012, but now via YouTube. So the students are not losing anything from that audiovisual stimulation that some of them really seem to like, as we saw some data in the previous presentation. So they have that entire same lecture experience they would have had with me.
Next, they carry out a series of exercises. Sometimes, when the flip is done, it's just a multiple choice quiz to make sure they watch the video, like those various compliance videos we all love to do. But we decided, the student's time is too valuable for that.
So what we did is we are giving them offline the exact same exercises that they normally would have experienced on their problem set. These, of course, are the easier warm-up exercises. But those are real problem set material.
The technology that enables us to do this is something called Learning Catalytics. What's nice about it is that it is an open-ended technology. The skills that we want to impart particularly in physics are being able to analyze a physical situation.
We require them to draw diagrams. They derive equations. They can enter these directly into this online app. And the computer can score that for them automatically. It gives them automatic feedback. We even give them two tries on it, so they can seek help if they feel they need it.
So after about a half hour of those types of exercises, then they are really primed to come into lecture. And in lecture, it's whole hog. 100%, we are doing physics problems the entire lecture right out of the main meat of what normally would have been our problem set.
Here, what the technology allows us to do-- again, we use this open-ended response system. But in the lecture hall, what we are able to do then is to take all of the students' responses-- for instance, all of their diagrams and overlay them on top of each other and project it onto the screen in front of the students. Likewise, we can categorize the different equations that they have entered.
And without giving away what the correct answer is, I am then enabled, as the expert, to look at those responses and critique that, point out various trends that I'm seeing in the responses that the students have given, and point out to them where they may want to rethink. And then the students can confer with their neighbors and the neighboring rows. And then I can reissue the question one or two more times as necessary for them to really get the material. So that's what's going on in lecture.
In discussion section, they are, again, doing now cooperative exercises in formal groups of three. We had done cooperative discussions in the past on paper. We, again, though, are using the same response system. The advantage is, it tells them right away if they are right or wrong.
In our system, we give them three tries. So they've already tried and failed twice before they will call on a TA. So now, the TA is talking to a set of students who have really engaged this problem. And they are primed for that input.
It also means that the TA time is not spent in a simple check that everybody is correct. It's not spent on people who haven't really engaged the material. His time is then able to be used much more effectively to enable the learning.
Finally then, they are sent home and given a problem set. But if you've been tracking what I've been saying, I don't want to expand any time and they are watching these videos online, so we did have to shorten the problem set. And you might worry, wait, maybe something's leaked out here. I may have lost something.
But of course, no. The problems that are missing from this problem set, they have now done. But they have done them in much more effective venues with that expert feedback in peer-learning type environments.
So that's basically what we had carried out. Oh, no. Good, all right. So let's see then how this works.
So we'll discuss the learning gains that we're seeing. So for us, again-- and I think we saw a result in the previous talk as well-- really, the measure, sine qua non, is for us what we see on that final exam. As practitioners of our respective fields, it is really the final exam instrument that we use to give the students the opportunity to exhibit the skills that we've been hoping to impart.
So here's how our comparison went on the final exam. So what I'm plotting for you here is I'm comparing a standard student histogram, percentage of students that fall in each bin. It was a long exam. So on a final exam, it's graded out of 130 points. And I'm comparing histograms in blue before the flip in 2012 and in red in 2015.
Now, I don't want to disclose too much in the public environment. But we have a special technique, where we can pretty much statistically ensure that these exams are really of identical difficulty. You can ask me offline how we carry that out.
But what you'll see is now, on identical exams, you see there is this radical upward shift in the gray distribution. It's quite significant. It's a 0.6 standard deviations. For us, that represents two grade subdivisions.
So a student who, in 2012 was giving me a B minus performance under the standard teaching techniques, in 2015 now is giving me a B plus performance on the identical difficulty final exam. Or a mid-range B student is now giving me an A level performance on that final exam. It's really a remarkable improvement.
The other thing that we're very proud of, if you look at the distribution here carefully, you'll notice, comparing red and blue, really what we've done is we've grabbed that blue distribution and we've shifted it fairly rigidly upwards. So we are servicing the entire cohort of the class. We're not just catering to, say, the top level or the bottom level or the middle level. Everybody is gaining significantly. And we're very proud of that.
So we can see now, the students really are learning much better. Now how about more deeply? Well, you can see, we did have a problem in 2015 that, well, they did max out my final exam now.
So what we did was we gave a much more difficult final exam last year in 2016, last spring. So what we did here is we, again, had our identically difficult exam. But what we did now is we made it significantly more difficult by removing all of the little hints that we typically give and removing the different subparts of these complex analyzes.
Rather than stepping them through it, the students now are asked to be much more mature, evaluate a much more complex physical situation, break it down the analysis into its component parts. And you can see that, even though they are now taking a much more difficult exam, they are still doing better in the flipped classroom format.
And once again, we are very proud that the lower end here has been depleted and pushed up into the higher end. So students at all ends of the spectrum are benefiting and able to show a much deeper knowledge and maturity with the material.
So these have been measures so far that compare students from different semesters before and after. We can also track the learning of individual students throughout the semester. This is a standard measure that is frequently used. There's something called a Hake gain that's associated with it.
The basic idea here is we've given the students an exam at the beginning of the semester. In this case, it's an open-ended instrument that we've developed. It's not a multiple choice. It's open-ended-- draw diagrams, input equations for your answers.
And I'm showing you the results before the semester, or during the first week before they've been exposed. After they've been exposed, they now have gone from about a 30 average to an 80 average on this instrument. And the Hake game finally is a measure of what they were not able to exhibit for us in terms of skills prior to their learning.
And then we ask, what fraction of that missing amount now has been filled in? And that fraction is around 70%, which is a very high Hake learning gain. And traditionally taught courses, measures often come out in the low to mid 20s. So this is 70. We're very proud of this.
The other thing you'll notice I put up here, which I thought was interesting to look at, was I disaggregated the data here by gender. And there's a very nice story here. It's beautiful actually. It's just what we would hope to see.
First on the pretest, this is really kudos to the Engineering College, who is giving us a nice, very well-balanced class, no gender distinctions at all. Only the fluctuations are all what you expect from statistical noise. And then post-test, that remains. And so between the genders, the technique is perfectly balanced, just the kind of thing that we would like.
Now, there's another measure that we've used on this. This is a measure that comes out of the literature. This is sort of external to the course. This comes from the outside. Again, it's a pre-, post-test format.
This particular instrument, I've personally had some reservations with, just because the questions to me seem kind of more like trick questions. I've never really liked it, because also it doesn't really reflect the skills that I feel are important in a physics context. So we'll see the results on this. Even though I'm downgrading the exam, we do do very well on it. But it tells an interesting story.
So on this external multiple choice instrument, we're getting a Hake gain of around 40, which is actually quite respectable in this is kind of a context. But there's a little bit of a cautionary tale I thought I'd throw out for discussion. You'll notice, we are now beginning to see some gender discrepancies. And I believe this is reflective of a bias in this particular instrument, because we see how well-balanced the genders are on those skills that I actually valued.
In terms of-- how we doing on time? Oh, here's those overall Hake gains on that instrument. And since we've flipped the course, those are increasing steadily. So we're very happy with that.
Finally, I guess, we'll take a quick look at student satisfaction. And then I think my time will be running out. Here is the standard course evaluation instrument that we're all familiar with scored out of five.
Here, we've picked out the categories that focus on basically student satisfaction-- how they view the lecture overall, the course overall, how it compares to other STEM courses at Cornell, how much the lectures instill interest in learning, at least as perceived by the students, and how they perceive the effort, organization, and communication of the faculty member involved.
In blue, I'm showing you the pre-CUPID results. So then, in our first semester in red, you can see we had a significant uptick in all of these relevant measures across the spectrum. In 2015, in the fall, at this point, we had worked all of our kinks out. And so you see another big uptick in the student satisfaction.
And then finally, in spring '16, we were able to maintain those differences. I'll just point out, in 2015, that's my colleague Kyle Shen. He's maybe a little more charismatic than I am. But it doesn't go down too much. But it shows it transfers between-- he's younger too-- between faculty members.
Now, the last thing I'll say, the last slide to show is that, OK, so I put a lot of emphasis on this open-ended technology. It's a web-based technology. The students, as we've been hearing, are very used to this kind of thing.
But still, there's a bit of a learning curve associated with it. It's certainly not as simple as hitting a multiple choice button. So you might wonder how the students feel about that. So that was the last survey question I'll share with you.
So we asked them a standard question on the course evaluations to rank things on a bipolar scale between strongly prefer multiple choice iclicker, which they had used in the previous semester, versus our open-ended Learning Catalytics, which we are using in the current semester. And you can see, there's very overwhelming support for that new learning technology. Among students with a preference, it's a 5 to 1 in favor of open-ended questioning.
So I throw something out there to the other folks, if they have an opportunity for more open-ended things, I would highly recommend that as well. And I think I've taken up a lot of time. So I'd just like to open things for questions.
[APPLAUSE]
Yeah?
SPEAKER 1: Actually, I was looking forward to the [INAUDIBLE].
RON HARRIS-WARRICK: Oh, what is it? Yeah, so it's a web-based response system. So it will run in any browser. So it'll run on your cell phone. It'll run on your laptop.
And then they can type in-- they have a WYSIWYG equation editor. You can sketch with a pen on your screen. It works like that. It's been very useful.
SPEAKER 4: [INAUDIBLE] can I follow up on that? Are you posing questions on this Learning Catalytics? And then how are you gathering responses?
RON HARRIS-WARRICK: Oh, they enter them into their device. So they'll either draw on their cell phone with their thumb, believe or not. They are amazing with this. You should see the type, right?
Yeah. Did I answer your question? I think-- yeah, yeah. And then it scores it for you automatically and colates the results for you, makes it very convenient.
Yeah. Oh, and it also doesn't just do nerdy physics things. One of my favorite things is a word cloud response, where they'll type in a short phrase. And then it'll display all the words that have been entered. But the more common ones are shown in brighter letters and bigger. So you get a sense of the entire class, what they're thinking.
So sometimes I'll ask a question, how should we analyze this particular situation? Like, what should we do? And then they'll say, momentum or energy. And you'll see how the class responds. So you can do quite a lot with it, besides just figures and the equations. Yes?
SPEAKER 1: Are there problems with all students having a device?
RON HARRIS-WARRICK: So we actually bought like three or four spare, little, cheap, tablets or whatever-- never been asked once. They've never wanted it. They all are so wired, it's not even funny. They just-- yeah, not a problem with the devices.
SPEAKER 2: So a few hundred students, right?
RON HARRIS-WARRICK: Oh, well, now, it's been over 1,000 students gone through this course. Not one request or concern with that. We do have them work in groups. And usually, at least one of them in lecture will bring their laptop, because it's a little easier to type the equations. And they manage it quite well. It's an excellent question, though. Good concern, yeah.
SPEAKER 5: You've already said you're putting videos of your lectures on line?
RON HARRIS-WARRICK: Yes.
SPEAKER 5: Lecture lose a little bit of vitality when you use videos. Are students really putting up with that? Are they watching all those lectures?
RON HARRIS-WARRICK: Yeah, they actually do seem to be. We've gotten a lot of interesting feedback actually. It's very nice. Sometimes, a student will ask you a question. And then you'll see this beautiful set of notes that they've taken meticulously down from the video.
But the more amusing comments are, if they miss a point, they explain how they like to back up for me, because I have a range of student abilities. Some think I explain things too slowly and put in too much detail. They just play it back on YouTube at 1.5 speed. And they shorten their lecture. Then they slow it down when they want to see it.
SPEAKER 1: So if you're talking, [INAUDIBLE].
RON HARRIS-WARRICK: No, I just hacked it together myself. I just got a free ware, a screengrab software, and I write like on a tablet. I don't have my little face there. I don't know, I know I'm going to go out of style. But my videos, hopefully-- because I love watching the old lab videos.
SPEAKER 1: You break it up?
RON HARRIS-WARRICK: Yeah, I break it up into topical chunks, about 8 minutes or so each topic is. And then they can-- yeah.
SPEAKER 1: That's similar to what [INAUDIBLE].
RON HARRIS-WARRICK: Yeah, it's like Panopto. And whatever it is, it works. They learn a lot better. So some like the lectures. Some might skip them. But it works. Yeah?
SPEAKER 6: I'm going to ask you what they asked me, which is, do you have any naysayers among your students? Do you get the negative reviews like we do occasionally?
RON HARRIS-WARRICK: Well, of course, in the first semester, it was nasty. I actually had the president of the Cornell debate team in my class, who hated what we were doing. Oh, man, I had to fend off like 20 well-constructed arguments. It was really scary.
But what I did in response to that was, that was actually what finally convinced me to go from just readings to the lecture videos. And that calmed a whole lot of it. So we haven't really had-- I mean, in 300 kids, there's always going to be a couple of complainers. But overall, the response has been very positive since we made that and a couple of other tweaks and changes. So yeah, there's not-- no more debate team meetings.
Three Cornell University faculty members in the College of Arts and Sciences shared their experiences of transforming their classrooms from traditional lectures to active learning spaces at an Oct. 25, 2016 workshop.
Four years ago, the College of Arts and Sciences launched the Active Learning Initiative (ALI) pilot project in three departments, in response to calls from government organizations and professional societies to improve college-level teaching in science and mathematics. In active learning classrooms, students enjoy more hands-on activities and more frequent student-student and student-instructor interactions through methods such as small discussion groups, partner sharing and the use of technology like iclickers and smartphone apps to enhance learning. Students gain the information from the traditional lecture through videos, readings, online exercises and quizzes they do in advance of class.
Learn more at: http://as.cornell.edu/active-learning-initiative
