share interactive transcript request transcript/captions live captions
|
MyPlaylist
SPEAKER 1: Welcome to a third installment of our series on ethical and social considerations in AI. This week's speaker's Kilian Weinberger. Kilian is a faculty member of the Computer Science Department here. He just joined us last year from the University of Washington, St. Louis.
Kilian is a leader in the area of deep learning, and that's what all the buzz is about these days, and that's what tonight's talk will be about. Kilian's going to be the Program Chair of the next big AI conference next year and was also Program Chair for the Machine Learning conference a year ago. So shows what the community thinks of him and how much of a leader he is. But rather than take his time, I'll let him start to talk about deep learning and interpretability.
KILIAN WEINBERGER: Hello.
[APPLAUSE]
Thanks. It's an honor to be here. One thing I see is a lot of people from my class here. That's awesome. That's really great. Some of the material's actually taken from my class, so just pretend you haven't seen it before. Please laugh at the jokes, just as if it's the first time.
[LAUGHTER]
OK, so I'm talking about interpretability today, but actually, I want to start with just a little overview of what machine learning actually is. Actually, can we switch this light off? Is there a way? Because it's really bright. No?
SPEAKER 3: We prefer to leave the lights on.
KILIAN WEINBERGER: OK, all right. Can you see this in the back? All right. So what is machine learning? And well, from a really, really high bird's-eye view, let me just tell you what traditional computer science is and how machine learning differs.
So traditional computer science-- you have a computer. That's this thing in the middle. And you have some data that you want to process on that computer, and you want it to generate an output. And the way you typically do this is you, as computer scientists, you write a program.
So for example, let's say the data is an MPEG4 file or something-- some video. Then you look up on Wikipedia what the file format of an MPEG4 is, how it's encoded. And over a long weekend, you write up some code that basically reads in that file and generates, basically, stuff on the screen-- that's actually the movie, right?
That's not the kind of data that I'm interested in or that I'm confronted with. So the kind of data that I see is something like this-- where someone from the medical school or some colleague comes to me and says, here's an MRI scan-- it's an fMRI scan-- of a person, and I would like to know does this person have Alzheimer's, yes or no? And so the problem is I have absolutely no idea how to write such a program.
If I knew, I would know a lot more about Alzheimer's than I do. I really don't know. In fact, nobody knows how to write this program, because we don't know how this actually works. But it turns out, well, we don't just have this one image.
Turns out, actually, doctors have taken many, many images of people with Alzheimer's. And so one thing we can do-- we can actually take these other images that we've taken in the past, and in those cases, we actually know what the answer is right. So in those cases, we know if the person had Alzheimer's or not.
So let's say five years ago someone took an fMRI scan of someone. Five years later, now it's actually obvious-- yes, that person actually, turns out, does have Alzheimer's are they should have been visible five years ago. So what we can do is the following.
We can take this data that was collected before-- so this is old data. But the nice thing about this data is we know the answer. We know yes, this image is of someone has Alzheimer's. This image is of someone who doesn't have Alzheimer's-- and so on.
And what machine learning does is you take this data for which you know what the output should be and you generate a program. So machine learning basically takes this input, the data, and the output and generates the programs. And that can literally be a program, right? So it can be [INAUDIBLE] or something. Most commonly it's not. Most commonly it's just a string of numbers that they can be interpreted as a program, but it could generate-- actually, interpret-- actually, not interpret. Oh, god.
And then what you do is, of course, then you go back to the image that you're actually interested in and you use the program that you just generated, and you stick it in, and now it generates as an output. And hopefully, if that image is similar to the images from what we call the training data, then it will give you a good prediction.
One criticism of machine learning-- and this is what we'll be talking a little bit about-- is that nobody's actually written that program, so we can't ask any humans questions about it. And this is what I will get into later-- is what can we say about these problems?
One of the early moments where people realized the potential of machine learning was in 1994, with TD-Gammon. So this is a famous moment with Gerry Tesauro, who worked at IBM, wrote a game player to play Backgammon. Backgammon is a game that's actually very, very popular. It's one of the most popular games in the world. It's not popular in the United States, but in other places, it's very popular.
And so in 1994, the way people wrote game players is you used the minimax algorithm. And to this day, it's a very popular algorithm. The minimax algorithm just takes-- basically what it does is it just tries every single moves and then flips the board and said, if I don't make this move, then you could make this move, and then I would make this move-- and it basically goes through it like 10 times, and at the end, said, well, that would look better for you. And so it tries out all possible moves.
That's an algorithm that's really, really slow and has to-- basically, because it has to try so many combinations, takes forever. So Gerry did something else. He took an artificial neural network and let the artificial neural network decide which move to take. So it programmed all legal moves, and then the neural network chose which move to take.
And let it make moves and played against it, and the neural network was horrible. Of course-- how would it know how to play? But then he did something that was really, really awesome. He took the neural network and let it play against itself.
So basically, two copies of the identical program, and let it play against itself. And then one of them always wins and one always lose, and so what he did is the one that wins gets positive feedback, and the one that loses gets negative feedback. And so he put that in an infinite loop and went home.
And the next day he came back, and the program had now played several hundred thousand games against itself. And he stops it and starts playing against it, and the amazing thing is he has no chance. The neural network beats him. And he goes around his office to ask other people-- look at this. Play against my neural network. And nobody has a chance.
So he gets really, really excited. So what he did is, first of all, he continued that infinite loop and let it play a few more million games against itself. But then, actually, he contacted the world championship organization in Backgammon and asked them, I have this program, and I think it's really, really amazing. Can I get time with the world champion in Backgammon?
And they were curious, and they gave him time with the world champion at the time. He met up with him, and it turned out the program won. And people were amazed. They were like, wow, how does it do it? And he's like, I have no idea. It learned it itself.
In fact, Gerry doesn't know how to play Backgammon that well. The program really learned it itself. And since then, people have used machine learning for many, many, many different tasks. And you probably use them every single day-- for search engines, mail spam filters, et cetera.
So machine learning is now everywhere in your life, and it's not, now, just in everyone's lives now. It's also, actually, in science. So that's what we call data-driven science. So historically, what people do in science is they try to explain some complicated phenomena and to come up with some model. And then what they do is they make experiments-- conduct experiments-- collect data, and they try to verify the model that they came up with. So this is what people have been using ever since Galileo.
Data-driven science-- that's exactly the opposite-- once again. So basically what you do is you collect data, and then from this data, you try to construct a model. And this can literally work, actually. There's a famous paper, actually-- here from Cornell-- from Hod Lipson, where you actually take a pendulum and let it swing and let a camera observe that pendulum and let the computer, by itself, figure out fundamental laws of physics-- by literally just watching the pendulum.
By the way, if there's any questions, please interrupt me. Please feel free to interrupt me. So the question is, how does machine learning work? And I want to give you a three-minute introduction on, actually, specific deep learnings.
There's many different ways of machine learning. There's statistical machine learning. There's machine learning that's based on optimization. And what I will look at is machine learning in terms of function approximation, and something that people, lately, call deep learning.
That goes back-- it was actually invented a long time ago, and it was invented here at Cornell, actually-- by Frank Rosenblatt in 1957. This was the very first neural network. It wasn't called that at the time. It was called a perceptron. And arguably, it was the first classification algorithm.
Here's how it works. Imagine you have data. And so I'm giving you an example, now, that obviously Frank Rosenblatt didn't have back then. So you have emails, and you want to know, is an email spam or is it not spam? And so what we do in machine learning-- we typically take our data, and we represent it as a vector in some very high-dimensional space.
So for example, with emails, what you can do is you can construct a long vector that counts how many times a given word is present in your email. So there's, maybe, a 1 million-dimensional vector or something. And you give them two different labels. Plus 1 are the ones that have spam, and minus 1 are the ones that are not spam.
So what does the perceptron do? The perceptron is a very simple algorithm that says, in this high-dimensional space, let's position a hyperplane-- so in 2D, this is just a line-- such that all the points of one class lie on one side and all the points of the other class lie on the other side.
And the magic behind the perceptron was-- and this was, at the time, really revolutionary-- that Rosenblatt came up with an algorithm that actually provably finds such a hyperplane if it exists. And it turns out, in very high-dimensional spaces, actually, that always exists, pretty much.
So you use the perceptron algorithm to position this hyperplane, and then when a new email comes along, you just have to look into which side of this hyperplane do you lie on? And so in mathematics, it's just the hyperplane is defined by vector. You just compute the inner product with that vector.
So that's positive if you're on this side. If it's negative, you lie on that side. And that's all there is to it. So during test time, it's a very, very fast algorithm. And actually, it's used to this day.
So just to explain this in terms of-- people draw this as a neural network. If you draw it as a neural network, and basically what you do is you have the different input dimensions of your vector X-- which are these dots here. So for example, these could be the counts of the different words in your email. And what you do is you multiply them by some weight, and you sum that up. That gives you the output. That's the same thing as this in our product notation.
OK, good. So this was in 1957, and it was a big deal. It was all over the news, et cetera. But actually, it's pretty limited. And why is it limited? Because in your low-dimensional space, drawing a hyperplane just a line may not be enough.
You could easily imagine that the blue point lies here. Well, then, actually, there's no line in the world that would separate the red points from the blue points, because you have a blue point over here. So what are you going to do?
So Rosenblatt thought about this for a couple years, and he extended it. And he called it the multi-layer perceptron. The multi-layer perceptron is exactly the same thing as an artificial neural network or as a deep learning algorithm or a deep net. These are all just different words for the same thing.
So what he did is very simple. He basically said, well, if I want to extend this to nonlinear decision boundaries, what I do is I take my input x, and I multiply it by a matrix. So basically, you have an [INAUDIBLE] transformation.
What this does, in some sense, is take certain dimensions in the space, and it stretches them. And others it pushes together. That's all that's going on. And then I have some function here that is non-linear. And what people nowadays use-- just take the max with 0.
So everything that's negative, I just forget about. Only keep them at positive values. That's called a rectified linear unit. He, back then, did something else. He, back then, used the sigmoid function, because he wanted to build something like the human brain, so he tried to model this. Turns out, actually, it's much, much simpler just to take the max with 0.
And then, after this, you just stick it in the regular perceptrons. Then you just do a linear update. And again, there's an algorithm that's called back propagation that was actually invented a couple years later to learn this efficiently-- that what this algorithm does-- it learns this matrix and this matrix such that, basically, you learn a decision boundary so that points of one class on one side, and points of the other class are on the other side.
This is, in some sense-- most machine learning algorithms can actually be written in one form or another in this kind of paradigm, as some kind of neural network. And so machine learning people are always interested in how can you make machine learning algorithms more powerful? And so for the last, let's say, 15 years-- let's say 10 years, until 5 years ago-- so during the early 2000's-- when I did my PhD, people tried to make these algorithms more powerful by making this layer here wider. And this is actually what support vector machines are.
So in case you've heard of support vector machines, they actually make this here infinitely large. So that is a mathematical trick-- how we can still actually compute it even if you have infinitely many dimensions here. And that makes networks really, really powerful.
That was the paradigm until, well, five years ago-- maybe a little bit before that. And then something interesting happened. And what happened was that GPUs came out. GPUs became very, very cheap. So GPUs are basically graphic cards for gamers, initially. But actually, turns out that they are very, very well-suited for matrix modification.
And so in here, I have a matrix vector modification which can be done really, really fast on one of those gaming cards. And so what, actually, Geoff Hinton famously did-- he was one of the first to realize the potential of this, and he bought a whole bunch of these GPUs and actually spun off a company. And basically said, well, with all these GPUs, we can do a lot of matrix modifications.
In this case, it doesn't actually pay off anymore to make this really, really wide. What we should do instead is we should make this really, very deep. So what we do is we actually-- we don't just have one layer. Here we actually have many different layers, so we multiply with many, many matrices over and over again.
This is something that Rosenblatt had thought of before-- that you can actually introduce many, many layers. But it turns out there's a famous theorem that shows that you never have to do this. Theoretically, there's no need for you to ever do this.
Everything you can learn with such a network you can always learn with just one layer that's really, really wide. So it seemed much more reasonable to just make these really, really wide. But now the GPUs came around and were so cheap!
Actually, this was no longer expensive. You could actually compute these deep networks very, very fast, and suddenly became very fashionable. And well, if you've been reading the news in the last couple of years-- or I'm sure you've seen any of these algorithms-- that's basically the crazy success that deep learning has had.
So suddenly, problems that used to be really, really hard suddenly became really, really easy. And I'm not going to go into why is this the case. It turns out, actually, if you have many of these layers, it does actually help you a lot in practice. In theory, actually, it doesn't help you.
Theoretically, there's no proof that shows that that's actually an advantage, but actually we are very good at intuitions to why it helps you in practical settings. And that depends on-- there's many good reasons why that is the case-- especially, actually, for images. Yep? Sorry, yeah?
AUDIENCE: Isn't there a concern about overfitting?
KILIAN WEINBERGER: There is a concern about overfitting. So what is overfitting? So overfitting is-- initially I had this example with fMRI, MRI scans, and I said, well, I show you images of people where I know this person has Alzheimer's and this person doesn't have Alzheimer's.
Well, one thing the machine learning algorithm could do is it could just memorize which person had Alzheimer's, which person doesn't have Alzheimer's, for example, based on the form of the skull. That works, but it wouldn't generalize for any new people.
So usually-- you're totally right-- machine learning theory says the more parameters you're fitting, the more likely you are to overfit, and you have to do something against it. Now, turns out the way these deep neural networks-- and they have many, many parameters, you're totally right. But the way they are trained is actually with an algorithm called stochastic gradient descent. And what that does is-- it gets a little bit technical now, but what it basically says is ultimately it doesn't find any solution here. It actually finds a very specific solution that is very constrained.
And so it's not actually that these matrices can take on every possible value. You actually effectively have much fewer possibilities than you should have. Does that make-- yeah, and we can talk about it more offline. That's a good question, though. Any other questions about deep learning?
OK, so this baby says it's getting boring. So let me show you a demo. One second, just need to-- OK, so what I thought is I'd just show you quick examples of neural networks, just so that you can visualize it a little bit.
So the first thing I want to show you is actually just a little example of the perceptron. This is the algorithm that was invented in 1957. And so here's what we do. We just take some data points in 2D.
These are the positive points. These were the negative points. Here are the positive points. And what the perceptron does, now, it basically-- one second. So it basically starts with some hyperplane.
Here is a plane that says, OK, I would like to find a line such that all the positive points are on one side, all the negative points are on the other side. This is clearly not correct. This guy is wrong, and this guy's wrong.
So it picks one point that is wrong and makes an update so that this point is now classified correctly. So it moves the hyperplane, and that's what it does. And actually, in this case, it's already done, because this is a really simple example.
So now, actually, all the positive points are on one side. All the negative points are on the other side. And the beauty about the perceptron is that you can prove that if a hyperplane exists, you will actually stop after a finite number of iterations.
So it will not go infinitely, which is an important property. Just to, actually, get the-- here's one more example, just to actually introduce this concept. This was classification, so you try to find a hyperplane such that the positive set points are in one side, the negative points are on the other side.
Another setting that's been known in statistics for many, many years is regression, where you actually say, I have data points, and I would like to find a hyperplane-- a function that basically best fits these data points. So in this case, I have no two classes. I just have one class, and I would like to find some function that explains this class.
In this case, basically, you find a line, as here. I think my data points basically roughly lie on this line. So the relationship between x and y is basically roughly the relationship of this line. These algorithms are tightly related, and so one is called regression, and the other one's called classification.
So let me show you an example of, actually, a neural network. So-- [INAUDIBLE]. OK, so here's the same problem. So I'm trying to have these data points here.
I just draw them anywhere I want, and now I can say fit a neural network-- basically, this multi-layer perceptron-- to explain these points. And I can run it, and this is what it gives me. And so what the neural network does, in this case-- it actually finds many, many linear functions that approximate this data set.
So the numbers I have here is basically I have 40 hidden neurons in the middle, and then another 40-- so that's two layers of 40-- neurons. Any questions about this demo? Yeah?
AUDIENCE: So what do the hidden parts in the email versus in this--
KILIAN WEINBERGER: They basically make these lines. There's not just one straight line. So what about-- these hidden nodes, in some sense, say that we have many different lines that we can switch on and switch off. And by doing this, you can actually say, here's my first line. Now the second line comes in, which now makes it a little bit steeper, and so on.
So all of these, basically, are functions. And because I always have this max operator-- max with 0-- whenever they become negative, I just set them to 0, which essentially just means I switched them off. So I have many, many functions I can switch on and off. And this way I can actually combine them to take any shape. That's the high-level idea.
I can maybe show this, actually, on a-- here. 1d plus-- all right, here we go. This is the demo. Here I have a data set, and what I'm doing is I actually increase the capacity of the network. So initially I just have two nodes.
So in this case, actually, you have a very, very simple neural network. So it tries to explain this data, and all it can do is basically have some linear function here, some linear function here, and some linear function here, and that's all it does. That's all it's capable of.
Now I can increase the number of neurons I have to four, and it already has a much, much better fit of this data-- maybe. This could also just be noise. Who knows? And what's the scary part is this part here. So what scares people is in this part, where there's no data, the function sometimes can do anything it wants. And that's why people would like to get more interpretability.
All right, so as I increase the capacity, it keeps going up and fits the data more and more. I actually have one more-- more fun-- application. This is all-- I mean, I look at you, and you're like, well, this is just some dots.
[LAUGHTER]
What is this? All right, so let me show you something more interesting. OK, so I thought of some really-- I hope I'm not offending anybody, because I'm on a really challenging part. This is on ethics, so let's do something ethically questionable.
[LAUGHTER]
I hope Joe and Barb will still talk to me afterwards. So here's the idea. So I thought, well, let's come up with some really, really hard task. This is not going to convince you that neural networks are really, really awesome. So here's a really, really hard task.
So here's what I did. So turns out, there's this web page where people can upload their own photos, and other people can go to the back page. And they can say if you're hot or if you're not hot. And so what I thought is, well, this is actually great.
So now we have these images of people, and we actually have thousands of votes if the person is hot or if they're not hot. And so I thought, well, can we train the neural network to predict if someone is good-looking or not? This is a really, really hard task. I usually can't do it.
[LAUGHTER]
So here we have a data set. So there's several thousand images taken from this data set-- from this web page. And so what I did is I actually cropped the faces out. I'm only looking at the face. Some guys cheat. They go in a muscle shirt like this or something. So I'm just look at the face.
And so on this data set, actually-- on the data set that I showed it, the data points-- it gets 98.16 correct. That's not surprising, because these are the images where I showed this image of someone who's hot. This person's not-so-hot, et cetera. And by the way, I just binarized it. Just said above-average or below-average.
So on the data set that I showed it, it does really, really well. But on data points that I didn't show it, it still gets 73% accuracy, which is very impressive. This is an extremely hard task. So what I thought is I can now actually-- I can test this. Let's just see how this works.
So the first thing is-- this is Heidi Klum. So Heidi Klum is a supermodel. I don't know if you know her. She's from Germany. But--
[LAUGHTER]
So the question is, what does the neural network-- what would it predict if it sees Heidi Klum? And just to be clear, the labels here are minus 1 for not good-looking and plus 1 for good-looking. So we can now test out this image, and let's see what the neural network thinks. And it's about 1. It's off the charts.
[LAUGHTER]
Has the hots for Heidi Klum. It's not bad, it's bad at all. So then I thought, well, what about a guy? I don't know who's good-looking, so I asked my wife. And she didn't skip a beat. She's like, George Clooney.
[LAUGHTER]
It razed me a little that he looks nothing like me, but that's OK.
[LAUGHTER]
All right, so let's see how George Clooney fares. So I can stick him in, and 1.2, 1.3. It's not bad. Not bad. So of course, I couldn't resist, so--
[LAUGHTER]
How am I doing? So finally someone is telling me the truth. All right, watch George Clooney going down. Let's see how it does-- and all right, never mind. Never mind.
[LAUGHTER]
How am I doing compared to the-- i need someone who's worse than me. How am I doing compared to the ugliest person on the internet? And it's actually easy to find. You just type "ugliest person on the internet" into Google, and this is the image you get.
[LAUGHTER]
That was the first hit. So as I go-- let's try this out. All right, oh, never mind.
[LAUGHTER]
All right, back to PowerPoint. Back to PowerPoint.
[LAUGHTER]
So clearly, there's something we don't understand about neural networks. I just want to point that out. Made a lot of mistakes. Any questions about the demo? Yeah?
AUDIENCE: Once you had a set of hidden nodes that were larger than the number of points on that two-dimensional model, why was it not just fitting directly between all the points?
KILIAN WEINBERGER: Why was it not fitting--
AUDIENCE: Just drawing a line directly between the points [INAUDIBLE]?
KILIAN WEINBERGER: It depends on how I trained-- I don't remember. It probably was that there was some regularization going on, yeah. Any other questions?
AUDIENCE: [INAUDIBLE] score higher than [INAUDIBLE]?
KILIAN WEINBERGER: I have not tried this. I've given up at this point.
[LAUGHTER]
Any other questions? All right. OK, good. So why does deep learning work so well on images?
AUDIENCE: How robust is the algorithm? So suppose you take the same person from different aspects--
KILIAN WEINBERGER: Oh, I have not tested this, actually. I have not. Yeah, so actually, one thing I keep telling myself-- this was not my best picture.
[LAUGHTER]
Could have done better.
OK, so one thing about deep nets is that they work surprisingly well on images. And really, most of the successes, actually, that you've seen in the last couple years have been on images-- either object recognition, or even, actually, when you look at the Go player-- AlphaGo.
Actually, one trick they did is they took the board and encoded it as an image, actually-- as a multidimensional image. So it was, like, a 10-dimensional image. Not just red, green, and blue, not just three dimensions.
So the one thing that seems very, very clear is that neural networks work particularly well on images. And there was one paper that I thought was very interesting where we got a lot of attention, and that was in 2014 by Szegedy, et al. And they did some very interesting experiment.
So they took an image-- there's this ImageNet competition. There's a million images. And so they trained a neural network on the competition data set and then tested it on some images, and these are images that it all got correct.
So this here's, I think, a speaker system. This is a praying mantis. This is a laptop. And then one thing they did is they said, OK, well, what is the smallest change I would have to make such that these images get misclassified-- not just as anything, but as an ostrich? And ostrich was an arbitrary thing.
So these are the changes that they had to make. So they found them-- and it's very easy to find them-- and these images here are basically the output. So this image here actually gets classified as an ostrich. So this here is the speaker system. This here's an ostrich. So this here;s a praying mantis.
According to the neural network, this here's an ostrich. And so all of these images are ostriches. And not just a little bit, but with very, very high confidence. And so this got a lot of attention. And people went, oh, this is really scary. Clearly this is not an ostrich. So what's happening here? Any questions about this?
And it scared me too, but for a different reason. I was interested in this setting, where you go to a doctor and the doctor uses a machine learning algorithm and says, congratulations, our machine learning algorithm predicts you have high risk of dying of terrible disease x, y, z. And I'm like, well, that doesn't sound so great. What can I do to change that prediction?
Well, what we just learned is you can change a lot to change the prediction, but it doesn't change jack. You're still going to die of the disease. It's just that the algorithm is going to think that you're healthy now, but really nothing changed. I mean, this actually did not turn into an ostrich. I don't know if people in the back can see this.
OK, so I thought about this a little bit, and I came up with some conclusion. And this is my conclusion. Well, this is my explanation. So one reason that neural networks work so-- and by the way, this is a very new result, actually. It actually just got accepted to conference. So you're the first big audience to see this.
One reason that neural networks work so well on images is because images lie on some Riemannian manifold. So what is the manifold? The manifold is some surface that's curled up in some high-dimensional space. So this here, for example, would be-- it's like this.
Why do I know this? Well, it's quite simple. So if you take the space of all images-- if I just take an image and I basically look at the pixels, obviously not everything in that space is a natural image. If it was, then I could just take an image, randomly assign colors to the pixels, and I would get an image of something-- maybe my brother or something.
Well, I wouldn't. It would never, ever happen. You can spend your entire life randomly sampling images, you will never get anything that's real. Why is this? Because the space of possible pixels-- this 16 million-dimensional space of 16, or whatever, 8 megapixel images-- is gigantic, but actually, only a tiny, tiny part of it is actually the surface of natural images.
And so why I think it's a Riemannian manifold? Why is it curled? Well, I know it's not a subspace. Why is it not a subspace? Because if I take two images of people that are far away from each other and I would average them-- imagine if you think an image of Hillary Clinton and Donald Trump and you average them. You're not going to get anything meaningful, because the people look too different.
So somehow there's some smooth transition that curls up in the space-- in this very, very high-dimensional space-- that goes all the way from Hillary Clinton to Donald Trump. Well, if you just look at bearded people versus not bearded men-- so clearly there's a smooth transition between them, but if you just take two people that are far away-- yeah?
AUDIENCE: What makes you think it's smooth?
KILIAN WEINBERGER: Why do I think it's smooth? OK, because I take two-- good question. So she says, OK, all we know here is that globally it's not a subspace. But why is it smooth? Well, the idea is the following.
If I take two images that are very, very similar-- for example, two twins or something-- and I align the faces and average them, I would actually get something meaningful. So that's basically-- or if you have a movie where you actually have very, very small changes, you can actually interpolate between them. So very, very locally-- if you're a little ant and you live on this thing and you're tiny, then, actually, you think it's locally Euclidena. Yeah?
AUDIENCE: You claim that it's for images in general, but your examples are faces. Is the claim actually for any--
KILIAN WEINBERGER: So the claim is for natural images that we observe in real world. Here I only took faces, just because it's very-- I'll actually show you in a minute. It's very easy to get data sets of faces, and it's basically a sub-manifold.
By the way, they don't all have to be connected. There could be a manifold of man and manifold of women-- which, there may not be a transition between them. But basically, every image lies some manifold of some images.
OK, so that's basically my claim. And then this is something, actually, that's-- I didn't come up with this. This is something-- there's pattern theory. This is something that's been known or hypothesized for a long time.
But my claim is that neural networks-- what they do is they take this complicated manifold and they map it internally into a subspace. And so in this subspace, now, things suddenly become easy. Transition between very complex settings-- do we have a beard or not? This is just the example I picked here. Suddenly become, actually, a linear transformation of space. That's what I claim.
So I told people this. I said, that explains what's going on with these Szegedy things. So with these Szegedy-- this paper that I showed here-- if images lie on a manifold, what's happening is if you change the image ever so slightly, you actually move it off the manifold. So what we're doing here is we take this image of a lapdog, and we're adding some features of an ostrich on it. And so we're moving off the manifold, and that's something that the neural network has never seen, and basically off that subspace, and weird things | happen.
So people go like, well, prove it. So I thought of an experiment how to prove it. So here's the area. So I take neural network, and basically, this neural network's trained on some data sets. And it's not even trained by me. It's been trained by someone else.
And-- OK, sorry, one more thing. Why do I actually believe that it's a subspace? Because in this space, actually, now you can train linear classifiers that work really, really well. So you can actually train a perceptron, for example. It just learns a hyperplane to classify bearded men from not-bearded men-- which is a really, really complicated concept in this original manifold.
So you could never do this in pixel space. Suddenly these really complicated problems become easy, and by easy I mean linear. OK, so here's my experiment. What I want to do is I take a new image that I know that the neural network has never seen before, and I stick it in here.
So there's myself. So this picture was taken after the neural network was trained, so this is definitely not in here. And I don't have a beard, but what I could do is I could take a lot of people with beards and move myself into the direction of the people with beards. And if this actually works-- if I'm actually right-- then what should happen is I should get a beard. So that's the experiment.
So let's try this. So here's the idea. So I take my image. I propagate it through the neural network. Now I represent this image as all the nodes of the neural network-- just concatenated-- for some really, really high-dimensional vector.
That's this point here. That encodes what the neural network thinks about me. I did not train this neural network. This is a neural network that I downloaded that other people at Oxford University trained.
So this here's this point, now. This is my face. Now, here's what I do. I take people-- a man-- without beards, then I take men with beards, and I also put them through the neural network, and they'll do exactly the same thing. So I would present each one of them as all the knowledge of the neural network about this image. This is a long, long vector.
And now here's the trick. Now I take all the people with beards and all the people not beards, and I average their representation. So now I basically get the average bearded man and the average non-bearded man. And if I look at these two vectors, this gives me a direction. And what is that direction?
The direction's the direction from non-bearded man to bearded man. And if this is really a subspace, then I can walk along that direction, and I could add beards to people. So let's try this.
So I take my data point, and if I now move it to the right, it should get a beard. Here it is.
[LAUGHTER]
Not that easily fooled, all right. So here's what I can do-- is I can now take this data point-- this is just a whole-- it's, like, millions of numbers. Well, hundred thousands, actually.
The question is, can I actually reconstruct an image? And it turns out, actually, that there is a way, introduced by Geddes et al a couple years ago, how to actually find an image that corresponds to a point in the neural network. So we can do exactly this.
We can now take this point that I just constructed, which is just me plus some vector, and go backwards and generate the image that corresponds to this point. And are you ready? Let's do it. And here we go.
[LAUGHTER]
I was actually amazed that this actually worked. So we can actually-- we don't have to just do this with me. We can actually do this with many things. So here's a bunch of people that we randomly picked from some database of images.
There's Berlusconi. These are famous people. There's Ralf Schumacher. And we can just add beards to them. But there's nothing about beards that is special, so we can just actually make them look older. We can make their mouth open.
You can open their eyes. You can make them smiling. You can add glasses to them-- which is actually quite amazing that that works. And the amazing thing is there's really-- I just take a point, and I'm just linearly interpreting it. So in some sense-- this is a cool tool to play around with, and then my students have a lot of fun with my face, actually.
[LAUGHTER]
But really, I'm making a point here. The point is that there's a subspace. I can actually even do this in high resolution. So this is a really high-resolution image. And I can now actually show you the movie as I'm moving this along this direction of becoming more bearded. And so here's this guy, basically, I'm moving back and forth in this space, and adding beard or subtracting it.
Here's two pictures of me that my students had fun with. And so this, basically-- what you see, going to the right, is I'm going further and further and adding more and more beard. You can also make me look older, so this is what I-- when I started the faculty job, this is what I looked like. This is what I'm--
[LAUGHTER]
--where I'm heading.
AUDIENCE: [INAUDIBLE].
KILIAN WEINBERGER: And-- yeah, my students couldn't resist. So here's a picture of a little boy. You can give him a beard.
[LAUGHTER]
But it doesn't actually always work, and so here's some failure cases. It's interesting to see why it failed. So the projecter, unfortunately, is not bright enough. So there's a person who's Caucasian, and then we add him a beard. Turns out he becomes African-American.
[LAUGHTER]
And so we thought about this. Turns out, actually, basically what happens is, here-- this is very simple-- we picked bald men, because we didn't want to add hair. So it turns out, actually, that this is more a fashion in the African-American community, so actually, what we did is we had added a vector of not just adding beard, but also actually making them more African-American.
One thing we also found is that it doesn't work as well for women. And the reason here is actually-- it's quite interesting that, actually, the manifold of women is much, much more complicated-- pictures of women.
[LAUGHTER]
There's actually a very good reason for this, because men actually all look pretty much the same.
[LAUGHTER]
There's only a few looks that men sport, whereas, actually, women-- they have very different hair and very different-- more makeup and so on. And especially, actually, most images that you find online, actually-- celebrities. So celebrities actually use a lot of makeup, and then so on. And that actually breaks the manifold. So you would need a lot more data from them, which we don't have, so you actually get weird artifacts.
Any questions about this? Yeah? Sorry, yeah.
AUDIENCE: So that's your hypothesis. Have you applied it to other domains than faces-- for example language?
KILIAN WEINBERGER: You know, we have actually. We have. And it does work. Basically, what we did is we took positive Amazon reviews and then tried to make them negative. And it works, but it breaks the language.
So the one thing is, with pixels, we're not very sensitive to small changes here. But for words, if you just add one word that doesn't make any sense, you immediately see, oh, the sentence is garbage. I mean, at the beginning what it does, it takes a sentence-- "I love this razor--" and it says, "I love this razor not."
But then I see, as you move it more, it breaks the grammar of it. And that's just because that's too subtle. Yeah, good question. Any more questions? OK, good. So now we can get to interpretability.
So I basically tried to make two points here, actually. The first point was when I talked about how neural networks work. That actually, the building block of neural networks is really, really simple.
It's just a hyperplane. It's just a linear [INAUDIBLE]. And what we do is we just modify these matrices. What does that do? Well, in some sense, you're just having many hyperplanes. You're just stretching different dimensions.
So we just basically-- it's like LEGO blocks. They're very, very simple, but you can build really complicated things out of them. So the individual building block of a neural network is crystal clear. There's no question about what it does.
So when you look at that demo, basically what it is-- just took these linear functions and just cuts them together-- concatenates them-- to basically learn some complicated function. But the actual function that it learns at the end is often hard to understand. And so for example, it completely failed to classify me as beautiful.
[LAUGHTER]
So lately there's been a lot of talk about we should-- people are scared of neural networks, because now they can do so many things. And so in particular, self-driving cars-- what they do actually have is they have a neural network at the beginning, and at the very front, that actually does image segmentation. So basically what it does-- it detects pedestrians, detects the road, et cetera-- all the stuff. And it freaks people out, because they don't know how it works.
And so there's been a big cry for more interpretability in neural networks. And so the question is-- I tried to sum up the reasons why do people want machine learning algorithms to be more interpretable? And so ultimately I found four high-level reasons why that's the case.
The first one is to learn more about the data-- to basically say, well, my classifier can do something. So if you go back to the very, very beginning of the example with Alzheimer's, let's say I actually can learn an algorithm that predicts if a person is Alzheimer's, yes or no.
Well, the neuroscientists, they don't actually want the algorithm. They want to know how it works, because they want their science paper-- that actually explains the mechanism of Alzheimer's. So that's the first part. Another reason why you want interpretability is debugging-- is because, basically, something doesn't work. Car drives against the wall.
Then, of course, is accountability. What if your neural network makes a mistake? Who is guilty? Is the person who trained it, the person who wrote it, et cetera? And finally, there's trust-- and I think that's a big one.
So people, before they get into this car-- the first time, they'd be really scared. So how do you know that you can trust this thing? So let me just go over each one of these.
So the first one is learn about the data. So this is something that I have a lot of experience with, because I work a lot with people in the medical-- used to work a lot with people in medical school. And this was exactly their setting. So basically, they give me some data, and then me and my grad students-- we retrain some neural networks to make predictions. And then at the end, if it worked right, they really, really want to know, how does it do it?
One thing I claim is that actually, in this case, it's not very hard to figure out how it does it. And the reason is because you're working with these domain experts, and so basically come up with a hypothesis. And then basically, what you say is-- let's say you predict something in the brain. What you basically do is you can just say, OK, well, let's retrain the algorithm one more time, and let's take out this part of the brain.
Let's make sure the algorithm cannot access it. And let's see if it can still predict it. And you keep doing this, and eventually, you basically figure out, OK, well, this here is the essential part of the brain, and only if that's included does it work. And if it's not included, it doesn't work. And so there must be something in here.
And you can do some other tests. So if you're allowed to retrain the algorithm many times and just make changes to the data-- well, one thing you do is you take some of the dimensions of the data and you just shuffle them randomly. And so now they're nonsense. And you see if, actually, that affects it or not. So then if it doesn't affect it, you know this was not necessary.
So I think that we are actually at a level of interpretability that is sufficient for science, because you can retrain the system. You can try out hypotheses. I don't think that's a big issue, and I've never really seen many cries there from trying to understand better how the system works.
Second thing is debugging. Now, that's something that's scary. So this here is actually a picture of the person who got fatally injured because he drove his Tesla on autopilot, and the Tesla actually ran into a truck or-- I think it was a truck, a trailer-- that was actually orthogonal to the street.
And so in this particular case, of course, the question's how could that possibly happen, and how can we prevent these things in the future? Now, actually, let me just actually go into this particular setting, just because you may have heard about it. In this particular setting, actually, people have analyzed what happened, and the machine learning algorithm did very well classify this object as something on the road. It just thought it was so unlikely that there would be a truck perpendicular to the road that it basically just decided not to brake.
And that's actually an interesting decision, because these things will happen all the time. It actually is very, very unlikely that there's a truck in front of you that's orthogonal to the street. It doesn't happen very often.
And so it's much, much more like there's actually a bridge and you shouldn't slam on the brake. And so there's basically a trade-off that someone had to make about, well, what if something really, really unlikely-- may be the case-- that could be really bad? Should I step on the brake, yes or no? And there's some threshold where you have to do this.
And the danger is that if you slam on the brake too often, the next time you drive 80 miles down the freeway and there's some bridge and the classifier's not sure is that an object, or is that a bridge? It's going to slam on the brakes suddenly, and you will cause more accidents that way-- potentially.
So actually, it turned out, in this case-- actually, ultimately, the decision was that it wasn't a bug. It was the kind of mistake that these algorithms make. And it's important to put that in perspective. There's some mistakes that algorithms make that humans would never make, but there's also mistakes that humans make that algorithms never make.
So for example, humans get drunk and then get into their car and are sure they can still drive home. Or humans try to impress their girlfriend and drive 120 miles an hour downtown or something. So algorithms don't do this.
So in this particular case, ultimately, there may be some responsibility on Tesla's part, in particular because they didn't actually use a LIDAR. So if they had used a LIDAR, it would have never happened, because LIDAR detects objects. But it's pretty clear what happened, so I wouldn't consider this a bug.
But let me get back to debugging. So the problem is we have some really complex machine learning algorithm that makes decisions, and you want to now make sure this works right, and you want to test it. But how do you do this?
Well, how do you test other systems? Let's say you write an operating system-- you write some big, big program with 50,000 lines of code, or you have an Intel processor. Let's use the Intel Sandy Bridge processor-- 1 to 2 billion transistors.
So how are these things tested? It's actually not that someone goes through every single one of these transistors and checks, is this correct? Is this correct? What they do is they actually have benchmark tests that they run them over, and they make sure that everything is correct. And if it's not correct, then they basically go back and look into it. And then they have new benchmark sets.
Well, you can do exactly the same thing with machine learning. So this is exactly the kind of thing you can do really, really well. You can actually run tests and run it, and if it fails them, you can retrain and do these kind of steps.
So ultimately, my point is that you have to treat machine learning algorithms as a really, really complex program-- which is essentially what it is. It's exactly what it is. Which typically also gets debugged by basically testing it repeatedly and making sure it works on these unit tests.
The nice thing about machine learning, actually, in contrast to traditional programming is that you can retrain it. So after, basically, you see it fails in certain cases, you don't have to go back and read through 50,000 lines of code and see, where does it fail? You just throw in these examples into your training set, and you retrain. So debugging may actually become easier. Yeah, Greg?
AUDIENCE: So on the other hand, you have higher level abstraction compared to the [INAUDIBLE] for the [INAUDIBLE] or the [INAUDIBLE] program. And many of the-- so it's easier to debug the tests [INAUDIBLE] level of your analogy.
KILIAN WEINBERGER: That's totally true. So basically, what he's saying is for these, sure, you wouldn't go through the transistors, but actually, what you do is you have high-level representations. You say, this part is the ALU-- so basically, this is the part that adds numbers or something. And this here's the memory, et cetera. And I can test each one of these.
So machine learning algorithms-- it depends. So you don't really have this kind of level of abstraction. On the other hand, the bug is very rarely in the code. The bug is different.
The bug is more that the training data does not look like your testing data. And that is something you can test very effectively, actually. You can train a classifier to predict, does this sample look like my training data, yes or no? And most of the cases-- for example, this was actually a case where this was a crazy outlier. And you have to deal with those effectively, but it's a very different way of thinking, I think. Does that answer the question? Yeah. Yeah, Fred?
AUDIENCE: Well, these program systems-- like hardware and software-- they were independence assumptions that you made. You may have characterized it as abstraction, but the essence is you can infer, from carefully understanding one piece in isolation, the unit test, how it will work in context.
And it seems that the whole essence of machine learning is that you're not doing isolation, but the big win is the fact that you're combining things. And therefore, your question about whether the data in the real world corresponds to the training data is equivalent to saying exactly because we don't have independence, we have the leverage we get from machine learning, and that suggests that there's not a good analogy between how we debug complex systems and [INAUDIBLE].
KILIAN WEINBERGER: I agree. I agree completely. So this is not perfect. I guess my main point was is that there certainly are these very complex systems that are also very hard to debug, but you're right.
So basically, what you could do is you could just take this part by itself and debug that part and just say, well, that works in isolation. You may have--
AUDIENCE: It will work in the context.
KILIAN WEINBERGER: Yeah. You may have some of these things, though, with local regions of the data. So basically, you could test your classifier on bridges right. And it doesn't mean that it works on trucks, but you know roads and bridges afterwards. So in some sense, when you looked at this-- remember this little demo, where I showed basically it has these little patches-- these linear patches. So if you know what works well in a certain region of the space, it doesn't tell you anything about the other part, but you could debug that.
OK, one thing that's important is accountability. So the question's-- let's say something breaks. In this case, who is guilty? And that I don't really have a great answer for, because ultimately, I think it's a legal question, and our legal system is just so made for humans that it's hard, when machines make decisions, to decide these things.
I think what it will lead to is something like this, where you actually have a human who's responsible. This is actually the solution that Tesla came up with, where they basically said actually, the autopilot-- despite we call it autopilot-- is not an autopilot. It is just an assistance, and you are still responsible as the driver, and you should never, ever take your hands off the steering wheel right. And so ultimately, you know it's clear who is the responsible person in this case.
There's two things I just want to highlight. Sorry, yeah? Actually, that's why, in Europe, it cannot be called autopilot. It's called something else. There's also two things I want to highlight.
So one thing is outliers. So machine learning algorithms-- where they will fail, where they're more likely to fail, is for data points that are really different-- they are really, really rare, and they're probably not in your trained data. One possible solution in terms of legal terms is something that was done with vaccinations many decades ago.
So with vaccinations, at some point, the US government made the decision that it's recommended that children get vaccinated-- which is in the interest of everybody-- but there's a very, very small chance, whenever you get vaccinated for smallpox or something, that you actually get the disease. And this could be really terrible.
So the solution they actually had is that everybody pays in some money into a big pot, and whenever something goes wrong, that pot of money then was used to actually compensate the victims. So you could imagine something similar for self-driving cars, for example, where there's basically a small chance that it hits you because there's some really, really bad situation. But in some sense, we're averaging out the risk of everybody. It's just one solution.
Well, one thing that, I guess, Jon Kleinberg talked about a few weeks ago, I think, is systematic biases. So that is a problem. And you've seen this in applications in the criminal justice system.
The question was, convicted felons-- are they likely to commit future crimes? And one thing that is very apparent is that, actually, there was racial biases in these predictions that the classifiers made. And that, I would still say, is an open research question, and figuring out how to fight against this-- that these algorithms pick up these biases from the data.
All right, last thing you should have is trust. So there is this trust issue, and it seems scary to get into a car-- and I just have this as a running example, but-- or to get surgery done by some robot that you don't know how it operates. But I think ultimately, this is something where numbers will speak. I came to make a little example.
So imagine you have to go up for surgery, and this is a dangerous surgery. It's on your heart. And if you go to the hospital, there's a 90% chance it succeeds, but in 10% of the cases, you die. So there's a 10% mortality rate.
You could, instead of having your [INAUDIBLE] surgeon-- you could also have a robotic surgeon that only has a 1% mortality rate. So it's much, much more accurate, but you have no idea what it does. Who would pick the robot? Raise your hands.
All right, so--
AUDIENCE: We have no idea what the human does.
KILIAN WEINBERGER: That's right. That's exactly right. And actually, that's a very good point. But one thing is you could, of course-- it could be-- let me just tell you some more.
Let's say the human usually fails by making some mistake in the heart. The robot sometimes is just totally off. We have failure where, instead of actually cutting out your heart, it actually cuts out your eye.
[LAUGHTER]
But it's very rare, but it occasionally happens. The human would never do this. Who would still go with the robot? Let's raise your hand. So what if I tell you-- OK, one last one.
What if I tell you that the robot was actually trained on people over 60 years old and does really, really well on people over 60 years old, but if you're under 60-- especially under 40-- there's a 50% chance it cuts your eye out. Who would still do it?
[LAUGHTER]
So your odds just went up. So I think those are the questions we have to ask. The question, basically, is are you representative of the training data? And is the algorithm actually trained for you as a patient?
This will make an imperfect analogy, and I hope it doesn't offend anyone. But actually, when planes were first invented, only men were allowed to fly. And it took a long time for women to fly. And actually, when I was a kid-- I remember my parents talking about this, actually-- the first time German airlines allowed female pilots, some passengers walked out of the plane.
They were like, we're all for equal rights, but let's not risk anything. I'm out.
[LAUGHTER]
But the irony was that women pilots were actually safer than male pilots, and the reason is-- I mean, I don't know this for sure, but I could speculate that it was probably much, much, much harder for women to make it, and therefore, only the very best made it to become pilots. So those who did make it were actually safer pilots than men.
But the interesting part is that just because it was different, people actually were scared of it. Now this is, of course, not a great analogy, because computers make very different mistakes. The difference between men and women actually turned out-- wasn't real. But computers would actually make slightly different mistakes, but I think a lot of the trust issue is actually just having something new that you're not familiar with and that you have to get used to.
And I'm already four minutes over, but I have-- should I keep? OK, OK. So let me just talk about two efforts to make machine learning algorithms. Ordinarily my point of view is that, very often, you don't need interpretability, but sometimes you do. And some people have actually made some efforts to make algorithms interpretable. And let me highlight two papers that recently came out.
And this is something from Trevor Darrell's group in Berkeley. And this is an interesting algorithm. So here's what it does. I don't know if you can read this. This is actually trained on the Cornell Bridge Recognition data set from the Lab of Ornithology. And so here's what it does.
It predicts what that bird is. It's a Baltimore-- or I don't know what that is. Oriole or whatever. But not only this. It actually tells you why. So it actually-- I don't know if you can read this. It actually says this is a Baltimore-- I don't know how to pronounce that word-- because this is a--
AUDIENCE: "Or-ee-ole."
KILIAN WEINBERGER: "Or-ee-ole--" because it's a small bird with a black head, and orange body with black wings and tail. So that's a very, very good description of, actually, that exact bird. And then you can actually make a lot of sense of it.
That made quite a splash, and people were really excited about this. But actually, let me just be careful. So this works great on these examples, but here's, actually, the second-most-likely choice. It says this is a cliff swallow-- which it's not-- because this bird has a black crown-- which it does-- a black throat-- which it doesn't-- and a white belly-- which it doesn't have at all.
And the problem with these kind of algorithms is that if you read the sentence, for humans, this seems to suggest-- this sounds like something another human would tell you. And so you think, oh-- it seems to suggest to you that-- to us-- that this is actually how the algorithm really arrived at its decisions. Which actually, it doesn't at all.
So the algorithm was constrained to predict these birds and to come up with plausible explanations, but actually, there's no reason to believe that this is really, truly the reason it predicts this. It could actually have predicted this just because there's a green background and still come up with the same explanation. Does that make sense?
So just because it sounds like something that a human would say to describe their thought process does not actually mean that a machine actually has followed that thought process. So actually, [INAUDIBLE] these vectors-- they actually took trained data of birds, and they actually wrote explanations. And they basically trained the algorithm to predict the bird correctly and to come up with the same explanations.
AUDIENCE: So it's not doing anything besides identifying the bird? The bird is just paired with the sentence?
KILIAN WEINBERGER: That's right. The bird is paired. That's right. Yeah?
AUDIENCE: But it seems like these descriptions are slightly different each time.
KILIAN WEINBERGER: Oh, yeah, yeah. So it basically has some probability distribution over descriptions, and it draws one of them. And so in this case, actually, you could then verify this and say, actually, something is wrong here. It's a colorful bird with red belly-- which is actually not true. So you can actually conclude that maybe there's a mistake. But you have to be very, very careful, because it's really not a human who said this. It's just a probability distribution over possible sentences, which is not really as connected as you think to actually arriving at the final conclusion. Yeah?
AUDIENCE: It seems like it's just got to learn some correlations between pixels, pics, and the colors and stuff without having any understanding of the order in which these--
KILIAN WEINBERGER: That's right. So that's right. It's not--
AUDIENCE: What if we can-- is there anything which gives a cause more to like, OK, color causes this description?
KILIAN WEINBERGER: Yeah, so the problem is now you're-- that's a good question. Automatic machine learning is all about correlations. All you're doing is you're discovering correlations in the data.
So if you now find these causal things-- actually, Joe is actually the expert on this-- my fear would be that if you actually tried to inject, basically, these kind of very clear reasoning patterns into machine learning algorithms-- and people have tried this-- you take a massive hit in accuracy, because you're really constraining the algorithm.
Basically, you're forcing it to think like a human, which is really not the way these algorithms work. They're really just function approximators, and you pay a very high price. Yeah?
AUDIENCE: Is there any theoretical work in saying how there's actually a possibility for there to be human interpretable explanations of this?
KILIAN WEINBERGER: There's not.
AUDIENCE: Because wouldn't it be entirely possible that, although it arrives at the same conclusion, it's a very different reasoning--
KILIAN WEINBERGER: Oh, that's totally correct. Yeah, that's right. That's right. And I think it's actually-- that's very much a real possibility. Yeah?
AUDIENCE: A question about the debug. It sounded like you might have been arguing that a bug is really just a blind spot in the data. [INAUDIBLE] definition of a bug.
KILIAN WEINBERGER: So it's a good point. A bug is just a behavior that you did not intend. So bugs are different than the bugs in code, where you basically have the for loop, the index wrong or something. So I think typically, machine learning algorithms-- it's rare that you really have bugs in the code. The code is not all that complicated, actually.
It's mostly matrix modifications. And there we have these levels of abstraction that, actually, Greg mentioned earlier. It's relatively easy to write bug-free machine learning code, because it's always not very long. It's usually just maybe 100 lines of code or something.
AUDIENCE: Like an [INAUDIBLE] at the end of the day.
KILIAN WEINBERGER: But then the algorithm that is learned-- that could actually make mistakes that are surprising. So those are the kind of bugs that are hard to debug. Yeah?
AUDIENCE: So when you write programs, when you have exceptions [INAUDIBLE] exception handle and stuff, and we can handle those exceptions because we understand [INAUDIBLE]. So for machine learning algorithm, if we have exception, [INAUDIBLE] exception and point out outliers and these things. And we don't understand what caused them. So how do we handle-- how do you handle similar processes?
KILIAN WEINBERGER: So it's definitely. So I guess one thing-- you don't just have loops where you can put in print statements or something, where it actually goes, and then now it does this, now it does this. That's just not how these algorithms work.
One thing they are very good at, though-- and this is something I'm also working on but I think, at this point, is pretty mature, actually-- is giving well-calibrated confidences. So machine learning algorithms now, actually-- they don't just tell you this is a bird. They actually tell you, I believe that this is a bird. And then there's a distribution over the possible birds that it could be, et cetera.
That's certainly very, very helpful, because that's how you can detect outliers. So you can actually incorporate something the algorithm's really, really unsure about. Let's say you have a robotic surgery and the robot goes, I'm pretty sure is the heart, but it could also be your nose. Well, maybe at this point, stop.
So I think these are, I guess, similar to exceptions. So maybe there's some analogy, here, to exceptions and to traditional programming. But it can easily catch unlikely-- could catch situations where the algorithm is unsure, and then deal with those differently. And this is, by the way, actually, how, for example, flight simulators in real planes are now handled, actually.
If you have a machine learning algorithm, that's only legal if, actually, there's an apparatus around it that actually catches and goes over to traditional control theory, in case, actually, there's some setting that actually is too dangerous, et cetera. Yeah?
AUDIENCE: Could debugging also look like-- if we have, for example, these neurons that we expect to be doing something, but for some reason in our training their weights keep getting zeroed out-- so if there's something in our training system that's zeroing out this set of neurons, and we had expected these neurons to be contributing [INAUDIBLE]?
KILIAN WEINBERGER: Yeah, no. Actually, I'm generally-- so some dumb people do this. They look at individual neurons and say, what is this neuronic code, and should it be on or not? I'm really not a big fan of this personally, actually.
I think most of these just made great New York Times articles or something where people would say, oh, here's a cat neuron. But ultimately, this is just not how these spaces work. And then it's quite simple. I can tell you why.
Each neuron-- essentially, what it is one-dimensional space, but you could rotate the space any way you want. When you think a little bit about linear algebra, you could just multiply this with any matrix that rotates the space. And then, actually, this neuron would represent something totally different, but the function would be the same.
So this notion of a neuron representing something is ultimately misguided, I think. So even if you have sets of neurons, I'd be very careful with this kind of-- I mean, it's very tempting. It's very tempting to put interpretations on these neurons. But it's very dangerous, because it doesn't really work that way. Yeah?
AUDIENCE: So suppose you trained your algorithm that worked on two different data sets. For the people who claim that on certain neurons-- say there's a cat neuron-- does it train the same way on two different data sets?
KILIAN WEINBERGER: So if you train two neural networks on two different data sets, typically what you do is initially, you initialize these neurons randomly, and so you will get two different neural networks. They will behave pretty similar, but actually have learned to approximate the same function, basically, in different ways.
If you think about the demo that I showed, there's actually many different ways of connecting these dots. And so basically, depending on your random visualization, they will all be a little different.
AUDIENCE: Maybe one or two more questions?
KILIAN WEINBERGER: Actually, I'm going to put my last slide up here. Yeah?
AUDIENCE: I'm concerned about one aspect of the interpretability, combining two of the examples you gave. If we asked this last neural network why it was seeing an ostrich, a photo would tell us something that's virtually meaningless. And that gives me thought when I think about your [INAUDIBLE].
KILIAN WEINBERGER: So the question is-- there's one example I showed with this lapdog, who basically was classified as an ostrich with high confidence. So the one thing that's very important to remember about this example is that this is an adversarial example. Never in your lifetime would you find an image that actually is so perfectly created for that particular neural network to fool it.
So if you want to make neural networks safe against attacks of some adversary, I think that's a whole different ballgame. And they are very vulnerable to this kind of stuff. But actually, in the general case, I think, these probabilities are actually well-calibrated.
AUDIENCE: Maybe we can thank the speaker for a very hot talk.
[APPLAUSE]
KILIAN WEINBERGER: Thanks.
Recent years have seen a revival of deep neural networks in machine learning. Although this has lead to impressive reduction in error rates in some prominent machine learning tasks, it also raises the concern about interpretability of machine learning algorithms.
Kilian Weinberger, associate professor of computer science, describes the basics of deep learning algorithms and their building blocks. His talk is part of a series on "The Emergence of Intelligent Machines: Challenges and Opportunities."
Weinberger received his Ph.D. from the University of Pennsylvania in Machine Learning under the supervision of Lawrence Saul, and his undergraduate degree in Mathematics and Computer Science from the University of Oxford. During his career he has won several best paper awards at ICML, CVPR, AISTATS and KDD runner-up award. In 2011 he was awarded the Outstanding AAAI Senior Program Chair Award and in 2012 he received an NSF CAREER award. He was elected co-Program Chair for ICML 2016 and for AAAI 2018. Kilian Weinberger's research focuses on machine learning and its applications. In particular, he focuses on learning under resource constraints, metric learning, machine learned web-search ranking, computer vision and deep learning.
