Mark Schneider on Skills and Experiences That Matter (Live from SXSW)

Hey there. Our team had a great time last week at South by Southwest EDU. This week, we wanted to share a few of our sessions with you through the Getting Smart podcast feed. On this episode, we’re bringing you a recording of an intimate fireside chat between Tom Van Der Arc and Mark Schneider. They touch on all things innovation, higher ed, and much more. Let’s listen in.

Hi everybody. Welcome. This afternoon, we are excited to host a small and intimate conversation with Mark Schneider, the director of the Institute for Education Sciences and Tom Van Der Arc, obviously known innovator and leader at Getting Smart. I’m so excited for your conversation. I can’t wait to hear what you have to say. Thanks, y’all. Thank you. Thank you, Aaron. Welcome, Mark. Thank you. Good to see you in Austin. Live in a person.

I think I’ve been to all the South by’s and so it’s been a drag not to be here for the last two years, but it’s great to be back. It is. Except I wish the weather was better. It’s going to be a cold, kind of crummy week for South by. You’re probably staying busy now at IES. Well, so I’m four years into a six year term and my goal is to push as far in trying to modernize the Institute and education research. It’s been challenging the last two years because

of the pandemic has been exceptionally crazy. I mean, I go to work almost every day and their days we have an office that seats 170 people. The most I’ve ever seen was probably a dozen and there were times when I was there all on myself, which was actually the safest place to be. Yeah, I imagine. And now things are starting to pick up again. So the question is how far can we push in what dimensions? Mark, it’s we’re at such an interesting point in history.

And interesting in terms of education. We’re sort of at the end of the information age when we just as a human species began to use computers to calculate and to help us do our work. And we’re at the beginning of this new age where we’re beginning to use machine learning code that learns big data sets. It feels like we’re at the beginning of an era of educating in the age of big data that feels like it’s going to change not only what education looks like, but it’s certainly

going to change how we research, how we learn about learning. Does it feel like we’re at the beginning of something new? So, IAS is 20 years old this year. November will be our official 20th birthday anniversary. And I’ve commissioned three reports from the National Academies and all as part of the 20th year celebration. So the first one will be out on the 24th and that’s about the National Assessment of Education progress. So NAIT has many purposes, but and it’s a

reasonable assessment. It’s extremely expensive. It’s not too long and it’s a sample which is beautiful. It’s a sample and no person, no one student sits for the entire framework. So it’s a matrix sampling, but the question is it’s unbelievably expensive and is it really up to date? Does it, for example, we’ve gone through a multiple year effort to get something that everybody else uses, which is automated scoring of reading passages done. And this is going to take years and cost

millions of dollars when we ran a competition and we gave out a $50,000 prize. We got six winners, all of whom came within striking distance of a $6 million process for hand grading, human grading scores. So this is a government agency. This is a huge test. Everybody’s afraid to break trend. Everybody’s afraid to innovate. And as a result, a lot of the kinds of things you’re talking about, AI, automated scoring, automated question generation, adaptive testing, all the

things that the world is getting used to and thinking about and all of which are built on much better AI and much better algorithms are we are struggling to try to put them into NAIT. The second report is about the research center. We have two research centers, one called the National Center for Education Research, NCEER, and the other one, the National Center for Special Education Research. So together we spend, between the two, we spend about $200 million a year on

education research between the two centers. So this turns out to be also an incredibly difficult problem to modernize them because there’s so many things that go into that equation that need to be changed. So for example, peer review is fundamental to our business, right? And it’s one of the most stringent, reliable things that we must do to ensure quality. And what happens, is that almost all the peer reviewers are from R1 research universities and they are

status quo oriented and they like the way the research has been done for the last 20 years because most of them have won in the last 20 years. And so we, so trying to broaden that pool to get AI researchers in, for example, is extremely difficult, especially when you want innovators and you want people who have their own companies and people who are on the cutting edge of innovation. Well, guess what? They don’t have five days to serve on one of our panels, right?

So three days of prep and two days, usually two days of actual meetings. The other people that we can’t get are state officials, which would ground us in the reality of the work. So we have that problem. And then the third study, which will be out in about a month, is for the National Center of Education Statistics, which you would think, you would think that we would be doing big data. You would think that we are doing, you know, new data collections, you would think, etc., etc., etc.,

and you would be wrong. So the question is, how do we modernize our data collections? So we are, we’ve just gotten, we reorganized the front office to have a center for data science, that we call the Center for Excellence in Education Data Science. So we’re going to start hiring data scientists to, as an institute-wide innovation, to actually, you know, to bring us up to, I don’t know, 2018. This summer you’re, I think you’re training some new fellows on

research methods. Are there innovations in research methods that you’re excited about? Yes. So we, so we’re going to release an RFA, and I’m making this public for the first time, and my commissioner, who’s actually going to be Liz Albro, who’s responsible for, going to be responsible for making this RFA and writing this RFA. So hi, Liz, we’re going to do this training program in Big Data, right? So we have to, we have to do this, we have to do this. And, you know,

so we’ve been spending a lot of time and a lot of money on digital learning platforms. So we have an XPRIZE going on, that’s going pretty well, Erin’s on one of the judging panels, Erin Mote, and and we’re pushing really hard, really fast, and we’re running into some time constraints, but, but we, so it’s going to be an XPRIZE and digital learning platforms. We’ve just funded the $10 million network on, on digital learning platforms that include the, you know, many of the digital

promise, Neil Heffernan, etc. So we are in the process of funding platforms that have 100,000 or more users, and the, they’re generating a lot of information, and what we have to do is train the next generation of researchers who can now work with those data, who could understand the unstructured nature of a lot of those data, and how to turn them into usable, and, and usable research products. So that is, that, that’s fundamentally important. So the, the other thing actually is that the,

the standard business model for education research is an RCT, the gold standard. It takes five years, five million dollars, and it fails. So the 3Fs, five million, five years, fail. So we, the model that we’ve used for way too long is we do one investment, five years, five million dollars, fail. And we do another one. We do serially, right? Well, I mean, could you imagine if we had done that with the operation warp speed, right? If we’d given money to Novartis, and we said you have five

years, five million dollars, or maybe 50 million dollars, but, but, and it failed. And oh, that could work. And then we’ll do another one, and then we’ll do another one, and then we’ll do another one. And, you know, it’s not the way to do business. So what we’re trying to do in part with the platforms, especially with the X Prize, is to do fast experiments, fail fast, because most things are going to fail. The few things that work, the prize is going to be given to the platform that

could deliver experiments and replications. So the education sciences does not have a replication crisis like so many other sciences, because we don’t replicate anything. But if you don’t replicate, if you don’t replicate, you never drive down to the point that we need, which is to learn what works for whom, under what conditions. And the only way you could do that is to take the few things that work, replicate, don’t replicate once, replicate five times in different, in geographies or different

demographies, and then of those five, what worked? Replicate. And the process is that’s the only way we’re going to get down to identifying what works for whom, under what conditions. It looks, some things can’t, like a year’s learning gain has to be, takes a year to measure. But there are things that we could enter, there are things that we can innovate, there are things that we could experiment with at a short term, and we could run a five-week experiment and find out if it works. And if it

doesn’t work, boom, boom, boom, fail fast, move on. If I squint, I can see in the second half of this decade that most learners and employees have a, have a portable electronic record that they have some ability to, to, to, to curate and manage. Is that going to happen? Is that going to make it much easier to both personalize learning in real time, but also study what works longer term? So I believe that, I mean, you’re talking about something like blockchain, obviously.

So we have not invested in, sufficiently in blockchain. We have looked at some systems of blockchain. The Dallas County Community College is one of the best systems that I know, we’ve talked to them several times. We’re really excited about what’s happening there. It’s all part of the Dallas County Promise system. They’ve been working with Greenlight credentials and every high school student in Dallas County has a learner record that they’re able to manage and

permission out to college’s employers and scholarship institutions feels like a picture of where things are headed. Yes, and unfortunately, Joe May is retiring at the end of this year, but I mean, one of, one of my favorite people and has such insights, but you know, so here’s the question. Where else, who else is taking that system, which is now an existing system that we know works? Where else is it? It’s been slow. It’s been, it’s been slow. Even though Commissioner Mike Murow

made it available to every high school student in Texas for a year, he was an Essard funds, I think because it was a one year deal, didn’t quite as quickly catch on as I thought. I thought initially everybody in Texas would have it because it’s the benefits are so obvious, but they’re obvious to us. Yeah, right. So it’s always the question of getting something adopted. And again, I mean, every time I’ve talked to Joe, every time I’ve seen, we talked to Greenlight,

I mean, all this stuff is like amazed. Yeah, right. But these things are sticky. And we’re talking about the two sided market where you both have to train senders to send new information and the new receivers of colleges and employers. Scholarship, you have to train them on how to receive and use the information. So kickstarting this two sided market is challenging. It’s incredibly, incredibly difficult. So I want to talk about another two sided problem, something that we’ve been only for

a couple of months. I mean, I’m sorry, the problem we’ve known for a long time. So there are 14,000 school districts in the United States. So IAS is one of the biggest funders of education research in the country and maybe arguably in the world. So of those 14,000 school districts, I don’t even think 5% of them have ever been funded by us or ever been had rigorous research. There are 20,000 researchers that go to AERA. And we probably funded and this is not the fault of

IAS. This is what science looks like, right? We probably funded 200 of them, right? And our money keeps going to the same institutions and the same people. So we have 14,000 school districts, New York, Chicago, LA, right? You can name the five, six, eight school districts that get almost all the work and of the 20,000 researchers, right? Probably we funded a couple hundred. So how do we democratize that process? That to me is also a two sided problem. So I’ve talked to a fellow,

an organization, a company that started at an MIT, which is a market design company. Josh Ingers is one of the founders of it. And we’ve been talking about how to replicate, duplicate the match, the match, not match.com, but the match that matches, that matches medical students and hospitals. So hospitals need people for interns, people finishing medical school, need internships. So there’s something like 37,000 matches that take place.

The algorithm goes back to 1982. They create the algorithm that matches across these 37,000 students and internship possibilities. So the algorithm’s not the problem. The problem is exactly the same as we just talked about. It’s a two sided problem. The incentives don’t align at the current time. They just don’t align. Why would a school district contribute, say, oh, yeah, put me into this side of the match? These are my problems. And I’m looking for researchers who

could do X, Y, and Z. And here are 20,000 researchers, most of whom maybe couldn’t answer those problems. But among those 20,000, even if they have 5,000 of researchers who have skills and can’t get through our rigorous process, but if we could figure out how to get those 5,000 researchers, I’m making these numbers up, but you can see the concept, and 500, a thousand school districts, if we can get them into this match, we can match them and we could democratize, open

the education research process to more schools, more districts, more researchers. We could. There’s sort of an unmentionable problem in America that we inherited this bizarre, decentralized system of local control. And we have two or three times more school districts than anyone would ever in their right mind design. So if there’s a way that we could sort of right size districts and then work on the match. That’s above my pay grade. That’s a very difficult

problem. Let me quickly want to dive into another problem that you spend a lot of time thinking about, and that’s higher education. Before you went to IES, you spent 10 years thinking about college. Do college degrees still matter? And if they do, how would we know? So most college degrees don’t matter, right? I mean, increasingly, and I don’t think we came as far as I hope we would in terms of skills taking the place of degrees. So how would we know? Well, you know, so I work with

eight or nine states, and we had really good detailed information about the courses students, students took, the degrees, and then their earnings 10 years afterwards. And it was the pattern was so consistent, state after state. So the first state that we released the results from was Tennessee. And it turned out that on average, people getting two year technical degrees from community colleges, 10 years after completion, we’re making more than the average bachelor’s

degrees do. And I swear to God, like, we looked at this and we talented it and there has to be something wrong, right? I was a university faculty member for almost my entire life and like, wait a second, you’re telling me that a four year degree in political science subject I taught is less valuable than someone who gets a skill from a community college and out earns a bachelor’s degree in political science history fine arts. And and and we but the data we look at data and the

answer is yes. And then the next state came in, I think Virginia, St. Pattern, and then Texas, Florida, one after the other. So the lesson was technical degrees given by community colleges actually matter. And then I started working with Burning Glass, and to work to identify the the skills that were the the distinguishing characteristic made someone employable, and gave them that shot into really good wages. You know, and some of it was pretty simple, like Salesforce. I mean,

Salesforce on average was was worth like $20,000 a year in salaries. Computer computer data data science was like amazing, right, in terms of the the bump. And some of the some of the data science was not fancy stuff that we talk about. But like, could you do an Excel spreadsheet? Can you analyze an Excel spreadsheet? That’s worth money, right? So we were so I was I was hoping that that message would keep going. And I think part of what happened was that

the economy until the COVID was relatively strong, and everybody with a sociology degree or fine arts degree was getting a job. So the pressure to move from degrees to to skills, I think was was the pressure was a little bit. So the one thing that we’re trying to do right now. So IAS runs IPEDS, the integrated post secondary education data system. So we have just put in a package to OMB to and Joe was instrumental in this domain, was to start collecting data on non credit

activities. So many, many, many students, we don’t even know, but it could be as many as half of the enrollments in in community colleges are non credit activities. And most of those not credit activities are not underwater basket weaving, which everybody likes to cite. I’ve never seen a course on the water. But many of them, they may be not credit, they may roll into a certificate. Yeah, or a credential. Exactly the point. Exactly the point. And you know, so someone might take three

courses in non credit and auto body repair, right? Well, so how many of those kinds of credentials are being produced? And what’s the ROI on Texas could answer that question, because Joe may could answer that question, because he has records on non credit activity, he has credits, he has the information, especially since the Texas higher ed coordinating board collects data on wages. So we could actually and I’ve been talking to him and I hope we can get this launched

to do a couple of experiments on like identifying the high value non credit activities. I spotted what I think is a trend over the last two years towards skill based hiring. It feels like major employers got much smarter about the skills required for specific jobs and about how to identify them. Right. That feels like an important change. And those pathways to produce those skills often include credentials. It feels like degrees just got less valuable and

that new identifications of skills became more valuable. I feel like an important signal for higher education. I’m not sure. They’re going to need it. Yeah, right. Or they’re going to tend to it. Right. I mean, their business model is turning out, you know, degrees. Right. I think this is right. I’m right. And I think that as companies are getting much more sophisticated and testing for these skills, we may see some product like this and some change. But you know, I think

so when I was doing college measures of 10 years in between. So it was amazing when you showed these data to parents, right, who at kids in high school or early in college. And you’d start off and say, what do you want? I want my daughter to have a bachelor’s degree. That’s what I want. And then you show them the data. And it was like, their minds just said, you mean like a degree is not going to get them into the low class. That if they get the skills based credential, they will be

further along than if they get a bachelor’s degree in something that has no market value. And the conversation that just that data encouraged was actually amazing to watch. And we saw this with also students who we would sit down with our tool and you know, launch my career tool and they would play with this and they would say, you mean if I’m major in theater arts, I’m never going to have the lifestyle that I want. But if I major in business and minor in theater arts, I could

actually run a production company. Yeah, that’s right. Yeah, that signal is so important. I think guidance is more important than ever because it matters what degree, it matters what college, it matters what degree. And we have to make sure people don’t pay too much, particularly for degrees, not likely to be pathways to high wage employment. Right. So all the work we did was that there were a couple hundred schools in the United States that have a brand, right? So when we say get to

the right school, this is one of the traps that we fall into because I wanted my daughters to go to the Ivy League, they weren’t smart enough, so they both went to Wesleyan, which is no slouch school. But we think that when my daughters didn’t get into Princeton, I wanted to hit them. But you know what I mean, I was like, there’s something wrong with me because you couldn’t get into Princeton. So there’s some kind of prestige in our world. But most students go to

the local high, local college. And they may have regional reputations, but they don’t have the same kind of national market that students coming out of state university or don’t have that kind of market. And the advantage of the data that we have now is that we can drive it down to county level, for example. Right. So we could say that in your county, these are the skills that are in high demand. These are the jobs that are in high demand. And state labor departments, they do this, but

here’s the problem. And this is, I think, endemic to everything we’re talking about is we have done a terrible job in communicating what we know to teachers, to parents, to students. And we’ve been working really hard on this. And it is the hardest thing that I think all of us face. It’s one of the hardest things. I probably said five different things that were the hardest thing. But this is one of the hardest things that we face. It’s the last mile problem. And we’ve done a terrible job

on that last mile. So we accumulate information, we accumulate insights, we accumulate all kinds of things. And it’s an interesting time to be leading research at the country’s research institute. It’s been a fun ride. Mark Schneider heads up IES, America’s Education Research Center. Mark, thanks for being with us. Okay, my pleasure. Thanks for tuning into the Getting Smart podcast today. We want this podcast to be actionable

and insightful and a great way to learn about what’s next in learning. In order to stay on the cutting edge, we need people in the field to tell us what they’re hearing, what they’re wanting, and what they’re needing to learn more about. Got a topic or a guest in mind? Send your recommendations to me, Mason at GettingSmart.com. And if you like what you’re hearing, don’t forget to leave a review in Apple Podcasts or subscribe wherever you listen.

Feel free to share the podcast on social media using the hashtag GSPodcasts. Thanks so much.