Jennifer Neville是Microsoft的合作伙伴研究经理，他围绕理解和推进AI来进行现实世界的应用，她的早期职业生涯与她所研究的人类-AI互动一样
在与首席应用科学家查德·阿塔拉(Chad Atalla)的谈话中，她探索了评价在推动当今AI系统满足用户需要的性能界限以及模型测试超越传统基准测试时出现的"意外故障"中所起的作用。 Neville还分享了与当前AI系统合作的实用指导，并讨论了当结果无视预期时为何要仔细研究数据事项，以及几十年来AI的进步让她学会了预测接下来会发生什么。
从意想不到的职业轨迹到AI互动与学习的前沿
订阅 Microsoft 研究播客：
语录
[音乐]
有几次我觉得我在旋转我的车轮事情并不成功。 我会被吓到的然后，我们到了一个点，我们确实学到了一些东西，它只是，情感上的刺激，这才是真正吸引我的原因。 能够理解别人还不明白的事情。..
当然。..
因为它就在我们对事物的了解的前沿。
标准介绍：这是Microsoft Research Podcast，在那里，Microsoft 研究人员 — — 通过基础科技研究推动进步 — — 探索在计算和AI中谁、如何以及下一步是什么。
[音乐ENDS]
你好，欢迎。 我是Microsoft研究的应用科学家Chad Atalla。
以及塞缪尔·康特(Samuel D.
她的研究研究了机器学习和AI的交互领域和结构化数据，考察了AI系统所训练的数据点如何影响他们的行为，以及这如何与用户实际想要的一致。
在这项研究中，她发表了130多篇论文，引用了10 000多篇，并获得了国家科学基金会CAREER奖、AI列表中IEEE“10 to Watch”的插曲(在新标签中打开)以及数据采矿国际会议(在新标签中打开)和国际学习代表大会的最佳论文奖等荣誉。
Jen的作品让我印象最深的是她的远见和能力
珍，谢谢你加入我
谢谢你邀请我
赞极了嗯，我很想知道你是如何来到这里的。 让我们回到过去。 当我还是个孩子的时候，我想成为一个天体物理学家，但是，当然，我在这里有一个计算机科学背景。 你什么时候知道你想进入计算机科学？
这是一个很好的问题。 我。.. 当我长大的时候，我想做任何事情除了计算机科学，因为这就是我爸在，我想做任何事情，所以当我上大学的时候，我先是主修数学，然后是物理，但我真的无法与那些主修。
所以我退学了一段时间，当我又回到大学时，我主修认知科学，这对我来说也太粗糙了。 并没有足够的数学。 那时，如果有人告诉我，“ AI是你们应该做的事，因为它把认知科学与数学和计算思维结合起来了，”我想我会省下很多时间，[LAUGHTER]，但这并没有发生。
所以我工作了一段时间。 然后当我回到学校时，我决定主修计算机科学，因为我想做的是思考数据的工作和处理数据。 这时我找到了AI。 只是巧合罢了我甚至没有意识到这是计算机科学的一部分。
赞极了进入计算机科学并研究数据或人工智能是一回事，那么，什么样的问题或大想法激发了你的研究驱动力？
这其实也是个有趣的故事。 我进入研究是因为我在计算机科学学位的荣誉课程，作为荣誉课程的一部分，你必须做一个研究项目[LAUGHTER]。
这是强制性的，是的。
这是强制性的。 所以当我和教授们谈论研究计划应该是什么时，他们其实是称，“嗯，你得决定你想研究什么主题。 ”
那时，我对数据感兴趣，对AI感兴趣，我思考了很多。 我读了很多书我决定要调查的是如何在网络数据上进行数据挖掘数据是互相连接的所以。.. 当我决定这个问题是我想研究的时我被指向了一位特定的教职员
不错。
刚开始在这个新生领域工作的人在当时被称为统计关系学习。 我们做了我的项目，我在车间发表了一篇论文，我有点迷上了。
好吧，没错。
所以我没有打算继续念研究生
对
让我想继续念研究生
不错。 你觉得你上钩了？做研究的刺激感吗？这是会议的环境吗？学术出版是什么样子？
这确实是研究的刺激和过程。 这是一个很长的项目。 有几次我觉得自己在旋转我的车轮事情并不成功。 我会被吓到的但是，我的顾问会像，你知道，支持和肯定地说，“不，继续前进。 我们在学习一些东西。 然后，然后，我们到了一个点，我们确实学到了一些东西，它只是情感上的刺激，这就是真正吸引我的原因。 能够理解别人还不明白的事情。..
当然
..... [笑声] 因此，它让我在研究这么长，同样的，同样的。.. 追寻同样的感觉。
抓到他们了好吧，爱它。 是啊，你加入了Microsoft 2021年从学术界。 事实上，你还是个教授，你教书和提供建议已经20年了。 是什么促使你转向部分职业生活成为工业研究，你会如何描述工业研究和学术研究的区别？
我的研究从理论到应用都跨越光谱我上任后第一次休假时很多同事。.. 都在同一时间。.. 我想到我想走的方向，要么更多理论，要么更多应用
所以我去了伯克利的西蒙斯学院做我的安息日，这是非常理论性的。 这是一个伟大的经验，但它是一种无缝过渡回学术界。 在我第二次安息时，我走了另一条路，那就是来到MSR[Microsoft研究]这里做一个安息日。 当然，在理论上的算法是怎样的，某种意义上的，在橡胶达到的路面，所以我现在仍然在这里。
抓到他们了
所以，我认为学术界和工业界研究的区别取决于。.. 是否真的受到目标的影响，目标在哪里，目标在哪里。 所以我认为，从根本上来说，这感觉非常相似，在学术界和工业界，在工业界，你在真实数据上应用它的能力与学术界大不相同，在学术界，我认为你最后提出的问题更抽象一些，涵盖了很多不同领域的应用。 国家科学基金会(NSF ) 等。
但是在工业中，简单一点，你知道的，手弄脏，在真正的系统里做。 然后，你，那种，目标是产品和公司的利益。 作为。.. 这种变化，可能你问或调查的问题类型。 所以，我觉得这其实是伟大的。..
对
在两地，对不对？比如，两者之间有很多协同效应，我认为有很多机会去工业，回到学术界，或者开始工业，进入学术界。 但现在，如果你在做人工智能系统的工作，我认为工业才是真正的地方。
抓到他们了对给那些决定如何在自己的职业生涯中做出这一决定的人提供有益的建议。
我在Microsoft工作了六年多
我最初看到了你的工作，并意识到你的工作，因为你的部分工作为其中一些问题提出了一些解决方案，并有助于指出这些问题。 我立刻的反应是，我们有像你这样的好人来研究这些东西，你会为我们照顾它。 我非常欣赏你如何平衡对人工智能的评价，并提供解决方案。
但我明白这只是你更广泛的研究日程的一小段。 那么，你将如何描述你所领导的团队目前的研究章程？
我们的团队被称为AI互动与学习团队。 我们真正关注的是试图在现实的工作环境中推动这些人工智能系统的行为前沿因此，这需要研究的是当前系统性能的界限在哪里，用户如何在实际中体验到真正的工作流程，然后我们如何改进和推动模型的性能，以完成用户正在完成的这类复杂任务。
我们注重评价的原因是，我们发现，ML和AI人评价的标准方式是通过基准测试，这些方法对于人们如何实际使用它们相当简单。 因此我们发现，你真正需要做的第一件事就是问，我们想要从这些系统中得到什么？并且设计实用的评价将这些系统置于这种环境中。 因此，这意味着诸如多回合行为，合作环境，长视野任务等，用户正在做的工作。
然后，一旦我们可以看到这些评估中发生的性能差距或问题在哪里[是]，那么这也让我们知道我们需要改进这些系统在哪里、在理论上或算法上。
因此我们从评估开始，但最终，我们正在努力开发更好的算法，模型，估计方法。
当然
.推动模型的性能。
(笑声) (掌声) (掌声) (掌声) (掌声) (掌声) (掌声)
本文由 AI图鉴自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Jennifer Neville is a partner research manager at Microsoft who’s built a career around understanding and advancing AI for real-world use, and much like the human-AI interactions she’s been studying, her early-career path was multiturn: math, then physics; cognitive science, then work; and finally computer science—despite her best efforts to avoid the field.
In this conversation with Principal Applied Scientist Chad Atalla, she explores the role evaluation plays in pushing the performance boundaries of today’s AI systems to meet user needs and the “surprising failures” that emerge when models are tested beyond traditional benchmarks. Neville also shares practical guidance for working with current AI systems and discusses why looking closely at data matters when results defy expectations, and what decades of AI progress have taught her about predicting what comes next.
From an unexpected career trajectory to the frontier of AI interaction and learning, this episode asks a larger question: what can we learn when the path—whether human or artificial—doesn’t unfold the way we expect?
Subscribe to the Microsoft Research Podcast:
Transcript
[MUSIC]
JENNIFER NEVILLE: There were times that I felt like I was spinning my wheels. Things weren’t working. I would get, kind of, dejected. And then we’d get to a point where we did learn something, and it was just the, the emotional thrill of that was, that was really what hooked me. The being able to, kind of, understand something that no one else understood yet …
CHAD ATALLA: Sure …
NEVILLE: … because it’s just at the frontier of what we know about things.
STANDARD INTRODUCTION: This is the Microsoft Research Podcast, where Microsoft researchers—driving advancement through fundamental science and technology research—explore the who, how, and what’s next in computing and AI.
[MUSIC ENDS]
CHAD ATALLA: Hello, and welcome. I’m Chad Atalla, an applied scientist here at Microsoft Research.
Today, I’m joined by Jennifer Neville, a partner research manager at Microsoft Research and the Samuel D. Conte chair professor of computer science and statistics at Purdue.
Her research examines machine learning and AI for interactive domains and structured data, looking at how the data points that AI systems are trained on affect their behaviors and how that aligns with what users actually want.
Across that research, she has published more than 130 papers with over 10,000 citations and received honors such as a National Science Foundation CAREER Award, a spot on IEEE’s “10 to Watch” in AI list (opens in new tab), and best paper awards from the International Conference on Data Mining (opens in new tab) and the International Conference on Learning Representations.
What amazes me most about Jen’s work is her foresight and ability to have a through line that stays consistent as the field of AI develops, and I’m excited to hear more about how she got to where she is today.
Jen, thank you for joining me.
JENNIFER NEVILLE: Thanks for having me.
ATALLA: Awesome. Well, I would love to learn about how you got here today. And let’s rewind way back. When I was a kid, I wanted to be an astrophysicist, but of course, here I am with a computer science background. When did you know that you wanted to go into computer science?
NEVILLE: That’s a good question. I … when I was growing up, I wanted to do anything but computer science because that’s what my dad was in, and I wanted to do anything but what my dad did. So when I went to college, first I majored in math and then in physics, but I really just couldn’t vibe with those majors.
So I dropped out of college for a while, and when I went back to college again, I majored in cognitive science instead, which also was too squishy for me. Didn’t have enough math in it. At that point, if somebody had told me, “AI is the thing you should be doing because it combines the cognitive science with the math and computational thinking,” I think I would have saved myself a lot of time, [LAUGHTER] but that didn’t happen.
And so I worked for a while. And then when I went back to school, I decided to major in computer science because what I wanted to do was think about working with data and dealing with data. And that’s when I found AI. Just by happenstance. I didn’t even really realize that it was part of computer science.
ATALLA: Awesome. Well, it’s one thing to go into computer science and want to work on data or AI and another thing to want to be involved in research on that front. So what sorts of questions or big ideas sparked your research drive?
NEVILLE: Yeah, that’s actually an interesting story, as well. I only got into research because I was in the honors program in my computer science degree, and as part of the honors program, you have to do a research project [LAUGHTER].
ATALLA: It’s mandatory, yeah.
NEVILLE: It’s mandatory. And so when I talked to professors about what the research project should be, they actually said, “Well, you have to decide what topic you want to work on.”
And at that point, I was interested in data and I was interested in AI, and I thought a lot. I did a lot of reading. And what I decided that I wanted to investigate was how to do data mining on web data that was interconnected, and so that … when I decided that was the question I wanted to work on, I got pointed to a particular faculty member …
ATALLA: Nice.
NEVILLE: … who had just started working in this nascent field at the time that was called statistical relational learning. And we did my project in that, I published a paper at a workshop, and I was, kind of, hooked.
ATALLA: OK, yeah.
NEVILLE: So I hadn’t planned to go on to grad school, but that experience …
ATALLA: Yeah.
NEVILLE: … made me want to go on to grad school and continue.
ATALLA: Nice. What part of it do you think hooked you? Was it, like, the thrill of doing the research? Was it the environment of the conference and what academic publishing looks like?
NEVILLE: It was really the thrill and the process of doing research. So it was a long project. There were times that I felt like I was spinning my wheels. Things weren’t working. I would get, kind of, dejected. But my adviser would be, kind of, like, you know, supportive and positive, saying, “No, keep going. We’re learning something.” And then, and then we’d get to a point where we did learn something, and it was just the emotional thrill of that was, that was really what hooked me. The being able to, kind of, understand something that no one else understood yet …
ATALLA: Sure.
NEVILLE: … because it’s just at the frontier of what we know about things was, uh, it’s just, it’s like a drug almost, right? [LAUGHTER] So it’s kept me in research for this long, that same, that same … chasing after that same feeling.
ATALLA: Gotcha. Well, love it. Yeah, you joined Microsoft in 2021 from academia. And in fact, you’re still a professor, and you’ve been teaching and advising for 20 years. What motivated the shift to part of your professional life being research in industry, and how would you characterize the difference between industry research and academia research?
NEVILLE: My research, kind of, spans the spectrum from theory to application. When I did my first sabbatical after I got tenure, a lot of colleagues that I had at the … that were at the same point in their career went off into industry labs for their sabbatical and never came back to Purdue. I thought about which direction I’d want to go, either more theory or more applied, and I thought at that point in my career, it’d be better to explore the theory side because then I might actually go back to Purdue.
And so I went to the Simons Institute in Berkeley for my sabbatical, and that was very theoretical. It was a great experience, but it was kind of seamless to transition back to academia. On my second sabbatical, I went the other way, which was to come to MSR [Microsoft Research] and do a sabbatical here. And of course, being able to see how algorithms in theory, kind of, hit the—where the rubber hits the road with respect to how they behave in practice and real systems and with real users and real data, that’s hard to turn back from. And so that’s now why I’m still here.
ATALLA: Gotcha.
NEVILLE: So I think the difference between research in academia and industry depends on … really is affected by the target of where, where you’re aiming the research. And so I think fundamentally, it feels very similar, the questions you would ask in academia and industry, but in industry, your ability to apply it at scale in real systems on real data is really very different from academia, and in academia, I think you end up asking questions that are more abstractions that cover applications across a lot of different domains at once. And that’s how you get funding from places like DARPA [Defense Advanced Research Projects Agency] and NSF [National Science Foundation].
But in industry, it’s a little easier to just, you know, actually get your hands dirty and, and do it in the real systems. And then your, kind of, target is the products and the company’s interests. And so as … that sort of changes, maybe the types of questions that you would ask or investigate. So I think it’s actually great to be …
ATALLA: Yeah.
NEVILLE: … in both places, right? Like, there’s lots of synergies across the two, and I think there’s lots of opportunities to go to industry and come back to academia or to go start in industry and go to academia. But right now, if you’re doing work on AI systems, I think that industry is really the place to be.
ATALLA: Gotcha. Yeah. Perhaps useful advice to folks who are deciding which way to take that decision in their career right now.
I’ve been here at Microsoft for a little over six years and have been working on AI evaluation for much of that time, and there are, of course, a number of daunting fundamental questions and challenges in the space.
And I initially came across your work and became aware of your work because part of your work proposes some solutions to some of these problems and helps to point out these problems. And my immediate reaction was relief that we have someone wonderful like you working on these things, and you’re going to take care of it for us. And I really appreciate how you balance critiquing AI evaluation and also providing solutions along the way.
But I understand that that’s just a small sliver of your broader research agenda. And so how would you describe the current research charter for the team that you’re leading here?
NEVILLE: Our team is called the AI Interaction and Learning team. What we really focus on is trying to push the frontier of behavior of these AI systems in realistic work environments. And so what that entails is studying where’s the boundary of the performance of the current systems, how do users experience that in practice with real workflows, and then how do we improve that and push the performance of the models for these kind of tasks—complex tasks—that users are working on.
The reason we focus on evaluation is that we find that the standard way that ML and AI people evaluate are through benchmarks that are fairly simple with respect to how people would actually use them in practice. And so we find that the first thing that you really need to do is ask, what do we want out of these systems? And design practical evaluations to put the systems in those kind of environments. So that means things like multiturn behavior, collaborative environments, long-horizon tasks that users are doing.
And then once we can see where the performance gaps or problems that happen in those evaluations [are], then that also gives us the knowledge of where, sort of, theoretically or algorithmically do we need to improve these systems.
So we start with evaluation, but ultimately, what we’re trying to do is develop better algorithms, models, estimation methods …
ATALLA: Sure.
NEVILLE: … to push the performance of the models.
ATALLA: Yeah, it’s a gateway to und
Auto-fetched and translated by AI Atlas. All rights belong to the original author.