WEBVTT

NOTE
Podcast: Data Stories
Episode: Data Conversations with Vidya Setlur
Publishing Date: 2022-12-12T15:07:01+01:00
Podcast URL: https://datastori.es
Episode URL: https://datastori.es/169-data-conversations-with-vidya-setlur/

00:00:00.080 --> 00:00:07.492
People want additional information in addition to something that specifically answers or satisfies their intent.

00:00:27.745 --> 00:00:28.366
Hi, everyone.

00:00:28.426 --> 00:00:30.528
Welcome to a new episode of Data Stories.

00:00:30.930 --> 00:00:36.332
My name is Moritz Stefaner. I'm an independent designer of data visualizations. In fact, I work

00:00:36.372 --> 00:00:41.373
as a self-employed truth and beauty operator out of my office here in the beautiful countryside

00:00:41.433 --> 00:00:47.054
in the north of Germany. And I am Enrico Bertini. I am an associate professor at Northeastern

00:00:47.114 --> 00:00:53.116
University in Boston, where I do research and teach data visualization. Exactly. And on this

00:00:53.156 --> 00:00:59.298
podcast together, we talk about data visualization, data analysis, and generally the role data plays

00:00:59.338 --> 00:00:59.938
in our lives.

00:01:00.358 --> 00:01:00.818
Thank you.

00:01:00.798 --> 00:01:03.020
Usually we do that with a guest we invite on the show.

00:01:03.341 --> 00:01:06.883
Yes, but before we start with our guest, a quick note.

00:01:07.524 --> 00:01:11.427
Our podcast is listener supported, so there's no ads.

00:01:11.467 --> 00:01:17.532
If you enjoy the show, please consider supporting us with recurring payments on patreon.com

00:01:17.732 --> 00:01:18.893
slash datastories.

00:01:19.714 --> 00:01:23.397
Or if you prefer, you can also send one-time donations on PayPal.

00:01:24.159 --> 00:01:30.184
by following the link on to paypal.me slash data stories.

00:01:30.724 --> 00:01:31.605
Yeah, in the last few days,

00:01:31.825 --> 00:01:34.166
we've just seen a few new donations coming in.

00:01:34.306 --> 00:01:36.368
That's always a great feeling and always wonderful,

00:01:36.468 --> 00:01:37.369
much appreciated,

00:01:37.789 --> 00:01:40.291
especially if you send us a little funny note along with it.

00:01:40.431 --> 00:01:41.452
It always makes us smile.

00:01:41.532 --> 00:01:43.033
Yes, thank you, thank you.

00:01:43.053 --> 00:01:44.854
Yeah, and also if you can afford it

00:01:44.954 --> 00:01:46.735
and do the recurring thing on Patreon,

00:01:46.755 --> 00:01:48.116
this definitely keeps us going

00:01:48.216 --> 00:01:51.679
and it's again much appreciated.

00:01:52.881 --> 00:02:00.086
So let's get started. We have a cool topic today. We'll talk about language and data visualization

00:02:00.466 --> 00:02:07.610
and all that surrounds that. But before we dive into that, quick note on the Informationist

00:02:07.690 --> 00:02:12.473
Beautiful Awards. So some of you might remember, we had Amanda McCulloch from the Data Visualization

00:02:12.493 --> 00:02:15.916
in Society on the show a few weeks, months, can't remember, ago.

00:02:17.178 --> 00:02:22.142
And we mentioned that the awards are being rebooted, they're a bit like the mini Oscars

00:02:22.322 --> 00:02:29.167
of data visualization, and really try to celebrate all the diversity and variety and excellence

00:02:29.247 --> 00:02:29.807
in the field.

00:02:29.887 --> 00:02:31.469
And now the winners are announced.

00:02:31.649 --> 00:02:33.530
We didn't win anything, Enrico, right?

00:02:33.990 --> 00:02:35.011
Yeah.

00:02:35.091 --> 00:02:37.873
But a lot of other great people, which is awesome, too.

00:02:39.636 --> 00:02:42.418
And yeah, on the website you can see all the winners.

00:02:43.299 --> 00:02:46.962
Shout out to our podcast colleague Ali Turvan, who won Community Leader.

00:02:47.182 --> 00:02:48.002
Much deserved.

00:02:48.042 --> 00:02:48.663
Congratulations.

00:02:49.323 --> 00:02:50.464
Lots of other great people.

00:02:50.644 --> 00:02:52.906
Yeah, it's great to see this event is back.

00:02:53.146 --> 00:02:53.887
I'm here.

00:02:53.907 --> 00:02:54.207
Yeah.

00:02:54.347 --> 00:02:57.089
It's nice to have a celebration, you know?

00:03:04.956 --> 00:03:11.481
Yeah, but let's go to the main topic. Our guest today is Vidya Settler. Hi,

00:03:11.501 --> 00:03:15.984
Vidya. Thanks for joining us. Hi, Moritz and Enrico. Thanks for having me.

00:03:16.484 --> 00:03:22.149
Yeah, welcome on the show. And thanks so much for joining us. Can you tell us a bit about you?

00:03:22.669 --> 00:03:26.512
What's your background? What are you excited about? What are you currently working on?

00:03:27.394 --> 00:03:34.699
Sure. So I'm Vidya Settler. I head Tableau Research. I've been with Tableau for 10 years.

00:03:35.740 --> 00:03:41.224
I got a PhD from Northwestern. My background is in natural language processing and computer

00:03:41.264 --> 00:03:46.928
graphics. So I'm generally interested in the problem space of understanding the semantics

00:03:47.008 --> 00:03:53.193
of data and how that can help inform meaningful visual depiction of information. So Tableau has

00:03:53.233 --> 00:03:53.593
has been.

00:03:53.835 --> 00:03:56.976
the perfect place for leveraging that skill set.

00:03:57.997 --> 00:03:59.718
Before I joined Tableau,

00:03:59.858 --> 00:04:02.500
I was at Nokia Research for seven years.

00:04:03.540 --> 00:04:05.761
I manage a wonderful team

00:04:05.821 --> 00:04:07.783
of interdisciplinary research scientists

00:04:07.863 --> 00:04:11.425
who work on problems in the space of applied ML,

00:04:11.685 --> 00:04:14.727
multimodal interfaces, HCI techniques,

00:04:15.067 --> 00:04:16.708
augmented reality, and so forth.

00:04:18.028 --> 00:04:20.010
Already back in the day, augmented reality.

00:04:20.690 --> 00:04:22.571
I know, it's all coming back full circle.

00:04:23.392 --> 00:04:24.013
I know.

00:04:24.033 --> 00:04:27.374
It's all coming back full circle.

00:04:27.394 --> 00:04:27.795
Oh yeah.

00:04:27.815 --> 00:04:28.115
Oh yeah.

00:04:28.135 --> 00:04:28.235
Yeah.

00:04:28.255 --> 00:04:32.977
It was nice too today is that we have a true long distance call all around the world.

00:04:33.637 --> 00:04:37.079
Enrico is on the east coast in the US, I'm in Europe.

00:04:37.239 --> 00:04:39.020
Where are you based right now?

00:04:39.201 --> 00:04:42.222
Right now I'm actually in Mumbai in India.

00:04:43.002 --> 00:04:45.762
I'm teaching at the GEO Institute,

00:04:46.722 --> 00:04:49.303
teaching a course on data visualization,

00:04:49.863 --> 00:04:54.044
but incorporating aspects of semantics and intent,

00:04:54.164 --> 00:04:57.324
some of the NLP with data visualization stuff

00:04:57.924 --> 00:05:00.005
to a class of 60 students.

00:05:00.065 --> 00:05:02.065
So it's been a really great experience.

00:05:02.985 --> 00:05:03.685
Nice, yeah.

00:05:04.005 --> 00:05:05.966
So that's a little break from your usual

00:05:06.086 --> 00:05:07.786
like research work at Tableau.

00:05:08.627 --> 00:05:16.856
Sort of. I mean, I'm still kind of doing both, but it's been really refreshing to interact with the

00:05:17.116 --> 00:05:24.743
local population. And for me, it's just a way of sharing what I have learned over the years

00:05:24.884 --> 00:05:29.769
with people from my home country, India, and

00:05:30.431 --> 00:05:35.492
just helping them understand the value of data visualization and why it's important.

00:05:35.572 --> 00:05:43.174
So, I feel like India has reached that stage now where everybody understands the value of data visualization,

00:05:43.294 --> 00:05:49.195
but they don't necessarily have the training and the access to resources.

00:05:49.635 --> 00:05:50.175
Um, so

00:05:57.267 --> 00:06:03.292
Yeah, there's so much going on in India. I actually visited last summer.

00:06:03.373 --> 00:06:07.436
And for the first time, yes, for a wedding. So for an Indian wedding.

00:06:07.496 --> 00:06:10.418
So that was really, really special.

00:06:10.458 --> 00:06:11.739
It's a good way to get started.

00:06:11.900 --> 00:06:14.942
The full package.

00:06:20.485 --> 00:06:28.728
Yes, exactly. So Vidya, I think one reason why we wanted to have you on the show is because

00:06:28.808 --> 00:06:35.370
you recently published this book together with Bridget Cogley called Functional Aesthetics.

00:06:36.050 --> 00:06:42.492
And reading these books, me and Moritz realized that there's a lot in the book that is related

00:06:42.552 --> 00:06:46.173
to semantics and language, which interestingly is not.

00:06:46.954 --> 00:06:52.736
that much, discussed that much in the world of data visualization, right, and data analysis as

00:06:52.796 --> 00:07:00.599
well. So we thought we would focus this specific episode on the relationship between semantics and

00:07:00.639 --> 00:07:07.701
language and data visualization. So maybe we can start by exploring or understanding what are the

00:07:07.781 --> 00:07:12.743
main connections between data visualization and language and how did you get into this specific

00:07:12.803 --> 00:07:13.123
topic.

00:07:13.844 --> 00:07:19.390
Yeah, I honestly got into this topic kind of serendipitously.

00:07:21.394 --> 00:07:30.582
In general, I think information is meaningful when you understand what it's about, the context of it,

00:07:31.863 --> 00:07:37.708
it gets enriched by additional information, and when you have a better understanding of

00:07:37.748 --> 00:07:43.553
that information, you can figure out meaningful ways of communicating what that information is

00:07:43.573 --> 00:07:43.973
it's about.

00:07:44.616 --> 00:07:49.039
And given that we are very strong visual creatures,

00:07:49.459 --> 00:07:55.323
a very logical place to start is conveying that information visually to a user.

00:07:56.263 --> 00:08:00.006
So a lot of my previous work before I joined Tableau

00:08:00.406 --> 00:08:02.347
was exploring the space of.

00:08:04.170 --> 00:08:13.736
figuring out how to effectively communicate large pixel, megapixel imagery on small mobile screen devices.

00:08:13.816 --> 00:08:18.699
This was a time when smartphones were big and that was primarily the work I did at Nokia.

00:08:19.881 --> 00:08:26.886
And there are a lot of connections to data visualization from the graphics space.

00:08:27.486 --> 00:08:33.410
In a lot of respects, I would say, it's a much more tractable problem because in the

00:08:33.450 --> 00:08:39.354
In the graphics world, you're trying to find semantics within pixels of information.

00:08:40.236 --> 00:08:41.836
But with data visualization,

00:08:41.916 --> 00:08:44.037
there's a lot more structure and

00:08:44.057 --> 00:08:46.998
built-in semantics already that is provided.

00:08:47.478 --> 00:08:48.799
We know the chart type,

00:08:48.839 --> 00:08:49.819
we know the mark type,

00:08:49.919 --> 00:08:51.099
we know the attributes,

00:08:51.139 --> 00:08:54.180
we might know the attribute types.

00:08:54.520 --> 00:08:56.601
So there's a lot that we can start off with.

00:08:56.941 --> 00:09:02.523
So it really gives an extra foothold in terms of

00:09:02.583 --> 00:09:05.084
coming up with semantically meaningful ways

00:09:05.184 --> 00:09:06.644
of depicting that information.

00:09:07.785 --> 00:09:15.148
So I kind of started my journey at Tableau exploring ways in which we could provide smarter

00:09:15.208 --> 00:09:22.791
defaults to data visualization. So we noticed that on Tableau Public, users spent a lot of time

00:09:23.972 --> 00:09:28.053
going to the web, trying to find images, creating icons.

00:09:28.795 --> 00:09:32.198
and associating them with sports visualizations,

00:09:32.258 --> 00:09:33.759
with their favorite sports teams,

00:09:33.859 --> 00:09:37.022
or flags, or other types of icons.

00:09:37.763 --> 00:09:40.245
So, this was a project that I started with

00:09:40.305 --> 00:09:43.167
Jock McKinlay when I joined Tableau is,

00:09:43.628 --> 00:09:46.350
can we actually figure out a way to use

00:09:46.731 --> 00:09:49.833
natural language techniques to effectively query

00:09:50.494 --> 00:09:52.075
image databases to

00:09:52.157 --> 00:09:57.741
get back these images and use graphics techniques to come up with a visually consistent,

00:09:58.282 --> 00:10:04.606
yet aesthetic palette of icons and automatically suggest that to the user. So these were these

00:10:04.787 --> 00:10:11.091
little, you know, micro problems along the flow of analysis where I was exploring how can we

00:10:11.151 --> 00:10:16.795
provide reasonable defaults to reduce the friction so that people don't have to step out of their

00:10:17.276 --> 00:10:17.856
workflow.

00:10:18.378 --> 00:10:21.780
do these manually intensive tasks and come back.

00:10:22.281 --> 00:10:24.142
And so that's where I started.

00:10:24.622 --> 00:10:25.843
And then moving on,

00:10:25.903 --> 00:10:29.886
I started exploring natural language interaction with data,

00:10:31.327 --> 00:10:33.769
which we can talk about in more detail in a bit.

00:10:35.170 --> 00:10:39.473
But there is this general structure of

00:10:39.693 --> 00:10:43.435
using language and grammar for both defining charts,

00:10:43.456 --> 00:10:44.096
you

00:10:44.037 --> 00:10:51.386
and also asking questions of charts because at the end of the day data visualizations are a

00:10:51.446 --> 00:10:56.432
manifestation of visual communication. So communication is the underlying theme

00:10:56.953 --> 00:11:00.377
and the best way to think about communication is through language.

00:11:00.820 --> 00:11:04.587
Yeah, I think that's, yeah, there's so many cross connections there, right? It's like,

00:11:15.492 --> 00:11:18.254
Have a conversation.

00:11:18.274 --> 00:11:19.855
Even the metaphors that we use.

00:11:20.035 --> 00:11:20.856
Yeah, exactly.

00:11:20.876 --> 00:11:25.079
And that all shows us already that implicitly this is happening.

00:11:25.119 --> 00:11:29.082
And I think what's fascinating is that your work makes it explicit and really creates

00:11:29.142 --> 00:11:35.767
that tight link between language and visuals and it's very, very interesting.

00:11:35.867 --> 00:11:41.631
And I guess EVISA is a project that was sort of seminal there, both for you personally,

00:11:41.671 --> 00:11:43.772
but then also probably for the field.

00:11:43.792 --> 00:11:43.892
Yeah.

00:11:44.314 --> 00:11:49.816
For the first time allowing, in the paper it says, it's a prototype system that enables

00:11:49.876 --> 00:11:53.858
a user to have a conversation with their data using natural language.

00:11:54.178 --> 00:11:57.020
That sounds like really exciting.

00:11:58.560 --> 00:12:01.722
Can you tell us a bit about it, how it came about and how it developed?

00:12:02.483 --> 00:12:03.664
Yeah, for sure.

00:12:04.164 --> 00:12:11.506
So the name, actually, Eviza is inspired by the MIT chatbot

00:12:11.526 --> 00:12:14.668
assistant, Eliza.

00:12:15.268 --> 00:12:19.810
And I love the project so much that my vanity license plate

00:12:19.850 --> 00:12:20.910
is also called Eviza.

00:12:20.990 --> 00:12:27.052
So true Bay Area techie at heart.

00:12:27.793 --> 00:12:29.813
You should send us a picture.

00:12:30.334 --> 00:12:38.039
put it in the blog post. Yeah. And I, I bump into people randomly at grocery stores saying,

00:12:38.099 --> 00:12:42.362
Oh, I saw your car in the parking lot, so you must be somewhere in the store. Um, so it's great.

00:12:42.382 --> 00:12:53.069
Yeah. So, um, I, the, the underlying premise of what I think, what made, uh, advisor interesting

00:12:53.129 --> 00:12:53.389
was.

00:12:54.871 --> 00:13:01.313
Communicating with data and asking questions of data using natural language was not a new problem.

00:13:01.594 --> 00:13:05.475
People have done it for years. I mean the database community started with it

00:13:05.975 --> 00:13:08.216
where they had these pseudo-SQL statements

00:13:08.296 --> 00:13:09.976
and they were asking questions of their data.

00:13:11.478 --> 00:13:19.101
What made EVISA unique, and I want to acknowledge that it was a collaboration with other members

00:13:19.181 --> 00:13:27.524
of the research team, was we were really looking at not just single queries to data and getting

00:13:27.564 --> 00:13:28.445
back a response.

00:13:28.865 --> 00:13:32.847
As you indicated, Moritz, it's an actual conversation with data.

00:13:33.267 --> 00:13:38.529
So for example, if I am talking with you and I'm talking about, you know, my evening here

00:13:38.589 --> 00:13:41.850
in Mumbai, you will respond and you will respond.

00:13:41.211 --> 00:13:45.814
you will ask me a follow-up question. You might say, what is the time? And I might say, you know,

00:13:46.074 --> 00:13:52.779
it's 8.30 in the night. So there's this back and forth exchange where questions are not restated

00:13:52.839 --> 00:13:58.543
every time. We build on the previous context and knowledge and have this sort of natural

00:13:58.603 --> 00:14:03.606
progression of dialogue between me and and yourselves.

00:14:04.428 --> 00:14:10.832
And so I was really interested in emulating that human-to-human conversation behavior

00:14:11.293 --> 00:14:18.377
with respect to a computer around data. So I looked at what we call language pragmatics,

00:14:18.498 --> 00:14:22.240
which is looking at the context of the previous query. So if I ask,

00:14:22.660 --> 00:14:25.382
show me the large earthquakes near california

00:14:26.204 --> 00:14:29.385
My follow-up question is, how about near Texas?

00:14:30.386 --> 00:14:33.467
I'm not restating, show me large earthquakes near Texas,

00:14:33.547 --> 00:14:35.327
because that's kind of unnatural.

00:14:35.727 --> 00:14:38.308
So that was kind of the first, I would say,

00:14:38.828 --> 00:14:42.170
novel contribution of that work, is just really implementing

00:14:42.310 --> 00:14:44.890
this pragmatics model in the context

00:14:44.930 --> 00:14:46.571
of an analytical conversation.

00:14:47.131 --> 00:14:52.433
The second insight with building the system which actually got

00:14:52.473 --> 00:14:54.434
Tableau leadership excited was.

00:14:55.535 --> 00:15:01.259
There was a previous concern that these natural language systems needed to be perfect.

00:15:01.739 --> 00:15:04.721
They needed to completely understand what the user was saying,

00:15:05.162 --> 00:15:07.904
and they needed to generate a perfect answer,

00:15:08.084 --> 00:15:10.386
otherwise, people would get annoyed.

00:15:10.926 --> 00:15:14.248
I think some of it largely came from systems like

00:15:14.308 --> 00:15:18.051
Microsoft Clippy and other places where if

00:15:18.071 --> 00:15:21.534
the system constantly made an error,

00:15:21.774 --> 00:15:23.175
then people would get annoyed.

00:15:23.957 --> 00:15:30.812
But if you actually think about the nature of human-to-human conversation, our conversation

00:15:23.975 --> 00:15:24.095
You

00:15:30.852 --> 00:15:32.155
patterns are not perfect.

00:15:33.380 --> 00:15:38.082
I might, I mean, well, Enrico, you thought I was in Bangalore, and I said, no, I'm actually

00:15:38.122 --> 00:15:38.742
in Mumbai.

00:15:38.862 --> 00:15:41.803
And I corrected you, but I was not offensive, right?

00:15:41.864 --> 00:15:43.504
I mean, it was just a conversation.

00:15:43.884 --> 00:15:45.025
Hopefully I didn't offend you.

00:15:45.565 --> 00:15:46.926
Thank you.

00:15:55.276 --> 00:15:59.861
Maybe the system comes up with a reasonable guesstimate of what a large earthquake might

00:16:00.021 --> 00:16:04.266
mean and might set it up to a magnitude of six and greater on the Richter scale.

00:16:05.489 --> 00:16:09.011
and surfaces that reasoning back to me and says,

00:16:09.091 --> 00:16:10.532
hey, you know, you said large,

00:16:10.632 --> 00:16:12.013
I wasn't quite sure what you meant,

00:16:12.133 --> 00:16:14.054
but I took a guess and I, you know,

00:16:14.414 --> 00:16:15.795
assumed you said six or greater.

00:16:15.835 --> 00:16:18.176
And I might, you know, come back to the system and say,

00:16:18.236 --> 00:16:19.857
no, it's actually, you know,

00:16:19.917 --> 00:16:22.338
I live in California right by the earthquake fault

00:16:22.398 --> 00:16:23.299
and I'm not joking.

00:16:23.759 --> 00:16:25.980
So, you know, large earthquakes to me

00:16:26.060 --> 00:16:27.721
is anything four or greater.

00:16:27.881 --> 00:16:29.742
And I might tweak the slider

00:16:29.982 --> 00:16:31.583
that the computer produces to me.

00:16:32.905 --> 00:16:38.026
To me, when we actually tested it with users,

00:16:38.326 --> 00:16:41.507
A, users were actually not bothered when they had to

00:16:41.567 --> 00:16:45.289
correct the system because it was just a small tweak in a slider,

00:16:46.189 --> 00:16:50.010
and B, the system actually remembered that setting so that when

00:16:50.050 --> 00:16:51.030
a user asked

00:16:51.751 --> 00:16:58.034
you know, about large in a subsequent interaction, the system remembered that and the users were

00:16:58.074 --> 00:17:01.676
delighted by that. It's like, oh, you know, actually, you remember the context, right? So

00:17:02.057 --> 00:17:06.839
if Enrico remembers that I'm actually in Mumbai, I'd be like, you know, that's great, Enrico,

00:17:06.859 --> 00:17:08.300
So you have a great memory, right?

00:17:08.340 --> 00:17:10.061
I mean, I'm actually going to compliment you.

00:17:10.761 --> 00:17:10.942
So...

00:17:12.184 --> 00:17:15.787
It turned out people actually love that sort of back and forth interaction.

00:17:16.127 --> 00:17:22.152
So something that was previously viewed as a hindrance or a limitation was actually

00:17:22.212 --> 00:17:27.597
a strength in terms of modeling human to human dialogue, in terms of a human

00:17:27.617 --> 00:17:29.859
to computer dialogue with respect to data.

00:17:30.459 --> 00:17:31.600
And as I indicated,

00:17:32.781 --> 00:17:36.484
You know, data is a lot more tractable because it's not...

00:17:36.906 --> 00:17:39.667
trying to make a model of the entire universe,

00:17:40.107 --> 00:17:43.247
the universe for the system are the bounds of the data.

00:17:43.467 --> 00:17:47.048
So all of that made it a very tractable problem

00:17:47.188 --> 00:17:50.229
to move the work forward, including productization.

00:17:51.169 --> 00:17:54.150
Yeah, that's one aspect that I really like

00:17:54.210 --> 00:17:57.390
of the type of research that you and your team

00:17:57.590 --> 00:18:00.231
have done over the years that is not,

00:18:00.771 --> 00:18:02.912
I mean, the technical contribution is remarkable,

00:18:02.992 --> 00:18:04.732
but what is really interesting is that you have done

00:18:04.772 --> 00:18:05.632
a lot of also...

00:18:06.693 --> 00:18:11.456
human factors type of research on top of it and trying to understand what happens when people

00:18:12.596 --> 00:18:17.679
interact with this type of quote-unquote intelligent systems, right? And I think

00:18:17.699 --> 00:18:21.421
that there's so much to do in that direction and really understanding how

00:18:21.441 --> 00:18:25.704
how users' behavior also changes as they learn how to...

00:18:26.645 --> 00:18:29.528
how to work with these machines. One thing that comes to mind is, like,

00:18:30.068 --> 00:18:33.651
I guess even the way we use Google, we just adapt because we know that...

00:18:34.151 --> 00:18:38.335
We don't ask Google the questions in the same way we would ask to a person.

00:18:38.975 --> 00:18:40.276
Or to Alexa, even.

00:18:40.496 --> 00:18:41.997
Or to Alexa, exactly.

00:18:42.017 --> 00:18:42.398
That's right.

00:18:43.338 --> 00:18:51.185
So I was wondering if you observed the same kind of behavior where after a while people just know

00:18:51.285 --> 00:18:55.708
how to ask the thing so that they will get the information.

00:18:55.830 --> 00:18:56.990
or the results that they want?

00:18:57.791 --> 00:19:00.372
That's a really excellent question and insight.

00:19:01.473 --> 00:19:03.354
So what we have found over the years,

00:19:04.614 --> 00:19:09.117
even with search systems and natural language interfaces

00:19:09.817 --> 00:19:12.899
is people don't like to be wrong.

00:19:13.559 --> 00:19:14.820
They don't like failure.

00:19:15.180 --> 00:19:19.703
So they will adapt to the computer speak of the system.

00:19:19.883 --> 00:19:22.224
Humans really understand very quickly

00:19:22.544 --> 00:19:24.565
what a system can or cannot do.

00:19:25.727 --> 00:19:32.552
and readjust their model and their view of their interactions with the system,

00:19:33.933 --> 00:19:40.778
which is good in a way because then you don't have too many frustrated users, hopefully,

00:19:41.259 --> 00:19:44.961
and they sort of adapt to whatever the system can support.

00:19:45.362 --> 00:19:47.123
But there is a downside to it.

00:19:48.786 --> 00:19:55.371
When you're looking at the telemetry data of how people are using systems,

00:19:56.612 --> 00:19:59.195
the window of opportunity to

00:19:59.335 --> 00:20:04.039
understand what are the types of queries the system cannot support,

00:20:04.900 --> 00:20:08.162
and what would make for useful extensions and

00:20:08.222 --> 00:20:11.105
improvements to the system is a very narrow time window.

00:20:12.167 --> 00:20:15.149
what we have found is within a couple of days,

00:20:15.749 --> 00:20:19.190
people will change from asking questions

00:20:19.250 --> 00:20:21.111
that the system does not understand

00:20:21.191 --> 00:20:24.373
or partially understand to only the questions

00:20:24.613 --> 00:20:28.755
and the repertoire of analytical functions

00:20:28.835 --> 00:20:30.295
that the system can support.

00:20:30.956 --> 00:20:32.076
And this includes search.

00:20:32.736 --> 00:20:36.538
And so you immediately see a drop-off of failure rates

00:20:36.878 --> 00:20:37.979
for all these users.

00:20:38.039 --> 00:20:38.339
So you...

00:20:38.700 --> 00:20:45.465
we really have to catch these new users within a one to three day window, which makes it

00:20:45.585 --> 00:20:52.451
kind of challenging when we're using customer data and telemetry data to help inform how

00:20:52.491 --> 00:20:53.592
we can improve the system.

00:20:54.112 --> 00:20:54.633
Yeah, yeah.

00:20:54.913 --> 00:20:58.936
Yeah, that's a tricky aspect, is the discoverability in terms of...

00:20:59.759 --> 00:21:04.303
What can I even ask, right? It's a bit like with gesture interfaces where you just try and figure

00:21:04.323 --> 00:21:09.307
out, okay, maybe swiping works, maybe zooming, right? It's the same with language because

00:21:10.288 --> 00:21:14.772
first probably you ask anything and then you realize, yeah, it can't quite do everything.

00:21:14.812 --> 00:21:18.395
And then you're like, yeah, that makes sense. You know, there must be some limitation, but then

00:21:18.916 --> 00:21:21.498
you need to figure out what is the vocabulary I can use.

00:21:21.518 --> 00:21:21.879
Yeah, exactly.

00:21:27.479 --> 00:21:35.320
So one part I'm really interested in is also this transfer between research and then putting

00:21:35.360 --> 00:21:36.300
it into the product.

00:21:36.400 --> 00:21:45.462
And yeah, EVISA in some way found its place in Tableau as Ask Data or probably some further

00:21:45.482 --> 00:21:48.623
development but I guess it all started with EVISA, right?

00:21:48.703 --> 00:21:52.104
And so yeah, I'm curious, like how does this work?

00:21:52.304 --> 00:21:58.065
How did this specific prototype and piece of research make it into being part of a product

00:21:58.085 --> 00:21:58.545
Temporal.

00:21:59.446 --> 00:21:59.746
Yeah.

00:21:59.846 --> 00:22:03.587
What's the process there and how did this work and, you know, I want to learn all about

00:22:03.607 --> 00:22:04.087
the process.

00:22:05.727 --> 00:22:06.008
Sure.

00:22:07.288 --> 00:22:12.970
So, as mentioned, I mean, when I joined Tableau back in the day, I was fortunate enough to

00:22:14.010 --> 00:22:19.972
interact with Chris Dolty and Christian Chabot and Pat Hanrahan, you know, the co-founders

00:22:20.032 --> 00:22:20.672
of the company.

00:22:21.873 --> 00:22:27.396
And we were having these conversations when I joined Tableau, where, you know, Chris Dolty said,

00:22:27.476 --> 00:22:32.539
you know, NLP is really hard, Vidya. I don't know how we can actually crack that nut. And

00:22:33.059 --> 00:22:37.662
even if we do crack the nut in research, how are we actually going to productize it so it works

00:22:37.702 --> 00:22:43.285
for any customer, right? Because we are not, Tableau doesn't cater to a very specific vertical.

00:22:44.025 --> 00:22:48.688
And there were, there are previous systems like IBM Watson that have catered to very

00:22:49.509 --> 00:22:57.774
verticals, including chess or healthcare, but Tableau, in a sense, is more of a generalist

00:22:57.834 --> 00:22:58.714
type of platform.

00:22:58.754 --> 00:23:00.155
So that was kind of the problem.

00:23:01.057 --> 00:23:09.800
But when I demoed Eviza, it's kind of funny because the story there was I was in California

00:23:10.801 --> 00:23:18.884
and everybody else was in Seattle. And back then, for various reasons, I was using voice to talk

00:23:18.924 --> 00:23:23.345
with the advice. I just thought it would be so cool to talk to the data with voice. I mean...

00:23:24.627 --> 00:23:25.767
you could do text or voice.

00:23:25.787 --> 00:23:28.188
That was not kind of the main part of the story.

00:23:28.208 --> 00:23:30.149
But I was like, yeah, I'm going to demo this with voice.

00:23:30.209 --> 00:23:32.210
And I asked everybody to mute themselves.

00:23:33.110 --> 00:23:34.611
And I did my whole demo.

00:23:34.671 --> 00:23:36.712
And I said, don't interrupt me in between,

00:23:36.752 --> 00:23:39.213
because it's going to catch your voice and mess up my demo.

00:23:40.393 --> 00:23:42.734
So I did my demo for 10 minutes.

00:23:42.954 --> 00:23:47.056
And then I had to rush off to a doctor's appointment.

00:23:47.096 --> 00:23:48.056
So I just said bye.

00:23:48.117 --> 00:23:50.397
I didn't even wait for Q&A. And I didn't make it.

00:23:51.359 --> 00:23:53.120
I didn't really think about it, right?

00:23:53.500 --> 00:23:56.982
And late in the night, I get an email from Chris Dolte.

00:23:57.022 --> 00:24:00.844
It's like, OK, can you demo this to Tom Walker, who

00:24:00.864 --> 00:24:06.847
at that time was making the call in terms of which teams

00:24:06.908 --> 00:24:08.869
are funded and stuff like that.

00:24:08.969 --> 00:24:12.351
So long story short, Tableau leadership

00:24:12.411 --> 00:24:14.352
got really excited about the project.

00:24:15.172 --> 00:24:15.893
And, um,

00:24:16.694 --> 00:24:21.699
they decided, okay, I think it's time that we invest in NLP. And this was also a time where

00:24:21.799 --> 00:24:27.685
the tech industry was hedging its bets in this space. And this is, I think, one of the things,

00:24:27.745 --> 00:24:35.493
And I don't think it's one criteria that indicates or dictates what, you know, what's going on.

00:24:35.154 --> 00:24:37.796
you know, what makes a successful tech transfer.

00:24:37.956 --> 00:24:39.437
It's a multitude of factors.

00:24:39.897 --> 00:24:43.218
You need to have some sort of perspective

00:24:43.319 --> 00:24:45.680
that you can show that it works

00:24:45.820 --> 00:24:47.881
and the market needs to be ready for that.

00:24:47.961 --> 00:24:49.802
The customers need to be ready for that

00:24:50.042 --> 00:24:51.743
and the company needs to be ready.

00:24:51.783 --> 00:24:52.923
So there's so much that has,

00:24:52.983 --> 00:24:54.664
I mean, all the stars need to align.

00:24:55.304 --> 00:24:58.086
And we were fortunate enough that everything did align.

00:24:58.586 --> 00:25:00.227
But the problem back then was

00:25:00.587 --> 00:25:00.687
Um,

00:25:01.288 --> 00:25:03.729
Tableau was not known to be an NLP company.

00:25:03.789 --> 00:25:06.571
I was the only NLP person in the entire company,

00:25:06.651 --> 00:25:08.631
and it was a very small startup.

00:25:09.792 --> 00:25:11.893
So how do you actually convince people

00:25:12.073 --> 00:25:14.854
who are working in companies like Google

00:25:14.954 --> 00:25:17.795
to come and join Tableau and work on, you know, NLP.

00:25:18.736 --> 00:25:25.140
So we made the decision to look for smallish startups in the Bay Area, because I'm based in

00:25:25.160 --> 00:25:32.624
the Bay Area, to try to see if we can acquire some seed technology as foundation and then

00:25:32.644 --> 00:25:37.927
then I could help build some of the stuff that I had in Eviza on top of that platform.

00:25:39.128 --> 00:25:45.532
So after, you know, several months of vetting various startups that I was involved in,

00:25:45.852 --> 00:25:51.755
and this is the other piece, you know, when you're talking about doing tech transfer research

00:25:51.815 --> 00:25:58.198
projects, you have to be all in. So that might involve doing technical due diligence for startups,

00:25:58.338 --> 00:26:02.940
interviewing people. You have to be all in. You have to show that you are

00:26:03.982 --> 00:26:11.146
motivated to make it work. There is kind of a larger sociological aspect to this.

00:26:12.247 --> 00:26:19.792
So, long story short, we identified this startup called ClearGraph, and they came on board

00:26:20.132 --> 00:26:20.312
and

00:26:21.655 --> 00:26:30.186
I moved from research and joined the product team, starting first as a lead engineer,

00:26:31.267 --> 00:26:33.330
working on production quality code.

00:26:38.198 --> 00:26:44.403
to write production quality code with your peers, including junior people critiquing

00:26:44.443 --> 00:26:51.488
and reviewing your code. But it was really good for my soul. And I definitely improved

00:26:51.568 --> 00:26:58.052
my coding chops after that. And soon enough, I became the engineering manager on the team.

00:26:58.753 --> 00:27:01.355
So, I was responsible for that.

00:27:01.276 --> 00:27:06.621
you know, all the release planning, working closely with the PMs, doing code reviews, writing code myself,

00:27:07.862 --> 00:27:16.730
sneaking in features from Eviza into the product, and writing papers. Melanie Torrey, who is currently at

00:27:16.770 --> 00:27:19.792
Northeastern, was a very close collaborator of mine.

00:27:19.892 --> 00:27:21.694
And so she and I would...

00:27:22.737 --> 00:27:27.661
you know, buddy up and in figuring out the research aspects of the project, you know,

00:27:28.222 --> 00:27:34.528
how do we actually support intent understanding and semantics so that we can build on top of this

00:27:34.608 --> 00:27:41.014
platform. And so I stayed on, stayed in the team for a few releases of AskData and then

00:27:41.214 --> 00:27:43.016
ultimately move back to research.

00:27:45.520 --> 00:27:51.482
And like the way it's implemented now, what can you ask AskData, like let's say the earthquake

00:27:51.502 --> 00:27:57.023
data set, right? What are some of the questions you could ask and where you would get a meaningful

00:27:57.063 --> 00:28:08.607
chart back? Yeah, so AskData connects natural language utterances to Tableau's visql command

00:28:08.627 --> 00:28:12.548
stack and the feature leverages show me.

00:28:13.429 --> 00:28:19.495
to display the top recommended chart that ShowMe suggests.

00:28:20.015 --> 00:28:26.021
So we do leverage some of the Tableau underlying architecture

00:28:26.401 --> 00:28:31.086
and combine it with the parser and the natural language input

00:28:31.106 --> 00:28:32.147
that's part of Astida.

00:28:32.849 --> 00:28:41.437
So that being said, we support five basic analytical functions that are tied to Tableau's

00:28:41.718 --> 00:28:47.663
core analytical stack. You know, you can ask questions about aggregations and groupings

00:28:47.803 --> 00:28:49.605
and drill downs, filters.

00:28:50.207 --> 00:28:58.058
and sorting and various combinations of that. We also have vague concepts, kind of like the

00:28:58.258 --> 00:29:01.102
the large and the near in the earthquake examples.

00:29:17.615 --> 00:29:22.757
It, you know, also supports these follow-up questions.

00:29:22.817 --> 00:29:25.017
So some of the pragmatics behavior as well.

00:29:25.277 --> 00:29:25.497
Wow.

00:29:25.838 --> 00:29:25.998
Yeah.

00:29:26.378 --> 00:29:27.398
That sounds quite useful.

00:29:30.099 --> 00:29:33.360
I was, it was interesting for me to hear that initially

00:29:33.420 --> 00:29:36.961
the prototype was meant to be used through voice.

00:29:37.761 --> 00:29:38.082
Yeah.

00:29:38.362 --> 00:29:40.122
I don't know if, if it's actually possible

00:29:40.162 --> 00:29:41.383
to do that with Ask Data.

00:29:42.345 --> 00:29:43.927
Or is it only through typing?

00:29:43.947 --> 00:29:45.870
We decided to do text.

00:29:46.912 --> 00:29:49.135
You know, from a business standpoint, it makes sense.

00:29:49.195 --> 00:29:52.040
I mean, if people are all at their desk talking to their data,

00:29:52.280 --> 00:29:54.664
You know, it may annoy, it may annoy you.

00:30:01.113 --> 00:30:07.295
No, I mean, to me, voice-to-text translation is more or less a solved problem.

00:30:08.135 --> 00:30:13.996
I was just using an off-the-shelf API to do that, so it was more of a footnote, and I

00:30:14.036 --> 00:30:18.457
just thought it would be cool to use my voice and talk to the data.

00:30:18.517 --> 00:30:19.698
as more of a cool effect.

00:30:21.339 --> 00:30:27.762
But, you know, I am exploring, you know, voice with data stories, actually,

00:30:28.842 --> 00:30:35.005
now as part of my research portfolio, just in the space of text and charts.

00:30:35.986 --> 00:30:38.567
So it'll probably come back to me full circle.

00:30:39.127 --> 00:30:40.548
Yeah.

00:30:40.530 --> 00:30:42.773
And I was also curious to hear from you.

00:30:43.074 --> 00:30:48.042
I really like the story that you started in research and then went to engineering.

00:30:48.102 --> 00:30:51.287
And I guess now you're back full time in research.

00:30:51.487 --> 00:30:51.948
Correct.

00:30:52.549 --> 00:30:52.810
Yeah.

00:31:05.280 --> 00:31:09.442
The last mile is probably really hard.

00:31:09.462 --> 00:31:10.783
The mini-miles.

00:31:10.983 --> 00:31:12.023
The mini-miles.

00:31:12.483 --> 00:31:14.644
The mini-miles, right.

00:31:14.745 --> 00:31:16.185
It's a big hike.

00:31:17.686 --> 00:31:20.067
How many years did it take all together, kicking it off?

00:31:22.249 --> 00:31:23.550
Um, it was about two years.

00:31:24.110 --> 00:31:31.094
Um, it was about, I mean, we shipped, we shipped our first, uh, feature on time.

00:31:31.114 --> 00:31:33.676
I would say a little over a year.

00:31:34.556 --> 00:31:34.996
And then...

00:31:35.016 --> 00:31:35.617
Mm-hmm.

00:31:51.522 --> 00:31:57.084
You have to be all in, so if you want the right type of sausage to come out, you need

00:31:57.124 --> 00:31:59.625
to be part of the messy sausage-making.

00:32:00.285 --> 00:32:08.988
It's very hard to sit in research and try to get the team to build a product.

00:32:09.949 --> 00:32:16.478
based on what you would like to see you have to roll up your sleeves and be part of that process

00:32:18.100 --> 00:32:24.208
for a lot of reasons, because unlike research, engineering works very fast.

00:32:24.891 --> 00:32:31.074
every day there are stand-ups, every day there are decisions being made. By sitting outside the team

00:32:31.174 --> 00:32:36.917
you're not part of that conversation, you're not part of those serendipitous decisions where it's

00:32:36.957 --> 00:32:42.860
like, oh there's a bug, how do we fix it? Oh by the way, if we fix this we can add this additional

00:32:42.900 --> 00:32:50.023
feature. That doesn't happen by setting up a formal meeting every week. So you lose out on a lot of

00:32:50.083 --> 00:32:53.665
that if you're not part of the team. And then the second thing is

00:32:54.627 --> 00:32:57.749
when you're part of the team and you're writing code

00:32:57.829 --> 00:33:00.611
and being part of the process as a team member,

00:33:01.272 --> 00:33:04.694
there is a whole notion of trust

00:33:05.155 --> 00:33:07.136
that is an credibility that is built.

00:33:08.217 --> 00:33:11.539
I will say that, you know, engineers are great people,

00:33:11.920 --> 00:33:14.622
but they are very skeptical of us researchers.

00:33:14.662 --> 00:33:17.564
They're like, yeah, you know, you guys are all smart.

00:33:17.584 --> 00:33:19.405
You write papers, but you know,

00:33:19.465 --> 00:33:22.668
when rubber has to hit the road, I don't know that man.

00:33:24.130 --> 00:33:31.034
So, you know, you have to spend some time with the team and building that credibility

00:33:31.615 --> 00:33:38.519
in order for them to take you seriously. Get the street cred. Exactly. And I think it's a

00:33:38.579 --> 00:33:46.224
really good opportunity, especially when you are in industry research, to work on code.

00:33:46.444 --> 00:33:52.208
You know, there's nothing as empowering as being able to have an idea and being able to make it

00:33:52.288 --> 00:33:52.528
real.

00:33:53.430 --> 00:33:55.532
It may not be the perfect prototype,

00:33:56.112 --> 00:33:59.935
but it's super power if you can actually take an idea

00:34:00.116 --> 00:34:01.837
and implement something and show it

00:34:02.377 --> 00:34:03.718
beyond a PowerPoint deck.

00:34:05.340 --> 00:34:07.601
And when you join a team like that,

00:34:08.202 --> 00:34:12.545
you understand the complexities and processes

00:34:12.605 --> 00:34:14.407
that go into shipping a feature.

00:34:15.147 --> 00:34:17.109
It's much more than what meets the eye.

00:34:17.589 --> 00:34:19.150
There's so much that goes into stake,

00:34:19.230 --> 00:34:20.071
the performance.

00:34:20.673 --> 00:34:21.974
You can't have regressions.

00:34:22.014 --> 00:34:24.855
You've got to build a very sophisticated test suite.

00:34:26.455 --> 00:34:28.156
And you have to listen to customers.

00:34:28.857 --> 00:34:30.857
You have to be very careful of release planning.

00:34:31.418 --> 00:34:33.418
So there's so much that goes on, and you kind of

00:34:33.479 --> 00:34:37.580
develop a profound respect for the engineering profession.

00:34:38.281 --> 00:34:40.982
And it's a way of bringing that back into research.

00:34:43.104 --> 00:34:46.945
And how do you then balance not getting lost in the weeds?

00:34:47.526 --> 00:34:51.388
Like this is very much like, you know, in the trenches and like the frog's eye perspective

00:34:51.448 --> 00:34:52.068
basically.

00:34:52.568 --> 00:34:58.291
But how do you keep that that vision then and not get lost in the, yeah, this doesn't

00:34:58.331 --> 00:35:03.213
work and that doesn't work and that's broken, you know, because engineering is fundamentally

00:35:03.253 --> 00:35:08.616
a lot about fixing something that's broken.

00:35:08.656 --> 00:35:10.137
How do you keep that big picture with you?

00:35:10.278 --> 00:35:14.901
I think it's very important to have a good working relationship with the PM on the team

00:35:15.182 --> 00:35:22.067
because at the end of the day, the PM is the one that really hashes out the roadmap for

00:35:22.087 --> 00:35:27.271
the feature and what does it look like over the years and being involved in those conversations

00:35:29.173 --> 00:35:37.159
and going back to that document that a PM would write on the roadmap and the strategy

00:35:37.199 --> 00:35:39.260
of the feature kind of keeps you honest.

00:35:41.104 --> 00:35:42.985
And, I think it's just a calibration.

00:35:43.065 --> 00:35:47.549
There are some days when I would be lost in the weeds because my code wouldn't work and

00:35:47.589 --> 00:35:49.811
you just can't sleep when your code doesn't work.

00:35:49.851 --> 00:35:50.832
You want to get it working.

00:35:50.852 --> 00:35:51.672
That's not good.

00:35:52.815 --> 00:36:01.099
But we made a conscious decision where, you know, I continued to publish while being on

00:36:01.139 --> 00:36:08.584
an engineering team and, in fact, actually getting some engineers to be on papers and

00:36:08.644 --> 00:36:09.584
patents with me.

00:36:19.124 --> 00:36:25.148
you know, the research aspects of it, and you have to articulate that in prose,

00:36:25.848 --> 00:36:29.470
that really helps you look at the bigger picture holistically.

00:36:29.811 --> 00:36:30.811
And that combined with...

00:36:30.831 --> 00:36:35.654
What makes this unique, or what are the fundamental mechanisms and so on.

00:36:35.714 --> 00:36:36.515
Yes, yes.

00:36:41.450 --> 00:36:48.117
I think it's a double word score when you have a feature and it is backed and supported by

00:36:48.937 --> 00:36:54.263
well-done research, right? I think that's the best of both worlds.

00:36:54.283 --> 00:36:54.623
Yeah.

00:36:55.404 --> 00:36:55.684
Yeah.

00:36:56.045 --> 00:36:57.906
Or it invites new research, right?

00:36:57.946 --> 00:36:58.147
Yeah.

00:36:58.267 --> 00:36:58.707
Yeah.

00:36:59.048 --> 00:36:59.168
Yeah.

00:37:10.793 --> 00:37:15.516
I think what is really interesting is that when I think about Tableau, to me,

00:37:16.096 --> 00:37:20.379
Tableau is like the thing that basically introduced

00:37:20.439 --> 00:37:26.343
a specific type of interaction for people to create visualizations, right?

00:37:26.383 --> 00:37:29.966
Before Tableau, it was basically Excel and a few other things.

00:37:30.886 --> 00:37:33.748
And Tableau introduced this idea that you basically

00:37:35.469 --> 00:37:35.890
select

00:37:36.852 --> 00:37:42.396
attributes and you drag them into a panel and and now you have some visualizations out there,

00:37:42.856 --> 00:37:48.981
right? And then you have now with with a visa and as data, you have a new type of interaction.

00:37:50.482 --> 00:37:57.708
I was wondering, I always felt that this space we're trying to figure out how to express what

00:37:57.748 --> 00:38:01.310
we have it in our mind so that the machine can understand it and produce.

00:38:02.893 --> 00:38:07.937
data analytics and visualizations is a really interesting space.

00:38:08.898 --> 00:38:11.199
And at the same time, it doesn't seem to be as,

00:38:12.520 --> 00:38:14.682
I would say it's not as much explored

00:38:14.762 --> 00:38:16.504
as the visual side of things,

00:38:17.144 --> 00:38:19.766
but it seems to me that is at least equally relevant,

00:38:19.826 --> 00:38:21.067
if not even more relevant.

00:38:21.387 --> 00:38:23.189
How do we express our intent?

00:38:23.269 --> 00:38:26.471
How do we tell a machine what we want, right?

00:38:30.435 --> 00:38:36.900
two really strong milestones.

00:38:36.920 --> 00:38:39.682
I would say these are two milestones in this sense

00:38:39.742 --> 00:38:42.444
because they introduce completely new paradigms

00:38:43.164 --> 00:38:46.086
on how to express what is my intent

00:38:46.186 --> 00:38:50.209
and what I would like the machine to do for me, basically.

00:38:50.289 --> 00:38:51.150
Right, right.

00:38:52.551 --> 00:38:55.233
Yeah, and I think there is a spectrum

00:38:55.553 --> 00:38:57.254
of the type of analytical intents that people express

00:38:57.275 --> 00:38:58.235
This is the end of the video.

00:38:58.315 --> 00:39:01.718
If you liked the video, please subscribe to the channel and click on the bell icon to

00:39:01.399 --> 00:39:02.660
and what is supported.

00:39:01.738 --> 00:39:02.638
receive notifications of my latest videos.

00:39:02.720 --> 00:39:04.201
So to your point,

00:39:04.961 --> 00:39:07.423
something like a direct manipulation type interface

00:39:07.443 --> 00:39:10.185
like Tableau can be very concrete, right?

00:39:10.265 --> 00:39:11.826
I click on a bunch of attributes

00:39:11.906 --> 00:39:13.547
and I drag them to rows and shelves.

00:39:14.268 --> 00:39:16.630
There's very little ambiguity

00:39:16.950 --> 00:39:20.052
in terms of what the user wants to see.

00:39:21.396 --> 00:39:21.477
and

00:39:43.080 --> 00:39:50.211
Now, with language, you can definitely be a bit much more abstract in terms of how you

00:39:50.271 --> 00:39:51.332
express your intent.

00:39:51.593 --> 00:39:52.474
It can be vague.

00:39:52.955 --> 00:39:53.836
It can be fuzzy.

00:39:55.641 --> 00:39:59.163
and there could also be new paradigms of analytical intent.

00:39:59.183 --> 00:40:03.085
So, you know, one of the one of my papers that was

00:40:03.585 --> 00:40:07.768
presented at VIS with, you know, where Eamon, Gaba and Cindy and others

00:40:08.989 --> 00:40:13.111
collaborated with me was just exploring the language of comparisons

00:40:14.171 --> 00:40:20.015
and how you can actually interpret comparisons and show meaningful

00:40:20.055 --> 00:40:20.995
representations.

00:40:22.378 --> 00:40:30.465
And I would argue that through language, the whole space of comparisons can be pretty complex

00:40:30.545 --> 00:40:33.688
and nuanced when compared to direct manipulation.

00:40:33.829 --> 00:40:36.611
There's cardinality, you can compare one to one.

00:40:37.454 --> 00:40:45.443
1 to N, N to N. You can be very specific or vague about the concepts.

00:40:45.703 --> 00:40:48.507
If I say, when is the safest time to fly?

00:40:49.988 --> 00:40:51.930
The data might indicate the morning.

00:40:52.674 --> 00:41:00.078
But there is an implicit intent there that I not only want morning as my answer, but I probably

00:41:00.118 --> 00:41:06.561
want to see how much safer is it to fly in the morning compared to other times of the day, right?

00:41:06.621 --> 00:41:12.865
I mean, if flying in the afternoon is slightly worse than the morning, I want to see that, right?

00:41:12.925 --> 00:41:18.208
I want to see the whole distribution. So there's so many explicit and implicit ways of expressing

00:41:18.288 --> 00:41:19.529
intent that

00:41:20.591 --> 00:41:26.154
One could do more easily with language as opposed to a direct manipulation interface,

00:41:26.174 --> 00:41:30.097
which kind of opens up a range of interesting problems and opportunities.

00:41:31.299 --> 00:41:36.084
Yeah. And I guess if I remember well, you also have kind of like a mix of,

00:41:36.925 --> 00:41:40.268
I don't remember if this is true for Revisa or us data,

00:41:40.328 --> 00:41:44.371
but I have seen systems where there's a natural language component,

00:41:44.472 --> 00:41:49.276
but then you can also interact with some elements of the

00:41:49.576 --> 00:41:53.420
sentence through direct manipulation, is that correct?

00:41:53.522 --> 00:42:00.809
That's right. So, in general, we have found that the pattern of developing mixed-initiative systems

00:42:01.530 --> 00:42:07.616
tends to be effective, because there are certain things that I just want to tweak directly. You

00:42:07.636 --> 00:42:14.022
know, going back to the earthquake example, if the system comes back with a Richter magnitude

00:42:14.062 --> 00:42:15.023
of Sixth and Greater.

00:42:15.846 --> 00:42:22.769
I may not want to type saying, actually, I want large to be interpreted as four and greater.

00:42:23.269 --> 00:42:28.591
It's a lot easier for me to just grab the slider and tweak it down to four, right?

00:42:28.651 --> 00:42:31.372
So there are certain things that.

00:42:32.053 --> 00:42:36.714
lends itself a lot more convenient to just click on marks. If I just want to see

00:42:38.655 --> 00:42:43.936
some data in a particular region and I see an outlier, it's just easier for me to click on

00:42:43.956 --> 00:42:49.457
the outlier rather than asking a question or I might click on it. It's called deictic referencing

00:42:49.497 --> 00:42:54.438
where you circle, maybe you lasso select some points and say, tell me more about this.

00:42:56.178 --> 00:42:58.678
And this is a very complex concept.

00:42:58.738 --> 00:42:59.619
What does this mean?

00:43:00.559 --> 00:43:05.462
It's hard to express that in language. I mean, we do this all the time as humans, right?

00:43:06.042 --> 00:43:11.305
We point at things and say this. We don't explain and describe the object that we are pointing to.

00:43:12.285 --> 00:43:18.728
So I think there is a place for every sort of interaction and the key challenge is to figure

00:43:18.828 --> 00:43:26.172
out which flavors of interaction lend itself better for certain types of questions and intent.

00:43:26.372 --> 00:43:27.813
Yeah, that's so interesting.

00:43:30.654 --> 00:43:34.197
Yeah, I'm now curious, new ideas pop in my mind.

00:43:35.098 --> 00:43:36.459
So going back to voice,

00:43:36.579 --> 00:43:38.180
I'm wondering if you ever explored

00:43:39.121 --> 00:43:41.783
the voice channel as an output, right?

00:43:41.843 --> 00:43:44.105
So what if I want to communicate data

00:43:44.746 --> 00:43:46.347
through say Alexa, right?

00:43:46.407 --> 00:43:49.149
Alexa is telling me, rather than visualizing something,

00:43:49.189 --> 00:43:52.111
it's telling me something about the data.

00:43:52.151 --> 00:43:54.693
How do you communicate data verbally?

00:43:55.696 --> 00:44:04.344
Yeah, so we actually had a paper at CHI last year, where we explored analytical chatbots

00:44:05.345 --> 00:44:11.471
using this Grecian model of cooperative conversation. And we essentially studied

00:44:11.631 --> 00:44:15.555
the behavior of three flavors of chatbots using Slack.

00:44:16.418 --> 00:44:22.381
where it's text-based input and the output is a combination of text and images.

00:44:23.742 --> 00:44:26.283
We had just a text chatbot,

00:44:26.483 --> 00:44:31.085
which did not have any provision for any imagery or charts,

00:44:31.125 --> 00:44:35.988
Then we had a pure voice-based chatbot that was using the Echo device.

00:44:36.749 --> 00:44:42.434
And so we first did kind of Wizard of Oz studies to just understand the expectations

00:44:43.435 --> 00:44:48.180
with respect to these various mediums. It's kind of the Marshall McLuhan paradigm of

00:44:48.320 --> 00:44:54.065
medium is the message, right? And it was very interesting, particularly with voice,

00:44:54.966 --> 00:44:57.128
but not surprising if you think about it.

00:44:58.089 --> 00:45:00.411
So people's working memory is very limited.

00:45:01.494 --> 00:45:09.720
Um, so if I ask what are the most expensive wines in Napa and this voice chat bot goes

00:45:10.180 --> 00:45:16.805
on and gives me a speech about all the, you know, the top five wines by the time it tells

00:45:16.865 --> 00:45:19.087
me the last one, I would have forgotten the first one.

00:45:20.429 --> 00:45:26.394
And so what we realized is people, and this is something that has been studied in

00:45:26.794 --> 00:45:31.738
kind of general voice chat bots, where it will give you a snippet of information saying, you

00:45:31.758 --> 00:45:37.582
know, the most expensive wine is this. Do you want to hear about the others? And so it's like a

00:45:37.702 --> 00:45:42.746
follow-up question, almost like this back and forth. And you're like, sure, tell me the next one.

00:45:43.206 --> 00:45:46.129
And so it's like conversation chunking.

00:45:47.751 --> 00:45:52.214
The other thing that we noticed was the issue around trust.

00:45:52.854 --> 00:45:54.815
When people, we were using,

00:45:54.875 --> 00:45:56.656
so we implemented these chatbots

00:45:56.756 --> 00:45:59.458
and they all shared a common parser.

00:46:00.539 --> 00:46:03.100
So while we knew that the performance

00:46:03.140 --> 00:46:05.722
of all these chatbots were comparable to one another,

00:46:06.342 --> 00:46:10.024
we found that people trusted voice chatbots less

00:46:10.585 --> 00:46:12.306
when compared to the text ones.

00:46:12.826 --> 00:46:16.948
So they wanted the voice chatbot to repeat the question.

00:46:18.230 --> 00:46:19.150
Just to make sure.

00:46:19.550 --> 00:46:20.591
Just to make sure.

00:46:20.611 --> 00:46:21.491
They got it right.

00:46:21.611 --> 00:46:23.352
They got it, exactly.

00:46:23.892 --> 00:46:25.152
So it's very interesting.

00:46:26.093 --> 00:46:33.135
So we found a lot of differences, and we also found the types of intents that people tend

00:46:33.175 --> 00:46:37.056
to ask of text versus voice chatbots to be different.

00:46:38.277 --> 00:46:44.339
With voice, it's almost always these fact-finding questions with single responses.

00:46:46.060 --> 00:46:48.501
You know, kind of like how you ask Google,

00:46:48.541 --> 00:46:50.442
tell me what the weather is going to be, right?

00:46:52.063 --> 00:46:53.664
So it's like these single ones.

00:46:53.704 --> 00:46:55.785
You know, what are the sales looking

00:46:55.865 --> 00:46:57.326
like today for this region?

00:46:58.987 --> 00:47:01.508
But with other types of chatbots,

00:47:01.928 --> 00:47:04.630
the questions are definitely much more complex.

00:47:05.170 --> 00:47:06.751
They are much more compound.

00:47:06.791 --> 00:47:09.573
People tend to chain multiple questions together.

00:47:10.093 --> 00:47:13.635
So it kind of goes back to that humans adapting to the computer

00:47:13.675 --> 00:47:14.595
speak of the system.

00:47:15.917 --> 00:47:23.401
But the new insight we learned was humans also adapt to their own limitations of interacting

00:47:23.421 --> 00:47:26.823
with the system, because they know that they have a limited working memory.

00:47:27.904 --> 00:47:32.926
And so they figure out, okay, to circumvent my limited working memory,

00:47:33.066 --> 00:47:36.909
let me simplify the types of questions that I will ask of these voice chat bots.

00:47:39.171 --> 00:47:45.137
Very interesting, yeah. And do you think, like when, I think we touched a bit on that, but

00:47:45.638 --> 00:47:51.343
like some of your systems respond with a chart and then others respond with an answer or like a fact?

00:47:51.984 --> 00:47:52.304
Yeah.

00:47:52.344 --> 00:47:58.671
Like how, when, when is what being asked for or what was better when?

00:47:58.691 --> 00:48:00.352
Yeah, I mean, yeah.

00:48:00.372 --> 00:48:00.793
depend on the

00:48:02.998 --> 00:48:04.338
I think it's all of the above.

00:48:05.699 --> 00:48:12.582
When you have the luxury of a text interface with real estate, people want the single answer,

00:48:13.502 --> 00:48:17.764
But they might want additional context that supports the answer.

00:48:18.906 --> 00:48:25.349
They might want the distribution of other regions, how well it's like, you know, especially

00:48:25.389 --> 00:48:31.071
these superlative questions, you know, the cheapest wine or the best selling product.

00:48:31.391 --> 00:48:38.154
People want to know how much better or worse is that answer when compared to the rest of

00:48:38.174 --> 00:48:40.855
the data distribution is very important to people.

00:48:41.337 --> 00:48:44.998
It's the same when you search for a product, you always get these big tables with the feature

00:48:45.018 --> 00:48:48.159
comparison and still the winner is marked, right?

00:48:48.179 --> 00:48:53.261
But you want to see the full matrix, just know that everything's correct, even though

00:48:53.301 --> 00:48:56.802
everything is made up anyways on these sites.

00:48:56.922 --> 00:48:57.362
Yeah.

00:48:57.522 --> 00:49:00.663
So this is not a new paradigm.

00:49:00.723 --> 00:49:05.705
I mean, search systems have been and recommendation systems have been doing it for ages, right?

00:49:06.165 --> 00:49:10.286
Like if I go to Google and I type in my flight status number.

00:49:11.167 --> 00:49:15.770
it's a fact-finding question, right? I type in my flight number and it'll get me the status,

00:49:16.431 --> 00:49:22.255
but there are other documents beneath it. The page just doesn't contain only my answer.

00:49:22.395 --> 00:49:28.259
There are articles that support, you know, my query and there's other stuff going on.

00:49:28.979 --> 00:49:33.822
The same thing with something like Amazon. I might be very specific and I want like

00:49:34.383 --> 00:49:38.485
Like a polka dot, you know, pink shoe made out of this brand, right?

00:49:39.547 --> 00:49:44.348
And maybe there's only one pink polka dot pair of shoes,

00:49:45.009 --> 00:49:48.329
but Amazon will show other related stuff.

00:49:48.589 --> 00:49:51.350
And you could argue that's just their business model

00:49:51.370 --> 00:49:53.631
because they want to entice you to buy more stuff,

00:49:54.411 --> 00:49:56.651
but it's also just context.

00:49:56.951 --> 00:50:00.012
People want additional information in addition

00:50:00.072 --> 00:50:02.353
to something that specifically answers

00:50:02.693 --> 00:50:04.373
or satisfies their intent.

00:50:04.714 --> 00:50:05.094
Yeah.

00:50:05.474 --> 00:50:11.318
There's also some generosity in providing a bit more around it than just literally what

00:50:11.358 --> 00:50:11.918
you ask for.

00:50:11.958 --> 00:50:12.278
Right.

00:50:12.619 --> 00:50:17.041
It's like the thing you actually ask for is like the bare minimum, basically.

00:50:17.161 --> 00:50:20.884
And it's good to provide something extra with it.

00:50:21.384 --> 00:50:21.504
Yeah.

00:50:21.564 --> 00:50:23.205
If somebody asks you, how are you doing?

00:50:23.265 --> 00:50:24.266
And you say, I'm fine.

00:50:24.906 --> 00:50:26.007
And you just stop there.

00:50:26.908 --> 00:50:27.288
It's weird.

00:50:27.448 --> 00:50:28.829
It's a German way of answering.

00:50:28.849 --> 00:50:30.450
That's a German way of answering.

00:50:34.334 --> 00:50:39.636
And then we're back to these flows and the processes and there's actually a mutual process

00:50:39.696 --> 00:50:40.356
going on.

00:50:40.456 --> 00:50:43.418
And I think that's so fascinating.

00:50:43.438 --> 00:50:43.558
You

00:50:53.303 --> 00:50:58.848
Yeah, it's called cooperative conversation and Grice came up with these, you know, principles

00:50:58.908 --> 00:51:02.972
around cooperative conversation. There is a back and forth, there is a give and take,

00:51:04.433 --> 00:51:12.160
there is a notion of manner and relation and the quality of the content and the amount of content,

00:51:12.180 --> 00:51:14.542
and keeping it very relevant, but...

00:51:15.465 --> 00:51:16.687
making sure you're polite.

00:51:16.947 --> 00:51:19.250
So there's so much that's going on,

00:51:19.710 --> 00:51:23.335
and we have found that to a large extent,

00:51:23.355 --> 00:51:26.959
a lot of those cooperative conversation Maxines

00:51:27.540 --> 00:51:30.623
hold good even in the context of analytical chatbots.

00:51:31.927 --> 00:51:32.468
Yeah.

00:51:32.528 --> 00:51:33.149
That's interesting.

00:51:33.189 --> 00:51:33.349
Yeah.

00:51:33.369 --> 00:51:38.595
Also in the book, you mentioned this idea of register and tone, like in natural language

00:51:38.635 --> 00:51:45.602
is not just what is said, but also how it's said and certain like, yeah, choice of formulations

00:51:45.642 --> 00:51:46.083
of words.

00:51:46.483 --> 00:51:49.827
just a certain maybe social status.

00:51:49.348 --> 00:51:53.896
social status between the participants or you know all these little things between the

00:51:53.936 --> 00:51:54.817
lines basically.

00:51:54.857 --> 00:51:55.198
Yeah.

00:51:55.238 --> 00:51:55.378
Yeah.

00:52:06.247 --> 00:52:12.813
But I've also been very lately interested in, you know, data storytelling and the types of words

00:52:13.053 --> 00:52:19.718
and the choice of words that you use in these stories can also set the tone.

00:52:19.738 --> 00:52:23.402
And if you have any questions, please feel free to contact me.

00:52:22.042 --> 00:52:23.602
It depends on the audience.

00:52:23.722 --> 00:52:27.463
I mean, the running joke when we wrote the book, actually,

00:52:28.583 --> 00:52:30.383
and Bridget will laugh about this,

00:52:30.504 --> 00:52:31.604
is she would always tell me,

00:52:31.684 --> 00:52:33.524
Vidya, you need to lower the register

00:52:33.564 --> 00:52:34.884
when you're writing your chapter

00:52:34.984 --> 00:52:37.765
because this is not just a PhD audience.

00:52:37.825 --> 00:52:40.706
So I appreciated her candid feedback

00:52:40.766 --> 00:52:43.246
and I use it all the time now.

00:52:43.366 --> 00:52:46.847
It's like, you have to figure out who your audience is

00:52:47.127 --> 00:52:50.148
and find the right level of information

00:52:50.228 --> 00:52:51.828
and how that is communicated.

00:52:51.911 --> 00:52:52.794
back to the user.

00:52:54.709 --> 00:52:59.997
So it's both about content, but also again, the form in which the framing, right, the

00:53:00.017 --> 00:53:00.438
It's creamy.

00:53:20.976 --> 00:53:24.978
you know, just giving me a meta table and saying,

00:53:25.078 --> 00:53:26.639
what question would you like to ask?

00:53:26.719 --> 00:53:28.600
That's a very different sort of tone.

00:53:29.020 --> 00:53:32.022
And it really influences how people interact

00:53:32.062 --> 00:53:34.664
with these systems and the systems expectations.

00:53:35.164 --> 00:53:37.625
So Vidya, one thing I'm curious about,

00:53:39.006 --> 00:53:42.448
do you see a role of language as a way to,

00:53:43.409 --> 00:53:47.591
say, help people interpret data visualizations,

00:53:47.671 --> 00:53:51.994
maybe through annotations or some sort of guidance.

00:53:52.555 --> 00:53:56.417
I believe there is, I can't recall if you're doing that type of research, but I've seen

00:53:57.017 --> 00:54:03.819
similar things, either from Tableau or other researchers, and, um, this idea that you can

00:54:03.839 --> 00:54:03.999
generate...

00:54:04.019 --> 00:54:05.840
Accessibility, obviously, right?

00:54:06.000 --> 00:54:10.422
Providing alternative, uh, means of accessing the same info.

00:54:10.602 --> 00:54:10.802
Yeah.

00:54:11.042 --> 00:54:11.362
Right.

00:54:11.582 --> 00:54:19.085
So, um, actually when we were writing the book, um, we had a chapter on, um, you know,

00:54:19.145 --> 00:54:19.685
in charts.

00:54:21.006 --> 00:54:25.209
And, you know, we were curious whether, because, you know, a lot of the practitioners,

00:54:26.169 --> 00:54:30.171
and even we have been told about, you know, this whole data-ink ratio, right? I mean,

00:54:30.491 --> 00:54:35.914
thanks to Tufti and other books. It's like, oh, you know, you don't want to add too much text.

00:54:35.974 --> 00:54:41.638
The charts want to speak for themselves, just add a little bit. And then we did this, you know,

00:54:41.678 --> 00:54:47.701
project, you know, which was a vis paper that was presented by Chase Stokes from UC Berkeley.

00:54:48.302 --> 00:54:50.886
where we we really wanted to understand.

00:54:48.321 --> 00:54:49.362
Thank you.

00:54:50.906 --> 00:54:52.308
Oh yeah, that's a super interesting one.

00:54:52.588 --> 00:54:52.909
Right?

00:54:52.969 --> 00:54:53.069
you

00:54:55.734 --> 00:54:59.897
Is there a notion of too much text is too bad?

00:55:00.458 --> 00:55:03.880
And it turns out, no, people actually love text.

00:55:05.041 --> 00:55:10.286
And quite often, you know, there are so many semantic levels of text, right?

00:55:10.326 --> 00:55:15.130
There's, you know, text that describes very low-level statistical properties, all the

00:55:15.150 --> 00:55:18.793
way to text that describes key takeaways, and, you know, there's been some work that

00:55:18.833 --> 00:55:21.155
I've done with Manish in that space as well.

00:55:22.357 --> 00:55:30.200
But, yeah, to your point, I think there is a very exciting opportunity around figuring

00:55:30.300 --> 00:55:35.642
out how text can be used along with charts as a first-class citizen for people to understand

00:55:35.682 --> 00:55:36.162
their data.

00:55:36.262 --> 00:55:43.985
And they can be pretty effective, text annotations and titles and descriptions as scaffolds to

00:55:44.025 --> 00:55:44.425
the user.

00:55:44.485 --> 00:55:48.647
And this kind of comes back to something I think that Moritz, you were alluding to, that

00:55:48.587 --> 00:55:55.934
at the beginning of this podcast was this cold slate problem where users struggle in

00:55:55.974 --> 00:55:57.856
terms of what they can ask of the data.

00:55:58.636 --> 00:56:04.482
And you can use text in conjunction with recommendations saying, you know what, you asked about this,

00:56:04.802 --> 00:56:08.845
This is the answer, but there are all these other things that you can ask.

00:56:10.348 --> 00:56:16.590
And so, yeah, I think it can really be an effective way of scaffolding the conversation.

00:56:16.710 --> 00:56:23.512
It's also very helpful for answering more complex analytical intents,

00:56:23.572 --> 00:56:27.093
especially these why questions. I think why questions are very hard.

00:56:27.113 --> 00:56:28.773
that we haven't really cracked that nut.

00:56:29.133 --> 00:56:31.194
It's like, why is this phenomena happening?

00:56:32.695 --> 00:56:38.338
If I am looking at the pound and U.S. dollar fluctuation around Brexit and I say,

00:56:38.618 --> 00:56:44.580
why is this drop happening, the data does not really have an answer and I have to go outside,

00:56:45.001 --> 00:56:50.323
you know, probably use natural language, go to access web articles and come up with

00:56:51.185 --> 00:56:54.609
a summary of my understanding of what happened and bring that

00:56:54.669 --> 00:56:58.313
back and formulate a very pithy annotation to the user.

00:56:59.094 --> 00:57:03.379
So supporting these why questions is going to be a place where

00:57:03.440 --> 00:57:07.725
text is definitely going to play a very important role.

00:57:09.208 --> 00:57:14.310
How do you feel about the whole new generation of deep learning-based models in this context

00:57:14.350 --> 00:57:16.370
like GPT-3, the transformer model?

00:57:16.410 --> 00:57:23.013
So there's a whole – seems to be a whole paradigm shift in that space and like GPT-3

00:57:23.093 --> 00:57:28.495
is a large language model trained on all of the internet as it seems, that seems to be

00:57:28.595 --> 00:57:31.436
able to answer like a lot of these common sense...

00:57:31.636 --> 00:57:31.996
Yeah.

00:57:42.733 --> 00:57:45.795
What's your feeling?

00:57:45.855 --> 00:57:54.299
Language models have really come a long way over the recent years, and GPT-3 has been

00:57:54.479 --> 00:58:00.582
extremely promising for answering and learning certain types of intents and questions and

00:58:00.602 --> 00:58:06.245
being able to reason about information and even just learning stuff that goes above and

00:58:06.285 --> 00:58:08.126
beyond the initial training set.

00:58:09.528 --> 00:58:14.229
What I have observed, though, with these large language

00:58:14.289 --> 00:58:19.151
models when it comes to data analysis and data exploration

00:58:19.251 --> 00:58:23.013
is that their capabilities around numeracy

00:58:23.953 --> 00:58:26.974
and understanding the numeracy aspects of data

00:58:27.434 --> 00:58:28.475
is still very limited.

00:58:31.437 --> 00:58:35.440
I am sure with systems like DALI, the graphicacy aspect

00:58:35.680 --> 00:58:38.182
might be teachable at some point.

00:58:38.282 --> 00:58:40.403
We might get to a point where we ask a question,

00:58:40.463 --> 00:58:42.545
and it'll learn to generate an effective chart.

00:58:44.226 --> 00:58:48.909
So DALI generates images based on prompts, actual texts.

00:58:49.170 --> 00:58:53.052
But the numerical understanding of data phenomena

00:58:53.393 --> 00:58:55.514
is still a hard problem to learn,

00:58:55.594 --> 00:58:57.215
because it's extremely nuanced.

00:58:57.235 --> 00:58:58.216
You

00:58:59.298 --> 00:59:02.839
But I don't know, it might head to a place where it is.

00:59:02.959 --> 00:59:08.941
So that being said, being able to take these traditional

00:59:09.021 --> 00:59:12.722
grammar-based approaches that I have used as well in a lot of

00:59:12.782 --> 00:59:17.224
my research and retrofitting it with large language models.

00:59:17.885 --> 00:59:26.392
can satisfy a good number of analytical intents. But at this point, I have found that you still

00:59:26.472 --> 00:59:33.077
have to augment these language models with additional logic that might come from these

00:59:33.357 --> 00:59:40.443
grammars or heuristics to solve numerical understanding of data that these models simply

00:59:41.164 --> 00:59:44.907
are unable to do or can do in a limited fashion.

00:59:45.529 --> 00:59:45.809
Yeah.

00:59:46.209 --> 00:59:46.390
Yeah.

00:59:46.850 --> 00:59:46.970
Yeah.

00:59:46.990 --> 00:59:54.317
But they could add that aspect of providing somebody hints with when does the gold price

00:59:54.417 --> 00:59:56.359
drop generally, right?

00:59:56.499 --> 01:00:01.363
And might that be related to that specific piece of data I have here?

01:00:01.383 --> 01:00:01.684
That's right.

01:00:02.144 --> 01:00:02.524
You're right.

01:00:02.564 --> 01:00:05.647
Probably the synthesis needs to happen.

01:00:05.888 --> 01:00:06.088
Yeah.

01:00:06.750 --> 01:00:08.091
So I think that is where we are,

01:00:08.351 --> 01:00:11.552
but if we talk in a couple of years,

01:00:11.792 --> 01:00:12.733
it might be different.

01:00:12.793 --> 01:00:15.474
I mean, this area is pretty cutting edge

01:00:15.534 --> 01:00:17.615
and things have been evolving quite quickly,

01:00:17.675 --> 01:00:18.495
which is exciting.

01:00:19.996 --> 01:00:20.196
Yeah.

01:00:19.997 --> 01:00:24.561
Yeah, this is a space where I feel like something's going to happen, something's

01:00:24.641 --> 01:00:25.962
going to happen, right?

01:00:25.982 --> 01:00:29.445
It's like, you can almost feel it in the air, right?

01:00:29.505 --> 01:00:29.805
Yeah.

01:00:30.466 --> 01:00:32.868
So it's, or maybe not, but I don't know.

01:00:32.888 --> 01:00:34.169
Yeah, it's fascinating.

01:00:34.189 --> 01:00:39.153
Yeah, because as you say, it feels like, oh, there must be like an amazing application

01:00:39.173 --> 01:00:39.793
for all this stuff.

01:00:39.834 --> 01:00:40.494
Yeah.

01:00:44.961 --> 01:00:49.285
know that you need, but you just get stuck at the 95, right?

01:00:49.506 --> 01:00:54.711
It's like, it's a bit of smoking mirrors, but I will say that, you know, these language models

01:00:55.352 --> 01:01:00.297
have surpassed the initial notion of just being smoking mirrors. They actually

01:01:00.357 --> 01:01:01.498
worked pretty well for...

01:01:09.400 --> 01:01:15.403
I think the challenge and the opportunity is to get them to really understand data and

01:01:15.943 --> 01:01:18.464
just the numerical understanding of data.

01:01:19.485 --> 01:01:27.809
But as I indicated, I think it's a very tractable problem because, you know, it's already, you

01:01:27.990 --> 01:01:35.814
figure it out to understand the knowledge of the world, right? So when you know certain properties

01:01:35.894 --> 01:01:43.098
and attributes of the data, you know, figuring out the numerical understanding of concepts

01:01:43.839 --> 01:01:45.840
It's probably teachable.

01:01:46.021 --> 01:01:46.141
Yeah.

01:01:46.561 --> 01:01:50.445
And this combination of systems is interesting, I think, to say, like, maybe there's different

01:01:51.185 --> 01:01:55.809
modules or agents that keep each other in check, or like, there's a, like a statistical

01:01:55.869 --> 01:02:02.934
like professor module that makes sure all the outputs are statistically correct.

01:02:03.015 --> 01:02:03.475
That's right.

01:02:03.735 --> 01:02:07.958
Otherwise sends a new prompt that forces the model to update or something like that.

01:02:07.978 --> 01:02:08.239
Yeah.

01:02:10.844 --> 01:02:11.485
Fascinating.

01:02:11.725 --> 01:02:15.525
I thought maybe we could wrap it up with our last question for Vidya.

01:02:16.226 --> 01:02:19.766
Maybe you can tell our listeners if they want to learn more about

01:02:21.187 --> 01:02:23.747
how to use language in data visualization,

01:02:23.807 --> 01:02:26.208
what are the interesting directions?

01:02:26.508 --> 01:02:28.868
And I would say even practically, right?

01:02:28.988 --> 01:02:33.149
Say I'm a database designer practitioner listening to this.

01:02:34.229 --> 01:02:39.070
How do I learn to use, to even be more aware of the role of language

01:02:39.510 --> 01:02:40.691
Edge in data visualization.

01:02:41.051 --> 01:02:46.796
Yeah. I will say your book, I guess there was an easy one. Just read the book because the book

01:03:00.128 --> 01:03:06.236
you know, various language disciplines, starting from kind of traditional NLP and information

01:03:06.296 --> 01:03:11.122
literature all the way to American Sign Language, because that's Bridget's background.

01:03:11.142 --> 01:03:13.185
That's fascinating too, by the way.

01:03:13.365 --> 01:03:13.885
Right, right.

01:03:22.354 --> 01:03:28.717
some of the recent work that's come in the space. I mean, there's a sizable number of papers in the

01:03:28.757 --> 01:03:36.340
space of language and visual analysis that is being published at conferences like VIZ and CHI,

01:03:37.201 --> 01:03:39.522
even Euroviz, so that's another

01:03:39.742 --> 01:03:40.402
recommendation.

01:03:42.984 --> 01:03:49.389
A lot of people ask me about just kind of core NLP stuff, but there's plenty of courses

01:03:49.469 --> 01:03:53.953
on Coursera to just understand language models and intent.

01:03:54.373 --> 01:04:00.178
So if people are very serious about implementing or trying out natural language algorithms,

01:04:00.498 --> 01:04:02.119
There are Python libraries.

01:04:02.741 --> 01:04:08.003
There's course lectures, including, you know, some really good ones by Chris Manning from Stanford.

01:04:08.804 --> 01:04:16.367
So I would start with those and then kind of work one's way down to reading the state-of-the-art

01:04:16.427 --> 01:04:19.829
literature in this space concerning visual analysis.

01:04:20.629 --> 01:04:25.151
Cool. It's a whole world out there to discover. It's one of these things once you open that box.

01:04:25.932 --> 01:04:27.492
It's a Pandora's box, right?

01:04:27.772 --> 01:04:28.573
Fits into it.

01:04:28.953 --> 01:04:29.253
Yeah.

01:04:30.174 --> 01:04:30.955
Yeah.

01:04:32.015 --> 01:04:32.296
Yeah.

01:04:33.636 --> 01:04:34.997
But a really fascinating one.

01:04:35.037 --> 01:04:39.540
And I think one that has been underappreciated, like everybody's just focused on the visuals

01:04:39.600 --> 01:04:46.045
and the visual encoding and perception and the language is more than half of what we

01:04:46.085 --> 01:04:46.465
do, right?

01:04:46.485 --> 01:04:50.508
Like a chart with zero labels is nothing, right?

01:04:50.528 --> 01:04:53.650
So it needs that language content.

01:04:53.631 --> 01:04:57.734
Yeah, exactly. And what I tell people is when people think about NLP and Viz,

01:04:57.934 --> 01:05:01.197
they always almost think about chatbots and talking with data.

01:05:01.818 --> 01:05:05.501
But there's so many other opportunities in the analytical workflow

01:05:05.961 --> 01:05:08.343
where natural language processing can help.

01:05:08.863 --> 01:05:12.126
It can help with intelligent data transformations under the hood.

01:05:12.766 --> 01:05:16.710
It can help with joining of tables, especially semantic joins,

01:05:16.810 --> 01:05:21.313
where the columns of values may not be identical, but they are related.

01:05:21.333 --> 01:05:22.795
Thank you.

01:05:22.596 --> 01:05:27.999
And, you know, coming up with meaningful encodings for data.

01:05:28.079 --> 01:05:34.302
So there's so many sub-processes as part of that analytical workflow, where semantics

01:05:34.922 --> 01:05:40.785
and language understanding is very useful, in addition to kind of the more obvious applications

01:05:40.845 --> 01:05:41.686
that we talked about.

01:05:41.826 --> 01:05:43.627
So you're not going to get bored anytime soon?

01:05:44.147 --> 01:05:46.749
No, no, I'm going strong here.

01:05:48.250 --> 01:05:48.650
Awesome.

01:05:48.670 --> 01:05:51.471
Yeah.

01:05:51.971 --> 01:05:54.371
No, it's, I think that's really great advice.

01:05:54.451 --> 01:05:58.692
And yeah, hopefully our listeners will now always look for the conversations and the

01:05:58.712 --> 01:06:04.273
dialogues and the register and the tone, because once you've seen it, you can't unsee it.

01:06:04.953 --> 01:06:05.613
Exactly.

01:06:05.753 --> 01:06:06.373
Exactly.

01:06:07.694 --> 01:06:07.994
Yeah.

01:06:08.134 --> 01:06:08.594
Wonderful.

01:06:08.694 --> 01:06:09.094
Yeah.

01:06:09.154 --> 01:06:11.374
That was really nice.

01:06:11.754 --> 01:06:17.315
for shedding light on all this fascinating stuff and especially this

01:06:18.336 --> 01:06:23.260
It was a great talking about it with you both.

01:06:23.500 --> 01:06:25.081
I enjoyed the conversation.

01:06:25.441 --> 01:06:25.881
Wonderful.

01:06:25.962 --> 01:06:29.184
And we'll check back in a few years how things have developed, right?

01:06:31.205 --> 01:06:31.725
Wonderful.

01:06:31.786 --> 01:06:32.286
That's right.

01:06:33.086 --> 01:06:33.547
Okay.

01:06:33.587 --> 01:06:34.167
Thanks so much.

01:06:35.108 --> 01:06:35.568
See you soon.

01:06:35.668 --> 01:06:35.868
Yeah.

01:06:35.908 --> 01:06:37.289
Thank you so much for having me.

01:06:37.629 --> 01:06:39.331
And I really enjoyed it.

01:06:39.891 --> 01:06:44.934
And I look forward to listening to it and all the other podcasts that you come out with.

01:06:44.974 --> 01:06:45.815
So I'm a big fan.

01:06:45.995 --> 01:06:46.876
So thank you very much.

01:06:46.997 --> 01:06:47.438
Thank you.

01:06:47.559 --> 01:06:48.220
Thanks so much.

01:06:48.240 --> 01:06:48.982
Thank you.

01:06:49.042 --> 01:06:49.844
All right, take care.

01:06:50.045 --> 01:06:50.266
Bye.

01:06:56.334 --> 01:06:58.576
Hey, folks, thanks for listening to Data Stories again.

01:06:59.316 --> 01:07:00.938
Before you leave, a few last notes.

01:07:01.238 --> 01:07:03.760
This show is crowdfunded, and you can support us

01:07:03.840 --> 01:07:06.942
on Patreon at patreon.com slash datastories,

01:07:07.083 --> 01:07:10.065
where we publish monthly previews of upcoming episodes

01:07:10.105 --> 01:07:10.986
for our supporters.

01:07:11.786 --> 01:07:14.048
Or you can also send us a one-time donation

01:07:14.429 --> 01:07:17.651
via paypal at paypal.me slash datastories.

01:07:18.433 --> 01:07:20.254
Or, as a free way to support the show,

01:07:20.654 --> 01:07:23.335
if you can spend a couple of minutes rating us on iTunes,

01:07:23.395 --> 01:07:24.995
that would be very helpful as well.

01:07:26.075 --> 01:07:28.016
And here's some information on the many ways

01:07:28.056 --> 01:07:29.536
you can get news directly from us.

01:07:30.017 --> 01:07:32.117
We are on Twitter, Facebook, and Instagram,

01:07:32.557 --> 01:07:34.658
so follow us there for the latest updates.

01:07:35.238 --> 01:07:36.959
We have also a Slack channel

01:07:37.359 --> 01:07:38.999
where you can chat with us directly.

01:07:39.599 --> 01:07:43.441
And to sign up, go to our homepage at datastory.es,

01:07:43.961 --> 01:07:46.381
and there you'll find a button at the bottom of the page.

01:07:46.922 --> 01:07:51.187
And there you can also subscribe to our email newsletter if you want to get news directly

01:07:51.227 --> 01:07:54.551
into your inbox and be notified whenever we publish a new episode.

01:07:54.791 --> 01:07:58.875
That's right. And we love to get in touch with our listeners. So let us know if you want to

01:07:58.935 --> 01:08:03.500
suggest a way to improve the show or know any amazing people you want us to invite

01:08:04.121 --> 01:08:07.124
or even have any project you want us to talk about.

01:08:07.507 --> 01:08:13.713
Yeah, absolutely. Don't hesitate to get in touch. Just send us an email at mail at datastory.es.

01:08:14.234 --> 01:08:16.016
That's all for now. See you next time,

01:08:16.216 --> 01:08:17.978
and thanks for listening to Data Stories.