{"version":"1.0.0","segments":[{"speaker":null,"startTime":0.08,"endTime":7.492,"body":"People want additional information in addition to something that specifically answers or satisfies their intent."},{"speaker":null,"startTime":27.745,"endTime":28.366,"body":"Hi, everyone."},{"speaker":null,"startTime":28.426,"endTime":30.528,"body":"Welcome to a new episode of Data Stories."},{"speaker":null,"startTime":30.93,"endTime":36.332,"body":"My name is Moritz Stefaner. I'm an independent designer of data visualizations. In fact, I work"},{"speaker":null,"startTime":36.372,"endTime":41.373,"body":"as a self-employed truth and beauty operator out of my office here in the beautiful countryside"},{"speaker":null,"startTime":41.433,"endTime":47.054,"body":"in the north of Germany. And I am Enrico Bertini. I am an associate professor at Northeastern"},{"speaker":null,"startTime":47.114,"endTime":53.116,"body":"University in Boston, where I do research and teach data visualization. Exactly. And on this"},{"speaker":null,"startTime":53.156,"endTime":59.298,"body":"podcast together, we talk about data visualization, data analysis, and generally the role data plays"},{"speaker":null,"startTime":59.338,"endTime":59.938,"body":"in our lives."},{"speaker":null,"startTime":60.358,"endTime":60.818,"body":"Thank you."},{"speaker":null,"startTime":60.798,"endTime":63.02,"body":"Usually we do that with a guest we invite on the show."},{"speaker":null,"startTime":63.341,"endTime":66.883,"body":"Yes, but before we start with our guest, a quick note."},{"speaker":null,"startTime":67.524,"endTime":71.427,"body":"Our podcast is listener supported, so there's no ads."},{"speaker":null,"startTime":71.467,"endTime":77.532,"body":"If you enjoy the show, please consider supporting us with recurring payments on patreon.com"},{"speaker":null,"startTime":77.732,"endTime":78.893,"body":"slash datastories."},{"speaker":null,"startTime":79.714,"endTime":83.397,"body":"Or if you prefer, you can also send one-time donations on PayPal."},{"speaker":null,"startTime":84.159,"endTime":90.184,"body":"by following the link on to paypal.me slash data stories."},{"speaker":null,"startTime":90.724,"endTime":91.605,"body":"Yeah, in the last few days,"},{"speaker":null,"startTime":91.825,"endTime":94.166,"body":"we've just seen a few new donations coming in."},{"speaker":null,"startTime":94.306,"endTime":96.368,"body":"That's always a great feeling and always wonderful,"},{"speaker":null,"startTime":96.468,"endTime":97.369,"body":"much appreciated,"},{"speaker":null,"startTime":97.789,"endTime":100.291,"body":"especially if you send us a little funny note along with it."},{"speaker":null,"startTime":100.431,"endTime":101.452,"body":"It always makes us smile."},{"speaker":null,"startTime":101.532,"endTime":103.033,"body":"Yes, thank you, thank you."},{"speaker":null,"startTime":103.053,"endTime":104.854,"body":"Yeah, and also if you can afford it"},{"speaker":null,"startTime":104.954,"endTime":106.735,"body":"and do the recurring thing on Patreon,"},{"speaker":null,"startTime":106.755,"endTime":108.116,"body":"this definitely keeps us going"},{"speaker":null,"startTime":108.216,"endTime":111.679,"body":"and it's again much appreciated."},{"speaker":null,"startTime":112.881,"endTime":120.086,"body":"So let's get started. We have a cool topic today. We'll talk about language and data visualization"},{"speaker":null,"startTime":120.466,"endTime":127.61,"body":"and all that surrounds that. But before we dive into that, quick note on the Informationist"},{"speaker":null,"startTime":127.69,"endTime":132.473,"body":"Beautiful Awards. So some of you might remember, we had Amanda McCulloch from the Data Visualization"},{"speaker":null,"startTime":132.493,"endTime":135.916,"body":"in Society on the show a few weeks, months, can't remember, ago."},{"speaker":null,"startTime":137.178,"endTime":142.142,"body":"And we mentioned that the awards are being rebooted, they're a bit like the mini Oscars"},{"speaker":null,"startTime":142.322,"endTime":149.167,"body":"of data visualization, and really try to celebrate all the diversity and variety and excellence"},{"speaker":null,"startTime":149.247,"endTime":149.807,"body":"in the field."},{"speaker":null,"startTime":149.887,"endTime":151.469,"body":"And now the winners are announced."},{"speaker":null,"startTime":151.649,"endTime":153.53,"body":"We didn't win anything, Enrico, right?"},{"speaker":null,"startTime":153.99,"endTime":155.011,"body":"Yeah."},{"speaker":null,"startTime":155.091,"endTime":157.873,"body":"But a lot of other great people, which is awesome, too."},{"speaker":null,"startTime":159.636,"endTime":162.418,"body":"And yeah, on the website you can see all the winners."},{"speaker":null,"startTime":163.299,"endTime":166.962,"body":"Shout out to our podcast colleague Ali Turvan, who won Community Leader."},{"speaker":null,"startTime":167.182,"endTime":168.002,"body":"Much deserved."},{"speaker":null,"startTime":168.042,"endTime":168.663,"body":"Congratulations."},{"speaker":null,"startTime":169.323,"endTime":170.464,"body":"Lots of other great people."},{"speaker":null,"startTime":170.644,"endTime":172.906,"body":"Yeah, it's great to see this event is back."},{"speaker":null,"startTime":173.146,"endTime":173.887,"body":"I'm here."},{"speaker":null,"startTime":173.907,"endTime":174.207,"body":"Yeah."},{"speaker":null,"startTime":174.347,"endTime":177.089,"body":"It's nice to have a celebration, you know?"},{"speaker":null,"startTime":184.956,"endTime":191.481,"body":"Yeah, but let's go to the main topic. Our guest today is Vidya Settler. Hi,"},{"speaker":null,"startTime":191.501,"endTime":195.984,"body":"Vidya. Thanks for joining us. Hi, Moritz and Enrico. Thanks for having me."},{"speaker":null,"startTime":196.484,"endTime":202.149,"body":"Yeah, welcome on the show. And thanks so much for joining us. Can you tell us a bit about you?"},{"speaker":null,"startTime":202.669,"endTime":206.512,"body":"What's your background? What are you excited about? What are you currently working on?"},{"speaker":null,"startTime":207.394,"endTime":214.699,"body":"Sure. So I'm Vidya Settler. I head Tableau Research. I've been with Tableau for 10 years."},{"speaker":null,"startTime":215.74,"endTime":221.224,"body":"I got a PhD from Northwestern. My background is in natural language processing and computer"},{"speaker":null,"startTime":221.264,"endTime":226.928,"body":"graphics. So I'm generally interested in the problem space of understanding the semantics"},{"speaker":null,"startTime":227.008,"endTime":233.193,"body":"of data and how that can help inform meaningful visual depiction of information. So Tableau has"},{"speaker":null,"startTime":233.233,"endTime":233.593,"body":"has been."},{"speaker":null,"startTime":233.835,"endTime":236.976,"body":"the perfect place for leveraging that skill set."},{"speaker":null,"startTime":237.997,"endTime":239.718,"body":"Before I joined Tableau,"},{"speaker":null,"startTime":239.858,"endTime":242.5,"body":"I was at Nokia Research for seven years."},{"speaker":null,"startTime":243.54,"endTime":245.761,"body":"I manage a wonderful team"},{"speaker":null,"startTime":245.821,"endTime":247.783,"body":"of interdisciplinary research scientists"},{"speaker":null,"startTime":247.863,"endTime":251.425,"body":"who work on problems in the space of applied ML,"},{"speaker":null,"startTime":251.685,"endTime":254.727,"body":"multimodal interfaces, HCI techniques,"},{"speaker":null,"startTime":255.067,"endTime":256.708,"body":"augmented reality, and so forth."},{"speaker":null,"startTime":258.028,"endTime":260.01,"body":"Already back in the day, augmented reality."},{"speaker":null,"startTime":260.69,"endTime":262.571,"body":"I know, it's all coming back full circle."},{"speaker":null,"startTime":263.392,"endTime":264.013,"body":"I know."},{"speaker":null,"startTime":264.033,"endTime":267.374,"body":"It's all coming back full circle."},{"speaker":null,"startTime":267.394,"endTime":267.795,"body":"Oh yeah."},{"speaker":null,"startTime":267.815,"endTime":268.115,"body":"Oh yeah."},{"speaker":null,"startTime":268.135,"endTime":268.235,"body":"Yeah."},{"speaker":null,"startTime":268.255,"endTime":272.977,"body":"It was nice too today is that we have a true long distance call all around the world."},{"speaker":null,"startTime":273.637,"endTime":277.079,"body":"Enrico is on the east coast in the US, I'm in Europe."},{"speaker":null,"startTime":277.239,"endTime":279.02,"body":"Where are you based right now?"},{"speaker":null,"startTime":279.201,"endTime":282.222,"body":"Right now I'm actually in Mumbai in India."},{"speaker":null,"startTime":283.002,"endTime":285.762,"body":"I'm teaching at the GEO Institute,"},{"speaker":null,"startTime":286.722,"endTime":289.303,"body":"teaching a course on data visualization,"},{"speaker":null,"startTime":289.863,"endTime":294.044,"body":"but incorporating aspects of semantics and intent,"},{"speaker":null,"startTime":294.164,"endTime":297.324,"body":"some of the NLP with data visualization stuff"},{"speaker":null,"startTime":297.924,"endTime":300.005,"body":"to a class of 60 students."},{"speaker":null,"startTime":300.065,"endTime":302.065,"body":"So it's been a really great experience."},{"speaker":null,"startTime":302.985,"endTime":303.685,"body":"Nice, yeah."},{"speaker":null,"startTime":304.005,"endTime":305.966,"body":"So that's a little break from your usual"},{"speaker":null,"startTime":306.086,"endTime":307.786,"body":"like research work at Tableau."},{"speaker":null,"startTime":308.627,"endTime":316.856,"body":"Sort of. I mean, I'm still kind of doing both, but it's been really refreshing to interact with the"},{"speaker":null,"startTime":317.116,"endTime":324.743,"body":"local population. And for me, it's just a way of sharing what I have learned over the years"},{"speaker":null,"startTime":324.884,"endTime":329.769,"body":"with people from my home country, India, and"},{"speaker":null,"startTime":330.431,"endTime":335.492,"body":"just helping them understand the value of data visualization and why it's important."},{"speaker":null,"startTime":335.572,"endTime":343.174,"body":"So, I feel like India has reached that stage now where everybody understands the value of data visualization,"},{"speaker":null,"startTime":343.294,"endTime":349.195,"body":"but they don't necessarily have the training and the access to resources."},{"speaker":null,"startTime":349.635,"endTime":350.175,"body":"Um, so"},{"speaker":null,"startTime":357.267,"endTime":363.292,"body":"Yeah, there's so much going on in India. I actually visited last summer."},{"speaker":null,"startTime":363.373,"endTime":367.436,"body":"And for the first time, yes, for a wedding. So for an Indian wedding."},{"speaker":null,"startTime":367.496,"endTime":370.418,"body":"So that was really, really special."},{"speaker":null,"startTime":370.458,"endTime":371.739,"body":"It's a good way to get started."},{"speaker":null,"startTime":371.9,"endTime":374.942,"body":"The full package."},{"speaker":null,"startTime":380.485,"endTime":388.728,"body":"Yes, exactly. So Vidya, I think one reason why we wanted to have you on the show is because"},{"speaker":null,"startTime":388.808,"endTime":395.37,"body":"you recently published this book together with Bridget Cogley called Functional Aesthetics."},{"speaker":null,"startTime":396.05,"endTime":402.492,"body":"And reading these books, me and Moritz realized that there's a lot in the book that is related"},{"speaker":null,"startTime":402.552,"endTime":406.173,"body":"to semantics and language, which interestingly is not."},{"speaker":null,"startTime":406.954,"endTime":412.736,"body":"that much, discussed that much in the world of data visualization, right, and data analysis as"},{"speaker":null,"startTime":412.796,"endTime":420.599,"body":"well. So we thought we would focus this specific episode on the relationship between semantics and"},{"speaker":null,"startTime":420.639,"endTime":427.701,"body":"language and data visualization. So maybe we can start by exploring or understanding what are the"},{"speaker":null,"startTime":427.781,"endTime":432.743,"body":"main connections between data visualization and language and how did you get into this specific"},{"speaker":null,"startTime":432.803,"endTime":433.123,"body":"topic."},{"speaker":null,"startTime":433.844,"endTime":439.39,"body":"Yeah, I honestly got into this topic kind of serendipitously."},{"speaker":null,"startTime":441.394,"endTime":450.582,"body":"In general, I think information is meaningful when you understand what it's about, the context of it,"},{"speaker":null,"startTime":451.863,"endTime":457.708,"body":"it gets enriched by additional information, and when you have a better understanding of"},{"speaker":null,"startTime":457.748,"endTime":463.553,"body":"that information, you can figure out meaningful ways of communicating what that information is"},{"speaker":null,"startTime":463.573,"endTime":463.973,"body":"it's about."},{"speaker":null,"startTime":464.616,"endTime":469.039,"body":"And given that we are very strong visual creatures,"},{"speaker":null,"startTime":469.459,"endTime":475.323,"body":"a very logical place to start is conveying that information visually to a user."},{"speaker":null,"startTime":476.263,"endTime":480.006,"body":"So a lot of my previous work before I joined Tableau"},{"speaker":null,"startTime":480.406,"endTime":482.347,"body":"was exploring the space of."},{"speaker":null,"startTime":484.17,"endTime":493.736,"body":"figuring out how to effectively communicate large pixel, megapixel imagery on small mobile screen devices."},{"speaker":null,"startTime":493.816,"endTime":498.699,"body":"This was a time when smartphones were big and that was primarily the work I did at Nokia."},{"speaker":null,"startTime":499.881,"endTime":506.886,"body":"And there are a lot of connections to data visualization from the graphics space."},{"speaker":null,"startTime":507.486,"endTime":513.41,"body":"In a lot of respects, I would say, it's a much more tractable problem because in the"},{"speaker":null,"startTime":513.45,"endTime":519.354,"body":"In the graphics world, you're trying to find semantics within pixels of information."},{"speaker":null,"startTime":520.236,"endTime":521.836,"body":"But with data visualization,"},{"speaker":null,"startTime":521.916,"endTime":524.037,"body":"there's a lot more structure and"},{"speaker":null,"startTime":524.057,"endTime":526.998,"body":"built-in semantics already that is provided."},{"speaker":null,"startTime":527.478,"endTime":528.799,"body":"We know the chart type,"},{"speaker":null,"startTime":528.839,"endTime":529.819,"body":"we know the mark type,"},{"speaker":null,"startTime":529.919,"endTime":531.099,"body":"we know the attributes,"},{"speaker":null,"startTime":531.139,"endTime":534.18,"body":"we might know the attribute types."},{"speaker":null,"startTime":534.52,"endTime":536.601,"body":"So there's a lot that we can start off with."},{"speaker":null,"startTime":536.941,"endTime":542.523,"body":"So it really gives an extra foothold in terms of"},{"speaker":null,"startTime":542.583,"endTime":545.084,"body":"coming up with semantically meaningful ways"},{"speaker":null,"startTime":545.184,"endTime":546.644,"body":"of depicting that information."},{"speaker":null,"startTime":547.785,"endTime":555.148,"body":"So I kind of started my journey at Tableau exploring ways in which we could provide smarter"},{"speaker":null,"startTime":555.208,"endTime":562.791,"body":"defaults to data visualization. So we noticed that on Tableau Public, users spent a lot of time"},{"speaker":null,"startTime":563.972,"endTime":568.053,"body":"going to the web, trying to find images, creating icons."},{"speaker":null,"startTime":568.795,"endTime":572.198,"body":"and associating them with sports visualizations,"},{"speaker":null,"startTime":572.258,"endTime":573.759,"body":"with their favorite sports teams,"},{"speaker":null,"startTime":573.859,"endTime":577.022,"body":"or flags, or other types of icons."},{"speaker":null,"startTime":577.763,"endTime":580.245,"body":"So, this was a project that I started with"},{"speaker":null,"startTime":580.305,"endTime":583.167,"body":"Jock McKinlay when I joined Tableau is,"},{"speaker":null,"startTime":583.628,"endTime":586.35,"body":"can we actually figure out a way to use"},{"speaker":null,"startTime":586.731,"endTime":589.833,"body":"natural language techniques to effectively query"},{"speaker":null,"startTime":590.494,"endTime":592.075,"body":"image databases to"},{"speaker":null,"startTime":592.157,"endTime":597.741,"body":"get back these images and use graphics techniques to come up with a visually consistent,"},{"speaker":null,"startTime":598.282,"endTime":604.606,"body":"yet aesthetic palette of icons and automatically suggest that to the user. So these were these"},{"speaker":null,"startTime":604.787,"endTime":611.091,"body":"little, you know, micro problems along the flow of analysis where I was exploring how can we"},{"speaker":null,"startTime":611.151,"endTime":616.795,"body":"provide reasonable defaults to reduce the friction so that people don't have to step out of their"},{"speaker":null,"startTime":617.276,"endTime":617.856,"body":"workflow."},{"speaker":null,"startTime":618.378,"endTime":621.78,"body":"do these manually intensive tasks and come back."},{"speaker":null,"startTime":622.281,"endTime":624.142,"body":"And so that's where I started."},{"speaker":null,"startTime":624.622,"endTime":625.843,"body":"And then moving on,"},{"speaker":null,"startTime":625.903,"endTime":629.886,"body":"I started exploring natural language interaction with data,"},{"speaker":null,"startTime":631.327,"endTime":633.769,"body":"which we can talk about in more detail in a bit."},{"speaker":null,"startTime":635.17,"endTime":639.473,"body":"But there is this general structure of"},{"speaker":null,"startTime":639.693,"endTime":643.435,"body":"using language and grammar for both defining charts,"},{"speaker":null,"startTime":643.456,"endTime":644.096,"body":"you"},{"speaker":null,"startTime":644.037,"endTime":651.386,"body":"and also asking questions of charts because at the end of the day data visualizations are a"},{"speaker":null,"startTime":651.446,"endTime":656.432,"body":"manifestation of visual communication. So communication is the underlying theme"},{"speaker":null,"startTime":656.953,"endTime":660.377,"body":"and the best way to think about communication is through language."},{"speaker":null,"startTime":660.82,"endTime":664.587,"body":"Yeah, I think that's, yeah, there's so many cross connections there, right? It's like,"},{"speaker":null,"startTime":675.492,"endTime":678.254,"body":"Have a conversation."},{"speaker":null,"startTime":678.274,"endTime":679.855,"body":"Even the metaphors that we use."},{"speaker":null,"startTime":680.035,"endTime":680.856,"body":"Yeah, exactly."},{"speaker":null,"startTime":680.876,"endTime":685.079,"body":"And that all shows us already that implicitly this is happening."},{"speaker":null,"startTime":685.119,"endTime":689.082,"body":"And I think what's fascinating is that your work makes it explicit and really creates"},{"speaker":null,"startTime":689.142,"endTime":695.767,"body":"that tight link between language and visuals and it's very, very interesting."},{"speaker":null,"startTime":695.867,"endTime":701.631,"body":"And I guess EVISA is a project that was sort of seminal there, both for you personally,"},{"speaker":null,"startTime":701.671,"endTime":703.772,"body":"but then also probably for the field."},{"speaker":null,"startTime":703.792,"endTime":703.892,"body":"Yeah."},{"speaker":null,"startTime":704.314,"endTime":709.816,"body":"For the first time allowing, in the paper it says, it's a prototype system that enables"},{"speaker":null,"startTime":709.876,"endTime":713.858,"body":"a user to have a conversation with their data using natural language."},{"speaker":null,"startTime":714.178,"endTime":717.02,"body":"That sounds like really exciting."},{"speaker":null,"startTime":718.56,"endTime":721.722,"body":"Can you tell us a bit about it, how it came about and how it developed?"},{"speaker":null,"startTime":722.483,"endTime":723.664,"body":"Yeah, for sure."},{"speaker":null,"startTime":724.164,"endTime":731.506,"body":"So the name, actually, Eviza is inspired by the MIT chatbot"},{"speaker":null,"startTime":731.526,"endTime":734.668,"body":"assistant, Eliza."},{"speaker":null,"startTime":735.268,"endTime":739.81,"body":"And I love the project so much that my vanity license plate"},{"speaker":null,"startTime":739.85,"endTime":740.91,"body":"is also called Eviza."},{"speaker":null,"startTime":740.99,"endTime":747.052,"body":"So true Bay Area techie at heart."},{"speaker":null,"startTime":747.793,"endTime":749.813,"body":"You should send us a picture."},{"speaker":null,"startTime":750.334,"endTime":758.039,"body":"put it in the blog post. Yeah. And I, I bump into people randomly at grocery stores saying,"},{"speaker":null,"startTime":758.099,"endTime":762.362,"body":"Oh, I saw your car in the parking lot, so you must be somewhere in the store. Um, so it's great."},{"speaker":null,"startTime":762.382,"endTime":773.069,"body":"Yeah. So, um, I, the, the underlying premise of what I think, what made, uh, advisor interesting"},{"speaker":null,"startTime":773.129,"endTime":773.389,"body":"was."},{"speaker":null,"startTime":774.871,"endTime":781.313,"body":"Communicating with data and asking questions of data using natural language was not a new problem."},{"speaker":null,"startTime":781.594,"endTime":785.475,"body":"People have done it for years. I mean the database community started with it"},{"speaker":null,"startTime":785.975,"endTime":788.216,"body":"where they had these pseudo-SQL statements"},{"speaker":null,"startTime":788.296,"endTime":789.976,"body":"and they were asking questions of their data."},{"speaker":null,"startTime":791.478,"endTime":799.101,"body":"What made EVISA unique, and I want to acknowledge that it was a collaboration with other members"},{"speaker":null,"startTime":799.181,"endTime":807.524,"body":"of the research team, was we were really looking at not just single queries to data and getting"},{"speaker":null,"startTime":807.564,"endTime":808.445,"body":"back a response."},{"speaker":null,"startTime":808.865,"endTime":812.847,"body":"As you indicated, Moritz, it's an actual conversation with data."},{"speaker":null,"startTime":813.267,"endTime":818.529,"body":"So for example, if I am talking with you and I'm talking about, you know, my evening here"},{"speaker":null,"startTime":818.589,"endTime":821.85,"body":"in Mumbai, you will respond and you will respond."},{"speaker":null,"startTime":821.211,"endTime":825.814,"body":"you will ask me a follow-up question. You might say, what is the time? And I might say, you know,"},{"speaker":null,"startTime":826.074,"endTime":832.779,"body":"it's 8.30 in the night. So there's this back and forth exchange where questions are not restated"},{"speaker":null,"startTime":832.839,"endTime":838.543,"body":"every time. We build on the previous context and knowledge and have this sort of natural"},{"speaker":null,"startTime":838.603,"endTime":843.606,"body":"progression of dialogue between me and and yourselves."},{"speaker":null,"startTime":844.428,"endTime":850.832,"body":"And so I was really interested in emulating that human-to-human conversation behavior"},{"speaker":null,"startTime":851.293,"endTime":858.377,"body":"with respect to a computer around data. So I looked at what we call language pragmatics,"},{"speaker":null,"startTime":858.498,"endTime":862.24,"body":"which is looking at the context of the previous query. So if I ask,"},{"speaker":null,"startTime":862.66,"endTime":865.382,"body":"show me the large earthquakes near california"},{"speaker":null,"startTime":866.204,"endTime":869.385,"body":"My follow-up question is, how about near Texas?"},{"speaker":null,"startTime":870.386,"endTime":873.467,"body":"I'm not restating, show me large earthquakes near Texas,"},{"speaker":null,"startTime":873.547,"endTime":875.327,"body":"because that's kind of unnatural."},{"speaker":null,"startTime":875.727,"endTime":878.308,"body":"So that was kind of the first, I would say,"},{"speaker":null,"startTime":878.828,"endTime":882.17,"body":"novel contribution of that work, is just really implementing"},{"speaker":null,"startTime":882.31,"endTime":884.89,"body":"this pragmatics model in the context"},{"speaker":null,"startTime":884.93,"endTime":886.571,"body":"of an analytical conversation."},{"speaker":null,"startTime":887.131,"endTime":892.433,"body":"The second insight with building the system which actually got"},{"speaker":null,"startTime":892.473,"endTime":894.434,"body":"Tableau leadership excited was."},{"speaker":null,"startTime":895.535,"endTime":901.259,"body":"There was a previous concern that these natural language systems needed to be perfect."},{"speaker":null,"startTime":901.739,"endTime":904.721,"body":"They needed to completely understand what the user was saying,"},{"speaker":null,"startTime":905.162,"endTime":907.904,"body":"and they needed to generate a perfect answer,"},{"speaker":null,"startTime":908.084,"endTime":910.386,"body":"otherwise, people would get annoyed."},{"speaker":null,"startTime":910.926,"endTime":914.248,"body":"I think some of it largely came from systems like"},{"speaker":null,"startTime":914.308,"endTime":918.051,"body":"Microsoft Clippy and other places where if"},{"speaker":null,"startTime":918.071,"endTime":921.534,"body":"the system constantly made an error,"},{"speaker":null,"startTime":921.774,"endTime":923.175,"body":"then people would get annoyed."},{"speaker":null,"startTime":923.957,"endTime":930.812,"body":"But if you actually think about the nature of human-to-human conversation, our conversation"},{"speaker":null,"startTime":923.975,"endTime":924.095,"body":"You"},{"speaker":null,"startTime":930.852,"endTime":932.155,"body":"patterns are not perfect."},{"speaker":null,"startTime":933.38,"endTime":938.082,"body":"I might, I mean, well, Enrico, you thought I was in Bangalore, and I said, no, I'm actually"},{"speaker":null,"startTime":938.122,"endTime":938.742,"body":"in Mumbai."},{"speaker":null,"startTime":938.862,"endTime":941.803,"body":"And I corrected you, but I was not offensive, right?"},{"speaker":null,"startTime":941.864,"endTime":943.504,"body":"I mean, it was just a conversation."},{"speaker":null,"startTime":943.884,"endTime":945.025,"body":"Hopefully I didn't offend you."},{"speaker":null,"startTime":945.565,"endTime":946.926,"body":"Thank you."},{"speaker":null,"startTime":955.276,"endTime":959.861,"body":"Maybe the system comes up with a reasonable guesstimate of what a large earthquake might"},{"speaker":null,"startTime":960.021,"endTime":964.266,"body":"mean and might set it up to a magnitude of six and greater on the Richter scale."},{"speaker":null,"startTime":965.489,"endTime":969.011,"body":"and surfaces that reasoning back to me and says,"},{"speaker":null,"startTime":969.091,"endTime":970.532,"body":"hey, you know, you said large,"},{"speaker":null,"startTime":970.632,"endTime":972.013,"body":"I wasn't quite sure what you meant,"},{"speaker":null,"startTime":972.133,"endTime":974.054,"body":"but I took a guess and I, you know,"},{"speaker":null,"startTime":974.414,"endTime":975.795,"body":"assumed you said six or greater."},{"speaker":null,"startTime":975.835,"endTime":978.176,"body":"And I might, you know, come back to the system and say,"},{"speaker":null,"startTime":978.236,"endTime":979.857,"body":"no, it's actually, you know,"},{"speaker":null,"startTime":979.917,"endTime":982.338,"body":"I live in California right by the earthquake fault"},{"speaker":null,"startTime":982.398,"endTime":983.299,"body":"and I'm not joking."},{"speaker":null,"startTime":983.759,"endTime":985.98,"body":"So, you know, large earthquakes to me"},{"speaker":null,"startTime":986.06,"endTime":987.721,"body":"is anything four or greater."},{"speaker":null,"startTime":987.881,"endTime":989.742,"body":"And I might tweak the slider"},{"speaker":null,"startTime":989.982,"endTime":991.583,"body":"that the computer produces to me."},{"speaker":null,"startTime":992.905,"endTime":998.026,"body":"To me, when we actually tested it with users,"},{"speaker":null,"startTime":998.326,"endTime":1001.507,"body":"A, users were actually not bothered when they had to"},{"speaker":null,"startTime":1001.567,"endTime":1005.289,"body":"correct the system because it was just a small tweak in a slider,"},{"speaker":null,"startTime":1006.189,"endTime":1010.01,"body":"and B, the system actually remembered that setting so that when"},{"speaker":null,"startTime":1010.05,"endTime":1011.03,"body":"a user asked"},{"speaker":null,"startTime":1011.751,"endTime":1018.034,"body":"you know, about large in a subsequent interaction, the system remembered that and the users were"},{"speaker":null,"startTime":1018.074,"endTime":1021.676,"body":"delighted by that. It's like, oh, you know, actually, you remember the context, right? So"},{"speaker":null,"startTime":1022.057,"endTime":1026.839,"body":"if Enrico remembers that I'm actually in Mumbai, I'd be like, you know, that's great, Enrico,"},{"speaker":null,"startTime":1026.859,"endTime":1028.3,"body":"So you have a great memory, right?"},{"speaker":null,"startTime":1028.34,"endTime":1030.061,"body":"I mean, I'm actually going to compliment you."},{"speaker":null,"startTime":1030.761,"endTime":1030.942,"body":"So..."},{"speaker":null,"startTime":1032.184,"endTime":1035.787,"body":"It turned out people actually love that sort of back and forth interaction."},{"speaker":null,"startTime":1036.127,"endTime":1042.152,"body":"So something that was previously viewed as a hindrance or a limitation was actually"},{"speaker":null,"startTime":1042.212,"endTime":1047.597,"body":"a strength in terms of modeling human to human dialogue, in terms of a human"},{"speaker":null,"startTime":1047.617,"endTime":1049.859,"body":"to computer dialogue with respect to data."},{"speaker":null,"startTime":1050.459,"endTime":1051.6,"body":"And as I indicated,"},{"speaker":null,"startTime":1052.781,"endTime":1056.484,"body":"You know, data is a lot more tractable because it's not..."},{"speaker":null,"startTime":1056.906,"endTime":1059.667,"body":"trying to make a model of the entire universe,"},{"speaker":null,"startTime":1060.107,"endTime":1063.247,"body":"the universe for the system are the bounds of the data."},{"speaker":null,"startTime":1063.467,"endTime":1067.048,"body":"So all of that made it a very tractable problem"},{"speaker":null,"startTime":1067.188,"endTime":1070.229,"body":"to move the work forward, including productization."},{"speaker":null,"startTime":1071.169,"endTime":1074.15,"body":"Yeah, that's one aspect that I really like"},{"speaker":null,"startTime":1074.21,"endTime":1077.39,"body":"of the type of research that you and your team"},{"speaker":null,"startTime":1077.59,"endTime":1080.231,"body":"have done over the years that is not,"},{"speaker":null,"startTime":1080.771,"endTime":1082.912,"body":"I mean, the technical contribution is remarkable,"},{"speaker":null,"startTime":1082.992,"endTime":1084.732,"body":"but what is really interesting is that you have done"},{"speaker":null,"startTime":1084.772,"endTime":1085.632,"body":"a lot of also..."},{"speaker":null,"startTime":1086.693,"endTime":1091.456,"body":"human factors type of research on top of it and trying to understand what happens when people"},{"speaker":null,"startTime":1092.596,"endTime":1097.679,"body":"interact with this type of quote-unquote intelligent systems, right? And I think"},{"speaker":null,"startTime":1097.699,"endTime":1101.421,"body":"that there's so much to do in that direction and really understanding how"},{"speaker":null,"startTime":1101.441,"endTime":1105.704,"body":"how users' behavior also changes as they learn how to..."},{"speaker":null,"startTime":1106.645,"endTime":1109.528,"body":"how to work with these machines. One thing that comes to mind is, like,"},{"speaker":null,"startTime":1110.068,"endTime":1113.651,"body":"I guess even the way we use Google, we just adapt because we know that..."},{"speaker":null,"startTime":1114.151,"endTime":1118.335,"body":"We don't ask Google the questions in the same way we would ask to a person."},{"speaker":null,"startTime":1118.975,"endTime":1120.276,"body":"Or to Alexa, even."},{"speaker":null,"startTime":1120.496,"endTime":1121.997,"body":"Or to Alexa, exactly."},{"speaker":null,"startTime":1122.017,"endTime":1122.398,"body":"That's right."},{"speaker":null,"startTime":1123.338,"endTime":1131.185,"body":"So I was wondering if you observed the same kind of behavior where after a while people just know"},{"speaker":null,"startTime":1131.285,"endTime":1135.708,"body":"how to ask the thing so that they will get the information."},{"speaker":null,"startTime":1135.83,"endTime":1136.99,"body":"or the results that they want?"},{"speaker":null,"startTime":1137.791,"endTime":1140.372,"body":"That's a really excellent question and insight."},{"speaker":null,"startTime":1141.473,"endTime":1143.354,"body":"So what we have found over the years,"},{"speaker":null,"startTime":1144.614,"endTime":1149.117,"body":"even with search systems and natural language interfaces"},{"speaker":null,"startTime":1149.817,"endTime":1152.899,"body":"is people don't like to be wrong."},{"speaker":null,"startTime":1153.559,"endTime":1154.82,"body":"They don't like failure."},{"speaker":null,"startTime":1155.18,"endTime":1159.703,"body":"So they will adapt to the computer speak of the system."},{"speaker":null,"startTime":1159.883,"endTime":1162.224,"body":"Humans really understand very quickly"},{"speaker":null,"startTime":1162.544,"endTime":1164.565,"body":"what a system can or cannot do."},{"speaker":null,"startTime":1165.727,"endTime":1172.552,"body":"and readjust their model and their view of their interactions with the system,"},{"speaker":null,"startTime":1173.933,"endTime":1180.778,"body":"which is good in a way because then you don't have too many frustrated users, hopefully,"},{"speaker":null,"startTime":1181.259,"endTime":1184.961,"body":"and they sort of adapt to whatever the system can support."},{"speaker":null,"startTime":1185.362,"endTime":1187.123,"body":"But there is a downside to it."},{"speaker":null,"startTime":1188.786,"endTime":1195.371,"body":"When you're looking at the telemetry data of how people are using systems,"},{"speaker":null,"startTime":1196.612,"endTime":1199.195,"body":"the window of opportunity to"},{"speaker":null,"startTime":1199.335,"endTime":1204.039,"body":"understand what are the types of queries the system cannot support,"},{"speaker":null,"startTime":1204.9,"endTime":1208.162,"body":"and what would make for useful extensions and"},{"speaker":null,"startTime":1208.222,"endTime":1211.105,"body":"improvements to the system is a very narrow time window."},{"speaker":null,"startTime":1212.167,"endTime":1215.149,"body":"what we have found is within a couple of days,"},{"speaker":null,"startTime":1215.749,"endTime":1219.19,"body":"people will change from asking questions"},{"speaker":null,"startTime":1219.25,"endTime":1221.111,"body":"that the system does not understand"},{"speaker":null,"startTime":1221.191,"endTime":1224.373,"body":"or partially understand to only the questions"},{"speaker":null,"startTime":1224.613,"endTime":1228.755,"body":"and the repertoire of analytical functions"},{"speaker":null,"startTime":1228.835,"endTime":1230.295,"body":"that the system can support."},{"speaker":null,"startTime":1230.956,"endTime":1232.076,"body":"And this includes search."},{"speaker":null,"startTime":1232.736,"endTime":1236.538,"body":"And so you immediately see a drop-off of failure rates"},{"speaker":null,"startTime":1236.878,"endTime":1237.979,"body":"for all these users."},{"speaker":null,"startTime":1238.039,"endTime":1238.339,"body":"So you..."},{"speaker":null,"startTime":1238.7,"endTime":1245.465,"body":"we really have to catch these new users within a one to three day window, which makes it"},{"speaker":null,"startTime":1245.585,"endTime":1252.451,"body":"kind of challenging when we're using customer data and telemetry data to help inform how"},{"speaker":null,"startTime":1252.491,"endTime":1253.592,"body":"we can improve the system."},{"speaker":null,"startTime":1254.112,"endTime":1254.633,"body":"Yeah, yeah."},{"speaker":null,"startTime":1254.913,"endTime":1258.936,"body":"Yeah, that's a tricky aspect, is the discoverability in terms of..."},{"speaker":null,"startTime":1259.759,"endTime":1264.303,"body":"What can I even ask, right? It's a bit like with gesture interfaces where you just try and figure"},{"speaker":null,"startTime":1264.323,"endTime":1269.307,"body":"out, okay, maybe swiping works, maybe zooming, right? It's the same with language because"},{"speaker":null,"startTime":1270.288,"endTime":1274.772,"body":"first probably you ask anything and then you realize, yeah, it can't quite do everything."},{"speaker":null,"startTime":1274.812,"endTime":1278.395,"body":"And then you're like, yeah, that makes sense. You know, there must be some limitation, but then"},{"speaker":null,"startTime":1278.916,"endTime":1281.498,"body":"you need to figure out what is the vocabulary I can use."},{"speaker":null,"startTime":1281.518,"endTime":1281.879,"body":"Yeah, exactly."},{"speaker":null,"startTime":1287.479,"endTime":1295.32,"body":"So one part I'm really interested in is also this transfer between research and then putting"},{"speaker":null,"startTime":1295.36,"endTime":1296.3,"body":"it into the product."},{"speaker":null,"startTime":1296.4,"endTime":1305.462,"body":"And yeah, EVISA in some way found its place in Tableau as Ask Data or probably some further"},{"speaker":null,"startTime":1305.482,"endTime":1308.623,"body":"development but I guess it all started with EVISA, right?"},{"speaker":null,"startTime":1308.703,"endTime":1312.104,"body":"And so yeah, I'm curious, like how does this work?"},{"speaker":null,"startTime":1312.304,"endTime":1318.065,"body":"How did this specific prototype and piece of research make it into being part of a product"},{"speaker":null,"startTime":1318.085,"endTime":1318.545,"body":"Temporal."},{"speaker":null,"startTime":1319.446,"endTime":1319.746,"body":"Yeah."},{"speaker":null,"startTime":1319.846,"endTime":1323.587,"body":"What's the process there and how did this work and, you know, I want to learn all about"},{"speaker":null,"startTime":1323.607,"endTime":1324.087,"body":"the process."},{"speaker":null,"startTime":1325.727,"endTime":1326.008,"body":"Sure."},{"speaker":null,"startTime":1327.288,"endTime":1332.97,"body":"So, as mentioned, I mean, when I joined Tableau back in the day, I was fortunate enough to"},{"speaker":null,"startTime":1334.01,"endTime":1339.972,"body":"interact with Chris Dolty and Christian Chabot and Pat Hanrahan, you know, the co-founders"},{"speaker":null,"startTime":1340.032,"endTime":1340.672,"body":"of the company."},{"speaker":null,"startTime":1341.873,"endTime":1347.396,"body":"And we were having these conversations when I joined Tableau, where, you know, Chris Dolty said,"},{"speaker":null,"startTime":1347.476,"endTime":1352.539,"body":"you know, NLP is really hard, Vidya. I don't know how we can actually crack that nut. And"},{"speaker":null,"startTime":1353.059,"endTime":1357.662,"body":"even if we do crack the nut in research, how are we actually going to productize it so it works"},{"speaker":null,"startTime":1357.702,"endTime":1363.285,"body":"for any customer, right? Because we are not, Tableau doesn't cater to a very specific vertical."},{"speaker":null,"startTime":1364.025,"endTime":1368.688,"body":"And there were, there are previous systems like IBM Watson that have catered to very"},{"speaker":null,"startTime":1369.509,"endTime":1377.774,"body":"verticals, including chess or healthcare, but Tableau, in a sense, is more of a generalist"},{"speaker":null,"startTime":1377.834,"endTime":1378.714,"body":"type of platform."},{"speaker":null,"startTime":1378.754,"endTime":1380.155,"body":"So that was kind of the problem."},{"speaker":null,"startTime":1381.057,"endTime":1389.8,"body":"But when I demoed Eviza, it's kind of funny because the story there was I was in California"},{"speaker":null,"startTime":1390.801,"endTime":1398.884,"body":"and everybody else was in Seattle. And back then, for various reasons, I was using voice to talk"},{"speaker":null,"startTime":1398.924,"endTime":1403.345,"body":"with the advice. I just thought it would be so cool to talk to the data with voice. I mean..."},{"speaker":null,"startTime":1404.627,"endTime":1405.767,"body":"you could do text or voice."},{"speaker":null,"startTime":1405.787,"endTime":1408.188,"body":"That was not kind of the main part of the story."},{"speaker":null,"startTime":1408.208,"endTime":1410.149,"body":"But I was like, yeah, I'm going to demo this with voice."},{"speaker":null,"startTime":1410.209,"endTime":1412.21,"body":"And I asked everybody to mute themselves."},{"speaker":null,"startTime":1413.11,"endTime":1414.611,"body":"And I did my whole demo."},{"speaker":null,"startTime":1414.671,"endTime":1416.712,"body":"And I said, don't interrupt me in between,"},{"speaker":null,"startTime":1416.752,"endTime":1419.213,"body":"because it's going to catch your voice and mess up my demo."},{"speaker":null,"startTime":1420.393,"endTime":1422.734,"body":"So I did my demo for 10 minutes."},{"speaker":null,"startTime":1422.954,"endTime":1427.056,"body":"And then I had to rush off to a doctor's appointment."},{"speaker":null,"startTime":1427.096,"endTime":1428.056,"body":"So I just said bye."},{"speaker":null,"startTime":1428.117,"endTime":1430.397,"body":"I didn't even wait for Q&A. And I didn't make it."},{"speaker":null,"startTime":1431.359,"endTime":1433.12,"body":"I didn't really think about it, right?"},{"speaker":null,"startTime":1433.5,"endTime":1436.982,"body":"And late in the night, I get an email from Chris Dolte."},{"speaker":null,"startTime":1437.022,"endTime":1440.844,"body":"It's like, OK, can you demo this to Tom Walker, who"},{"speaker":null,"startTime":1440.864,"endTime":1446.847,"body":"at that time was making the call in terms of which teams"},{"speaker":null,"startTime":1446.908,"endTime":1448.869,"body":"are funded and stuff like that."},{"speaker":null,"startTime":1448.969,"endTime":1452.351,"body":"So long story short, Tableau leadership"},{"speaker":null,"startTime":1452.411,"endTime":1454.352,"body":"got really excited about the project."},{"speaker":null,"startTime":1455.172,"endTime":1455.893,"body":"And, um,"},{"speaker":null,"startTime":1456.694,"endTime":1461.699,"body":"they decided, okay, I think it's time that we invest in NLP. And this was also a time where"},{"speaker":null,"startTime":1461.799,"endTime":1467.685,"body":"the tech industry was hedging its bets in this space. And this is, I think, one of the things,"},{"speaker":null,"startTime":1467.745,"endTime":1475.493,"body":"And I don't think it's one criteria that indicates or dictates what, you know, what's going on."},{"speaker":null,"startTime":1475.154,"endTime":1477.796,"body":"you know, what makes a successful tech transfer."},{"speaker":null,"startTime":1477.956,"endTime":1479.437,"body":"It's a multitude of factors."},{"speaker":null,"startTime":1479.897,"endTime":1483.218,"body":"You need to have some sort of perspective"},{"speaker":null,"startTime":1483.319,"endTime":1485.68,"body":"that you can show that it works"},{"speaker":null,"startTime":1485.82,"endTime":1487.881,"body":"and the market needs to be ready for that."},{"speaker":null,"startTime":1487.961,"endTime":1489.802,"body":"The customers need to be ready for that"},{"speaker":null,"startTime":1490.042,"endTime":1491.743,"body":"and the company needs to be ready."},{"speaker":null,"startTime":1491.783,"endTime":1492.923,"body":"So there's so much that has,"},{"speaker":null,"startTime":1492.983,"endTime":1494.664,"body":"I mean, all the stars need to align."},{"speaker":null,"startTime":1495.304,"endTime":1498.086,"body":"And we were fortunate enough that everything did align."},{"speaker":null,"startTime":1498.586,"endTime":1500.227,"body":"But the problem back then was"},{"speaker":null,"startTime":1500.587,"endTime":1500.687,"body":"Um,"},{"speaker":null,"startTime":1501.288,"endTime":1503.729,"body":"Tableau was not known to be an NLP company."},{"speaker":null,"startTime":1503.789,"endTime":1506.571,"body":"I was the only NLP person in the entire company,"},{"speaker":null,"startTime":1506.651,"endTime":1508.631,"body":"and it was a very small startup."},{"speaker":null,"startTime":1509.792,"endTime":1511.893,"body":"So how do you actually convince people"},{"speaker":null,"startTime":1512.073,"endTime":1514.854,"body":"who are working in companies like Google"},{"speaker":null,"startTime":1514.954,"endTime":1517.795,"body":"to come and join Tableau and work on, you know, NLP."},{"speaker":null,"startTime":1518.736,"endTime":1525.14,"body":"So we made the decision to look for smallish startups in the Bay Area, because I'm based in"},{"speaker":null,"startTime":1525.16,"endTime":1532.624,"body":"the Bay Area, to try to see if we can acquire some seed technology as foundation and then"},{"speaker":null,"startTime":1532.644,"endTime":1537.927,"body":"then I could help build some of the stuff that I had in Eviza on top of that platform."},{"speaker":null,"startTime":1539.128,"endTime":1545.532,"body":"So after, you know, several months of vetting various startups that I was involved in,"},{"speaker":null,"startTime":1545.852,"endTime":1551.755,"body":"and this is the other piece, you know, when you're talking about doing tech transfer research"},{"speaker":null,"startTime":1551.815,"endTime":1558.198,"body":"projects, you have to be all in. So that might involve doing technical due diligence for startups,"},{"speaker":null,"startTime":1558.338,"endTime":1562.94,"body":"interviewing people. You have to be all in. You have to show that you are"},{"speaker":null,"startTime":1563.982,"endTime":1571.146,"body":"motivated to make it work. There is kind of a larger sociological aspect to this."},{"speaker":null,"startTime":1572.247,"endTime":1579.792,"body":"So, long story short, we identified this startup called ClearGraph, and they came on board"},{"speaker":null,"startTime":1580.132,"endTime":1580.312,"body":"and"},{"speaker":null,"startTime":1581.655,"endTime":1590.186,"body":"I moved from research and joined the product team, starting first as a lead engineer,"},{"speaker":null,"startTime":1591.267,"endTime":1593.33,"body":"working on production quality code."},{"speaker":null,"startTime":1598.198,"endTime":1604.403,"body":"to write production quality code with your peers, including junior people critiquing"},{"speaker":null,"startTime":1604.443,"endTime":1611.488,"body":"and reviewing your code. But it was really good for my soul. And I definitely improved"},{"speaker":null,"startTime":1611.568,"endTime":1618.052,"body":"my coding chops after that. And soon enough, I became the engineering manager on the team."},{"speaker":null,"startTime":1618.753,"endTime":1621.355,"body":"So, I was responsible for that."},{"speaker":null,"startTime":1621.276,"endTime":1626.621,"body":"you know, all the release planning, working closely with the PMs, doing code reviews, writing code myself,"},{"speaker":null,"startTime":1627.862,"endTime":1636.73,"body":"sneaking in features from Eviza into the product, and writing papers. Melanie Torrey, who is currently at"},{"speaker":null,"startTime":1636.77,"endTime":1639.792,"body":"Northeastern, was a very close collaborator of mine."},{"speaker":null,"startTime":1639.892,"endTime":1641.694,"body":"And so she and I would..."},{"speaker":null,"startTime":1642.737,"endTime":1647.661,"body":"you know, buddy up and in figuring out the research aspects of the project, you know,"},{"speaker":null,"startTime":1648.222,"endTime":1654.528,"body":"how do we actually support intent understanding and semantics so that we can build on top of this"},{"speaker":null,"startTime":1654.608,"endTime":1661.014,"body":"platform. And so I stayed on, stayed in the team for a few releases of AskData and then"},{"speaker":null,"startTime":1661.214,"endTime":1663.016,"body":"ultimately move back to research."},{"speaker":null,"startTime":1665.52,"endTime":1671.482,"body":"And like the way it's implemented now, what can you ask AskData, like let's say the earthquake"},{"speaker":null,"startTime":1671.502,"endTime":1677.023,"body":"data set, right? What are some of the questions you could ask and where you would get a meaningful"},{"speaker":null,"startTime":1677.063,"endTime":1688.607,"body":"chart back? Yeah, so AskData connects natural language utterances to Tableau's visql command"},{"speaker":null,"startTime":1688.627,"endTime":1692.548,"body":"stack and the feature leverages show me."},{"speaker":null,"startTime":1693.429,"endTime":1699.495,"body":"to display the top recommended chart that ShowMe suggests."},{"speaker":null,"startTime":1700.015,"endTime":1706.021,"body":"So we do leverage some of the Tableau underlying architecture"},{"speaker":null,"startTime":1706.401,"endTime":1711.086,"body":"and combine it with the parser and the natural language input"},{"speaker":null,"startTime":1711.106,"endTime":1712.147,"body":"that's part of Astida."},{"speaker":null,"startTime":1712.849,"endTime":1721.437,"body":"So that being said, we support five basic analytical functions that are tied to Tableau's"},{"speaker":null,"startTime":1721.718,"endTime":1727.663,"body":"core analytical stack. You know, you can ask questions about aggregations and groupings"},{"speaker":null,"startTime":1727.803,"endTime":1729.605,"body":"and drill downs, filters."},{"speaker":null,"startTime":1730.207,"endTime":1738.058,"body":"and sorting and various combinations of that. We also have vague concepts, kind of like the"},{"speaker":null,"startTime":1738.258,"endTime":1741.102,"body":"the large and the near in the earthquake examples."},{"speaker":null,"startTime":1757.615,"endTime":1762.757,"body":"It, you know, also supports these follow-up questions."},{"speaker":null,"startTime":1762.817,"endTime":1765.017,"body":"So some of the pragmatics behavior as well."},{"speaker":null,"startTime":1765.277,"endTime":1765.497,"body":"Wow."},{"speaker":null,"startTime":1765.838,"endTime":1765.998,"body":"Yeah."},{"speaker":null,"startTime":1766.378,"endTime":1767.398,"body":"That sounds quite useful."},{"speaker":null,"startTime":1770.099,"endTime":1773.36,"body":"I was, it was interesting for me to hear that initially"},{"speaker":null,"startTime":1773.42,"endTime":1776.961,"body":"the prototype was meant to be used through voice."},{"speaker":null,"startTime":1777.761,"endTime":1778.082,"body":"Yeah."},{"speaker":null,"startTime":1778.362,"endTime":1780.122,"body":"I don't know if, if it's actually possible"},{"speaker":null,"startTime":1780.162,"endTime":1781.383,"body":"to do that with Ask Data."},{"speaker":null,"startTime":1782.345,"endTime":1783.927,"body":"Or is it only through typing?"},{"speaker":null,"startTime":1783.947,"endTime":1785.87,"body":"We decided to do text."},{"speaker":null,"startTime":1786.912,"endTime":1789.135,"body":"You know, from a business standpoint, it makes sense."},{"speaker":null,"startTime":1789.195,"endTime":1792.04,"body":"I mean, if people are all at their desk talking to their data,"},{"speaker":null,"startTime":1792.28,"endTime":1794.664,"body":"You know, it may annoy, it may annoy you."},{"speaker":null,"startTime":1801.113,"endTime":1807.295,"body":"No, I mean, to me, voice-to-text translation is more or less a solved problem."},{"speaker":null,"startTime":1808.135,"endTime":1813.996,"body":"I was just using an off-the-shelf API to do that, so it was more of a footnote, and I"},{"speaker":null,"startTime":1814.036,"endTime":1818.457,"body":"just thought it would be cool to use my voice and talk to the data."},{"speaker":null,"startTime":1818.517,"endTime":1819.698,"body":"as more of a cool effect."},{"speaker":null,"startTime":1821.339,"endTime":1827.762,"body":"But, you know, I am exploring, you know, voice with data stories, actually,"},{"speaker":null,"startTime":1828.842,"endTime":1835.005,"body":"now as part of my research portfolio, just in the space of text and charts."},{"speaker":null,"startTime":1835.986,"endTime":1838.567,"body":"So it'll probably come back to me full circle."},{"speaker":null,"startTime":1839.127,"endTime":1840.548,"body":"Yeah."},{"speaker":null,"startTime":1840.53,"endTime":1842.773,"body":"And I was also curious to hear from you."},{"speaker":null,"startTime":1843.074,"endTime":1848.042,"body":"I really like the story that you started in research and then went to engineering."},{"speaker":null,"startTime":1848.102,"endTime":1851.287,"body":"And I guess now you're back full time in research."},{"speaker":null,"startTime":1851.487,"endTime":1851.948,"body":"Correct."},{"speaker":null,"startTime":1852.549,"endTime":1852.81,"body":"Yeah."},{"speaker":null,"startTime":1865.28,"endTime":1869.442,"body":"The last mile is probably really hard."},{"speaker":null,"startTime":1869.462,"endTime":1870.783,"body":"The mini-miles."},{"speaker":null,"startTime":1870.983,"endTime":1872.023,"body":"The mini-miles."},{"speaker":null,"startTime":1872.483,"endTime":1874.644,"body":"The mini-miles, right."},{"speaker":null,"startTime":1874.745,"endTime":1876.185,"body":"It's a big hike."},{"speaker":null,"startTime":1877.686,"endTime":1880.067,"body":"How many years did it take all together, kicking it off?"},{"speaker":null,"startTime":1882.249,"endTime":1883.55,"body":"Um, it was about two years."},{"speaker":null,"startTime":1884.11,"endTime":1891.094,"body":"Um, it was about, I mean, we shipped, we shipped our first, uh, feature on time."},{"speaker":null,"startTime":1891.114,"endTime":1893.676,"body":"I would say a little over a year."},{"speaker":null,"startTime":1894.556,"endTime":1894.996,"body":"And then..."},{"speaker":null,"startTime":1895.016,"endTime":1895.617,"body":"Mm-hmm."},{"speaker":null,"startTime":1911.522,"endTime":1917.084,"body":"You have to be all in, so if you want the right type of sausage to come out, you need"},{"speaker":null,"startTime":1917.124,"endTime":1919.625,"body":"to be part of the messy sausage-making."},{"speaker":null,"startTime":1920.285,"endTime":1928.988,"body":"It's very hard to sit in research and try to get the team to build a product."},{"speaker":null,"startTime":1929.949,"endTime":1936.478,"body":"based on what you would like to see you have to roll up your sleeves and be part of that process"},{"speaker":null,"startTime":1938.1,"endTime":1944.208,"body":"for a lot of reasons, because unlike research, engineering works very fast."},{"speaker":null,"startTime":1944.891,"endTime":1951.074,"body":"every day there are stand-ups, every day there are decisions being made. By sitting outside the team"},{"speaker":null,"startTime":1951.174,"endTime":1956.917,"body":"you're not part of that conversation, you're not part of those serendipitous decisions where it's"},{"speaker":null,"startTime":1956.957,"endTime":1962.86,"body":"like, oh there's a bug, how do we fix it? Oh by the way, if we fix this we can add this additional"},{"speaker":null,"startTime":1962.9,"endTime":1970.023,"body":"feature. That doesn't happen by setting up a formal meeting every week. So you lose out on a lot of"},{"speaker":null,"startTime":1970.083,"endTime":1973.665,"body":"that if you're not part of the team. And then the second thing is"},{"speaker":null,"startTime":1974.627,"endTime":1977.749,"body":"when you're part of the team and you're writing code"},{"speaker":null,"startTime":1977.829,"endTime":1980.611,"body":"and being part of the process as a team member,"},{"speaker":null,"startTime":1981.272,"endTime":1984.694,"body":"there is a whole notion of trust"},{"speaker":null,"startTime":1985.155,"endTime":1987.136,"body":"that is an credibility that is built."},{"speaker":null,"startTime":1988.217,"endTime":1991.539,"body":"I will say that, you know, engineers are great people,"},{"speaker":null,"startTime":1991.92,"endTime":1994.622,"body":"but they are very skeptical of us researchers."},{"speaker":null,"startTime":1994.662,"endTime":1997.564,"body":"They're like, yeah, you know, you guys are all smart."},{"speaker":null,"startTime":1997.584,"endTime":1999.405,"body":"You write papers, but you know,"},{"speaker":null,"startTime":1999.465,"endTime":2002.668,"body":"when rubber has to hit the road, I don't know that man."},{"speaker":null,"startTime":2004.13,"endTime":2011.034,"body":"So, you know, you have to spend some time with the team and building that credibility"},{"speaker":null,"startTime":2011.615,"endTime":2018.519,"body":"in order for them to take you seriously. Get the street cred. Exactly. And I think it's a"},{"speaker":null,"startTime":2018.579,"endTime":2026.224,"body":"really good opportunity, especially when you are in industry research, to work on code."},{"speaker":null,"startTime":2026.444,"endTime":2032.208,"body":"You know, there's nothing as empowering as being able to have an idea and being able to make it"},{"speaker":null,"startTime":2032.288,"endTime":2032.528,"body":"real."},{"speaker":null,"startTime":2033.43,"endTime":2035.532,"body":"It may not be the perfect prototype,"},{"speaker":null,"startTime":2036.112,"endTime":2039.935,"body":"but it's super power if you can actually take an idea"},{"speaker":null,"startTime":2040.116,"endTime":2041.837,"body":"and implement something and show it"},{"speaker":null,"startTime":2042.377,"endTime":2043.718,"body":"beyond a PowerPoint deck."},{"speaker":null,"startTime":2045.34,"endTime":2047.601,"body":"And when you join a team like that,"},{"speaker":null,"startTime":2048.202,"endTime":2052.545,"body":"you understand the complexities and processes"},{"speaker":null,"startTime":2052.605,"endTime":2054.407,"body":"that go into shipping a feature."},{"speaker":null,"startTime":2055.147,"endTime":2057.109,"body":"It's much more than what meets the eye."},{"speaker":null,"startTime":2057.589,"endTime":2059.15,"body":"There's so much that goes into stake,"},{"speaker":null,"startTime":2059.23,"endTime":2060.071,"body":"the performance."},{"speaker":null,"startTime":2060.673,"endTime":2061.974,"body":"You can't have regressions."},{"speaker":null,"startTime":2062.014,"endTime":2064.855,"body":"You've got to build a very sophisticated test suite."},{"speaker":null,"startTime":2066.455,"endTime":2068.156,"body":"And you have to listen to customers."},{"speaker":null,"startTime":2068.857,"endTime":2070.857,"body":"You have to be very careful of release planning."},{"speaker":null,"startTime":2071.418,"endTime":2073.418,"body":"So there's so much that goes on, and you kind of"},{"speaker":null,"startTime":2073.479,"endTime":2077.58,"body":"develop a profound respect for the engineering profession."},{"speaker":null,"startTime":2078.281,"endTime":2080.982,"body":"And it's a way of bringing that back into research."},{"speaker":null,"startTime":2083.104,"endTime":2086.945,"body":"And how do you then balance not getting lost in the weeds?"},{"speaker":null,"startTime":2087.526,"endTime":2091.388,"body":"Like this is very much like, you know, in the trenches and like the frog's eye perspective"},{"speaker":null,"startTime":2091.448,"endTime":2092.068,"body":"basically."},{"speaker":null,"startTime":2092.568,"endTime":2098.291,"body":"But how do you keep that that vision then and not get lost in the, yeah, this doesn't"},{"speaker":null,"startTime":2098.331,"endTime":2103.213,"body":"work and that doesn't work and that's broken, you know, because engineering is fundamentally"},{"speaker":null,"startTime":2103.253,"endTime":2108.616,"body":"a lot about fixing something that's broken."},{"speaker":null,"startTime":2108.656,"endTime":2110.137,"body":"How do you keep that big picture with you?"},{"speaker":null,"startTime":2110.278,"endTime":2114.901,"body":"I think it's very important to have a good working relationship with the PM on the team"},{"speaker":null,"startTime":2115.182,"endTime":2122.067,"body":"because at the end of the day, the PM is the one that really hashes out the roadmap for"},{"speaker":null,"startTime":2122.087,"endTime":2127.271,"body":"the feature and what does it look like over the years and being involved in those conversations"},{"speaker":null,"startTime":2129.173,"endTime":2137.159,"body":"and going back to that document that a PM would write on the roadmap and the strategy"},{"speaker":null,"startTime":2137.199,"endTime":2139.26,"body":"of the feature kind of keeps you honest."},{"speaker":null,"startTime":2141.104,"endTime":2142.985,"body":"And, I think it's just a calibration."},{"speaker":null,"startTime":2143.065,"endTime":2147.549,"body":"There are some days when I would be lost in the weeds because my code wouldn't work and"},{"speaker":null,"startTime":2147.589,"endTime":2149.811,"body":"you just can't sleep when your code doesn't work."},{"speaker":null,"startTime":2149.851,"endTime":2150.832,"body":"You want to get it working."},{"speaker":null,"startTime":2150.852,"endTime":2151.672,"body":"That's not good."},{"speaker":null,"startTime":2152.815,"endTime":2161.099,"body":"But we made a conscious decision where, you know, I continued to publish while being on"},{"speaker":null,"startTime":2161.139,"endTime":2168.584,"body":"an engineering team and, in fact, actually getting some engineers to be on papers and"},{"speaker":null,"startTime":2168.644,"endTime":2169.584,"body":"patents with me."},{"speaker":null,"startTime":2179.124,"endTime":2185.148,"body":"you know, the research aspects of it, and you have to articulate that in prose,"},{"speaker":null,"startTime":2185.848,"endTime":2189.47,"body":"that really helps you look at the bigger picture holistically."},{"speaker":null,"startTime":2189.811,"endTime":2190.811,"body":"And that combined with..."},{"speaker":null,"startTime":2190.831,"endTime":2195.654,"body":"What makes this unique, or what are the fundamental mechanisms and so on."},{"speaker":null,"startTime":2195.714,"endTime":2196.515,"body":"Yes, yes."},{"speaker":null,"startTime":2201.45,"endTime":2208.117,"body":"I think it's a double word score when you have a feature and it is backed and supported by"},{"speaker":null,"startTime":2208.937,"endTime":2214.263,"body":"well-done research, right? I think that's the best of both worlds."},{"speaker":null,"startTime":2214.283,"endTime":2214.623,"body":"Yeah."},{"speaker":null,"startTime":2215.404,"endTime":2215.684,"body":"Yeah."},{"speaker":null,"startTime":2216.045,"endTime":2217.906,"body":"Or it invites new research, right?"},{"speaker":null,"startTime":2217.946,"endTime":2218.147,"body":"Yeah."},{"speaker":null,"startTime":2218.267,"endTime":2218.707,"body":"Yeah."},{"speaker":null,"startTime":2219.048,"endTime":2219.168,"body":"Yeah."},{"speaker":null,"startTime":2230.793,"endTime":2235.516,"body":"I think what is really interesting is that when I think about Tableau, to me,"},{"speaker":null,"startTime":2236.096,"endTime":2240.379,"body":"Tableau is like the thing that basically introduced"},{"speaker":null,"startTime":2240.439,"endTime":2246.343,"body":"a specific type of interaction for people to create visualizations, right?"},{"speaker":null,"startTime":2246.383,"endTime":2249.966,"body":"Before Tableau, it was basically Excel and a few other things."},{"speaker":null,"startTime":2250.886,"endTime":2253.748,"body":"And Tableau introduced this idea that you basically"},{"speaker":null,"startTime":2255.469,"endTime":2255.89,"body":"select"},{"speaker":null,"startTime":2256.852,"endTime":2262.396,"body":"attributes and you drag them into a panel and and now you have some visualizations out there,"},{"speaker":null,"startTime":2262.856,"endTime":2268.981,"body":"right? And then you have now with with a visa and as data, you have a new type of interaction."},{"speaker":null,"startTime":2270.482,"endTime":2277.708,"body":"I was wondering, I always felt that this space we're trying to figure out how to express what"},{"speaker":null,"startTime":2277.748,"endTime":2281.31,"body":"we have it in our mind so that the machine can understand it and produce."},{"speaker":null,"startTime":2282.893,"endTime":2287.937,"body":"data analytics and visualizations is a really interesting space."},{"speaker":null,"startTime":2288.898,"endTime":2291.199,"body":"And at the same time, it doesn't seem to be as,"},{"speaker":null,"startTime":2292.52,"endTime":2294.682,"body":"I would say it's not as much explored"},{"speaker":null,"startTime":2294.762,"endTime":2296.504,"body":"as the visual side of things,"},{"speaker":null,"startTime":2297.144,"endTime":2299.766,"body":"but it seems to me that is at least equally relevant,"},{"speaker":null,"startTime":2299.826,"endTime":2301.067,"body":"if not even more relevant."},{"speaker":null,"startTime":2301.387,"endTime":2303.189,"body":"How do we express our intent?"},{"speaker":null,"startTime":2303.269,"endTime":2306.471,"body":"How do we tell a machine what we want, right?"},{"speaker":null,"startTime":2310.435,"endTime":2316.9,"body":"two really strong milestones."},{"speaker":null,"startTime":2316.92,"endTime":2319.682,"body":"I would say these are two milestones in this sense"},{"speaker":null,"startTime":2319.742,"endTime":2322.444,"body":"because they introduce completely new paradigms"},{"speaker":null,"startTime":2323.164,"endTime":2326.086,"body":"on how to express what is my intent"},{"speaker":null,"startTime":2326.186,"endTime":2330.209,"body":"and what I would like the machine to do for me, basically."},{"speaker":null,"startTime":2330.289,"endTime":2331.15,"body":"Right, right."},{"speaker":null,"startTime":2332.551,"endTime":2335.233,"body":"Yeah, and I think there is a spectrum"},{"speaker":null,"startTime":2335.553,"endTime":2337.254,"body":"of the type of analytical intents that people express"},{"speaker":null,"startTime":2337.275,"endTime":2338.235,"body":"This is the end of the video."},{"speaker":null,"startTime":2338.315,"endTime":2341.718,"body":"If you liked the video, please subscribe to the channel and click on the bell icon to"},{"speaker":null,"startTime":2341.399,"endTime":2342.66,"body":"and what is supported."},{"speaker":null,"startTime":2341.738,"endTime":2342.638,"body":"receive notifications of my latest videos."},{"speaker":null,"startTime":2342.72,"endTime":2344.201,"body":"So to your point,"},{"speaker":null,"startTime":2344.961,"endTime":2347.423,"body":"something like a direct manipulation type interface"},{"speaker":null,"startTime":2347.443,"endTime":2350.185,"body":"like Tableau can be very concrete, right?"},{"speaker":null,"startTime":2350.265,"endTime":2351.826,"body":"I click on a bunch of attributes"},{"speaker":null,"startTime":2351.906,"endTime":2353.547,"body":"and I drag them to rows and shelves."},{"speaker":null,"startTime":2354.268,"endTime":2356.63,"body":"There's very little ambiguity"},{"speaker":null,"startTime":2356.95,"endTime":2360.052,"body":"in terms of what the user wants to see."},{"speaker":null,"startTime":2361.396,"endTime":2361.477,"body":"and"},{"speaker":null,"startTime":2383.08,"endTime":2390.211,"body":"Now, with language, you can definitely be a bit much more abstract in terms of how you"},{"speaker":null,"startTime":2390.271,"endTime":2391.332,"body":"express your intent."},{"speaker":null,"startTime":2391.593,"endTime":2392.474,"body":"It can be vague."},{"speaker":null,"startTime":2392.955,"endTime":2393.836,"body":"It can be fuzzy."},{"speaker":null,"startTime":2395.641,"endTime":2399.163,"body":"and there could also be new paradigms of analytical intent."},{"speaker":null,"startTime":2399.183,"endTime":2403.085,"body":"So, you know, one of the one of my papers that was"},{"speaker":null,"startTime":2403.585,"endTime":2407.768,"body":"presented at VIS with, you know, where Eamon, Gaba and Cindy and others"},{"speaker":null,"startTime":2408.989,"endTime":2413.111,"body":"collaborated with me was just exploring the language of comparisons"},{"speaker":null,"startTime":2414.171,"endTime":2420.015,"body":"and how you can actually interpret comparisons and show meaningful"},{"speaker":null,"startTime":2420.055,"endTime":2420.995,"body":"representations."},{"speaker":null,"startTime":2422.378,"endTime":2430.465,"body":"And I would argue that through language, the whole space of comparisons can be pretty complex"},{"speaker":null,"startTime":2430.545,"endTime":2433.688,"body":"and nuanced when compared to direct manipulation."},{"speaker":null,"startTime":2433.829,"endTime":2436.611,"body":"There's cardinality, you can compare one to one."},{"speaker":null,"startTime":2437.454,"endTime":2445.443,"body":"1 to N, N to N. You can be very specific or vague about the concepts."},{"speaker":null,"startTime":2445.703,"endTime":2448.507,"body":"If I say, when is the safest time to fly?"},{"speaker":null,"startTime":2449.988,"endTime":2451.93,"body":"The data might indicate the morning."},{"speaker":null,"startTime":2452.674,"endTime":2460.078,"body":"But there is an implicit intent there that I not only want morning as my answer, but I probably"},{"speaker":null,"startTime":2460.118,"endTime":2466.561,"body":"want to see how much safer is it to fly in the morning compared to other times of the day, right?"},{"speaker":null,"startTime":2466.621,"endTime":2472.865,"body":"I mean, if flying in the afternoon is slightly worse than the morning, I want to see that, right?"},{"speaker":null,"startTime":2472.925,"endTime":2478.208,"body":"I want to see the whole distribution. So there's so many explicit and implicit ways of expressing"},{"speaker":null,"startTime":2478.288,"endTime":2479.529,"body":"intent that"},{"speaker":null,"startTime":2480.591,"endTime":2486.154,"body":"One could do more easily with language as opposed to a direct manipulation interface,"},{"speaker":null,"startTime":2486.174,"endTime":2490.097,"body":"which kind of opens up a range of interesting problems and opportunities."},{"speaker":null,"startTime":2491.299,"endTime":2496.084,"body":"Yeah. And I guess if I remember well, you also have kind of like a mix of,"},{"speaker":null,"startTime":2496.925,"endTime":2500.268,"body":"I don't remember if this is true for Revisa or us data,"},{"speaker":null,"startTime":2500.328,"endTime":2504.371,"body":"but I have seen systems where there's a natural language component,"},{"speaker":null,"startTime":2504.472,"endTime":2509.276,"body":"but then you can also interact with some elements of the"},{"speaker":null,"startTime":2509.576,"endTime":2513.42,"body":"sentence through direct manipulation, is that correct?"},{"speaker":null,"startTime":2513.522,"endTime":2520.809,"body":"That's right. So, in general, we have found that the pattern of developing mixed-initiative systems"},{"speaker":null,"startTime":2521.53,"endTime":2527.616,"body":"tends to be effective, because there are certain things that I just want to tweak directly. You"},{"speaker":null,"startTime":2527.636,"endTime":2534.022,"body":"know, going back to the earthquake example, if the system comes back with a Richter magnitude"},{"speaker":null,"startTime":2534.062,"endTime":2535.023,"body":"of Sixth and Greater."},{"speaker":null,"startTime":2535.846,"endTime":2542.769,"body":"I may not want to type saying, actually, I want large to be interpreted as four and greater."},{"speaker":null,"startTime":2543.269,"endTime":2548.591,"body":"It's a lot easier for me to just grab the slider and tweak it down to four, right?"},{"speaker":null,"startTime":2548.651,"endTime":2551.372,"body":"So there are certain things that."},{"speaker":null,"startTime":2552.053,"endTime":2556.714,"body":"lends itself a lot more convenient to just click on marks. If I just want to see"},{"speaker":null,"startTime":2558.655,"endTime":2563.936,"body":"some data in a particular region and I see an outlier, it's just easier for me to click on"},{"speaker":null,"startTime":2563.956,"endTime":2569.457,"body":"the outlier rather than asking a question or I might click on it. It's called deictic referencing"},{"speaker":null,"startTime":2569.497,"endTime":2574.438,"body":"where you circle, maybe you lasso select some points and say, tell me more about this."},{"speaker":null,"startTime":2576.178,"endTime":2578.678,"body":"And this is a very complex concept."},{"speaker":null,"startTime":2578.738,"endTime":2579.619,"body":"What does this mean?"},{"speaker":null,"startTime":2580.559,"endTime":2585.462,"body":"It's hard to express that in language. I mean, we do this all the time as humans, right?"},{"speaker":null,"startTime":2586.042,"endTime":2591.305,"body":"We point at things and say this. We don't explain and describe the object that we are pointing to."},{"speaker":null,"startTime":2592.285,"endTime":2598.728,"body":"So I think there is a place for every sort of interaction and the key challenge is to figure"},{"speaker":null,"startTime":2598.828,"endTime":2606.172,"body":"out which flavors of interaction lend itself better for certain types of questions and intent."},{"speaker":null,"startTime":2606.372,"endTime":2607.813,"body":"Yeah, that's so interesting."},{"speaker":null,"startTime":2610.654,"endTime":2614.197,"body":"Yeah, I'm now curious, new ideas pop in my mind."},{"speaker":null,"startTime":2615.098,"endTime":2616.459,"body":"So going back to voice,"},{"speaker":null,"startTime":2616.579,"endTime":2618.18,"body":"I'm wondering if you ever explored"},{"speaker":null,"startTime":2619.121,"endTime":2621.783,"body":"the voice channel as an output, right?"},{"speaker":null,"startTime":2621.843,"endTime":2624.105,"body":"So what if I want to communicate data"},{"speaker":null,"startTime":2624.746,"endTime":2626.347,"body":"through say Alexa, right?"},{"speaker":null,"startTime":2626.407,"endTime":2629.149,"body":"Alexa is telling me, rather than visualizing something,"},{"speaker":null,"startTime":2629.189,"endTime":2632.111,"body":"it's telling me something about the data."},{"speaker":null,"startTime":2632.151,"endTime":2634.693,"body":"How do you communicate data verbally?"},{"speaker":null,"startTime":2635.696,"endTime":2644.344,"body":"Yeah, so we actually had a paper at CHI last year, where we explored analytical chatbots"},{"speaker":null,"startTime":2645.345,"endTime":2651.471,"body":"using this Grecian model of cooperative conversation. And we essentially studied"},{"speaker":null,"startTime":2651.631,"endTime":2655.555,"body":"the behavior of three flavors of chatbots using Slack."},{"speaker":null,"startTime":2656.418,"endTime":2662.381,"body":"where it's text-based input and the output is a combination of text and images."},{"speaker":null,"startTime":2663.742,"endTime":2666.283,"body":"We had just a text chatbot,"},{"speaker":null,"startTime":2666.483,"endTime":2671.085,"body":"which did not have any provision for any imagery or charts,"},{"speaker":null,"startTime":2671.125,"endTime":2675.988,"body":"Then we had a pure voice-based chatbot that was using the Echo device."},{"speaker":null,"startTime":2676.749,"endTime":2682.434,"body":"And so we first did kind of Wizard of Oz studies to just understand the expectations"},{"speaker":null,"startTime":2683.435,"endTime":2688.18,"body":"with respect to these various mediums. It's kind of the Marshall McLuhan paradigm of"},{"speaker":null,"startTime":2688.32,"endTime":2694.065,"body":"medium is the message, right? And it was very interesting, particularly with voice,"},{"speaker":null,"startTime":2694.966,"endTime":2697.128,"body":"but not surprising if you think about it."},{"speaker":null,"startTime":2698.089,"endTime":2700.411,"body":"So people's working memory is very limited."},{"speaker":null,"startTime":2701.494,"endTime":2709.72,"body":"Um, so if I ask what are the most expensive wines in Napa and this voice chat bot goes"},{"speaker":null,"startTime":2710.18,"endTime":2716.805,"body":"on and gives me a speech about all the, you know, the top five wines by the time it tells"},{"speaker":null,"startTime":2716.865,"endTime":2719.087,"body":"me the last one, I would have forgotten the first one."},{"speaker":null,"startTime":2720.429,"endTime":2726.394,"body":"And so what we realized is people, and this is something that has been studied in"},{"speaker":null,"startTime":2726.794,"endTime":2731.738,"body":"kind of general voice chat bots, where it will give you a snippet of information saying, you"},{"speaker":null,"startTime":2731.758,"endTime":2737.582,"body":"know, the most expensive wine is this. Do you want to hear about the others? And so it's like a"},{"speaker":null,"startTime":2737.702,"endTime":2742.746,"body":"follow-up question, almost like this back and forth. And you're like, sure, tell me the next one."},{"speaker":null,"startTime":2743.206,"endTime":2746.129,"body":"And so it's like conversation chunking."},{"speaker":null,"startTime":2747.751,"endTime":2752.214,"body":"The other thing that we noticed was the issue around trust."},{"speaker":null,"startTime":2752.854,"endTime":2754.815,"body":"When people, we were using,"},{"speaker":null,"startTime":2754.875,"endTime":2756.656,"body":"so we implemented these chatbots"},{"speaker":null,"startTime":2756.756,"endTime":2759.458,"body":"and they all shared a common parser."},{"speaker":null,"startTime":2760.539,"endTime":2763.1,"body":"So while we knew that the performance"},{"speaker":null,"startTime":2763.14,"endTime":2765.722,"body":"of all these chatbots were comparable to one another,"},{"speaker":null,"startTime":2766.342,"endTime":2770.024,"body":"we found that people trusted voice chatbots less"},{"speaker":null,"startTime":2770.585,"endTime":2772.306,"body":"when compared to the text ones."},{"speaker":null,"startTime":2772.826,"endTime":2776.948,"body":"So they wanted the voice chatbot to repeat the question."},{"speaker":null,"startTime":2778.23,"endTime":2779.15,"body":"Just to make sure."},{"speaker":null,"startTime":2779.55,"endTime":2780.591,"body":"Just to make sure."},{"speaker":null,"startTime":2780.611,"endTime":2781.491,"body":"They got it right."},{"speaker":null,"startTime":2781.611,"endTime":2783.352,"body":"They got it, exactly."},{"speaker":null,"startTime":2783.892,"endTime":2785.152,"body":"So it's very interesting."},{"speaker":null,"startTime":2786.093,"endTime":2793.135,"body":"So we found a lot of differences, and we also found the types of intents that people tend"},{"speaker":null,"startTime":2793.175,"endTime":2797.056,"body":"to ask of text versus voice chatbots to be different."},{"speaker":null,"startTime":2798.277,"endTime":2804.339,"body":"With voice, it's almost always these fact-finding questions with single responses."},{"speaker":null,"startTime":2806.06,"endTime":2808.501,"body":"You know, kind of like how you ask Google,"},{"speaker":null,"startTime":2808.541,"endTime":2810.442,"body":"tell me what the weather is going to be, right?"},{"speaker":null,"startTime":2812.063,"endTime":2813.664,"body":"So it's like these single ones."},{"speaker":null,"startTime":2813.704,"endTime":2815.785,"body":"You know, what are the sales looking"},{"speaker":null,"startTime":2815.865,"endTime":2817.326,"body":"like today for this region?"},{"speaker":null,"startTime":2818.987,"endTime":2821.508,"body":"But with other types of chatbots,"},{"speaker":null,"startTime":2821.928,"endTime":2824.63,"body":"the questions are definitely much more complex."},{"speaker":null,"startTime":2825.17,"endTime":2826.751,"body":"They are much more compound."},{"speaker":null,"startTime":2826.791,"endTime":2829.573,"body":"People tend to chain multiple questions together."},{"speaker":null,"startTime":2830.093,"endTime":2833.635,"body":"So it kind of goes back to that humans adapting to the computer"},{"speaker":null,"startTime":2833.675,"endTime":2834.595,"body":"speak of the system."},{"speaker":null,"startTime":2835.917,"endTime":2843.401,"body":"But the new insight we learned was humans also adapt to their own limitations of interacting"},{"speaker":null,"startTime":2843.421,"endTime":2846.823,"body":"with the system, because they know that they have a limited working memory."},{"speaker":null,"startTime":2847.904,"endTime":2852.926,"body":"And so they figure out, okay, to circumvent my limited working memory,"},{"speaker":null,"startTime":2853.066,"endTime":2856.909,"body":"let me simplify the types of questions that I will ask of these voice chat bots."},{"speaker":null,"startTime":2859.171,"endTime":2865.137,"body":"Very interesting, yeah. And do you think, like when, I think we touched a bit on that, but"},{"speaker":null,"startTime":2865.638,"endTime":2871.343,"body":"like some of your systems respond with a chart and then others respond with an answer or like a fact?"},{"speaker":null,"startTime":2871.984,"endTime":2872.304,"body":"Yeah."},{"speaker":null,"startTime":2872.344,"endTime":2878.671,"body":"Like how, when, when is what being asked for or what was better when?"},{"speaker":null,"startTime":2878.691,"endTime":2880.352,"body":"Yeah, I mean, yeah."},{"speaker":null,"startTime":2880.372,"endTime":2880.793,"body":"depend on the"},{"speaker":null,"startTime":2882.998,"endTime":2884.338,"body":"I think it's all of the above."},{"speaker":null,"startTime":2885.699,"endTime":2892.582,"body":"When you have the luxury of a text interface with real estate, people want the single answer,"},{"speaker":null,"startTime":2893.502,"endTime":2897.764,"body":"But they might want additional context that supports the answer."},{"speaker":null,"startTime":2898.906,"endTime":2905.349,"body":"They might want the distribution of other regions, how well it's like, you know, especially"},{"speaker":null,"startTime":2905.389,"endTime":2911.071,"body":"these superlative questions, you know, the cheapest wine or the best selling product."},{"speaker":null,"startTime":2911.391,"endTime":2918.154,"body":"People want to know how much better or worse is that answer when compared to the rest of"},{"speaker":null,"startTime":2918.174,"endTime":2920.855,"body":"the data distribution is very important to people."},{"speaker":null,"startTime":2921.337,"endTime":2924.998,"body":"It's the same when you search for a product, you always get these big tables with the feature"},{"speaker":null,"startTime":2925.018,"endTime":2928.159,"body":"comparison and still the winner is marked, right?"},{"speaker":null,"startTime":2928.179,"endTime":2933.261,"body":"But you want to see the full matrix, just know that everything's correct, even though"},{"speaker":null,"startTime":2933.301,"endTime":2936.802,"body":"everything is made up anyways on these sites."},{"speaker":null,"startTime":2936.922,"endTime":2937.362,"body":"Yeah."},{"speaker":null,"startTime":2937.522,"endTime":2940.663,"body":"So this is not a new paradigm."},{"speaker":null,"startTime":2940.723,"endTime":2945.705,"body":"I mean, search systems have been and recommendation systems have been doing it for ages, right?"},{"speaker":null,"startTime":2946.165,"endTime":2950.286,"body":"Like if I go to Google and I type in my flight status number."},{"speaker":null,"startTime":2951.167,"endTime":2955.77,"body":"it's a fact-finding question, right? I type in my flight number and it'll get me the status,"},{"speaker":null,"startTime":2956.431,"endTime":2962.255,"body":"but there are other documents beneath it. The page just doesn't contain only my answer."},{"speaker":null,"startTime":2962.395,"endTime":2968.259,"body":"There are articles that support, you know, my query and there's other stuff going on."},{"speaker":null,"startTime":2968.979,"endTime":2973.822,"body":"The same thing with something like Amazon. I might be very specific and I want like"},{"speaker":null,"startTime":2974.383,"endTime":2978.485,"body":"Like a polka dot, you know, pink shoe made out of this brand, right?"},{"speaker":null,"startTime":2979.547,"endTime":2984.348,"body":"And maybe there's only one pink polka dot pair of shoes,"},{"speaker":null,"startTime":2985.009,"endTime":2988.329,"body":"but Amazon will show other related stuff."},{"speaker":null,"startTime":2988.589,"endTime":2991.35,"body":"And you could argue that's just their business model"},{"speaker":null,"startTime":2991.37,"endTime":2993.631,"body":"because they want to entice you to buy more stuff,"},{"speaker":null,"startTime":2994.411,"endTime":2996.651,"body":"but it's also just context."},{"speaker":null,"startTime":2996.951,"endTime":3000.012,"body":"People want additional information in addition"},{"speaker":null,"startTime":3000.072,"endTime":3002.353,"body":"to something that specifically answers"},{"speaker":null,"startTime":3002.693,"endTime":3004.373,"body":"or satisfies their intent."},{"speaker":null,"startTime":3004.714,"endTime":3005.094,"body":"Yeah."},{"speaker":null,"startTime":3005.474,"endTime":3011.318,"body":"There's also some generosity in providing a bit more around it than just literally what"},{"speaker":null,"startTime":3011.358,"endTime":3011.918,"body":"you ask for."},{"speaker":null,"startTime":3011.958,"endTime":3012.278,"body":"Right."},{"speaker":null,"startTime":3012.619,"endTime":3017.041,"body":"It's like the thing you actually ask for is like the bare minimum, basically."},{"speaker":null,"startTime":3017.161,"endTime":3020.884,"body":"And it's good to provide something extra with it."},{"speaker":null,"startTime":3021.384,"endTime":3021.504,"body":"Yeah."},{"speaker":null,"startTime":3021.564,"endTime":3023.205,"body":"If somebody asks you, how are you doing?"},{"speaker":null,"startTime":3023.265,"endTime":3024.266,"body":"And you say, I'm fine."},{"speaker":null,"startTime":3024.906,"endTime":3026.007,"body":"And you just stop there."},{"speaker":null,"startTime":3026.908,"endTime":3027.288,"body":"It's weird."},{"speaker":null,"startTime":3027.448,"endTime":3028.829,"body":"It's a German way of answering."},{"speaker":null,"startTime":3028.849,"endTime":3030.45,"body":"That's a German way of answering."},{"speaker":null,"startTime":3034.334,"endTime":3039.636,"body":"And then we're back to these flows and the processes and there's actually a mutual process"},{"speaker":null,"startTime":3039.696,"endTime":3040.356,"body":"going on."},{"speaker":null,"startTime":3040.456,"endTime":3043.418,"body":"And I think that's so fascinating."},{"speaker":null,"startTime":3043.438,"endTime":3043.558,"body":"You"},{"speaker":null,"startTime":3053.303,"endTime":3058.848,"body":"Yeah, it's called cooperative conversation and Grice came up with these, you know, principles"},{"speaker":null,"startTime":3058.908,"endTime":3062.972,"body":"around cooperative conversation. There is a back and forth, there is a give and take,"},{"speaker":null,"startTime":3064.433,"endTime":3072.16,"body":"there is a notion of manner and relation and the quality of the content and the amount of content,"},{"speaker":null,"startTime":3072.18,"endTime":3074.542,"body":"and keeping it very relevant, but..."},{"speaker":null,"startTime":3075.465,"endTime":3076.687,"body":"making sure you're polite."},{"speaker":null,"startTime":3076.947,"endTime":3079.25,"body":"So there's so much that's going on,"},{"speaker":null,"startTime":3079.71,"endTime":3083.335,"body":"and we have found that to a large extent,"},{"speaker":null,"startTime":3083.355,"endTime":3086.959,"body":"a lot of those cooperative conversation Maxines"},{"speaker":null,"startTime":3087.54,"endTime":3090.623,"body":"hold good even in the context of analytical chatbots."},{"speaker":null,"startTime":3091.927,"endTime":3092.468,"body":"Yeah."},{"speaker":null,"startTime":3092.528,"endTime":3093.149,"body":"That's interesting."},{"speaker":null,"startTime":3093.189,"endTime":3093.349,"body":"Yeah."},{"speaker":null,"startTime":3093.369,"endTime":3098.595,"body":"Also in the book, you mentioned this idea of register and tone, like in natural language"},{"speaker":null,"startTime":3098.635,"endTime":3105.602,"body":"is not just what is said, but also how it's said and certain like, yeah, choice of formulations"},{"speaker":null,"startTime":3105.642,"endTime":3106.083,"body":"of words."},{"speaker":null,"startTime":3106.483,"endTime":3109.827,"body":"just a certain maybe social status."},{"speaker":null,"startTime":3109.348,"endTime":3113.896,"body":"social status between the participants or you know all these little things between the"},{"speaker":null,"startTime":3113.936,"endTime":3114.817,"body":"lines basically."},{"speaker":null,"startTime":3114.857,"endTime":3115.198,"body":"Yeah."},{"speaker":null,"startTime":3115.238,"endTime":3115.378,"body":"Yeah."},{"speaker":null,"startTime":3126.247,"endTime":3132.813,"body":"But I've also been very lately interested in, you know, data storytelling and the types of words"},{"speaker":null,"startTime":3133.053,"endTime":3139.718,"body":"and the choice of words that you use in these stories can also set the tone."},{"speaker":null,"startTime":3139.738,"endTime":3143.402,"body":"And if you have any questions, please feel free to contact me."},{"speaker":null,"startTime":3142.042,"endTime":3143.602,"body":"It depends on the audience."},{"speaker":null,"startTime":3143.722,"endTime":3147.463,"body":"I mean, the running joke when we wrote the book, actually,"},{"speaker":null,"startTime":3148.583,"endTime":3150.383,"body":"and Bridget will laugh about this,"},{"speaker":null,"startTime":3150.504,"endTime":3151.604,"body":"is she would always tell me,"},{"speaker":null,"startTime":3151.684,"endTime":3153.524,"body":"Vidya, you need to lower the register"},{"speaker":null,"startTime":3153.564,"endTime":3154.884,"body":"when you're writing your chapter"},{"speaker":null,"startTime":3154.984,"endTime":3157.765,"body":"because this is not just a PhD audience."},{"speaker":null,"startTime":3157.825,"endTime":3160.706,"body":"So I appreciated her candid feedback"},{"speaker":null,"startTime":3160.766,"endTime":3163.246,"body":"and I use it all the time now."},{"speaker":null,"startTime":3163.366,"endTime":3166.847,"body":"It's like, you have to figure out who your audience is"},{"speaker":null,"startTime":3167.127,"endTime":3170.148,"body":"and find the right level of information"},{"speaker":null,"startTime":3170.228,"endTime":3171.828,"body":"and how that is communicated."},{"speaker":null,"startTime":3171.911,"endTime":3172.794,"body":"back to the user."},{"speaker":null,"startTime":3174.709,"endTime":3179.997,"body":"So it's both about content, but also again, the form in which the framing, right, the"},{"speaker":null,"startTime":3180.017,"endTime":3180.438,"body":"It's creamy."},{"speaker":null,"startTime":3200.976,"endTime":3204.978,"body":"you know, just giving me a meta table and saying,"},{"speaker":null,"startTime":3205.078,"endTime":3206.639,"body":"what question would you like to ask?"},{"speaker":null,"startTime":3206.719,"endTime":3208.6,"body":"That's a very different sort of tone."},{"speaker":null,"startTime":3209.02,"endTime":3212.022,"body":"And it really influences how people interact"},{"speaker":null,"startTime":3212.062,"endTime":3214.664,"body":"with these systems and the systems expectations."},{"speaker":null,"startTime":3215.164,"endTime":3217.625,"body":"So Vidya, one thing I'm curious about,"},{"speaker":null,"startTime":3219.006,"endTime":3222.448,"body":"do you see a role of language as a way to,"},{"speaker":null,"startTime":3223.409,"endTime":3227.591,"body":"say, help people interpret data visualizations,"},{"speaker":null,"startTime":3227.671,"endTime":3231.994,"body":"maybe through annotations or some sort of guidance."},{"speaker":null,"startTime":3232.555,"endTime":3236.417,"body":"I believe there is, I can't recall if you're doing that type of research, but I've seen"},{"speaker":null,"startTime":3237.017,"endTime":3243.819,"body":"similar things, either from Tableau or other researchers, and, um, this idea that you can"},{"speaker":null,"startTime":3243.839,"endTime":3243.999,"body":"generate..."},{"speaker":null,"startTime":3244.019,"endTime":3245.84,"body":"Accessibility, obviously, right?"},{"speaker":null,"startTime":3246,"endTime":3250.422,"body":"Providing alternative, uh, means of accessing the same info."},{"speaker":null,"startTime":3250.602,"endTime":3250.802,"body":"Yeah."},{"speaker":null,"startTime":3251.042,"endTime":3251.362,"body":"Right."},{"speaker":null,"startTime":3251.582,"endTime":3259.085,"body":"So, um, actually when we were writing the book, um, we had a chapter on, um, you know,"},{"speaker":null,"startTime":3259.145,"endTime":3259.685,"body":"in charts."},{"speaker":null,"startTime":3261.006,"endTime":3265.209,"body":"And, you know, we were curious whether, because, you know, a lot of the practitioners,"},{"speaker":null,"startTime":3266.169,"endTime":3270.171,"body":"and even we have been told about, you know, this whole data-ink ratio, right? I mean,"},{"speaker":null,"startTime":3270.491,"endTime":3275.914,"body":"thanks to Tufti and other books. It's like, oh, you know, you don't want to add too much text."},{"speaker":null,"startTime":3275.974,"endTime":3281.638,"body":"The charts want to speak for themselves, just add a little bit. And then we did this, you know,"},{"speaker":null,"startTime":3281.678,"endTime":3287.701,"body":"project, you know, which was a vis paper that was presented by Chase Stokes from UC Berkeley."},{"speaker":null,"startTime":3288.302,"endTime":3290.886,"body":"where we we really wanted to understand."},{"speaker":null,"startTime":3288.321,"endTime":3289.362,"body":"Thank you."},{"speaker":null,"startTime":3290.906,"endTime":3292.308,"body":"Oh yeah, that's a super interesting one."},{"speaker":null,"startTime":3292.588,"endTime":3292.909,"body":"Right?"},{"speaker":null,"startTime":3292.969,"endTime":3293.069,"body":"you"},{"speaker":null,"startTime":3295.734,"endTime":3299.897,"body":"Is there a notion of too much text is too bad?"},{"speaker":null,"startTime":3300.458,"endTime":3303.88,"body":"And it turns out, no, people actually love text."},{"speaker":null,"startTime":3305.041,"endTime":3310.286,"body":"And quite often, you know, there are so many semantic levels of text, right?"},{"speaker":null,"startTime":3310.326,"endTime":3315.13,"body":"There's, you know, text that describes very low-level statistical properties, all the"},{"speaker":null,"startTime":3315.15,"endTime":3318.793,"body":"way to text that describes key takeaways, and, you know, there's been some work that"},{"speaker":null,"startTime":3318.833,"endTime":3321.155,"body":"I've done with Manish in that space as well."},{"speaker":null,"startTime":3322.357,"endTime":3330.2,"body":"But, yeah, to your point, I think there is a very exciting opportunity around figuring"},{"speaker":null,"startTime":3330.3,"endTime":3335.642,"body":"out how text can be used along with charts as a first-class citizen for people to understand"},{"speaker":null,"startTime":3335.682,"endTime":3336.162,"body":"their data."},{"speaker":null,"startTime":3336.262,"endTime":3343.985,"body":"And they can be pretty effective, text annotations and titles and descriptions as scaffolds to"},{"speaker":null,"startTime":3344.025,"endTime":3344.425,"body":"the user."},{"speaker":null,"startTime":3344.485,"endTime":3348.647,"body":"And this kind of comes back to something I think that Moritz, you were alluding to, that"},{"speaker":null,"startTime":3348.587,"endTime":3355.934,"body":"at the beginning of this podcast was this cold slate problem where users struggle in"},{"speaker":null,"startTime":3355.974,"endTime":3357.856,"body":"terms of what they can ask of the data."},{"speaker":null,"startTime":3358.636,"endTime":3364.482,"body":"And you can use text in conjunction with recommendations saying, you know what, you asked about this,"},{"speaker":null,"startTime":3364.802,"endTime":3368.845,"body":"This is the answer, but there are all these other things that you can ask."},{"speaker":null,"startTime":3370.348,"endTime":3376.59,"body":"And so, yeah, I think it can really be an effective way of scaffolding the conversation."},{"speaker":null,"startTime":3376.71,"endTime":3383.512,"body":"It's also very helpful for answering more complex analytical intents,"},{"speaker":null,"startTime":3383.572,"endTime":3387.093,"body":"especially these why questions. I think why questions are very hard."},{"speaker":null,"startTime":3387.113,"endTime":3388.773,"body":"that we haven't really cracked that nut."},{"speaker":null,"startTime":3389.133,"endTime":3391.194,"body":"It's like, why is this phenomena happening?"},{"speaker":null,"startTime":3392.695,"endTime":3398.338,"body":"If I am looking at the pound and U.S. dollar fluctuation around Brexit and I say,"},{"speaker":null,"startTime":3398.618,"endTime":3404.58,"body":"why is this drop happening, the data does not really have an answer and I have to go outside,"},{"speaker":null,"startTime":3405.001,"endTime":3410.323,"body":"you know, probably use natural language, go to access web articles and come up with"},{"speaker":null,"startTime":3411.185,"endTime":3414.609,"body":"a summary of my understanding of what happened and bring that"},{"speaker":null,"startTime":3414.669,"endTime":3418.313,"body":"back and formulate a very pithy annotation to the user."},{"speaker":null,"startTime":3419.094,"endTime":3423.379,"body":"So supporting these why questions is going to be a place where"},{"speaker":null,"startTime":3423.44,"endTime":3427.725,"body":"text is definitely going to play a very important role."},{"speaker":null,"startTime":3429.208,"endTime":3434.31,"body":"How do you feel about the whole new generation of deep learning-based models in this context"},{"speaker":null,"startTime":3434.35,"endTime":3436.37,"body":"like GPT-3, the transformer model?"},{"speaker":null,"startTime":3436.41,"endTime":3443.013,"body":"So there's a whole \u2013 seems to be a whole paradigm shift in that space and like GPT-3"},{"speaker":null,"startTime":3443.093,"endTime":3448.495,"body":"is a large language model trained on all of the internet as it seems, that seems to be"},{"speaker":null,"startTime":3448.595,"endTime":3451.436,"body":"able to answer like a lot of these common sense..."},{"speaker":null,"startTime":3451.636,"endTime":3451.996,"body":"Yeah."},{"speaker":null,"startTime":3462.733,"endTime":3465.795,"body":"What's your feeling?"},{"speaker":null,"startTime":3465.855,"endTime":3474.299,"body":"Language models have really come a long way over the recent years, and GPT-3 has been"},{"speaker":null,"startTime":3474.479,"endTime":3480.582,"body":"extremely promising for answering and learning certain types of intents and questions and"},{"speaker":null,"startTime":3480.602,"endTime":3486.245,"body":"being able to reason about information and even just learning stuff that goes above and"},{"speaker":null,"startTime":3486.285,"endTime":3488.126,"body":"beyond the initial training set."},{"speaker":null,"startTime":3489.528,"endTime":3494.229,"body":"What I have observed, though, with these large language"},{"speaker":null,"startTime":3494.289,"endTime":3499.151,"body":"models when it comes to data analysis and data exploration"},{"speaker":null,"startTime":3499.251,"endTime":3503.013,"body":"is that their capabilities around numeracy"},{"speaker":null,"startTime":3503.953,"endTime":3506.974,"body":"and understanding the numeracy aspects of data"},{"speaker":null,"startTime":3507.434,"endTime":3508.475,"body":"is still very limited."},{"speaker":null,"startTime":3511.437,"endTime":3515.44,"body":"I am sure with systems like DALI, the graphicacy aspect"},{"speaker":null,"startTime":3515.68,"endTime":3518.182,"body":"might be teachable at some point."},{"speaker":null,"startTime":3518.282,"endTime":3520.403,"body":"We might get to a point where we ask a question,"},{"speaker":null,"startTime":3520.463,"endTime":3522.545,"body":"and it'll learn to generate an effective chart."},{"speaker":null,"startTime":3524.226,"endTime":3528.909,"body":"So DALI generates images based on prompts, actual texts."},{"speaker":null,"startTime":3529.17,"endTime":3533.052,"body":"But the numerical understanding of data phenomena"},{"speaker":null,"startTime":3533.393,"endTime":3535.514,"body":"is still a hard problem to learn,"},{"speaker":null,"startTime":3535.594,"endTime":3537.215,"body":"because it's extremely nuanced."},{"speaker":null,"startTime":3537.235,"endTime":3538.216,"body":"You"},{"speaker":null,"startTime":3539.298,"endTime":3542.839,"body":"But I don't know, it might head to a place where it is."},{"speaker":null,"startTime":3542.959,"endTime":3548.941,"body":"So that being said, being able to take these traditional"},{"speaker":null,"startTime":3549.021,"endTime":3552.722,"body":"grammar-based approaches that I have used as well in a lot of"},{"speaker":null,"startTime":3552.782,"endTime":3557.224,"body":"my research and retrofitting it with large language models."},{"speaker":null,"startTime":3557.885,"endTime":3566.392,"body":"can satisfy a good number of analytical intents. But at this point, I have found that you still"},{"speaker":null,"startTime":3566.472,"endTime":3573.077,"body":"have to augment these language models with additional logic that might come from these"},{"speaker":null,"startTime":3573.357,"endTime":3580.443,"body":"grammars or heuristics to solve numerical understanding of data that these models simply"},{"speaker":null,"startTime":3581.164,"endTime":3584.907,"body":"are unable to do or can do in a limited fashion."},{"speaker":null,"startTime":3585.529,"endTime":3585.809,"body":"Yeah."},{"speaker":null,"startTime":3586.209,"endTime":3586.39,"body":"Yeah."},{"speaker":null,"startTime":3586.85,"endTime":3586.97,"body":"Yeah."},{"speaker":null,"startTime":3586.99,"endTime":3594.317,"body":"But they could add that aspect of providing somebody hints with when does the gold price"},{"speaker":null,"startTime":3594.417,"endTime":3596.359,"body":"drop generally, right?"},{"speaker":null,"startTime":3596.499,"endTime":3601.363,"body":"And might that be related to that specific piece of data I have here?"},{"speaker":null,"startTime":3601.383,"endTime":3601.684,"body":"That's right."},{"speaker":null,"startTime":3602.144,"endTime":3602.524,"body":"You're right."},{"speaker":null,"startTime":3602.564,"endTime":3605.647,"body":"Probably the synthesis needs to happen."},{"speaker":null,"startTime":3605.888,"endTime":3606.088,"body":"Yeah."},{"speaker":null,"startTime":3606.75,"endTime":3608.091,"body":"So I think that is where we are,"},{"speaker":null,"startTime":3608.351,"endTime":3611.552,"body":"but if we talk in a couple of years,"},{"speaker":null,"startTime":3611.792,"endTime":3612.733,"body":"it might be different."},{"speaker":null,"startTime":3612.793,"endTime":3615.474,"body":"I mean, this area is pretty cutting edge"},{"speaker":null,"startTime":3615.534,"endTime":3617.615,"body":"and things have been evolving quite quickly,"},{"speaker":null,"startTime":3617.675,"endTime":3618.495,"body":"which is exciting."},{"speaker":null,"startTime":3619.996,"endTime":3620.196,"body":"Yeah."},{"speaker":null,"startTime":3619.997,"endTime":3624.561,"body":"Yeah, this is a space where I feel like something's going to happen, something's"},{"speaker":null,"startTime":3624.641,"endTime":3625.962,"body":"going to happen, right?"},{"speaker":null,"startTime":3625.982,"endTime":3629.445,"body":"It's like, you can almost feel it in the air, right?"},{"speaker":null,"startTime":3629.505,"endTime":3629.805,"body":"Yeah."},{"speaker":null,"startTime":3630.466,"endTime":3632.868,"body":"So it's, or maybe not, but I don't know."},{"speaker":null,"startTime":3632.888,"endTime":3634.169,"body":"Yeah, it's fascinating."},{"speaker":null,"startTime":3634.189,"endTime":3639.153,"body":"Yeah, because as you say, it feels like, oh, there must be like an amazing application"},{"speaker":null,"startTime":3639.173,"endTime":3639.793,"body":"for all this stuff."},{"speaker":null,"startTime":3639.834,"endTime":3640.494,"body":"Yeah."},{"speaker":null,"startTime":3644.961,"endTime":3649.285,"body":"know that you need, but you just get stuck at the 95, right?"},{"speaker":null,"startTime":3649.506,"endTime":3654.711,"body":"It's like, it's a bit of smoking mirrors, but I will say that, you know, these language models"},{"speaker":null,"startTime":3655.352,"endTime":3660.297,"body":"have surpassed the initial notion of just being smoking mirrors. They actually"},{"speaker":null,"startTime":3660.357,"endTime":3661.498,"body":"worked pretty well for..."},{"speaker":null,"startTime":3669.4,"endTime":3675.403,"body":"I think the challenge and the opportunity is to get them to really understand data and"},{"speaker":null,"startTime":3675.943,"endTime":3678.464,"body":"just the numerical understanding of data."},{"speaker":null,"startTime":3679.485,"endTime":3687.809,"body":"But as I indicated, I think it's a very tractable problem because, you know, it's already, you"},{"speaker":null,"startTime":3687.99,"endTime":3695.814,"body":"figure it out to understand the knowledge of the world, right? So when you know certain properties"},{"speaker":null,"startTime":3695.894,"endTime":3703.098,"body":"and attributes of the data, you know, figuring out the numerical understanding of concepts"},{"speaker":null,"startTime":3703.839,"endTime":3705.84,"body":"It's probably teachable."},{"speaker":null,"startTime":3706.021,"endTime":3706.141,"body":"Yeah."},{"speaker":null,"startTime":3706.561,"endTime":3710.445,"body":"And this combination of systems is interesting, I think, to say, like, maybe there's different"},{"speaker":null,"startTime":3711.185,"endTime":3715.809,"body":"modules or agents that keep each other in check, or like, there's a, like a statistical"},{"speaker":null,"startTime":3715.869,"endTime":3722.934,"body":"like professor module that makes sure all the outputs are statistically correct."},{"speaker":null,"startTime":3723.015,"endTime":3723.475,"body":"That's right."},{"speaker":null,"startTime":3723.735,"endTime":3727.958,"body":"Otherwise sends a new prompt that forces the model to update or something like that."},{"speaker":null,"startTime":3727.978,"endTime":3728.239,"body":"Yeah."},{"speaker":null,"startTime":3730.844,"endTime":3731.485,"body":"Fascinating."},{"speaker":null,"startTime":3731.725,"endTime":3735.525,"body":"I thought maybe we could wrap it up with our last question for Vidya."},{"speaker":null,"startTime":3736.226,"endTime":3739.766,"body":"Maybe you can tell our listeners if they want to learn more about"},{"speaker":null,"startTime":3741.187,"endTime":3743.747,"body":"how to use language in data visualization,"},{"speaker":null,"startTime":3743.807,"endTime":3746.208,"body":"what are the interesting directions?"},{"speaker":null,"startTime":3746.508,"endTime":3748.868,"body":"And I would say even practically, right?"},{"speaker":null,"startTime":3748.988,"endTime":3753.149,"body":"Say I'm a database designer practitioner listening to this."},{"speaker":null,"startTime":3754.229,"endTime":3759.07,"body":"How do I learn to use, to even be more aware of the role of language"},{"speaker":null,"startTime":3759.51,"endTime":3760.691,"body":"Edge in data visualization."},{"speaker":null,"startTime":3761.051,"endTime":3766.796,"body":"Yeah. I will say your book, I guess there was an easy one. Just read the book because the book"},{"speaker":null,"startTime":3780.128,"endTime":3786.236,"body":"you know, various language disciplines, starting from kind of traditional NLP and information"},{"speaker":null,"startTime":3786.296,"endTime":3791.122,"body":"literature all the way to American Sign Language, because that's Bridget's background."},{"speaker":null,"startTime":3791.142,"endTime":3793.185,"body":"That's fascinating too, by the way."},{"speaker":null,"startTime":3793.365,"endTime":3793.885,"body":"Right, right."},{"speaker":null,"startTime":3802.354,"endTime":3808.717,"body":"some of the recent work that's come in the space. I mean, there's a sizable number of papers in the"},{"speaker":null,"startTime":3808.757,"endTime":3816.34,"body":"space of language and visual analysis that is being published at conferences like VIZ and CHI,"},{"speaker":null,"startTime":3817.201,"endTime":3819.522,"body":"even Euroviz, so that's another"},{"speaker":null,"startTime":3819.742,"endTime":3820.402,"body":"recommendation."},{"speaker":null,"startTime":3822.984,"endTime":3829.389,"body":"A lot of people ask me about just kind of core NLP stuff, but there's plenty of courses"},{"speaker":null,"startTime":3829.469,"endTime":3833.953,"body":"on Coursera to just understand language models and intent."},{"speaker":null,"startTime":3834.373,"endTime":3840.178,"body":"So if people are very serious about implementing or trying out natural language algorithms,"},{"speaker":null,"startTime":3840.498,"endTime":3842.119,"body":"There are Python libraries."},{"speaker":null,"startTime":3842.741,"endTime":3848.003,"body":"There's course lectures, including, you know, some really good ones by Chris Manning from Stanford."},{"speaker":null,"startTime":3848.804,"endTime":3856.367,"body":"So I would start with those and then kind of work one's way down to reading the state-of-the-art"},{"speaker":null,"startTime":3856.427,"endTime":3859.829,"body":"literature in this space concerning visual analysis."},{"speaker":null,"startTime":3860.629,"endTime":3865.151,"body":"Cool. It's a whole world out there to discover. It's one of these things once you open that box."},{"speaker":null,"startTime":3865.932,"endTime":3867.492,"body":"It's a Pandora's box, right?"},{"speaker":null,"startTime":3867.772,"endTime":3868.573,"body":"Fits into it."},{"speaker":null,"startTime":3868.953,"endTime":3869.253,"body":"Yeah."},{"speaker":null,"startTime":3870.174,"endTime":3870.955,"body":"Yeah."},{"speaker":null,"startTime":3872.015,"endTime":3872.296,"body":"Yeah."},{"speaker":null,"startTime":3873.636,"endTime":3874.997,"body":"But a really fascinating one."},{"speaker":null,"startTime":3875.037,"endTime":3879.54,"body":"And I think one that has been underappreciated, like everybody's just focused on the visuals"},{"speaker":null,"startTime":3879.6,"endTime":3886.045,"body":"and the visual encoding and perception and the language is more than half of what we"},{"speaker":null,"startTime":3886.085,"endTime":3886.465,"body":"do, right?"},{"speaker":null,"startTime":3886.485,"endTime":3890.508,"body":"Like a chart with zero labels is nothing, right?"},{"speaker":null,"startTime":3890.528,"endTime":3893.65,"body":"So it needs that language content."},{"speaker":null,"startTime":3893.631,"endTime":3897.734,"body":"Yeah, exactly. And what I tell people is when people think about NLP and Viz,"},{"speaker":null,"startTime":3897.934,"endTime":3901.197,"body":"they always almost think about chatbots and talking with data."},{"speaker":null,"startTime":3901.818,"endTime":3905.501,"body":"But there's so many other opportunities in the analytical workflow"},{"speaker":null,"startTime":3905.961,"endTime":3908.343,"body":"where natural language processing can help."},{"speaker":null,"startTime":3908.863,"endTime":3912.126,"body":"It can help with intelligent data transformations under the hood."},{"speaker":null,"startTime":3912.766,"endTime":3916.71,"body":"It can help with joining of tables, especially semantic joins,"},{"speaker":null,"startTime":3916.81,"endTime":3921.313,"body":"where the columns of values may not be identical, but they are related."},{"speaker":null,"startTime":3921.333,"endTime":3922.795,"body":"Thank you."},{"speaker":null,"startTime":3922.596,"endTime":3927.999,"body":"And, you know, coming up with meaningful encodings for data."},{"speaker":null,"startTime":3928.079,"endTime":3934.302,"body":"So there's so many sub-processes as part of that analytical workflow, where semantics"},{"speaker":null,"startTime":3934.922,"endTime":3940.785,"body":"and language understanding is very useful, in addition to kind of the more obvious applications"},{"speaker":null,"startTime":3940.845,"endTime":3941.686,"body":"that we talked about."},{"speaker":null,"startTime":3941.826,"endTime":3943.627,"body":"So you're not going to get bored anytime soon?"},{"speaker":null,"startTime":3944.147,"endTime":3946.749,"body":"No, no, I'm going strong here."},{"speaker":null,"startTime":3948.25,"endTime":3948.65,"body":"Awesome."},{"speaker":null,"startTime":3948.67,"endTime":3951.471,"body":"Yeah."},{"speaker":null,"startTime":3951.971,"endTime":3954.371,"body":"No, it's, I think that's really great advice."},{"speaker":null,"startTime":3954.451,"endTime":3958.692,"body":"And yeah, hopefully our listeners will now always look for the conversations and the"},{"speaker":null,"startTime":3958.712,"endTime":3964.273,"body":"dialogues and the register and the tone, because once you've seen it, you can't unsee it."},{"speaker":null,"startTime":3964.953,"endTime":3965.613,"body":"Exactly."},{"speaker":null,"startTime":3965.753,"endTime":3966.373,"body":"Exactly."},{"speaker":null,"startTime":3967.694,"endTime":3967.994,"body":"Yeah."},{"speaker":null,"startTime":3968.134,"endTime":3968.594,"body":"Wonderful."},{"speaker":null,"startTime":3968.694,"endTime":3969.094,"body":"Yeah."},{"speaker":null,"startTime":3969.154,"endTime":3971.374,"body":"That was really nice."},{"speaker":null,"startTime":3971.754,"endTime":3977.315,"body":"for shedding light on all this fascinating stuff and especially this"},{"speaker":null,"startTime":3978.336,"endTime":3983.26,"body":"It was a great talking about it with you both."},{"speaker":null,"startTime":3983.5,"endTime":3985.081,"body":"I enjoyed the conversation."},{"speaker":null,"startTime":3985.441,"endTime":3985.881,"body":"Wonderful."},{"speaker":null,"startTime":3985.962,"endTime":3989.184,"body":"And we'll check back in a few years how things have developed, right?"},{"speaker":null,"startTime":3991.205,"endTime":3991.725,"body":"Wonderful."},{"speaker":null,"startTime":3991.786,"endTime":3992.286,"body":"That's right."},{"speaker":null,"startTime":3993.086,"endTime":3993.547,"body":"Okay."},{"speaker":null,"startTime":3993.587,"endTime":3994.167,"body":"Thanks so much."},{"speaker":null,"startTime":3995.108,"endTime":3995.568,"body":"See you soon."},{"speaker":null,"startTime":3995.668,"endTime":3995.868,"body":"Yeah."},{"speaker":null,"startTime":3995.908,"endTime":3997.289,"body":"Thank you so much for having me."},{"speaker":null,"startTime":3997.629,"endTime":3999.331,"body":"And I really enjoyed it."},{"speaker":null,"startTime":3999.891,"endTime":4004.934,"body":"And I look forward to listening to it and all the other podcasts that you come out with."},{"speaker":null,"startTime":4004.974,"endTime":4005.815,"body":"So I'm a big fan."},{"speaker":null,"startTime":4005.995,"endTime":4006.876,"body":"So thank you very much."},{"speaker":null,"startTime":4006.997,"endTime":4007.438,"body":"Thank you."},{"speaker":null,"startTime":4007.559,"endTime":4008.22,"body":"Thanks so much."},{"speaker":null,"startTime":4008.24,"endTime":4008.982,"body":"Thank you."},{"speaker":null,"startTime":4009.042,"endTime":4009.844,"body":"All right, take care."},{"speaker":null,"startTime":4010.045,"endTime":4010.266,"body":"Bye."},{"speaker":null,"startTime":4016.334,"endTime":4018.576,"body":"Hey, folks, thanks for listening to Data Stories again."},{"speaker":null,"startTime":4019.316,"endTime":4020.938,"body":"Before you leave, a few last notes."},{"speaker":null,"startTime":4021.238,"endTime":4023.76,"body":"This show is crowdfunded, and you can support us"},{"speaker":null,"startTime":4023.84,"endTime":4026.942,"body":"on Patreon at patreon.com slash datastories,"},{"speaker":null,"startTime":4027.083,"endTime":4030.065,"body":"where we publish monthly previews of upcoming episodes"},{"speaker":null,"startTime":4030.105,"endTime":4030.986,"body":"for our supporters."},{"speaker":null,"startTime":4031.786,"endTime":4034.048,"body":"Or you can also send us a one-time donation"},{"speaker":null,"startTime":4034.429,"endTime":4037.651,"body":"via paypal at paypal.me slash datastories."},{"speaker":null,"startTime":4038.433,"endTime":4040.254,"body":"Or, as a free way to support the show,"},{"speaker":null,"startTime":4040.654,"endTime":4043.335,"body":"if you can spend a couple of minutes rating us on iTunes,"},{"speaker":null,"startTime":4043.395,"endTime":4044.995,"body":"that would be very helpful as well."},{"speaker":null,"startTime":4046.075,"endTime":4048.016,"body":"And here's some information on the many ways"},{"speaker":null,"startTime":4048.056,"endTime":4049.536,"body":"you can get news directly from us."},{"speaker":null,"startTime":4050.017,"endTime":4052.117,"body":"We are on Twitter, Facebook, and Instagram,"},{"speaker":null,"startTime":4052.557,"endTime":4054.658,"body":"so follow us there for the latest updates."},{"speaker":null,"startTime":4055.238,"endTime":4056.959,"body":"We have also a Slack channel"},{"speaker":null,"startTime":4057.359,"endTime":4058.999,"body":"where you can chat with us directly."},{"speaker":null,"startTime":4059.599,"endTime":4063.441,"body":"And to sign up, go to our homepage at datastory.es,"},{"speaker":null,"startTime":4063.961,"endTime":4066.381,"body":"and there you'll find a button at the bottom of the page."},{"speaker":null,"startTime":4066.922,"endTime":4071.187,"body":"And there you can also subscribe to our email newsletter if you want to get news directly"},{"speaker":null,"startTime":4071.227,"endTime":4074.551,"body":"into your inbox and be notified whenever we publish a new episode."},{"speaker":null,"startTime":4074.791,"endTime":4078.875,"body":"That's right. And we love to get in touch with our listeners. So let us know if you want to"},{"speaker":null,"startTime":4078.935,"endTime":4083.5,"body":"suggest a way to improve the show or know any amazing people you want us to invite"},{"speaker":null,"startTime":4084.121,"endTime":4087.124,"body":"or even have any project you want us to talk about."},{"speaker":null,"startTime":4087.507,"endTime":4093.713,"body":"Yeah, absolutely. Don't hesitate to get in touch. Just send us an email at mail at datastory.es."},{"speaker":null,"startTime":4094.234,"endTime":4096.016,"body":"That's all for now. See you next time,"},{"speaker":null,"startTime":4096.216,"endTime":4097.978,"body":"and thanks for listening to Data Stories."}]}