Business as Usual: COVIDcast

Transcription: 

And Shean learning scientists all walk into a bar. What happens next? that come up to the forefront? Tell me what happens.

I don't know what happens.

Okay, so you don't know the punch line taught under What? What's the punch line to that? Top? The site director, VP of Engineering and maniacal treadmill Walker at Google.

I think this is a phenomenally unfair question. I think we should ask not what happens next. But what should we have done differently? No, I'm sorry, I'm channeling.

Okay, we're gonna jump in. Welcome, Zach. I think there's a little bit about you that we need to know. So let's start with Zach. Welcome. And tell us a little bit about who you are. And if any of the things that I talked about in terms of you walk into a bar, were accurate.

Um, yeah, I'm a faculty member at Carnegie Mellon. I am. I've kind of diverse interest. So um, and I've wandered a bit over time, I guess. But right now I am a professor of machine learning and operations research. And apparently, I need to shut my browser window. So they stopped making

this an activity going on in there.

Yeah. Very popular guy.

So we're lucky to have you.

All right, we'll see. All right, hopefully, this is like a problem always like my phone is set up so that if I don't pick up my phone, my browser starts ringing or my Gmail starts ringing. But I always have like seven, Gmail is up and at the same time, and I can't, I'm down. So hopefully, that won't happen. Okay, I'm a, I'm a professor of operations research and machine learning at CMU. So I'm affiliated across departments. And I run a lab that we call the approximately intelligent, approximately correct machine intelligence lab. This is a play both off of proximately, probably approximately correct learning, which is like a framework for, you know, underlying, most like the theory of machine learning. But it's also sort of a play off the way machine learning as applied in the world is usually sort of, you know, only an approximate solution or solving sort of a surrogate problem for what you really care about.

And so we work on problems, including

our key application focus and problems in clinical healthcare. On a more theoretical or methodological side, we work on sort of what are the foundational principles for building systems that'll actually be robust in the real world. So people are really excited about AI thinking they're going to use it in all kinds of environments, and to drive all kinds of decisions. But when you look at what they're actually developing, it's technology that does one thing really well, which is prediction and usually prediction under the assumption that what you've seen in the past is perfectly representative of what you'll see in the future. So we try to sort of go beyond that formulation to to build technology that's maybe more robust in the real world. But that's kind of my my like, kind of professional background. In the context of delfy, though, basically, when when COVID hit a lot of us who had sort of related or adjacent interest or skill sets, sort of look to see how we could help out. And so professors, Roni Rosenfeld and Ryan tibshirani, co founded this epidemiologic sort of forecasting and tracking group called Delphi that traditionally focused on flu forecasting. And they're actually one of the leaders in this area. So they're like a perennial perennial top finisher and say, the CDC flu forecasting competitions. They're, they're a CDC Center of Excellence for flu forecasting. So obviously, once we found ourselves in an epidemic requiring tracking and forecasting of virus spread, there were sort of like natural focal points. So I've joined the Delphi team along with a bunch of other people, the team sort of swelled from eight people to maybe 30 people. And now with the addition of, at least, you know, for the next six months, have some, some generous support from Google, both financially and in the form of 13 of their really talented employees who sort of joined us to help sort of like turbocharge our operations. We're now like a 40 something person operation.

So that's awesome. But let me ask you why do you think I asked you that question about what happens when you walk into a bar because then I'm gonna pass the baton. I want to talk to Todd for a second.

I'm not 100% sure your headlines or your your punch lines going? But I guess something about my background sounds like a setup for a joke.

So your background is you were a jazz musician? I guess you still are once a jazz musician, always. And you played saxophone. How did you pivot into what you're doing now? Are you running it simultaneously?

Um, I at some point, I made a sacrifice. So I grew up. I think I think the the impulse to pursue music or at least the intellectual aspect of it is a lot more like research than it is Like what you do in coursework, so I think if you're a kid trapped in the school system just be horrifically bored, you know, eight hours a day, like there's something about the pursuit of jazz music, which is very much a sort of like autodidactic like, you know, you might take less and someone might point you in some directions, but it's mostly like you have to go decide what information to get from where, you know, it was playing, I also grew up at a time like I grew up in New Rochelle, New York and Branford Marsalis, I was playing, I was a saxophone player and bred for myself live in the same city as me. And I guess, like, the thing I was good at was the saxophone. And so I was able to get Bradford's attention and get lessons privately, like when I was growing up, and so, you know, I think like, when you're in that kind of environment, the person who's like, kind of like, sort of, like, you know, something of a god to you like that age is there, I think that kind of provides a certain kind of sense that you're, you know, like, got to get to meet people like Herbie Hancock and Wayne Shorter and stuff when I was a teenager.

So in terms of switching, I mean, I continued,

you know, I got a secular education. So I didn't go to Conservatory, I went to Columbia, and I studied math and economics. But I was playing music the whole time. And because I was in Colombia, it's easy, because you're in New York, and the, the music, the music thing tends to operate between 11 at night and four in the morning. And school tends to operate between like 9am and, or jam or something. So it kind of worked out, you know, although maybe I didn't get enough sleep. And it was really in my mid 20s, that like, I think I was happier with that balance of like, sort of creative and scholarly pursuits that I had in college. And I think in my mid 20s, something was missing. And I went out and visited a France I was like, I was like in New York, I was kind of broke playing playing gigs that you know, for like $50 like wherever the fat dad or the moldy fig. And I was like living in the way you live in New York when you have no money. So it's just like people vomiting on my sidewalk, and I was like, gonna rent stabilized apartment I was like, full of mold, where the landlord's like, key objective in life is to get us to leave the building, as like, you know, had some some kind of whatever ambivalence and I went out and visited a friend who is doing a PhD actually, in music. He's a composer was doing a PhD at UC Santa Cruz. But like my first exposure, you know, maybe not at first, but I also had some some contacts who were like, researchers in biology at Columbia, so I had some exposure to the scientific community. But then I went out and visited this friend at UC Santa Cruz, and which is just like heaven on earth, you know, it's just like, up on this beautiful Hill, and you're in the woods, and you're one mile away from the beach. And all the food is organic and delicious. And you know, it's kind of sunny enough, you're around, and I just kind of like went and you know, all the eight year olds Look, 30 years old, and everybody's fit. And I just thought, like, I had a sense of like, this is, you know, and then people are arguing with each other all the time, which is like a New York Jew with like, academic inclination is sort of appealing to me, you know, about the, you know, at least the PhD students were, yeah, so I just kind of, I just kind of went on this trip and was like, okay, that's I'm gonna do, I'm gonna do a PhD. And that that was like the first, the first consideration is to do a PhD. And then the second consideration was PhD and what? And so I taught myself how to program earlier and had some natural feeling towards competition, but I was like, maybe a decade or two behind where I like should be. I don't know, I think there's something kind of liberating about like, when you when you're when you come up doing something and you're like, kind of you expect it, you're like, excellent at it. Like it's the thing, you've made your identity. You you get kind of paranoid, you look over your shoulder, like as a musician growing up, you're very aware of like, Who's your age and what have they accomplished. And actually just like, committing a small act of like imposter ism, and just blitzing into a new field is quite liberating. Because the bar was, you know, the expectations for me were so low, it's just like, Oh, that's great. Like, when I published my first paper at a second tier conference, at CMU, that might have just been like, okay, whatever. But you know, for me, it was like, you know, it's sort of like, people are impressed, because it's like, oh, wait a minute, like, the saxophone player is writing a paper and like six months later, so there's something you know, it's like, when you learn a new language, like the bar is lower, but that also frees you up to, like, you're not like, I'm not gonna try unless I'm the best because you're not holding yourself to that you're just kind of having fun with it. So I basically just applied to a bunch of PhDs and I had some faculty from undergrad that had enough faith in me to like, write strong letters, even though I had no research accomplishments whatsoever in life. And I got lucky that when I when I actually faked my way into a Ph. D program and got there, I got there at the right place at the right time. You know, it was 2013, right as machine learning was starting to blow up. And, and it was also a time a period of change. Because in machine learning, we were going from a very, like super mathematical discipline focus primarily on a system of methods called kernel machines, to a very kind of like wild west where neural networks were taking over and nobody really knew how to get them to work and it was new to everybody. And so I was curious if transition are really an opportunity for for someone entering the field. Because, you know, everybody's in some sense, you know, those other skills obviously do come in useful for the people who are already experts at something. But at least in some sense, everybody's starting from scratch. And so I got to enter the field at a moment where everybody was starting from scratch. And there's even a way in which it's an advantage because like, you have no prior commitments, I'm not like, I'm attached to this other methodology. It's like, I just got here, I want to do something that's actually gonna work. And so being able to see things with fresh eyes at that moment, I think I just got one lucky that someone took a chance on me and I got into PhD, maybe too lucky that I got in right at that moment of kind of like a seismic change in that kind of AI world. And then, and then maybe, you know, there are some some execution involved in like actually

getting stuff done. Well, let me let me know because I want to, I want to bring Todd on. So Todd, I mentioned earlier, taut Underwood, he's the site manager, Vice President of Engineering, and maniacal treadmill Walker, as well as very a lot of civic muscle he exerts in our community. So we're thrilled to have him from Google on really quick. But Todd. So, you know, let's talk about this project. Let's talk about Google in the engagement here in terms of what Zach was sort of setting the stage for, so that we can dive in a little bit.

Yeah, thanks, Audrey. It. One of the things that I think for all of us, has been so infuriating and so frustrating about this pandemic, is the lack of understanding and the lack of knowledge in the sense that we are just adrift. And it's not unexpected. Those of us who know anything about science, who, you know, understand how progress in knowledge works, knows it's been just a heartbeat, we still don't understand who gets it who gets sick, how is it transmitted. But for those of us as we're living through this, it is maddening to know that, like, we don't understand, we don't know, we can't predict. And so I think in that context, there's been rightfully a pretty aggressive effort to try to give public policymakers and citizens like us some way of hanging on to, where is this going? Where is this bad? Where is it getting better? What are the factors that contribute to that? The sound sort of like the sounds dreamy, like, oh, we're gonna get public policymakers and dashboards, but it's also for us as like, as humans, as citizens as parents, like, should the schools open? Like, should we all wear masks? Are bars safe? Can I get takeout or not? Like, do I need to, like spray myself down with Clorox? If I touch the UPS box? Like these are all like, these are quite I mean, like, you all have lived through this as well as I have. And we've all gone through these phases of like, I'm terrified to touch things. Wait, the Atlantic says touching things isn't that bad? Like, I wasn't worried about the air. Now I'm terrified about the air like planes are dangerous planes are safe. planes are dangerous, right. So I think like, I think in that context, I have a lot of gratitude for the people who are spending some time trying to figure things out. And among the most exciting aspects about other projects that are trying to figure out where things are going. And so, you know, earlier in the year, Google AI did some effort to try to make some predictions and try to make some forecasts, there's a lot of data, there's a lot of information, right, there's a lot of data, you can get test rates, you can get positive test rates, you can get PCR cycle thresholds, in some cases, you can get hospitalizations, intubations deaths, you can get excess deaths. But putting all of that together in a way that is meaningful in any way is pretty scarce out there. And I know that like I'm not the only nerd on this call, who's built his own spreadsheet scraping data from places and putting it together the way I want, because I'm not satisfied that the county won't report positivity or that you can't get. So I think we're really excited about this project. This is this team is a story team, this team is like it. It's not a team that starting from scratch. It's like oh, maybe we should predict a thing. This is a team that's been analyzing data, understanding the way that data indicates progress of illness, and surfacing information about that for some time, they're respected, they know what they're doing. And this is a this is an extension of existing super promising work. And we're just we're really happy to support it. And I have to say, for our employees here in Pittsburgh and around the country, actually, you know, I think there's one who might be outside of the US as well. Like, we had a ton of applications from some amazing engineers, because they're frustrated too. Like you're sitting there like, it's great, you got a job, that's good, you're getting paid, that's good. You're not at risk, maybe your family's okay, but you're desperate to do anything to contribute to the response that society is having to this pandemic. And so we had a ton, it was very difficult my understanding to select, and I think that got some of the cream of the crop of Google program managers, product managers and software engineers, so really excited to see this happen,

and so are people Coming out of your team as well, local Yeah,

my personal team I lost, I lost two people thoughtfully contributed to this. And their motivation, three, I'm sorry, three. And they're like their motivation is exactly what I said is they, they're happy with their jobs. They love working for Google, they like the challenges they have. They're desperate to actually contribute to the most important thing that's happening to most of us right now. And so this is a this is a phenomenal way for somebody who's a program manager or software engineer or a data nerd to like actually get to do something that probably matters quite a lot.

That's really and so some of this information available now. I mean, is there a way now for us to see some of this because you can see people are asking questions, they want to know about rolling averages, etc. I mean, you really whetting our appetites? Because you articulated very well taught this is, this is a nightmare in terms of making any kind of intelligent sense out of it. So Zachary, is anything available? But,

uh, we we've had, we've actually had data available for, you know, since, yeah, since since I think, you know, early, you know, early into the epidemic, maybe April or something. So there's sort of two primary form factors. And I think they target maybe different audiences, there may be more than two, but two primary ones. One is, if you go to COVID, cast.cmu.edu, we have a live interactive map. And so one danger of this is that we're always, there's always a concern that I think people are so inundated with people, I've seen this kind of style, or some people call it a heat map. But really, it's it's not actually a heat map, it's a it's something called a chloroplast, where the the shade of each region corresponds to like the number associated with that region. So here, what we have is like a view at the county level, but filling in some background among, you know, pulling together, the counties don't have enough information to just give like, an average over the state. And so as long as people is looking at, and you're used to seeing so much data, visualizing more or less this way, whether it's state or county and MSA view, that people get a little bit inundated and distinct, they're all the same. But the key thing is, what is the data that you're visualizing. And so one of the things that we realized early on is like, basically, everyone's looking at, for the most part, like the same two data points, right, people looking at reported cases, confirmed cases, and people are looking at deaths, there's a bunch of other things that are reported in perhaps on reliable ways or varying degrees of reliability. And so we came into it. As you know, people with a background like, you know, Ryan's a famous theoretical statistician, you know, Ronis got a background, also here. So we've always had this, given the statistical strength of the of the group, you know, obviously, that they're, they're sort of practically minded, they've been working on real flute problems since you know, eight years ago, but at the same time, modeling has always been maybe at least a central, you know, like, maybe the central part of the push. And I think, you know, one, one realization here looking at the COVID situation on the fold is that as much as worth statisticians or ml people, we want to think that like stats or ml is like the the juiciest part of the story. Sometimes, like making progress on a real world problem requires a certain sort of humility to recognize that the thing that you think is the sexiest or the you know, whatever is maybe not what's needed right now, or maybe not the biggest choke point. And, and so one of the things you realize is that, that the actual data itself, like, like, we needed more sensors to make sense of the situation. And so when you look at case numbers, you know, people just sort of wind up at this impasse where you're like, Okay, cases are going up. But is it because infections are going up? Or is it because nothing's going up? And of course, you have, like, some people may be like political actors who want to tell you, oh, it's just because of testing. And you have other people who want to say, it's not because of testing at all, and it's just decision and then the truth is somewhere in the middle, more testing means case numbers go up. On the other hand, you know, in case numbers triple in a short period of time, that's probably not just because of testing, right? And so other other pieces of information, give you another window on this, right. So like, for example, imagine a number of cases went down. Is that because infections are going down? Or is that because of a testing shortage? If you're able to see the percent positivity of the test, that can shed and give you a little more confidence because the case numbers are going down, and test positivity is going down? commensurately. This might suggest the reason why things are going down is because the positivity rates, you know, actually less people are positive versus if positivity rate were going up in case numbers are going down. It might suggest that there was a case shortage and people are getting stingier with who they're testing.

Exactly What was that? One of the things that I think you all are doing that other people aren't doing as well as combining those data with other data sets. Can you talk A little bit about that, like, What are you looking at? How does it predict? How does it not? Like? What are maybe what's that? What is some of the most predictive data sets that you're looking at that are separate from the reporting? And what are some of the ones that looked promising that turned out to be useless?

Or? Okay, so just completely just complete. The thought first is just that, right? So we're looking at things like we're leveraging our partnerships with makers of diagnostic equipments, local health providers, and then huge tech companies like Google and Facebook are able to help us do things at a scale that we never could do with our own resources like, so the sources that we have include things like test positivity, fraction of doctor's visits that are due to patients due to COVID, like symptoms, fraction of hospital visits that are due to COVID, like symptoms, then we have these surveys, and the survey is give us another lens, among other things, it gives a unique lens to ask questions about not only disease, but also like things like compliance measures, things like mask wiring, things like and even just to get a completely independent read on read levels of symptoms that doesn't go through the normal reporting chains, like we get to ask people are you experiencing a loss of sense of smell and see, even if that's not by itself doesn't tell you how many cases are we can look track the change of that over time. Now, as far as combining those indicators, there's a number of ways that we combine this is one thing is to, you can do something called we report a combine indicator, that's something like a first principal component. So if you've got like some multivariate signal, and you want to say I want to, I want to distill from that a single signal that is like the signal that explains the most variants, and it's like higher dimensional, you know, so I've got this this group of signals that are moving over time, and I want to turn this into a single a single indicator that is sort of some synthesis of all of them. So that's one thing that we report on the map. So if you look on the covid caste map, you can see the combined indicator. But that's not the only way to combine them. Another thing that we do is we have certain indicators, that intuition tells us our sort of late indicators, and general and other indicators that we're hoping will be like early indicators will give us like a read before everybody knows, before everybody already would have known that there was a problem, things that would have been like, indicate, hey, things are getting bad now before it shows up, right? Like, like the log earlier in advance, we can know and take some kind of intervention or preparation or, you know, whether it's that we need a different lockdown measures are we need to, you know, acquire more ventilators. So one of the things that we do is we see how useful are these features in forecasting models for forecasting the conventional indicators. So for example, you could say, I want to take the survey data and use the survey data in a forecasting model whose job is to predict future deaths or future cases. And then we can look at it and say, one confirmation that the surveys, so the surveys we'd hoped would actually be an early indicator, because people might experience symptoms before they go to the hospital before they get tested. Right. And so we'd hope that like, if it outbreak from come sort of from nothing into an area, we might see it in our survey before, they would see it in the confirmed cases. And the confirmed cases tend to lag deaths by you know, somewhere between like 1420 days often, so and often much more, but there's some, so what we do is what we call like a lag correlation model, or like, we're trying to figure out how past K or you know, current cases of you know, current survey reports, current test positivity, current, you know, all these alternative data sources, forecast future deaths or future confirmed cases. And, you know, by showing correlations of our signals with each other, and by showing correlations of our signals, with, you know, future manifestations of like, you know, official trailing indicators, were able to, like, get more confidence, you know, any one indicator, one thing I'd point out that like any one indicator, it can be really brittle, right? Like, if you just look at confirmed, basically, they might spike on one day, because some cases for whatever reason, got like clogged in in the clerical system and just show it up, you want to know where they're really more cases? Or is this an artifact? And when you look at all these different signals together, you can often use, you know, n minus one as a check on the one you're suspicious of.

So what about the article than the washington post about the mask wearing? Can you talk about that?

Sure. So basically, um, one idea and this is work. You know, a lot of people involved and maybe spearheaded by my colleague, Alex Reinhart, who's a stats professor and one of our leaders and the delfy team. One of the one of the biggest things we have is this daily survey that runs all the time. So I tell you, we have this. We have this Facebook survey that or I say our survey, but the one that Facebook directs traffic to, which is sort of a weekly thing. This is like the the slow burn, where's the Google surveys like the like the fire hose, it's done. The Google survey at its peak got 1.2 million respondents in a single day. And so it's this is something that is costs a lot. And so we were sort of, you know, keep it in reserve over the summer, but we'll reactivate it in the fall, you know, what once Wednesdays is, of course, the daily survey that covers sort of every county every day. In this one, you know, we traditionally started asking people, are you experiencing certain things? Like, are you experiencing loss and smell? Do you know, anyone your community is, but then we started very recently, about a month ago, our plan for like, the next version of the survey was to add an indicator of add a question about compliance, like, like, are you? Are you you know, what fraction of times are you wearing a mask Are you or people in your community wearing masks, and we're able to track now, over time, now we have a month of data on comparative mask wearing over time. And one thing that's becomes very salient is that if you plot say, a chloroplast, like, of, of currents, you know, covid cases per 100,000 people, and you plot a current chloroplast of mask wearing, you know, self reported mask wearing via the survey, you find it, these are completely sort of complimentary or inverse. And in the Washington Post article, they show something, which is just the same thing, but plotted as a not as a chloroplast. But just as a, you know, a scatterplot, where you can see a clear trend line of more math less COVID. Now, of course, you know, I post these things, you know, they're these things are, are interesting for the very reason that they're fitting into their evidence that bears upon a causal story that we all are centrally concerned with, which is doesn't ask wearing. Matter, cause Right, exactly. And, you know, of course, it's a fraught issue of there's a number of confounders, mask wearing a we believe we have a good mechanism in our heads for why mass squaring would lead to less coronavirus. On the other hand, mass squaring is probably also correlated with general compliance with government orders, including other government, you know, including other mandates or other lockdown measures. So, but it's the strength of the correlation is sufficiently striking that I think for people that want us there to say there's, you know, not even a, you know, yeah, I think there's also a question sort of from the perspective of a policymaker a burden of proof, right of like, what, given that the cost of wearing a mask is your faces scratchy for a few hours, and the cost of not wearing a mask, if you're wrong, is you get COVID. There's a question of, do you have sufficient cause to make, you know, to conclude that if nothing else, based on the current evidence, you know, this this, this is the right recommendation for people?

Okay. So, Todd, you want to jump in?

Yeah, there was just a question on the chat. And I think it's an interesting one. Let's see who Greg asked a question about these various COVID tracking apps. And so the topic of contact tracing is pretty important. I think, you know, if we take a step back, what are we talking about here, we're talking about, you know, the closed loop of, there's a disease, the disease is caused by a virus, the virus is transmitted, and we're trying to track it we're trying to like, so Google is spending some money and some people and CMU is spending some money. And some people and people are investing their attention to aggregate data and predict it. But we also when we find a positive case, the next thing we want to do is stop the transmission as close to that positive cases we can so many people have read about these contact tracing apps. And pretty recently, Pennsylvania launched one, the first one that was launched in the world, I think, was Italy. So what happened is, back a little ways ago, China has launched an app the app had, it's, it's, it's difficult to say had no privacy protections, because it was anti privacy. And it's designed on purpose, like maybe that is suitable for the Chinese market. But I think Google and Apple looked at this and said, like, okay, so no one else is gonna put up with that, like, literally no one else is gonna sacrifice 100% of the information about themselves to some unknown parties with no controls. And so they put their, you know, there was a bunch of really smart people. And they created this API that works on all Android phones that are relatively recent, all iPhones that are relatively recent. It's completely privacy protecting because the phone itself, stores the contacts, the contacts are not there. It's not like it's anonymized and someone has the key, there's no way to connect your phone to you except for you. You're the only person who can do that. But your phone is out there trying to listen to other phones and just writing down the numbers. I heard this number I heard this number. I heard this number. I heard this number cool. Every so often, and it connects up to a server and says, Hey, have any of these numbers that I was close to for a while been reported infected? And if it gets a hit, it tells you it says, hey, guess what? You had a close contact of someone who is infected and you should go get a test

but not fun. Right? Well, that was not necessarily Which one?

No, no, it doesn't tell you anything. It's just like, hey.

So leads, like the friends that I've had that have had this happen or like, Who was it? And they spent the next like, you know, like two weeks trying to figure out I was at the cafe and I stood in line, it was more than 15 minutes. And there were three people never who knows. But so the question was, specifically, why is it in the us? We have one per state like that doesn't? So the only answer I have is because the US is not very well organized right now. And we can, you know, we can try to do better as a people, but one per state is way better than none at all, and so pretty strongly encouraged. So here's how this is rolling out. outside of the US, most nation states are doing one for their whole country, inside of the US, most states are doing one per state. And on the server side, there is currently a contact protocol that's being implemented to do the cross state reporting. One of the things most of us don't know i i tested positive. And it turns out, I never got any symptoms. So who knows, but you find out a lot about the public health infrastructure of how positive tests are reported, because I tested positive in Rhode Island. And two hours later, I had a call from a contact tracer in Allegheny County, because that actually works really well. Like they had my address. They knew that address was part of Allegheny County Health Department. And they're the people who called me up. And they do it they have they have privacy protecting. But so I think we're going to look at we're all collectively going to look at trying to get that kind of contact tracing on the apps working across jurisdictions. And I hope we get that working well. But for now, it's strongly include folks to install the COVID cast state application doesn't it works, it doesn't suck your battery out, you don't do anything. And you can everyday say I feel fine or not feel fine, or you can not do that. It's up to you. But every so often if, if you've been out and about and you've actually, unbeknownst to you how to close contact with somebody to test positive, you'll get more.

So are you is Google working anywhere else with any other organizations to partner to, you know, address the issues of COVID? And this data or so yeah.

So yeah, sorry. So like this particular grant, um, that they includes Zacks research group at CMU, also included like there were there were a ton of resources made available for a bunch of other folks, as well. So there's a I can I can go back to the press release. But yeah, it's a it's an international effort. Like there are obviously, we're seeing very different responses and very different data sets and the need to do different kinds of data. So there's a research nonprofit research group in Africa that's teaming with the University in the US. So there's Yeah, there's, there's quite a few other people. It was actually like Google spending a lot of money quite appropriately, I think, trying to get a handle on this right now.

No, that's great. And I really appreciate both of you being here. But here's the question. Here's a question winter's coming. So what's what's can? What's the prognosis for us? And where should we get look for the data?

Are you really like you just like doom and gloom? That's what you're going for here. Audrey is Oh, I want to go

I want to travel Baby, I wanna get out and see the world.

I don't like do you want to get the bad news? Or do you want me to like,

me, I think the first step is maybe to say that

people struggle to like,

being in contact with the state of, say, 14 day forecasting. And this is among the very best people in the world, that epidemiological forecasting the state of 14 day forecasting of what will be, you know, case counts on like, you know, fine geographic level, or whatever is, it's very hard to beat strongman models. And so the case is, like, you know, where, though, it's like, it's hard to know, for sure what's gonna happen before, but to say what's gonna happen in three months is, you know, requires some humility that we did, there's a huge amount of uncertainty. That said, it's like, there are things that, you know, we just can't rule out and need to be vigilant about, right like we, you know, like there's, there's, there's a number of unknowns that I'm part of the problem is that we actually have interventions and the choice of the choice of these interventions like what we do, how we lock down how we don't, how good we are at enforcing certain kind of measures how good we are at deploying a contact tracing and actually executing and following up, you know, it's, in some ways, it's a bit easier for countries maybe at lower volumes of cases to have dedicated contracts working contact tracing workers that are order of magnitude equal to the number of people that are infected, and it's a lot harder when you have too many infected people. All these choices are going to actually influence what happens so it's Like there is a possible future where we get this thing under control. I think I'm optimistic. I think there's also a possible future where it gets a lot worse. Um, one thing you know, there's a lot of things that are also unknown yet like so. So we looking at flu forecasting, one of the key things that we do if you look at flu forecasting, flu goes away so much during the summer, that people like some of our partners who provide diagnostic wouldn't we stopped getting data from them because they stopped even aggregating it for months at a time. And so there's some places where there are literally like zero tests for weeks or months at a time during the summer. On the other hand, we have data going back 2017 1615 1413 12 we have really good data from the same sources for flu or we can look at full complete season say we know what flu is like in winter, and like in summers, a flu You know, there's this huge seasonal effect. And we know this for a number of other respiratory diseases for Coronavirus. And again, I'm not an epidemiologist. I'm I'm a machine learning researcher who works on problems and Clinical Health Care and recently working on more the public health side in contacted in connection with Coronavirus. I'm not going to wait, you know, I don't want to be clear to not represent myself as a virologist. or. But But you know, I think just at a very high level, we don't know what a full season of this looks like. We sort of caught onto it towards the end of winter. And it sort of went down one up and went down went up. But we really don't know. Like, are we seeing like like if you know, if April, June, July, August, whatever, if that was all actually winter? Would this thing have been crazy? Or is it that this thing is actually slightly less seasonally sensitive than influenza? And we can't pretend that we know the answer. And I think this is something that people struggle with in general and policy is that everyone wants someone like people see, like, it is equivocating, if you give an honest account of uncertainty, but um, you know, there's a tendency like among politicians to represent like a false level of confidence. But the truth is that, if it turns out that this thing is as seasonally sensitive as influenza, and that we are taking more or less the exact same measures we're taking now with no improvement, and cases are already increasing dramatically. Now, that paints a bleak picture. And we have to we have to, we have to be real about that possibility. You could also, you know, imagine any number of other scenarios, you can imagine that this combines in the body with, you know, a number of other coronaviruses. And whatever version starts affecting people four months from now is less deadly. I can't rule out that possibility. I'm not I don't, I'm not the world's expert on the timescales of virus evolution. And I, I don't know that anybody is specifically in this case, certainly some people are greater experts than I am on those particular facets. So no, that's the kind of situation where I could imagine getting a better under control. But you could also imagine things getting really bleak if

Listen, I'm not trying to be bleak. I was trying to extract from the amount of time that we spent together so that people could have some sense. So my takeaway is, we got some smart people that are working on a complex set of issues, and that you're trying to integrate a whole set of data so that people like Todd and myself don't have to do spreadsheets and try to deduce from that some information that is probably wrong anyway. And so my hat's off to Google for having the leadership and partnering with Carnegie Mellon. And working on this stuff and being around really smart people who are not being as binary as I am right now. You know, I just want to yes or no, and I'm moving on. But that's I don't mean to minimize the importance of your work and the complexity of it all. So I want to give my hat's off to Google and Todd for just being a great champion for it's always seeing the opportunities for partnership, and being good champion and the work that's done here in Pittsburgh and making sure that Google has a seat at the table for all those things. So I really appreciate that. Todd, and, you know, it's really cool that you're you get an up front view of this work as well. So that's, that's probably even justice. Cool.

I think I think Pittsburgh is incredibly lucky to have a research institution like CMU, we have an amazing community of research institutions and universities. But CMU is clearly it's right. And so we're grateful for any opportunity to work together. It's

great. Well, thank you, thank you.

Thank you, Zack, Lipton, and if anyone wants to know any information about either either of these two fellows, it's easy to find. We will put some of these resources that have been shared up next to the archive of this and will stay will stay close to this. So appreciate you tolerating some of my you know, seeking for you know, Binary answer and appreciate your humor and and of course, your leadership. I really can't thank Google enough as well as Carnegie Mellon and that team there for being a part of this ecosystem. So I'm going to say, wear a mask, be safe. And that, you know, that's my scientific. That's my scientific advice. And, you know, pay attention to the data, participate in some of this research and download some of the stuff that we've shared out there. And you'll get a sense of what what we're working on. I'm actually optimistic. I believe that the more information that we're gathering, and I feel very honored to know that this is going on with Google and Carnegie Mellon, it helps me as I start to extract this information and scratch my head. So be safe. Thank you, Zachary Lipton. Thank you, Todd, for everything that you do with us. Thank you, Google and CMU. Thank you, Jonathan. And thank you, everyone. We're back on Monday with who's on the show Monday.

The Three Rivers venture fairs happening this week. So we wait, what's going to be a great show. So get your sleep and remind everyone tuned in to tech vibe radio tomorrow morning at 8am on ESPN 970. Because we're talking to the Pittsburgh space collaborative. I'm drinking my tank tomorrow, Audrey, it's gonna be fun.

Yeah, what about that?

One? Thanks. Thanks, Todd. Thanks, Zach.

Oh, and maybe one thing I might add for people if anybody's sort of more. So for for citizens out there who are interested in data, we have the map where you can see all the signals. But we actually have many more signals that are available programmatically via our API, and I posted a link in the chat. So it actually we expose the raw data that's used to construct the maps. So anybody who's out there who sort of knows how to access things, you know, run a few lines of Python or R can pull all of our signals and generate other plots that you know, with other things.

So we will share everything I've already asked Taylor to archive all the links out there, we'll put it right next to the interview. We'll share it and really appreciate your passion. Thank you both for the work that you do and spending the time with us. So thanks, everyone. Have a great weekend.

Great. Thanks for having us.

Transcribed by https://otter.ai