Inside DuckDuckGo

Duck Tales: The role of user research, A/B testing, and "dogfooding" at DuckDuckGo (Ep.31)

In this episode, Beah (Chief Product Officer) and Zac (SVP, Insights) discuss our approach to user research, how we run robust research while respecting user privacy, and why moving quickly matters. Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy. [https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsubstack.com%2Fprivacy&data=05%7C02%7Clroberts%40duckduckgo.com%7C9bf97bb22b8e4672114908ddecafdd9e%7C728892a04da94114b51152f75ee3bc3d%7C1%7C0%7C638926962285187494%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XENubLLde0FgfAkknFHydtBuSfaEZcwDQzp0v4pN48E%3D&reserved=0] Beah: Hello and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology, and people that help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering, or approach to AI. In today’s episode, we’re going to talk about user research. And so my guest here is Zac. Zac, do you want to introduce yourself? Zac: Yeah, hi, Zac Pappas. I am on the Insights team at DuckDuckGo. I’ve been here for coming up on 14 years and pretend to participate in a lot of market research, user research, and other conversations that we have where we try to infer what users want and whether or not we’re giving it to them. Beah: Sweet, thanks. And I’m Beah. If you haven’t met me, I’m on the product team at DuckDuckGo. All right, let’s launch into it. I have questions for you, Zac. Are you ready? Zac: Ready as you’ll ever be. Beah: All right, so let’s start. Why don’t you give me an example? Like when we’re talking about validating things with user research, just like let’s set up an example of what that means and what that would look like. Zac: I think so. Yeah. So I guess starting with what is user research and why do we do it? I think the obvious answer that people tend to think about are, you know, sitting in an interview or doing some sort of focus group and learning from people. But really, for us, user research is kind of all of that plus, you know, internal dialogue and real testing like A/B testing. So to put it, you know, simply, I guess, user research for us is just trying to be wrong very productively and being efficient at being wrong productively over time. So for us, everything really starts out as a guess or an assumption about users or their behavior about, you know, what is upcoming in the market and research checks that, you know, our assumptions are correct or at least as close to correct as possible. And we try to do that as cheaply as possible, meaning quickly without spending more time on it than what we’re gaining in terms of the product that we’re creating. So an example could be our homepage at DuckDuckGo.com because our homepage has been around for as long as I think DuckDuckGo has been around, but has gone through many, many, many forms. We’ve done tons of A/B testing on it, showing one version of it to some users and a different version to other users. We’ve done a lot of user testing on it, like sitting down with actual consumers of the homepage or users of the homepage. Everything from discussing with parents how they perceive the homepage. So a lot of subtlety that goes into what we think maybe appears as a pretty simple layout or design. Beah: I had a little connection issue there for a second, but I think it’s alright. Let’s forge ahead. Sorry, audience. So, the homepage, I think, as you pointed out, sort of like deceptively, seems deceptively simple, it’s not highly functional software, it’s a page, there’s a few boxes on it, a field, but like what makes it so tricky and worthy of user research. Zac: Yeah, good question. Because it is kind of subtle, or I guess it doesn’t seem very obvious. So if you think about it, the homepage is really where we have a big mix of different user types coming in. Making any assumption about a part of our product usually means that you have to infer who is using it, meaning are they younger or older? Do they have a lot of context about what they’re doing or very little? So our homepage is actually a mix of different user types. There’s a lot of returning users who are very familiar with DuckDuckGo, they’ve been coming to it for a very long time. And so when they come to the homepage, it’s a pretty like rote act for them. The other group that really comes to our homepage a lot are new users or new visitors to the brand and product space in general. So they have maybe very little context about what DuckDuckGo is or what products we have. And so you can imagine, you have these two very diametrically opposite groups that come in, people who have a kind of a strong habit formation and muscle memory on how to use the page and then others who don’t have anything and they kind of, you know, maybe say meander or explore more than they are, you know, directly going to the thing that they know that they want. So what we learned over years of A/B testing, user testing, and kind of regularly reviewing what the experience is like for real people, we found something kind of interesting, if I can just generalize it, which is new users — like kind of people first visiting DuckDuckGo for the first time — really look at the center of the page. And if you see DuckDuckGo.com today, it has these two prominent boxes in the middle. It depends on the browser that you’re in or the form factor that you’re on. But primarily what we know is new users will tend, their eyes kind of follow the middle of the page. And so when they land, we really want to show them an overview or a good sense of what DuckDuckGo can do for them. Whereas returning users, people who’ve been coming back for a long time, tend to want to find the search box. And so we can, or at least we have some confidence that we can move the search box around, you know, the top of the page where it’s at now, to the bottom, to the middle. And if you’re a returning user, because you kind of expect or know that it’s there, you will more easily be able to hunt and find it. So the configuration that you see today, and maybe changes as things develop, really does to us strike a very good balance between new and returning user needs, both something that a brand new user can figure out and use efficiently and kind of get a lay of the land of what DuckDuckGo is and what we do. And then returning users can quickly get to the search box or other more mechanical parts of the product without too much friction. Beah: So how did we actually learn that? Like what testing methodology did we use? Can you walk me through it? Zac: Yeah, a lot of, I’m sure, pretty standard things for people who are used to product development, but A/B testing primarily. A couple years ago we were running, I think, like at least one A/B test of the homepage, like one alternative version of our homepage every week for pretty much the entirety of 2024, 2025. That year we had a ton and tons of homepage tests. And then independent of the actual A/B experiments that we’re doing, we will regularly run diary studies. So recruiting 10 to 20 DuckDuckGo users or other different user types, even non-users, to give them some background on the product. And then really give them a chance to use it for multiples of weeks and have them do journal entries and take notes of the things that they’re running into that they’ve experienced that are emotionally supercharged or maybe had caused some great duress in how they tried to use the product. So anything that’s really notable to that user can come up in a diary study. What we’ve also done in the past is to try to create alternative versions of our homepage from designs or a design system that we can give to users to play around with in real time. So we can actually simulate what an alternative version of the homepage might look like and how it should act and give them a clickable prototype that they can use, get some feedback before we ever invest too heavily in the platform. So it’s been a mix of really just doing a lot of testing pretty much every week since the conception of the company on some run. The homepage was a big focus for us for the last few years to try to nail something that could eloquently show what all of our kind of product offering is between search and now the expansion into Duck AI and of course the apps and email and other things. Beah: Yeah, so like there’s like, sounds like we’ve been marrying, and I think we do this elsewhere in the company, a mix of like quantitative research where we get to hear how people think, you know, people who are opted into some kind of study with us, like we actually get to understand what they’re thinking, what they’re doing in some detail and then just like quantitatively looking at results in an A/B test. Zac: Yeah, absolutely. And probably the maybe thing I didn’t say was we were very active on our social media. We point most of our users, at least if you click around our website, a lot of things point to our subreddit, which is reddit.com slash r slash DuckDuckGo. If you have product feedback, if you run into a product issue. So we do try to, and we have lots of people here that every day are checking comments or posts to that subreddit just to see if we released something and it’s causing some undue user harm. Tons of feedback boxes around our SERP, like the search results page within the app. And so we’re constantly trying to encourage that if a user has a negative experience, they’re able to get it pretty frictionlessly by a message to us. So we’re looking at social media, we’re really running really structured user tests, like intentional user tests, A/B testing and just kind of a what they call dogfooding or like internally using the product a lot and trying to be very critical about our understanding of what users really need. Beah: Yeah, I’ve said it before and I’ll say it again. If you’re writing feedback to us, a human is probably not just reading it, but like taking it to heart and thinking about it. So, um, how about like, tell me what are the privacy considerations? Can we do these experiments in a way that respects user privacy anonymity? And does that throw any monkey wrenches into the testing mix? Zac: Yes, truly. Yeah, good question. Yes, it has, it continues to, but we continue to have a lot of really intelligent people here that think and care a lot about it. So over the years, we’ve gotten a lot better at developing methodologies that allow us to consistently protect and respect user privacy while getting some of the analytics that we would want from robust A/B testing. So we do this through local storage or through consumer opt-in notices. So for example, if we have a survey that we show you on our SERP, we will have like a pretty robust disclaimer there that says this is anonymous or the responses maybe are going through a third party if it’s something like SurveyMonkey that we use for collection of survey responses. So very upfront to the user and very much building the engineering that is required to do anything that’s an A/B test or more like metric tracking in a privacy respecting way. And we’ve gotten a lot better at this over the years, but it definitely takes a lot of toil and discussion internally with data science and really good product folks like yourself to understand what things you just can’t get from numbers, which is a lot of like how the product feels to people and that’s where we come back to the qualitative research a lot. Getting, you know, FaceTime with actual people to hear what their experience is like and hearing in their words gives us the why to the what that we get from the metrics. So for as much as we can measure something, we don’t really know why it’s changing or what that we were making about a user behavior and how it’s actually playing out. So for those, we do tend to get or try to get face-to-face with users and recruit them either through our own product. If you see a survey and have the option to, you can connect with us at the end if you’d like to share an email address that we can reach out to. And again, those are all done within the in accordance with our privacy policy or the policies of the platforms that we’re providing them. So it’s been a long road, but we think we’re in a pretty good position now to be able to test most of the things that we would ever want to test with a pretty high level of accuracy. Beah: Nice. You kind of touched on this, but like who’s the we? Like who’s actually doing running these tests, writing these questions. Zac: Yeah, so when we started the user insights team, as maybe the long answer to this, and we started the user insights team, it was because everybody was doing testing at DuckDuckGo with product folks, engineers, and we were, as I’m sure you know, but for the listeners, like we’re a very validation heavy company and we care a lot about the assumptions that we’re making to make sure that we’re working smart and we’re a small company relative to the other larger competitors out there. And so we have to be a little bit more intentional and I should say, like, careful about how we approach things. And so when we started the insights team, it was really on the basis of everybody at this company doing some level of validation. And we wanted to up level or make sure that there was a baseline level of excellence that went along with that. So the short answer to your question is everybody here does some level of user testing. I think you yourself have probably done more than I have, at least in the recent past. Because it is a part of every conversation that we have. And so we have this formal team that we call user insights that tends to do the higher risk or the more robust or deeper research that takes somebody’s full attention for maybe multiples of weeks. But if you’re a member of the product team or engineering or you’re on the copywriting team, you yourself should have, I hope, the capability and tools to go out and do any validation that you would have to do today. So I can take a guess. There are probably five to ten people I can think are running an active test right now on something like usertesting.com or PickFu. or something that’s a pretty light way to free up a future discussion if you can provide some data ahead of that meeting or wherever that discussion is taking place. Beah: Yeah. And one of things that I think is cool is the user research team, like part of their, the team members, part of their job is running the research, but part of their job is making other people, like enabling other people to run the research. So even like if I’m running a test, like I can get support or help from somebody on the team to like check it, make sure the work was good, to understand the tools that I need and so forth. Okay, I think that was a great, I bet that’s probably not true everywhere and a good, interesting answer, but before we close, anything we didn’t talk about that you want to talk about, especially anything that’s in the surprise column or might be unusual about how we work at DuckDuckGo. Zac: I think there’s a lot of stuff that we do and usually at DuckDuckGo I think most of it’s probably for the better and certainly the thing that I can keep reiterating is just how much we all care about users. Obviously the, you know, dedication to privacy that we have comes from a place of wanting to be treated better as people and consumers ourselves but in the way that we approach research I think the heart of it is getting to how people really feel about the product and whether or not it’s a compelling experience. So for all of the metrics that go up or for positive feedback that we’re getting, I think we really strive to know why. And some of the interesting things, I was just trying to rack my brain earlier about some things that have come up that might be worth sharing. And certainly, like, you know, we have a bunch of layers around the company that do either social media listening or user testing. But where I think this really comes out is when we’re shipping a product, we have a final stage in our internal process where we can kind of all see the changes that we’re making. We call them ship reviews, but we tend to dogpile in and see UX changes or copy changes that we’re making and have an opportunity to really look for confounding evidence or a contradiction that might help improve that thing. And so this process that we have actually does involve more people at the end than it does at the beginning sometimes because we’re asking that if you can improve something or if you have positive or if you have any meaningful feedback about something to jump in. So something that I think people would be surprised to hear is that even though teams tend to work on individual projects here, anybody from the company can show up during the process of creation of product and leave their own feedback and have like a real impact on the outcome of what that project is doing or the direction that that product is going, which is I think really interesting. Beah: Yeah, one thing that we’ve been doing. Yeah, no, go ahead. Go ahead. Zac, please. Zac: Independent of that, we found a lot of the subsegment. I’m sorry. There’s a little glimmer. Sorry, you’re, yeah, you’re, you got it. You’re, you were frozen. You go. I’ll stop. I’ll start my next table. Beah: Okay, the listeners are gonna love this part. I was gonna say in ship reviews at that sort of last mile before we ship, one thing that we’ve been doing that I think we’re building maybe more muscle around now that I think’s been really good is like if we have some internal difference of opinion on like the best way, you know, some detail that we’re like, oh, this didn’t make sense to me, but it made sense to me. User research team members or anybody really is like trying to jump in with some, as one of your team members coined it I think, punk rock research — like really quick, like you know three-hour turnaround, like let’s get some data points to help unstick the conversation and and like base this decision as much as possible in empiricism. And like, you know, doing perfect research is very, very hard, but doing like, you know, imperfect research with known caveats can be quite, quite fast and productive and I love that trend that’s happening now. Zac: Yeah, yeah. And the tools are getting a lot better. Certainly, I think as we see AI develop and get more integrated and things become automated, things that we’re looking at now are things like synthetic user panels, which not to get too technical is just using AI to simulate as best as it can what a group of or a segment of people might say. And so with that, you can imagine doing a lot more testing or even having it automated so that you don’t have to do it and you just wake up to the results from them. But of course you have to make sure that the output of this automation or of the AI is actually reflective of the underlying user base that it’s supposed to be representing. And that’s like a constant challenge for us. It’s the same thing that we’re dealing with here. When we ask or get results from something, you know, we can’t take it as truth. We have to take it even still with a grain of salt despite the validation and continuously validate. After we launch something, you shouldn’t be surprised to see that even if it was really positive, we might come back and try to change it again a couple months later if we have a really good idea. Beah: Yeah, or a couple of weeks later. When you’re talking about the synthetic user testing via AI, like user testing script, like what would you do on this screen, AI — like that’s a great question. Excellent point. I hadn’t thought of that. What I would do on the screen is… Zac: Yeah. It’s saying you’re not trying to do something on the screen, you’re trying to do something on the internet. That would be the phrasing. Beah: Alright, awesome. Well thank you, Zac. I think we covered a lot of ground. Appreciate it. Pleasure as always talking to you. See you around the DuckTales hood. Zac: Thank you. See you at the office. Bye. Beah: Bye. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com [https://insideduckduckgo.substack.com?utm_medium=podcast&utm_campaign=CTA_1]

Duck Tales: How DuckDuckGo provides useful “near me” search results, while keeping your location private.

In this episode, Greg and John (Search) discuss local search results, and how we help you find local cafes, restaurants, and business, without knowing your precise location. Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy. [https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsubstack.com%2Fprivacy&data=05%7C02%7Clroberts%40duckduckgo.com%7C9bf97bb22b8e4672114908ddecafdd9e%7C728892a04da94114b51152f75ee3bc3d%7C1%7C0%7C638926962285187494%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XENubLLde0FgfAkknFHydtBuSfaEZcwDQzp0v4pN48E%3D&reserved=0] Greg: Hello and welcome to DuckTales where we go behind the scenes at DuckDuckGo and discuss the stories, technology and people that help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering or approach to AI. I’m Greg. I work on the search engine here at DuckDuckGo. With me today is John. John, would you like to introduce yourself? John: Hi, I’m John. also work here at DuckDuckGo on the search engine and I’ve been focused on working on local search for most of my time here. Greg: Awesome. Yeah, that’s kind of what we wanted to talk about today is local search. And so I’m looking forward to chatting with you a little bit about how local search works, how we do it at DuckDuckGo and get into some of the challenges that we have as a private search engine and how we solve those. Local search generally, it’s kind of maps related searches. whether you’re searching for places, businesses, restaurants, those kinds of search queries we kind of call local searches. We have a couple different versions of a map that shows up on the search results page. When you do one of those searches, either you’re searching for something kind of in a city and you put the name of that city in your search query, or if you search for something near you, have ways to provide those results. still in a privacy preserving way. So maybe as a starting point, I thought it would be good to just, if you could walk us through a little bit of the life of a local search query, how it goes from kind of something the user has submitted to us to what we decide to show on the page. John: Yeah. So we start with a, we get a user query. So somebody type something in and sends the query to us. And we try to figure out whether that would benefit from some sort of mapping information. So maybe the query contains a number and a few words and then drive or street or road. And you know, that looks like an address. So we’ll show a map of that address. Other common patterns are, you know, common. types of businesses like restaurants, hotels, cafes, and those sometimes come in with location indications. know, cafes in New York City or cafes in Paris. And sometimes they don’t, sometimes it’s just cafes or cafes near me. So we kind of have to handle both those types of situations. And the ones that are nearby are really kind of the more interesting ones. because we have to find a way to figure out where Nearby is for the user. And we have to do that in a very privacy-respecting way because that’s our whole thing. So the browser, the web browser that you’re using will send a location. But we don’t just straight up use that because we deal with lot of partnerships where we have to get some data from our partners at Apple and TripAdvisor. And we don’t want to just send them raw user locations because that is a privacy issue. So what we do is we’ll kind of add some noise to it. So if you say you’re at, you know, this exact location in a city, instead of sending that, we’ll move you a little bit. Depending how big the city is, we might move you just a few hundred meters. If you’re way out in the country though, where maybe you’re our only user for miles, you know, we might move you a few miles. And so that... can affect your results, makes them a little less accurate, but we think we do it with as little noise as possible to get you a lot of privacy benefit while still giving you results that are hopefully very accurate to the location you’re interested in. Greg: Yeah, so there are a couple of things in there that I think are interesting to talk a little bit more about. when you kind of described how we add noise to a user’s location, there’s a lot of documentation, by the way, about this, if you want to read more on our help pages. But I’m wondering how we think about the kind of trade-off between making sure that the results are still good and the way that we’re obfuscating someone’s location is sort of sufficient from a privacy standpoint. So maybe you can talk about the first part first. How do we think about the quality of the results and whether or not we’re showing the right local results for someone when they do one of those nearby queries? John: Right, yeah, we have a lot of criteria that we check against before we show results to users. We’ll get a whole bunch of options that might be relevant to the query. And if you’re searching for a cafe, maybe a restaurant is relevant, but probably a hotel is not so relevant. And so we may make sure we get the category right first. And then we also want to make sure the location is correct. And because we’re dealing with partners, they might give us some locations that aren’t as relevant just because we don’t give them as accurate of a location. And so we’ll kind of refine those on our side without sharing some of that information with a partner to say, you you gave us this location that’s 100 miles from the user and that’s maybe because we gave you the wrong location on purpose. And so we’re not going to show that to the user, but we will show these other locations that are actually nearby and relevant to the users. Greg: Got it. Yeah, and then, I mean, even, you know, sometimes when there is a location in the query, it could be one of those that is still ambiguous to us, right? I think a lot about the difference between Paris, Texas and Paris, France. Users don’t always clarify what they mean. So we have to try to figure that out based on context too, right? John: Exactly. even some locations that are very well defined. If somebody says New York City, that’s just one place that we’re going to look. But New York City is huge. just determining which part of New York City is maybe the most relevant to send to the user is another challenge that we have to figure out. Greg: Yeah, and I mean, just we’re sort of riffing on these now, but, you know, if you search someone searches for New York pizza, right, that could be they’re looking for a pizza restaurant in New York City, or they’re looking for a restaurant near them that serves a New York style pizza. So, yeah, all kinds of things that we have to sort of contend with and try to figure out to give those right results. I’m wondering if maybe we could talk a little bit more than about. the different types of ways we actually display the results once they appear on the page because we have a couple different versions of our local modules that we show depending on the type of result and the number of results we get back. So I’m wondering if you’d say a couple things about that. John: Yeah, we have many different ways to display based on what the user is searching for. As I mentioned earlier, we have a special address display if you’re just looking for a map given a location. That’s a pretty bare result. It’s mostly just a map and the location label. Some more popular landmarks, we have a special display for that where you’ll get the map and if it’s the Eiffel Tower or something like that. Then you’ll get a picture of the Eiffel Tower and a little description that we sourced from. Wikipedia and then there’s a business searches is the next big category and those are kind of divided into two as well if you’re searching for a very specific single business like, you know Joe’s cafe and Las Vegas I don’t think there is a Joe’s cafe in Las Vegas maybe there is Then then we’ll just show you that one Joe’s cafe ideally, you know, we don’t want to show you Bob’s cafe That’s also in Vegas or Joe’s cafe. That’s in Idaho or something Greg: There might be. John: And then there’s the sort of category or chain type businesses where if you search for Walmart or restaurants, we’ll show you a big listing of businesses and where they are distributed on the map to let you kind of click in and explore more if you want to do details about each one individually. Greg: Got it. And then that’s gonna vary too based on the category of the result, right? I think we have some kind of special display for hotels as well. John: Yeah, we’ve just kind of introduced this and I can share this real quickly because I think it’s a pretty cool feature. So just recently we’ve added some pricing indicators. So if you’re searching for a hotel, you probably want to go stay at one of these hotels. And one thing that is important about staying in a hotel is how much is it going to cost and is it available at the time I’m traveling. So we’ve got a new interface. We just released this last month where you can put in your check-in dates, check-out dates, how many people, and it’ll tell you a pricing estimate of how much it’s going to cost for those dates. And then once you click into it, you can get different offers from different providers. So you can get expedia and go through and actually have really easy access to booking one of those hotels that you’re interested in. Greg: Awesome, yeah, I think this is a really good example of something where it’s a new feature that we’ve added that captures a particular use case that we know users have. mean, that we ourselves as users of a search engine have when you’re booking a hotel, trying to make it easier for users to complete the task that they’re doing, not necessarily just kind of see the different results. I’m wondering, maybe you could talk a little bit more about some of the other challenges around those additional use cases or business data that’s important for us to show and how we’re thinking about that and how we see that evolving over time. John: Yeah, we build this feature on a few different really important partnerships with Apple, Yelp, and TripAdvisor being the main three. And we use those partnerships to give us all sorts of data about different businesses. Yelp gives us lot of information about restaurants and new service businesses, shopping, stuff like that. And then TripAdvisor gives us hotel information and information more about that. attractions, know, museums, theme parks, stuff like that. And we’re trying to push more into using this data to give the user a lot richer experience. Some ideas, you know, including menu links for restaurants, giving the ability to order food or serve tables, call us a service business, you know, if you want to plumber, you can get it like kind of a quote directly. And all those are kind of ideas that were pushing for and hope to release soon. Greg: Yeah, definitely. There’s all kinds of different use cases that people have with businesses. know even just knowing the hours that the business is open, right, is so important. And these are things that users ask about all the time. I think if I had to pinpoint one big challenge with local in general, it’s just that the coverage for the business that the user cares about in that particular moment is so important. And so we’re constantly trying to figure out ways to expand that coverage and make sure we have the right information about as many businesses as possible. John: Yeah, exactly. We’ve done a lot recently also in terms of taking user feedback because, you know, we found users get very upset if we send them to a business and we got the hours wrong and the business is actually closed or they try to call a business and it’s the wrong phone number. So we’ve taken a lot of steps to submit or solicit feedback from users on when we get it wrong and then forward that upstream to our partners so that they can correct it and we can correct it and everybody gets a better experience. Greg: Of course. Yeah. Yeah. Yeah, definitely. I want to emphasize, if you as a user send us feedback that, know, hours are wrong or, you know, an address or a phone number, you know, we take all of that in and do our best to forward that to the source of that data and also make sure we’re not showing it to other users and kind of getting better and better at correcting that over time. Cool. Well, we’ve covered a lot of ground. Anything else you think is worth sharing and talking about with respect to local search? John: nothing comes to mind straight away. I’m hoping we can keep working on the product, making it better. think we’ve got some good expansions in the future. Some more, pie in the sky ideas we have of, you know, moving to something like a flight search or something might, come into later on. and yeah, just hoping to keep making the product better and getting users what they want and giving them the right information as quickly as, as easily as possible. Greg: Yeah, definitely. All right, well, it was really fun chatting with you about this today, John. Thanks for taking the time. yeah, maybe we’ll have a chance to talk more on a future episode about other search-related things. But yeah, thank you. John: Thanks for your talking. Greg: Bye. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com [https://insideduckduckgo.substack.com?utm_medium=podcast&utm_campaign=CTA_1]

Duck Tales: How DuckDuckGo uses data science to measure marketing effectiveness, privately (Ep.29)

In this episode, Cristina (CMO) and Baran (Data Science) discuss our privacy-first approach to marketing measurement, the role of incrementality, and how the team operates day to day. Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy. [https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsubstack.com%2Fprivacy&data=05%7C02%7Clroberts%40duckduckgo.com%7C9bf97bb22b8e4672114908ddecafdd9e%7C728892a04da94114b51152f75ee3bc3d%7C1%7C0%7C638926962285187494%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XENubLLde0FgfAkknFHydtBuSfaEZcwDQzp0v4pN48E%3D&reserved=0] Cristina: Hi, and welcome to DuckTales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology, and people that help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering approach to AI, or how we operate as a company. I’m Cristina on the marketing team, and I’m here today with Baran on the data science. Would you like to say hi? Baran: Hi everyone. Cristina: So for our viewers, if you’ve already listened to episode 9 with me and Chuck, you know that, like our product philosophy, privacy is core to the ethos of our marketing. And most of the common marketing practices we just don’t do, identifying and targeting individual users, retargeting, using behavioral data, using third-party cookies and pixels, all that, all hard nos. We have very thoughtfully developed privacy-respecting measurement that is a bit unique, but includes techniques that I think and hope all marketers can adopt. Thankfully, we’ve come a long way from when I was doing the campaign analysis. And thanks to everyone’s data science background, we have much more sophisticated techniques and models. And I’d really like to dive in on a slice of that today. So first question, most companies measure their marketing through attribution, tracking who clicked on what and then installed. We sometimes do a version of when it’s possible to in aggregate and anonymously of course, but can you explain how does it work for us and is that even the right question to be asking? Baran: Yeah, thank you, Cristina. On one hand, we have a self-built, simple and privacy-respecting attribution. And on the other hand, we have a few channels who do their own attribution. But with both of them, we have three issues. One, only a small percentage of our total impact through the campaigns is captured by attribution. Let’s take the most obvious one, TV campaigns. There might be a QR code on the screen, but most people would just go to the app or Play Store to download the app instead of scanning the QR code. And then it doesn’t work on all channels. Many channels only support the industry standards, more privacy in ways of attribution methods, not necessarily what we built. And third and most importantly, which is also valid for all other companies, is attribution is not the same as incrementality. It measures who clicked, but not which channel actually caused, let’s say, a metric like an install. And the real question is actually incrementality. Cristina: Okay, so then if attribution doesn’t answer the real question, how do we introduce incrementality and how does that answer the question? Baran: So ideally it would be A-B testing for incrementality for a channel within a channel’s advertising platform. But there is no way to do this. Channels don’t provide this. It could also be because this would make it very easy for marketing teams to understand the incremental impact. The closest thing we can do is randomized GeoLift tests. Cristina: Okay, so incrementality is a concept I think a lot of marketers are likely familiar with, but to make sure everyone’s on the same page, can you walk us through what one of these tests actually looks like for us? Baran: Yeah, we can take all the geos in a country, for example, if it’s the US, we have the DMAs, they are like New York, Los Angeles, we would randomize them into two groups of test and control. While doing this randomization, we will optimize for comparability between the groups in terms of the metric that we’re interested in measuring, such as installs. And then once we have run the test, let’s say the test would be launching a campaign in the test geos, we would use the pre-period of the campaign to measure the relationship between these tests and control groups aggregated metric. And then for the campaign periods, we would use that relationship. And then based on the control geos actual metrics in the campaign periods, we would estimate what we call the counterfactual, what we expect the metric in the test group to look like in the absence of a campaign. And the difference between the actual test results minus this counterfactual would give us the causal impact. Cristina: Okay, I think that’s pretty clear. When the test is done though, what kind of answer do we get? Is it as simple as it worked or it didn’t? Baran: Yeah, so it’s not necessarily binary significant or not. In this case, unlike product tests, significant would mean the channel had a significant impact on installs. But maybe it did, but maybe it was too expensive. What matters is, for example, metrics like a cost per install and especially an uncertainty around this metric. If the uncertainty is narrow, we would have high confidence in the result. If it’s wide, we would have lower confidence and take decisions based on this uncertainty. And we do these by using Bayesian methods, which is helping with better decisions than a simple yes or no. Cristina: Got it. So if we think this is the gold standard, why isn’t everyone doing it? Baran: I think the main reason is the operational complexity. A clean test requires ideally a full turn on or off in the test or control geos, and there’s multiple parties involved in this. Let’s say you have companies with a paid search team and you’d like to do a test on the paid search channel. It would mean they would have to turn off their channel and maybe they have goals that they have to hit. Cristina: Yeah, this is an interesting peek into our culture. So can you help our viewers understand why we typically don’t have these problems? Baran: Yeah, we have a small and unified marketing team. No one person or team is controlling a single channel. There are no, let’s say, siloed budgets per channel or channel group. There are no consequences if a channel stops. Let’s say if we don’t run a certain channel anymore based on test results, the people running these channels would switch to another channel, which makes more sense. And also lacking more sophisticated attribution make it even more important to run these randomized GeoLift tests. So in this case the necessity actually becomes a strength from the marketing data science point of view. Cristina: I like the way you position that we turn a bug into a feature. We’ve had to get creative and develop marketing and data science methods that respect user privacy while still giving us the insights we need to make data driven decisions. I think it’s not just about what we do. It’s how we do it. That sets us apart. Our approach to marketing is as unique as our product philosophy. So what was it like the first time you ran one of these at DuckDuckGo? Baran: First time it was also a new experience for me, because I was also used to from previous work the operational complexity part. When we first discussed about turning off a channel, I was very amazed when the team said, okay, let’s just do it, let’s do it like this week, next week. There was no conflict, even though it was a high scale channel, nobody worried about their goals. In the end, the results were not favorable. So we ended up significantly cutting spend and optimizing the campaign from those results, and since then we repeated this approach many times. Cristina: I think that’s a great example. I like to think of us as a somewhat nimble team. And yeah, one of our core values is questioning assumptions. And this marketing experiment is a clear example of that. I really appreciate how we’re not territorial. We’re very much of all on the same team kind of mindset. So you get the results, then what happens? Baran: So if it’s a new channel, let’s say we ran it for the first time, either if the results are favorable, we might scale it up, start running it further at scale, we might keep testing if the results require some optimizations, or if the results really out of hope, we will just stop further testing. And if it was an existing channel, let’s say we turned it off in the test geos, then depending on the results, if the results were not favorable, we might adjust spend, change our, let’s say, target cost per 1000 impressions or another metric that the channel allows, we might further optimize or if the results are more favorable than we thought, we might increase scale. Cristina: Sounds like a lot of work. Do you have to keep running these tests every time? Baran: Yeah, that will be very difficult, but quick answer no. Let’s say a channel, a digital channel has a meaningful attribution rate. Let’s say if a channel claims y attribution, it has, let’s say 100 installs. Then we run this test, we see, actually we measured 50. Then we would call this 50% the channel’s attribution rate. And as I mentioned, we were using Bayesian methods. We will also measure an uncertainty around this. Let’s say 50% is the point estimate. Our full uncertainty is from 40 to 60%. We would apply this ratio to ongoing attribution data that we get from the channel anyway. And we would monitor without retesting it. But of course, this attribution rate could be subject to change. If there’s a material change, let’s say a major shift in the creatives, a structural change in the placements of the channels in the age groups and so on, then we would retest. Cristina: Interesting. So what about for bigger campaigns like TV, where it’s not possible or cost efficient to turn it on and off in specific regions? Baran: Yeah, very interesting. Like if you’re running, if you would like to run long big TV campaign, we wouldn’t like to limit our scale by just using 50% of the geos or even let’s say using maybe a lower percentage as a control group could be very difficult due to how these channels operates. We use media mix models to measure the impact of such large channels. As other companies, we modulate spend from week to week. We have on and off periods, but the challenge there is TV is not the only channel that is running. We also have other smaller or digital channels that are too small to model within the media mix model directly. Here is again where the randomized GeoLift tests come into play. Same principle, we use a Bayesian model where we include digital channels using their attributed installs as the inputs. And we don’t need the model to guess everything. We already have our high-confidence measurements from the randomized GeoLift tests, and we can put those attribution rates along with the uncertainty as Bayesian priors into this model. So in one unified model, we capture everything. Cristina: So then how do all the pieces fit together? Baran: Yeah, this is, we are doing the same thing that the industry agrees. Attribution, randomized GeoLift tests and media mix models go in hand. One of them alone is not enough. Some channels have the attribution, but attribution is not telling the whole picture. GeoLift tests are great, but it’s impractical to random all the time. And media mix models are great, but many channels are too small to be measured there on their own. And we’re combining them all together. Cristina: Well, thank you very much, Baran, for explaining part of how we measure campaigns. I hope it’s helpful and interesting to folks. And then more importantly, for your ongoing work, evolving our techniques to get higher confidence insights and decisions. We only spend marketing dollars on what actually grows the product, and we do it without compromising privacy. And frankly, we couldn’t do it without you. So thank you again, and we intend to keep you busy for a while. I so appreciate your time today and that of everyone who watched. So stay tuned for more episodes and bye for now. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com [https://insideduckduckgo.substack.com?utm_medium=podcast&utm_campaign=CTA_1]

Duck Tales: The fire button, a delightful way to delete your browsing data (Ep.28)

In this episode, Beah (Chief Product Officer) and Sveta (Design) discuss the fire button, the recent redesign, and how it improves both user privacy and delight. Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy [https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsubstack.com%2Fprivacy&data=05%7C02%7Clroberts%40duckduckgo.com%7C9bf97bb22b8e4672114908ddecafdd9e%7C728892a04da94114b51152f75ee3bc3d%7C1%7C0%7C638926962285187494%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XENubLLde0FgfAkknFHydtBuSfaEZcwDQzp0v4pN48E%3D&reserved=0]. Beah: Hello, welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology, the people that help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering, or our approach to AI. In this episode, we’re going to talk about the fire button. And I will be your host today. My name is Bea and I’m on the product team. And we have as our guest Sveta. Sveta, do you want to just quickly introduce yourself? Sveta: Yeah, I’m Sveta. I work on mobile apps at DuckDuckGo for now five years. Beah: Sweet. Thanks. And you’ve been, you’re a designer by trade, right? Sveta: Yeah. I’m on the design team. Beah: And you’ve been doing a lot of work to improve the fire button on mobile. And so that’s what we’ll jump into today. Okay, so first of all, let’s start with the basics. Like, what is the fire button? What does it do? Sveta: Right, so the fire button is a button that’s located quite centrally in mobile UI and it looks like a fire. And when you press it, it removes, it deletes all of your tabs and all of the browsing data and all of the history. So pretty much removes everything, all of the browsing activity and you can start from clean slate. And it also shows a really cool fire animation when you do that. Beah: That seems like a good prompt to actually show it. So let me see if I can share my screen here. Can you see that, Sveta? Okay, then hopefully our audience members can as well. Alright, so this is the fire button here. I’m just on a website, on NASA’s website. I’m going to tap the fire button. And so I have, what are these two choices here? Sveta: So the first, the primary button will delete all of the tabs, all of this data for all sites. And the second button is a choice to delete activity for only this one tab. So it will close this tab and will delete all of the site data associated with the sites visited in this one tab. Beah: Alright, I’m going to choose that one here. Delete this tab. Awesome. All right. I just, I wanted to only delete the one because I also wanted to show like if you’re using, if you’re a Duck AI user, this is a new feature. But if you’re on a particular chat, for instance, a chat about what music your dog might enjoy, you can hit the fire button and now you have this opportunity or this option to just delete this chat. So remove it from your chat history. Sveta: Nice. Beah: Boom. All right. let me get it shared on. Okay, I don’t think we’re sharing anymore. Cool, so like what is the benefit of using the fire button? Sveta: Yes, the primary benefit is privacy reasons, because as you browse the internet and you visit sites, you accumulate different site data such as cookies or different user preferences that then tracking companies can, if you accumulate a lot of it, they can use it to identify you and connect your browsing activity to you. Sell that information to other companies to show you targeted ads. And our browsers, we actually block most of the trackers and most of the cookies, but there are still some cases when we can’t do it. And then that’s a good practice to delete all of this data every now and then. And then the second reason is just still schooled and it makes you, lets you start your next session from clean slate. Beah: Yeah, I like how those two, they’re like different reasons to use the fire button, but I kind of like how they converge too. Like part of what feels good about a clean slate and not having like stuff everywhere is, I think for me, like not having your data everywhere, you know? Like it’s just clean, it’s like a sense of control and I don’t know, care, I suppose, that you can apply more easily with fire button. Like you can tell me if this is how you would characterize it. Like you can delete data in most browsers, but it’s just a lot harder, right? Like you don’t have a button front and center with just the options that you need, like one or two taps away. Sveta: Exactly. It’s usually tucked like hidden somewhere in the settings. I think it’s actually pretty cool that we managed to turn this really mundane and boring technical thing in a really satisfying experience. Beah: Yeah, and one thing I didn’t show is that you can actually pick between, I think, three different animations. Like the fire is the default animation, but there’s like this kind of nice swirly animation and all like water animation. And then you can also just turn animations off if like what delights you is speed and efficiency and not an animation. That’s cool. Cool. Tell me a little bit about like how the fire button has changed over time. Sveta: Yeah, so what the confirmation that we’ve seen just now, that’s updated design that we’ve launched recently. But for years, I think since we first introduced the fire button in our mobile apps, we just had this system, little system confirmation to confirm, delete all of your tabs and data that was very small in the bottom of the screen and it was really easy to press. And it was of course good for users who were regular fire button users because it was quick. But then there was also a big risk that you could just accidentally press it and delete all of your tabs and data without intent. Beah: Yeah, so if your intent was to blow everything away all the time, fine, great, it works great for you. But if you wanted to be a little bit more surgical about it and keep some tabs around, then it neither worked for you. It didn’t really work for you, and if you accidentally used it, you might really regret it. Sveta: Yeah, we did get some that obviously there’s if you’ve accidentally deleted all of your tabs without wanting you would write us an angry feedback, which we sometimes received you open it and you see all caps that chances are from someone who just deleted their tabs. Beah: Yep. So take note, dear reader, if you are writing angry feedback, a kind human like Sveta is reading it and taking it to heart. And so in this case, what did we do about it? Sveta: So we’ve changed a lot how the confirmation dialogue looks like. It’s bigger, that you’ve seen it. It takes a lot of space on the screen, so it’s harder to miss. We’ve on purpose moved the buttons a little bit high up. So when you press the fire button and your thumb is right above the fire button, the confirmation buttons, they are not on the same spot. And there is an animation, pictogram that also attracts attention. it’s just everything is made for you to pause, read the label, and make an informed decision. And then another big change is that we’ve added a second option because before you could have only deleted everything, all of your tabs and all of your data. And now we have a second option that you can delete just one tab and the data for that tab. Beah: Yeah, yeah, I love, by the way, like, I’ve developed, you know, I’ve been using our mobile apps for a long time, but I’ve developed this kind of workflow recently now that we have the single tab, single chat burn, so handy, we have this kind of, we’re in the process of updating behavior when you open the app after an idle period, we take you to a new tab automatically, so you can just start typing your query or your chat or whatever. I like realize there are tabs that I want to keep around, the things I want to keep around, but there are a lot of things that I just want to do a quick search, a quick whatever, and it’s just, it’s throwaway. And like now I open the app, I’m in exactly where I need to be, I don’t have to think do I want a new tab or do I want to, you know, override this tab? And then like when I’m done, you know, five seconds later, just doop, tap, burn that tab, and it’s just, I don’t know, there’s, it’s a very satisfying and efficient workflow for me for the I don’t know, 70% of things I do that are just quick interactions with the app and don’t require any kind of historical retention. So I love it, Sveta. Thank you. Sveta: I think there’s something interesting about this approach to delete activity from one tab, because in other browsers, the way they solve this problem, the task of deleting some of the data, not all of the data, they use time. So you can delete an activity for, let’s say, last 20 minutes or last hour, but then it deletes it across other tabs, all tabs. While in our browser, like you said, you just did one thing in one tab, and then we delete that one tab without affecting any other tabs. Beah: Yeah, yeah. I think it’s a great shift. Do you want to talk at all about the design process that you used with the team to get to these improvements? Sveta: Yeah, so our first hypothesis or first direction that we explored for this new mobile confirmation was the same approach that we have on desktop browsers where we have a list of options for users. Every time you press the fire button, you can choose, do you want to delete tabs or keep tabs, delete the side data or keep it. And that is a good direction because it gives most flexibility, but it also, we realized that in mobile context, just looks, it looks heavy. It took over the whole screen, the confirmation just, we were asking users to make these choices every time. We, well, in our old design where you had to only press one button. There was something nice about that privacy package. We gave users the most private option with one button pressed. So we decided to instead just add another option so you could either delete everything. With one button press or delete one tab. Beah: Yeah, yeah, I think it’s like the fire button is such a great example of a feature that has like, sort of delight elements to it that make it really satisfying to use and sort of goes above and beyond like what, you know, the basic functionality that users might describe themselves as expecting from a browser, like, but it does provide this like really useful functionality. And it, I think it’s like the process to keep both of those things true, maximize functionality that people want and keep it delightful in its simplicity and not add cognitive load to the user experience is pretty tricky and seems like something that we navigated well in this case and got to a pretty good outcome, I think. Do you have, have we gotten feedback or do you have any indicators, how do you know if it’s a good outcome? Sveta: Yeah, we do surveys for our new users and the last survey that we did, one of the questions, asked the users a specific question about the fire button and we’ve seen that the delight. I don’t remember exactly the question, but we ask if this experience is delightful. And that has increased compared to the old design. I think we will, I also see less angry feedbacks in all caps from users who deleted their tabs without wanting to. Something that was also a challenge in this project is that this balancing, making this confirmation easy enough for old users to not lose this effect of confirming quickly and deleting data with one tap, but also adding some friction to the new user so they will not accidentally delete it. Of course, we also received some feedback from the old users who thought the new confirmation was a little too complicated. But that’s the reason now think people are liking the new design. Beah: Yeah, I feel like that’s an ongoing challenge with changing our mobile apps, right? It’s like we have this pretty big band of DuckDuckGo mobile app faithful who have a particular workflow that works for them. you know, whenever you change it, there’s some risk that it irritates people by kind of just like messing with their workflow, even if it’s a net positive improvement. Yeah, I think the fire button’s probably right at the center of that challenge because people who use it really love it and it’s like probably a big part of their experience of what makes the app special. Sveta: Yeah, and we have a lot, have millions of users who use our regular fire button users, but we want more people to feel that this is something for them. I am actually, I wasn’t a regular fire button user because it was too distractive like my tabs, but now I also use it more often myself. Beah: Yeah, yeah, same. Totally. rarely used it. Rarely did I want to destroy everything, and so same. I mean, I wish I were that person that was just like clean slate every day or every use, but of course I have several tabs sitting around that are supposed to be reminders for something, so yeah, I love it. Yeah, I was gonna ask you what your favorite thing personally about the fire button is, but you just touched on something you like about it now, which is per tab distraction. Anything else you would add? Sveta: Well, when a few times when I do actually use the delete all app option and delete all of my tabs, I have to say it really feels nice. It feels like Marie Kondo, she came and cleaned all of my tabs and it’s a good feeling. Beah: Yeah, so sometimes you look at your tabs and you ask yourself, do they bring me joy? And if the answer is no, burn them all down. Yeah, nice. So anything that you’re looking forward to, like any plans to continue improving the fire button in the future? Sveta: Yeah, I think there’s lots of things we could explore and improve. We’ve maybe trying to see if we can find a new flow or simplify the confirmation for the regular fire button users. We’re also trying out something interesting on Android now where we’ve just going to launch to release this update. switching the buttons around and we’ll have burning, deleting the one top as a primary button. See if this would make this fire button feel more, less scary, less destructive and hopefully will make more people feel like that’s something for them. That will be interesting to see how that works out. Beah: Nice. Awesome. Anything that we didn’t get to that you want to add, Sveta? Sveta: I think, so I don’t think a lot of people know, but on our iOS app, you can actually customize that fire button. If you tried it and you’re sure that you want to use it regularly or not as much, you can long press and swap it for something else. You can also go to settings, appearance and do it from there. And if you do hide the fire button from the main UI, you’ll still have it in the main view. think that’s good to share. also, yeah, just to everyone, try the fire button. Give us feedback. We read it. It will help us. Yeah. Beah: Yeah, nice. No need to use all caps, but we do read and appreciate the feedback. Yeah, awesome. Yeah, and if you were like a, if you discounted the fire button in the past because it was too destructive, let this be your sign to give it another try. Now that we’ve made these improvements to it, I think you might really like it. Cool. Well, thanks so much for joining, Sveta. I really appreciate it. And thanks to everyone for watching. Sveta: can try with one tap, yeah, small steps. Thank you, was fun. Beah: and see you later. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com [https://insideduckduckgo.substack.com?utm_medium=podcast&utm_campaign=CTA_1]

Duck Tales: How we privately sync bookmarks, passwords & AI chats across your devices, without you needing an account (Ep.27)

In this episode, Beah (Chief Product Officer) and Emanuele (Engineering) discuss Sync — which lets you have consistent bookmarks, passwords, and AI chats across devices — how it’s unique from a privacy perspective, and what’s coming next. Disclaimers: (1) The audio, video (above), and transcript (below) are unedited and may contain minor inaccuracies or transcription errors. (2) This website is operated by Substack. This is their privacy policy [https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsubstack.com%2Fprivacy&data=05%7C02%7Clroberts%40duckduckgo.com%7C9bf97bb22b8e4672114908ddecafdd9e%7C728892a04da94114b51152f75ee3bc3d%7C1%7C0%7C638926962285187494%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XENubLLde0FgfAkknFHydtBuSfaEZcwDQzp0v4pN48E%3D&reserved=0]. Beah: Hi, welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology and people that help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering or approach to AI. In this episode, we’re going to talk about our Sync feature. So today, Emanuele has joined me. I’m Beah. You haven’t met me before. I’m on the product team here at DuckDuckGo and Emanuele is, well, I’ll let you introduce yourself. Emanuele: Yeah, thanks. Hi, Beah. Welcome everyone. I am Emanuele. I have been at DuckDuckGo for about six plus years and I’ve been involved in Autofill and Sync almost from the start. Yeah. Beah: Cool. All right, well let’s dig in. I have a few questions for you. So first of all, what are we even talking about? What is Sync? Emanuele: Sync is a feature that lets you synchronize your data from one device to another. For example, you may have a collection of bookmarks on your desktop and you want to see the same bookmarks on your mobile device. You set up Sync and all your data is showing across both devices seamlessly. Beah: It sounds very handy. So you can sync bookmarks. What else can you sync across devices? Emanuele: Yeah, you can sync also autofill data, credentials, and credit cards. You can also sync, well, we said bookmarks. Recently, we added Chat Sync, so you can sync your Duck AI chats across devices. This is currently only available for our own first-party applications, but we are expanding to other browsers as well. Beah: Nice. So like, just to kind of paint a picture here, I have a MacBook Pro, I have a Windows laptop, I have an iPhone, and have an Android phone as well. And I’m like, you know, trying to log in to Reddit in all these places. If I save my Reddit credentials, my login, my username and my password on any of those devices, then it can automatically sync to all of those. So then the next day when I’m like out and about on my phone, if I logged in on my laptop, like the password just autofills and I can log right in. Is that right? And then now with that... Emanuele: Exactly. Yeah. Beah: Duck AI chats, as you’ve mentioned as well, if I wanted to like, so what’s the yeah, tell me about chats. What’s the use case there? Why do I care that it’s across my devices? Emanuele: Well, the use case, I use it a lot personally when I chats on the desktop and then I want to continue in the evening on the couch, I can resume the same chat and continue the conversation with Duck AI from there. I also use it for looking at older chats and basically maybe I chatted with Duck AI for something a while ago, and all of a sudden I remember, oh, well, there was this information that I already discussed with Duck AI, and I can go when I’m on the go outside with a friend and show my finding even when I’m on the go. Well, with Duck AI specifically, the Sync feature also increases the limits of how many chats you can store in Duck AI. The limits used to be 30, we bumped it at 100, but with Sync you have a lot more storage, so you can store a lot more chats with Sync activated. Beah: Nice, so if I’m like chatting on my phone, chatting on my computer, I’m like, you kind of need it because you potentially are like doubling or whatever, multiplying the volume of chats that you’re having and you don’t want to be like penalized, like lose those because they’re synced across devices. You just want them available. Emanuele: Right. Yep. Beah: Yeah, that makes sense. Tell me, like, let’s talk about how we do Sync from a privacy perspective. Like, are there any considerations there? How do we do it in a way that is not identifying and protects all of our users’ privacy? Emanuele: Yeah. First of all, all of the data in Sync is encrypted end to end, which means nobody but you can access your data, not even us. We don’t have the decryption key, so the information stays encrypted on our servers and nobody can access them. We went a step further in that we don’t really ask you for emails to sign up on anything. We don’t ask you for your passwords or anything. We just set up an account in a way that’s secure with basically a random string that is used to encrypt, I’m kind of simplifying things, but like a random string that is used to encrypt your data and then the string stays locally on your device. This ensures that, you know, whatever happens in the server, the data cannot be decrypted by a malicious actor. And we, as a company, cannot know who you are. We don’t have any information on you and we don’t have any way to tie your data with a specific person. Beah: How does that compare, like, how unique is that? Like, is that what everybody does with sync and password management or is that pretty unique and how so? Emanuele: The end-to-end encryption is a pretty common pattern in password managers specifically. Not as much maybe in browsers, I don’t think. But the fact that the account is... that you don’t need an email to log into your account, that’s pretty unique. I don’t know that there are many other companies that do anything like this. Beah: Yeah, I can’t think of any mainstream, well-adopted examples anyway where you don’t need an account. Cool. How hard has that been in practice? What technical challenges did it create? Or was it just hunky dory? Emanuele: Yeah, no, it’s not easy, especially when you create an account that doesn’t have an email associated with it, it means that certain things that users are accustomed to, that they expect, they’re not there. So you have to kind of create user experiences that allow users to use their accounts seamlessly without thinking about an account, but with all the features that they expect. Specifically, the recovery part is the complex part because we don’t have data. We don’t have any way to retrieve your data. Therefore, you have to save a recovery code to decrypt the data on your side. The complexities that we face are both in terms of UX, as explained, but also there are complexities on the engineering side, as we need to ensure every time that all the flows are accounted for in this sort of accountless account. And yeah. Beah: Yeah. Emanuele: When we start to connect certain things together, like the chats and Sync and, you know, so we will look into subscriptions as well. We’re basically connecting different systems that all are sort of blind to the user concept, but they still have to sort of contain this data. So that’s a high level. Beah: Yeah, as you were talking, I was thinking, this is like a great example of one of those things that’s like deeply, deeply complex. And like for it to work, you just have to like, you have to boil down that complexity and like, shield the user from that complexity and present a very simple, it works, very little room for human error, no requirement of deep understanding of how it works. And I think that is a really interesting challenge. Emanuele: Yeah, I think our UX for setting up the account has this advantage that you don’t have to type anything and you just set it up and when you have to transfer it to another account, you just scan a QR code and the data is automatically transferred via that QR code scan. I think that’s kind of handy in the setup step. But again, it poses a few challenges then in the recovery phase. Beah: Yeah, like you don’t have to remember a password or remember what email you used. You just have a device, another device, and then let them talk to each other. But getting there was really hard. I mean, and maybe there’s still places that we want to improve the UX, but I just, I know we’ve been like finessing it for a long time. Now I’m somebody who like has to sync things all the time because I’m constantly like uninstalling and reinstalling and trying new devices as part of my job, like testing and you know, understanding our products and looking for gaps and like, so I go through the Sync setup, you know, way more often than any normal user should have to. And it has gotten a lot better. Like I, there are moments when I’m like, that was easy, which is what we’re going for. Emanuele: Magic. Beah: Yeah, magic. Magic. Emanuele: Yeah. We are actually working right now on an improvement to the Sync screens. We are going to launch an experiment on iOS first. And we hope that the Sync flow will be easier, even easier for users to understand specifically the very first step when you decide whether you want a new account or you want to connect to an existing device, that should be easier. But they’re just improvements around the existing flows rather than a new different flow. Beah: Yeah, I think one of the things you’ve done really well as a leader of this work, Emanuele, is continuing to watch the data and poke holes in the flow and basically make sure that we’re simultaneously building new features and optimizing what we have and just making it better and better and better as we learn more. I’m curious if there are any kind of in that vein or you can leave that direction if you want. You can say whatever you want. If there are things that have surprised you in the process of building Sync and getting feedback and watching the data points and so forth. Emanuele: Yeah, I think one sort of surprising thing was that the people were not really finding the feature organically. As soon as we started building entry points where they were useful to the user flow, meaning when you are saving a password, when you are watching a specific screen, like you are looking at your bookmarks, for example, and we show at that point a prompt that says, do you want to move this bookmark to your other device? And we do that in several touch points. And again, these have been studied to be the least intrusive possible, but also to pop up at the right moment when the feature is more salient to the user case. When we started doing these promotions, we call them promotions, we can call them entry points, basically making the feature discoverable. User adoption shot up quite dramatically and basically we were very happy to see the trajectory of adoption once we start to put in Sync front and center for the users. Basically it allows them to use DuckDuckGo, like enter the ecosystem and sort of buy into the whole system. And your data is there, your browsing is easier, your bookmarks are there, you can save something on desktop and again read that on your commute on mobile. Yeah, these entry points have been really effective. Beah: Yeah, if you build it, they won’t come unless they see that it exists and you make it easy for them. Yeah, nice. All right. Tell me what, yeah, you kind of already touched on this, but anything that’s coming up, coming soon to Sync that you want to talk about? Emanuele: Exactly. Yep. Yeah, a couple of things we are working on. The first is allowing third party browsers to sync their chats as well. This is the first time that we are actually building Sync on a third party browser that is not integrated with our apps basically. So that’s coming fairly soon. The new screens, we just touched on the new onboarding screen for iOS. And we’re also starting to sync subscription status. So if you have a subscription, a DuckDuckGo subscription on your mobile device, currently to move it to your desktop device, it’s quite a convoluted process. Whereas with Sync, again, we move to the QR code scanning. So if you have Sync already active, the data will start syncing automatically. And if not, you will be able to set up Sync again with a QR code right from the subscription screens and move your subscription information across very easily, which means your advanced model for Duck AI, they will transfer automatically and you can resume your advanced model chats across devices. Beah: Nice. I am personally looking forward to that because I also go through the, you’re a subscriber, let’s retrieve your subscription flow somewhat regularly. Cool. Emanuele: Yeah, it’s a bit convoluted. And we will also look into tab sync, which is a pet peeve of mine. I’m looking forward to that, which means that your open tabs on one device will be accessible from another device, you know, seamlessly through Sync. Beah: Cool. Awesome. Well, I think we should wrap it up. Is there anything that you want to say that you haven’t already before we close? This is your moment, man. Go nuts. Yeah, that’s a good one. Set up Sync. It is awesome. I can vouch for that. Cool. Well, thanks so much for joining. Really appreciate it. Yeah, see you around DuckDuckGo. Emanuele: You know, set up Sync, it’s awesome. Thank you, Beah. Bye-bye. Beah: Thanks everyone. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit insideduckduckgo.substack.com [https://insideduckduckgo.substack.com?utm_medium=podcast&utm_campaign=CTA_1]