047 - How Yelp Integrates Data Science, Engineering, UX, and Product Management when Creating AI Products with Yelp’s Justin Norman

In part one of an excellent series on AI product management, LinkedIn Research Scientist Peter Skomoroch and O’Reilly VP of Content Strategy Mike Loukides explained the importance of aligning AI products with your business plans and strategies. In other words, they have to deliver value, and they have to be delivered on time. Unfortunately, this is much easier said than done. I was curious to learn more about what goes into the complex AI product development process, and so for answers I turned to Yelp VP of Data Science Justin Norman, who collaborated with Peter and Mike in the O’Reilly series of articles. Justin is a career data professional and data science leader with experience in multiple companies and industries, having served as director of research and data science at Cloudera Fast Forward Labs, head of applied machine learning at Fitbit, head of Cisco’s enterprise data science office, and as a big data systems engineer with Booz Allen Hamilton. He also served as a Marine Corps Officer with a focus in systems analytics. We covered: Justin’s definition of a successful AI product The two key components behind AI products The lessons Justin learned building his first AI platform and what insights he applied when he went to Yelp. Why AI projects often fail early on, and how teams can better align themselves for success. Who or what Beaker and Bunsen are and how they enable Yelp to test over 700 experiments at any one time. What Justin learned at an airline about approaching problems from a ML standpoint vs. a user experience standpoint—and what the cross-functional team changed as a result. How Yelp incorporates designers, UX research, and product management with its technical teams Why companies should analyze the AI, ML and data science stack and form a strategy that aligns with their needs. The critical role of AI product management and what consideration Justin thinks is the most important when building a ML platform How Justin would approach AI development if he was starting all over at a brand new company Justin’s pros and cons about doing data science in the government vs. the private sector. Quotes from Today’s Episode “[My non-traditional background] gave me a really broad understanding of the full stack [...] from the physical layer all the way through delivering information to a decision-maker without a lot of time, maybe in an imperfect form, but really packaged for what we're all hoping to have, which is that value-add information to be able to do something with.” - Justin “It's very possible to create incredible data science products that are able to provide useful intelligence, but they may not be fast enough; they may not be [...] put together enough to be useful. They may not be easy enough to use by a layperson.” -Justin   “Just because we can do things in AI space, even if they're automated, doesn't mean that it's actually beneficial or a value-add.” - Justin “I think the most important thing to focus on there is to understand what you need to be able to test and deploy rapidly, and then build that framework.” - Justin “I think it's important to have a product management team that understands the maturity lifecycle of building out these capabilities and is able to interject and say, ‘Hey, it's time for us to make a different investment, either in parallel, once we've reached this milestone, or this next step in the product lifecycle.’” - Justin “...When we talk about product management, there are different audiences. I think [Yelp’s] internal AI product management role is really important because the same concepts of thinking about design, and how people are going to use the service, and making it useful — that can apply to employees just as much as it can to the digital experience that you put out to your end customers.” -Brian “You hear about these enterprise projects in particular, where the only thing that ever gets done is the infrastructure. And then by the time they get something read