A Practical Guide for Product Managers Building AI-Powered Products.
The problem with artificial intelligence (AI) today is that the way we *currently design and train it often leads to it being highly effective at achieving its objectives, but without considering the consequences. For example, if you ask a human to get a cup of tea, they might ask questions like, “How much are you willing to pay?” or “What would you like if it’s out of stock?”. In contrast, an AI might stop at nothing to fulfill the request, even if it means paying $50 or even killing everyone in its way.
While this is an exaggerated example, it illustrates the point: AI can become too efficient in reaching goals without taking context or limits into account.This reflects a broader challenge faced by organizations: AI models that excel at achieving goals often lack the contextual awareness needed to make balanced decisions.
Footnote: I acknowledge that there have been significant advancements in training methods and computational power, which may eventually lead us to Artificial General Intelligence (AGI). However, that’s a topic for another day.
This is why product managers play a critical role as orchestrators, ensuring that UX designers, data scientists, engineers, and other business stakeholders (such as legal and marketing) work together to deliver an AI-powered product that benefits both consumers and the organization within specified constraints.
However, despite the critical role they play, product managers often struggle to identify the right AI solution, given the available resources (data, infrastructure, and capital). They may also lack a clear understanding of the problems that AI is best suited to solve and how to work with data scientists to fine-tune the model iteratively. This has led to failed product launches and disappointment among senior executives due to misaligned and unmanaged expectations.
To help product managers effectively navigate these complexities and guide their teams, I have developed a 7-step framework for creating AI-powered products.
1. Define the Problem and Objectives
Objective: Clearly identify the problem you’re trying to solve and how an AI solution can address it. Establish measurable goals and success criteria for the AI-powered product.
Why It’s Important: Ensures that AI is used where it adds real value, aligning the solution with business objectives and user needs.
Example: “Improve user engagement by providing personalized news article recommendations.”
Useful questions to ask:
- What specific problem are we trying to solve, and why is it important?
- Is this problem significant enough to warrant an AI solution for both users and the business?
- How will an AI solution add value compared to non-AI alternatives?
- Can the solution tolerate uncertainty or some error?
- What sort of fall-back or human-intervention is required here?
2. Assess Feasibility and Resources
Objective: Evaluate the availability of data, infrastructure, technical capabilities, budget, and timeline. Determine whether your organization has the necessary data, talent, and technology to develop the AI solution.
Why It’s Important: Prevents wasted resources by ensuring the project is achievable given current constraints.
Example: Assess whether you have sufficient historical data to train a recommendation model.
Useful questions to ask:
- Is it a strategic priority for the company to develop and train our own AI model?
- Do we have sufficient quality data to train an AI model effectively?
- What infrastructure (e.g., cloud services, hardware) is required to develop and deploy the solution?
- Are there legal or regulatory frameworks that restrict us from partnering with an AI solution vendor?
- Is there enough budget and time to develop, test, and launch the AI model?
If the answer to all of the above questions is “no,” it may still be worth exploring the possibility of partnering with an AI solution vendor because they can provide the necessary expertise, infrastructure, and pre-trained models, allowing your organization to leverage AI capabilities without the need for extensive in-house development. This approach can save time, reduce costs, and accelerate the implementation of AI-powered solutions.
3. Select the Right AI Approach
Objective: Choose the appropriate AI technique (supervised learning, unsupervised learning, deep learning, etc.) based on the problem and available data. Consider whether a rule-based, machine learning, or deep learning model best suits your needs.
Why It’s Important: The right approach ensures efficiency and effectiveness in achieving your objectives.
Example: Use collaborative filtering for personalized recommendations if you have enough user interaction data.
Useful questions to ask:
- Is the problem best suited for supervised, unsupervised, or reinforcement learning?
- Should we use a rule-based system, traditional machine learning, or deep learning model?
- What data preprocessing or feature engineering will be required?
- How do we manage biases in the data to ensure fair and accurate outcomes?
In my next article, I will delve deeper into AI training approaches (supervised and unsupervised), techniques (such as neural networks, collaborative filtering, and transformers), and how to fine-tune models to reduce bias. Understanding these techniques is crucial not only for selecting the right AI approach in step 3 but also for effectively building, validating, and integrating the model in steps 4 and 5.
4. Build and Validate the AI Model
Objective: Work with data scientists and engineers to build, train, and validate the AI model using iterative development. Use techniques like cross-validation to test and refine the model.
Why It’s Important: Ensures that the model performs well on both training and unseen data, reducing the risk of errors or overfitting.
Example: Train a model using historical user behavior data and validate its accuracy with a test set.
Useful questions to ask:
- How will we split the data into training, validation, and test sets?
- What techniques will we use to prevent overfitting or underfitting?
- What iterative process will we use to refine and improve the model?
- How will we provide insights into the model’s decision-making process to make it more understandable to stakeholders?
5. Integrate the Model into the Product
Objective: Collaborate with engineering and UX teams to integrate the AI model into the product, ensuring it fits seamlessly within the user experience and meets performance requirements.
Why It’s Important: The model’s success depends on how well it integrates with the product and how users interact with it.
Example: Embed the recommendation engine into the homepage of a news website to personalize the content users see.
Useful questions to ask:
- How should we communicate to users that AI is involved in a way that builds trust and transparency?
- How will we handle model updates and deployments without disrupting the user experience?
- What feedback mechanisms can we implement to capture user interactions with the AI features?
- Are there fallback options if the model’s predictions fail or are inaccurate?
6. Monitor Performance and Iterate
Objective: Continuously monitor the model’s performance using key metrics (accuracy, engagement rates, conversion rates, etc.) and iterate based on user feedback and changing data patterns.
Why It’s Important: AI models can degrade over time as user behavior changes, so regular monitoring and updates are crucial to maintaining effectiveness.
Example: Track how often users click on recommended articles and refine the model based on patterns of user interaction.
Useful questions to ask:
- Which metrics will we monitor to assess the model’s performance in production?
- How will we handle situations where model performance degrades over time?
- What feedback loops will we implement to gather insights from user behavior?
- Are there emerging trends or data patterns that require adjusting the model’s parameters?
7. Align with Stakeholders and Communicate Results
Objective: Regularly communicate progress, results, challenges, and insights with stakeholders (executives, legal, marketing, UX) to manage expectations and ensure alignment.
Why It’s Important: Keeps all parties informed, ensures that expectations are managed, and allows for strategic adjustments based on AI outcomes.
Example: Present a monthly report on how the AI-driven recommendation engine has improved user engagement and what steps are planned for further enhancement.
Useful questions to ask:
- How often should we update stakeholders on the progress and performance of the AI solution?
- What metrics and insights should be shared to demonstrate the value of the AI model?
- How will we manage stakeholder expectations regarding the capabilities and limitations of the AI solution?
- Are there potential risks or challenges that need to be communicated early on?
I hope this 7-step framework serves as a valuable guide for product managers in their product development process. Building AI-powered products is not a straightforward task — it requires a clear understanding of the problem, careful selection of the right AI approach, and continuous monitoring and refinement. By following this framework, product managers can lead their teams in developing AI solutions that truly add value.
As a product manager, your ability to coordinate the efforts of data scientists, engineers, designers, and business stakeholders is essential. It’s your responsibility to ensure that the AI-powered product not only meets the organization’s goals but also provides real value to users.
Ultimately, as the field of AI continues to evolve, developing AI-powered products is a continuous learning journey. By embracing this journey and adapting to new challenges, product managers have the opportunity to create products that not only meet their objectives but also make a meaningful impact on both users and the organization.