The AI PM Portfolio Guide: What to Include, How to Build It, and 3 Examples

# The AI PM Portfolio Guide: What to Include, How to Build It, and 3 Examples **TL;DR:** Your traditional PM portfolio doesn't work for AI PM roles. Hiring managers (like me) want to see eval specs, model tradeoff analysis, and evidence that you understand the unique challenges of shipping AI products. Here's exactly what to include, how to showcase AI work without violating your NDA, and a template with three example entries you can adapt. --- ## Why Your Current PM Portfolio Won't Land You an AI PM Role I've hired over a dozen AI PMs in the past two years. I've reviewed hundreds of portfolios. Here's the pattern I see: Most candidates show me beautiful case studies about user research, A/B tests, and feature launches. That's great. It tells me they're a competent product manager. But it tells me nothing about whether they can ship AI products. AI product management is a different discipline. The failure modes are different. The success metrics are different. The stakeholder communication is different. Your portfolio needs to reflect that. If you're applying for AI PM roles and your portfolio looks identical to a traditional PM portfolio, you're leaving your biggest differentiator on the table. Let me show you what I actually look for — and how to build a portfolio that gets you hired. ## What AI PM Portfolios Need (That Traditional Ones Don't) ### 1. Eval Specs This is the #1 thing that separates AI PMs from traditional PMs in my interviews. Can you define what "good" looks like for an AI system? An eval spec shows me you can: - Define measurable quality criteria for non-deterministic systems - Think about edge cases and failure modes - Balance precision, recall, and other domain-specific metrics - Create evaluation datasets and rubrics **What to include:** A sample eval spec for a real or hypothetical AI feature. Show the metrics, the thresholds, the dataset design, and the human evaluation protocol. ### 2. Model Comparison Documentation AI PMs make model selection decisions. This isn't "pick the one with the highest benchmark score." It's a multidimensional tradeoff analysis across quality, cost, latency, reliability, and capability fit. **What to include:** A model comparison document that shows how you evaluated multiple approaches for a specific use case. Include your criteria, your testing methodology, and your recommendation with reasoning. ### 3. Prototype Demonstrations Nothing beats showing that you can build. You don't need to be an engineer, but demonstrating that you can prototype with AI tools — using no-code platforms, API playgrounds, or simple scripts — shows you understand the medium you're managing. **What to include:** A working prototype, a video walkthrough, or a detailed technical design doc that shows you understand how the pieces fit together. ### 4. Impact Metrics (AI-Specific) Traditional PMs show conversion rates and revenue impact. AI PMs should show those too — but also metrics specific to AI systems: - Model accuracy improvements over iterations - Cost per inference and optimization results - Latency improvements - Hallucination rate reductions - Eval suite coverage expansion - A/B test results comparing AI vs. non-AI experiences **What to include:** Quantified impact that demonstrates you understand AI-specific success metrics, not just business metrics. ### 5. Stakeholder Communication Artifacts AI products require a different kind of stakeholder communication. You're explaining probabilistic systems to deterministic thinkers. Show me you can do that. **What to include:** A roadmap presentation, a launch decision document, or a stakeholder FAQ that demonstrates how you communicate uncertainty, model limitations, and AI-specific risks. ## How to Showcase AI Work Without Violating Your NDA This is the elephant in the room. Your best AI work is probably covered by an NDA. Here's how I navigate this — and what I advise candidates to do. ### Strategy 1: Abstract and Generalize Take your real work and strip out the identifying details. Change the company, the domain, and the specific numbers while preserving the thinking and methodology. **Instead of:** "At [Company], I improved our customer support AI's resolution rate from 67% to 84%." **Try:** "For a B2B SaaS support automation system, I designed an eval framework that improved automated resolution rates by 25% while reducing escalation false negatives by 40%." The methodology and thinking are yours. The specific numbers and context are generalized enough to not violate your NDA. When in doubt, check with your legal team — most are fine with this level of abstraction. ### Strategy 2: Build Side Projects This is the most bulletproof approach. Build something real with AI that you own entirely. Ideas that work well: - **Build an AI-powered tool** that solves a real problem. Document the product decisions, not just the code. - **Create an eval suite** for a public AI use case (e.g., evaluate chatbots for customer service using publicly available benchmarks). - **Write a model comparison** for a specific use case using publicly available models. - **Prototype a feature** using public APIs and document your product thinking. The bar isn't "production-quality software." It's "evidence of AI product thinking." ### Strategy 3: Contribute to Public Discourse Blog posts, conference talks, and open-source contributions are portfolio pieces. If you've written thoughtfully about AI product challenges, that demonstrates expertise. **What counts:** - Blog posts about AI PM methodology - Conference talks about shipping AI features - Open-source eval frameworks or tools - Detailed product analyses of public AI products ### Strategy 4: Use the Interview Project Many AI PM interviews include a take-home project or case study. Do exceptional work on these and (with permission) include them in your portfolio. This is purpose-built portfolio material with no NDA concerns. ## The AI PM Portfolio Template Here's the structure I recommend. Keep it focused — 3-5 entries max. Quality over quantity. ### Portfolio Structure ``` 1. Introduction (1 paragraph) - Who you are - Your AI PM thesis (what you believe about building AI products) - What you're looking for 2. Portfolio Entry 1: [Your Strongest AI Case Study] 3. Portfolio Entry 2: [A Different Type of AI Work] 4. Portfolio Entry 3: [A Side Project or Technical Demonstration] 5. Background - Relevant experience summary - Technical skills (specific models, tools, frameworks) - Link to resume ``` ### Individual Entry Template ``` ## [Project Name] **Context:** [1-2 sentences: company size/stage, product area, your role] **Problem:** [What user/business problem were you solving? Why was AI the right approach?] **My Contribution:** - [Specific thing you did #1] - [Specific thing you did #2] - [Specific thing you did #3] **AI-Specific Challenges:** - [Challenge 1: e.g., "Model hallucinated on edge cases involving..."] - [Challenge 2: e.g., "Cost per query exceeded budget by 3x initially"] - [How you addressed each challenge] **Methodology:** - Eval approach: [How did you measure quality?] - Model strategy: [What models did you use/evaluate? Why?] - Rollout: [How did you launch? What was your risk mitigation?] **Results:** - [Metric 1 with specific numbers] - [Metric 2 with specific numbers] - [Business impact] **Artifacts:** [Links to docs, prototypes, or detailed write-ups] **What I'd Do Differently:** [Shows self-awareness and learning] ``` ## 3 Example Portfolio Entries ### Example 1: Enterprise AI Feature (Generalized from Real Work) --- **Project: AI-Powered Document Intelligence for Enterprise Search** **Context:** Senior PM at a mid-market B2B SaaS company (~2,000 enterprise customers). Led a team of 6 engineers building AI-powered document understanding capabilities. **Problem:** Enterprise customers stored millions of documents but couldn't find relevant information efficiently. Keyword search failed for ambiguous queries, synonyms, and conceptual questions. Support tickets about "can't find what I need" were our #2 churn driver. **My Contribution:** - Defined the eval framework: 500-example benchmark dataset with relevance ratings from domain experts on a 4-point scale - Led model selection process: evaluated 4 embedding models and 3 reranking approaches across quality, cost, and latency dimensions - Designed the phased rollout: internal dogfood → 50 beta customers → percentage ramp to GA - Created the stakeholder communication framework for explaining probabilistic search quality to enterprise buyers **AI-Specific Challenges:** - **Eval dataset bias:** Our initial benchmark over-represented short, keyword-like queries. Real user queries were longer and more conceptual. I rebuilt the dataset using actual search logs (anonymized), which changed our model ranking entirely. - **Cost explosion at scale:** First approach cost $0.12 per query. At our query volume, that was $400K/year. Implemented a hybrid architecture (cheap embedding search → expensive reranking on top-50) that reduced cost to $0.03 per query with <2% quality degradation. - **Hallucination in snippets:** The system generated answer snippets that sometimes included information not in the source documents. Added a faithfulness eval and implemented source-grounding constraints. **Methodology:** - Eval: Automated relevance scoring (NDCG@10) plus weekly human eval sessions with 3 domain experts - Model strategy: Started with OpenAI embeddings, migrated to Cohere for cost; used GPT-4o mini for snippet generation with source grounding - Rollout: 4-week internal dogfood, 6-week beta with weekly eval reviews, 3-stage percentage ramp over 4 weeks **Results:** - Search relevance (NDCG@10): 0.62 → 0.84 (+35%) - "Can't find" support tickets: -47% in first quarter post-launch - Query-to-result latency: p95 < 650ms (target was 800ms) - Cost per query: $0.03 (target was $0.05) - Net retention impact: +2.3 points in cohort with feature enabled **What I'd Do Differently:** I would have invested in the eval infrastructure earlier. We spent our first 3 weeks building the feature and only then realized our eval suite was inadequate. Building evals first would have saved us a month of iteration. --- ### Example 2: Model Comparison Analysis (Side Project) --- **Project: LLM Evaluation for Customer Service Automation** **Context:** Independent analysis comparing frontier LLMs for automated customer service response generation. Built as a side project to demonstrate AI PM methodology. All data and code publicly available. **Problem:** Companies adopting LLMs for customer service lack rigorous comparison frameworks. Benchmark performance doesn't translate to domain-specific quality. I built a realistic evaluation to show how model selection should actually work. **My Contribution:** - Designed a 200-example eval dataset based on publicly available customer service scenarios (pulled from Twitter support threads and Reddit help forums) - Created a 5-dimension rubric: accuracy, helpfulness, tone appropriateness, conciseness, and safety - Evaluated 6 models: GPT-4o, GPT-4o mini, Claude Sonnet, Claude Haiku, Gemini Pro, Llama 3.1 70B - Built cost and latency models for each at various scale points - Published findings with full methodology **AI-Specific Challenges:** - **Rubric calibration:** Initial inter-rater reliability was low (Cohen's kappa = 0.43). Revised rubric with specific examples for each rating level, improved to 0.78. - **Prompt sensitivity:** Model rankings changed significantly with prompt variations. Tested 5 prompt variants per model to ensure robust comparison. - **Cost modeling complexity:** Comparing token-based pricing across models with different tokenizers and different verbosity levels required careful normalization. **Methodology:** - Eval: 3 human raters per example, majority vote with adjudication for disagreements - Testing: 5 prompt variants × 200 examples × 6 models = 6,000 evaluations - Cost analysis: Normalized to "cost per resolved ticket" using average token counts from real conversations **Results:** - Published finding: Claude Sonnet offered the best quality-to-cost ratio for this use case (87% quality score at $0.04/interaction vs. GPT-4o's 91% at $0.11/interaction) - Llama 3.1 70B self-hosted was cheapest at scale (>10K interactions/day) but required significant prompt engineering to match hosted model quality - GPT-4o mini was the sweet spot for companies prioritizing cost over marginal quality gains - [Link to full analysis and dataset] **What I'd Do Differently:** I'd include a longitudinal component — testing the same prompts monthly to track model quality changes over time. Single-point-in-time comparisons have a short shelf life. --- ### Example 3: AI Product Prototype (Side Project) --- **Project: AI Meeting Brief Generator** **Context:** Built a working prototype that generates pre-meeting briefs by combining calendar context, email history, and CRM data. Designed as a demonstration of AI product thinking, not just technical capability. **Problem:** PMs spend 15-30 minutes preparing for each customer meeting — reviewing past emails, checking CRM notes, reading recent support tickets. This is repetitive, high-value work that AI can augment. **My Contribution:** - Defined the product spec: user stories, acceptance criteria, and eval rubric - Built the prototype using Python, OpenAI API, and mock data sources - Designed the eval framework: compared AI-generated briefs against manually-created ones using a panel of 5 PM volunteers - Documented the full product thinking: why this approach, what the risks are, how to scale it **AI-Specific Challenges:** - **Context window management:** A customer with 2 years of email history easily exceeds context limits. Built a relevance-based retrieval system that selects the most pertinent 20 emails and 5 CRM notes. - **Hallucination risk in high-stakes context:** A brief that includes inaccurate information about a customer is worse than no brief at all. Implemented a citation system — every claim in the brief links to its source document. - **Staleness:** Meeting context changes. Emails arrive after the brief is generated. Designed an incremental update system that refreshes the brief 30 minutes before the meeting. **Methodology:** - Eval: 5 PMs rated AI briefs vs. their own manual briefs on completeness, accuracy, and usefulness (1-5 scale) - 30 test meetings across different customer types and meeting contexts - Measured time savings and subjective quality **Results:** - Average brief quality: 4.1/5 (vs. 4.4/5 for manual briefs — 93% of human quality) - Time savings: 18 minutes average per meeting - Accuracy: 96% of factual claims verified against source documents - PM feedback: 4/5 said they'd use this daily; 1/5 preferred manual prep for high-stakes meetings - [Link to prototype demo video and product spec] **What I'd Do Differently:** I'd start with a narrower scope — just email summarization for a specific meeting type — and expand from there. The prototype tried to do everything at once, which made eval harder and the results noisier. --- ## Final Advice ### For Career Switchers If you're moving into AI PM from traditional PM, your portfolio is your bridge. You don't need production AI experience to start — you need *evidence of AI product thinking*. Side projects, model comparisons, and eval framework designs all count. What matters is that you demonstrate understanding of what makes AI products different. ### For Experienced AI PMs Your challenge is showcasing work you can't fully share. Use the abstraction strategies above. And invest in at least one public-facing project — it gives you something you can discuss in full detail during interviews, which is more valuable than five redacted case studies. ### For Everyone The best AI PM portfolio I've ever reviewed had three entries: one production case study (generalized), one side project prototype, and one published model comparison. Total length: 6 pages. It told me everything I needed to know. Don't overthink it. Build it. Ship it. Iterate. --- ## Try This Week 1. **Audit your current portfolio.** Does it include any AI-specific artifacts (eval specs, model comparisons, cost analyses)? If not, you have a gap. 2. **Write one eval spec.** Pick any AI feature (a chatbot, a recommendation system, a search feature) and write a complete eval spec: metrics, thresholds, dataset design, and human eval protocol. This alone is a portfolio piece. 3. **Start a side project.** Use a public API to build something small. The bar is "demonstrates AI product thinking," not "production-ready software." A weekend project with a good write-up beats a polished deck with no substance. 4. **Generalize one real project.** Take your best AI work, strip the identifying details, and write it up using the template above. Have a trusted colleague confirm it doesn't violate your NDA. --- *I hire AI PMs and write about what separates good ones from great ones. [Subscribe to my newsletter](https://pmthebuilder.com/newsletter) for weekly, practitioner-level takes on AI product management.*