# Evaluating Agentic AI: DeepEval, RAGAS & TruLens Frameworks Compared

In this episode of Memriq Inference Digest - Engineering Edition, we explore the cutting-edge evaluation frameworks designed for agentic AI systems. Dive into the strengths and trade-offs of DeepEval, RAGAS, and TruLens as we unpack how they address multi-step agent evaluation challenges, production readiness, and integration with popular AI toolkits.

In this episode:

- Compare DeepEval’s extensive agent-specific metrics and pytest-native integration for development testing

- Understand RAGAS’s knowledge graph-powered synthetic test generation that slashes test creation time by 90%

- Discover TruLens’s production-grade observability with hallucination detection via the RAG Triad framework

- Discuss hybrid evaluation strategies combining these frameworks across the AI lifecycle

- Learn about real-world deployments in fintech, e-commerce, and enterprise conversational AI

- Hear expert insights from Keith Bourne on calibration and industry trends

Key tools & technologies mentioned:

DeepEval, RAGAS, TruLens, LangChain, LlamaIndex, LangGraph, OpenTelemetry, Snowflake, Datadog, Cortex AI, DeepTeam

Timestamps:

00:00 - Introduction to agentic AI evaluation frameworks

03:00 - Key metrics and evaluation challenges

06:30 - Framework architectures and integration

10:00 - Head-to-head comparison and use cases

14:00 - Deep technical overview of each framework

17:30 - Real-world deployments and best practices

19:30 - Open problems and future directions

Resources:

1. "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition
2. This podcast is brought to you by Memriq.ai - AI consultancy and content studio building tools and resources for AI practitioners.

Discover how the Model Context Protocol (MCP) is revolutionizing AI systems integration by simplifying complex multi-tool interactions into a scalable, open standard. In this episode, we unpack MCP’s architecture, adoption by industry leaders, and its impact on engineering workflows.

In this episode:

- What MCP is and why it matters for AI/ML engineers and infrastructure teams

- The M×N integration problem and how MCP reduces it to M+N

- Core primitives: Tools, Resources, and Prompts, and their roles in MCP

- Technical deep dive into JSON-RPC 2.0 messaging, transports, and security with OAuth 2.1 + PKCE

- Comparison of MCP with OpenAI Function Calling, LangChain, and custom REST APIs

- Real-world adoption, performance metrics, and engineering trade-offs

- Open challenges including security, authentication, and operational complexity

Key tools & technologies mentioned:

- Model Context Protocol (MCP)

- JSON-RPC 2.0

- OAuth 2.1 with PKCE

- FastMCP Python SDK, MCP TypeScript SDK

- agentgateway by Solo.io

- OpenAI Function Calling

- LangChain

Timestamps:

00:00 — Introduction to MCP and episode overview

02:30 — The M×N integration problem and MCP’s solution

05:15 — Why MCP adoption is accelerating

07:00 — MCP architecture and core primitives explained

10:00 — Head-to-head comparison with alternatives

12:30 — Under the hood: protocol mechanics and transports

15:00 — Real-world impact and usage metrics

17:30 — Challenges and security considerations

19:00 — Closing thoughts and future outlook

Resources:

• "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition
• This podcast is brought to you by Memriq.ai - AI consultancy and content studio building tools and resources for AI practitioners.

Unlock the secrets to evaluating Retrieval-Augmented Generation (RAG) pipelines effectively and efficiently with ragas, the open-source framework that’s transforming AI quality assurance. In this episode, we explore how to implement reference-free evaluation, integrate continuous monitoring into your AI workflows, and optimize for production scale — all through the lens of Keith Bourne’s comprehensive Chapter 9.

In this episode:

- Overview of ragas and its reference-free metrics that achieve 95% human agreement on faithfulness scoring

- Implementation patterns and code walkthroughs for integrating ragas with LangChain, LlamaIndex, and CI/CD pipelines

- Production monitoring architecture: sampling, async evaluation, aggregation, and alerting

- Comparison of ragas with other evaluation frameworks like DeepEval and TruLens

- Strategies for cost optimization and asynchronous evaluation at scale

- Advanced features: custom domain-specific metrics with AspectCritic and multi-turn evaluation support

Key tools and technologies mentioned:

- ragas (Retrieval Augmented Generation Assessment System)

- LangChain, LlamaIndex

- LangSmith, LangFuse (observability and evaluation tools)

- OpenAI GPT-4o, GPT-3.5-turbo, Anthropic Claude, Google Gemini, Ollama

- Python datasets library

Timestamps:

00:00 - Introduction and overview with Keith Bourne

03:00 - Why reference-free evaluation matters and ragas’s approach

06:30 - Core metrics: faithfulness, answer relevancy, context precision & recall

09:00 - Code walkthrough: installation, dataset structure, evaluation calls

12:00 - Integrations with LangChain, LlamaIndex, and CI/CD workflows

14:30 - Production monitoring architecture and cost considerations

17:00 - Advanced metrics and custom domain-specific evaluations

19:00 - Common pitfalls and testing strategies

20:30 - Closing thoughts and next steps

Resources:

- "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition

- Memriq AI: https://Memriq.ai

- ragas website: https://www.ragas.io/

- ragas GitHub repository: https://github.com/vibrantlabsai/ragas (for direct access to code and docs)

Tune in to build more reliable, scalable, and maintainable RAG systems with confidence using open-source evaluation best practices.

Is agent engineering the next big AI discipline or a repackaged buzzword? In this episode, we cut through the hype to explore what agent engineering really means for business leaders navigating AI adoption. From market growth and real-world impact to the critical role of AI memory and the evolving tool landscape, we provide a clear-eyed view to help you make strategic decisions.

In this episode:

- The paradox of booming agent engineering markets despite high AI failure rates

- Why agent engineering is emerging now and what business problems it solves

- The essential role of AI memory systems and knowledge graphs for real impact

- Comparing agent engineering frameworks and when to hire agent engineers vs ML engineers

- Real-world success stories and measurable business payoffs

- Risks, challenges, and open problems leaders must manage

Key tools and technologies mentioned: LangChain, LangMem, Mem0, Zep, Memobase, Microsoft AutoGen, Semantic Kernel, CrewAI, OpenAI GPT-4, Anthropic Claude, Google Gemini, Pinecone, Weaviate, Chroma, DeepEval, LangSmith

Timestamps:

00:00 – Introduction & Why Agent Engineering Matters

03:45 – Market Overview & The Paradox of AI Agent Performance

07:30 – Why Now: Technology and Talent Trends Driving Adoption

11:15 – The Big Picture: Managing AI Unpredictability

14:00 – The Memory Imperative: Transforming AI Agents

17:00 – Knowledge Graphs & Domain Expertise

19:30 – Framework Landscape & When to Hire Agent Engineers

22:45 – How Agent Engineering Works: A Simplified View

26:00 – Real-World Payoffs & Business Impact

29:15 – Reality Check: Risks and Limitations

32:30 – Agent Engineering In the Wild: Industry Use Cases

35:00 – Tech Battle: Agent Engineers vs ML Engineers

38:00 – Toolbox for Leaders: Strategic Considerations

41:00 – Book Spotlight & Sponsor Message

43:00 – Open Problems & Future Outlook

45:00 – Final Words & Closing Remarks

Resources:

• "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition
• This podcast is brought to you by Memriq.ai - AI consultancy and content studio building tools and resources for AI practitioners.

Thanks for tuning into Memriq Inference Digest - Engineering Edition. Stay curious and keep building!

Are your AI initiatives stalling in production? This episode uncovers the critical architectural shift brought by the Natural Language Understanding (NLU) layer and why treating AI as just another feature is setting CTOs up for failure. Learn how rethinking your entire stack—from closed-world deterministic workflows to open-world AI-driven orchestration—is essential to unlock real business value.

In this episode:

- Understand the fundamental difference between traditional deterministic web apps and AI-powered conversational interfaces

- Explore the pivotal role of the NLU layer as the "brain" that dynamically interprets, prioritizes, and routes user intents

- Discover why adding an orchestrator component bridges the gap between probabilistic AI reasoning and deterministic backend execution

- Dive into multi-intent handling, partial understanding, and strategies for graceful fallback and out-of-scope requests

- Compare architectural approaches and learn best practices for building production-grade AI chatbots

- Hear about real-world deployments and open challenges facing AI/ML engineers and infrastructure teams

Key tools & technologies mentioned:

- Large Language Models (LLMs)

- Structured function calling APIs

- Conversational AI orchestrators

- 99-intents fallback pattern

- Semantic caching and episodic memory

Timestamps:

00:00 – Introduction & Why This Matters

03:30 – The NLU Paradigm Shift Explained

07:45 – The Orchestrator: Bridging AI and Backend

11:20 – Handling Multi-Intent & Partial Understanding

14:10 – Turning Fallbacks into Opportunities

16:50 – Architectural Comparisons & Best Practices

19:30 – Real-World Deployments & Open Problems

22:15 – Final Takeaways & Closing

Resources:

• "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition
• This podcast is brought to you by Memriq.ai - AI consultancy and content studio building tools and resources for AI practitioners.

Unlock the next level of Retrieval-Augmented Generation with full memory integration in AI agents. In the previous 3 episodes, we secretly built up what amounts to a 4-part series on agentic memory. This is the final piece of that 4-part series that pulls it ALL together.

In this episode, we explore how combining episodic, semantic, and procedural memories via the CoALA architecture and LangMem library transforms static retrieval systems into continuously learning, adaptive AI.

This also concludes our book series, highlighting ALL of the chapters of the 2nd edition of "Unlocking Data with Generative AI and RAG" by Keith Bourne. If you want to dive even deeper into these topics and even try out extensive code labs, search for 'Keith Bourne' on Amazon and grab the 2nd edition today!

In this episode:

- How CoALAAgent unifies multiple memory types for dynamic AI behavior

- Trade-offs between LangMem’s prompt_memory, gradient, and metaprompt algorithms

- Architectural patterns for modular and scalable AI agent development

- Real-world metrics demonstrating continuous procedural strategy learning

- Challenges around data quality, metric design, and domain agent engineering

- Practical advice for building safe, adaptive AI agents in production

Key tools & technologies: CoALAAgent, LangMem library, GPT models, hierarchical memory scopes

Timestamps:

0:00 Intro & guest welcome

3:30 Why integrating episodic, semantic & procedural memory matters

7:15 The CoALA architecture and hierarchical learning scopes

10:00 Comparing procedural learning algorithms in LangMem

13:30 Behind the scenes: memory integration pipeline

16:00 Real-world impact & procedural strategy success metrics

18:30 Challenges in deploying memory-integrated RAG systems

20:00 Practical engineering tips & closing thoughts

Resources:

- "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition

- Memriq AI: https://memriq.ai

Unlock the power of procedural memory to transform your Retrieval-Augmented Generation (RAG) agents into autonomous learners. In this episode, we explore how LangMem leverages hierarchical learning scopes to enable AI agents that continuously adapt and improve from their interactions — cutting down manual tuning and boosting real-world performance.

In this episode:

- Why procedural memory is a game changer for RAG systems and the challenges it addresses

- How LangMem integrates with LangChain and OpenAI GPT-4.1-mini to implement procedural memory

- The architecture patterns behind hierarchical namespaces and momentum-based feedback loops

- Trade-offs between traditional RAG and LangMem’s procedural memory approach

- Real-world applications across finance, healthcare, education, and customer service

- Practical engineering tips, monitoring best practices, and open problems in procedural memory

Key tools & technologies mentioned:

- LangMem

- LangChain

- Pydantic

- OpenAI GPT-4.1-mini

Timestamps:

0:00 - Introduction & overview

2:30 - Why procedural memory matters now

5:15 - Core concepts & hierarchical learning scopes

8:45 - LangMem architecture & domain interface

12:00 - Trade-offs: Traditional RAG vs LangMem

14:30 - Real-world use cases & impact

17:00 - Engineering best practices & pitfalls

19:30 - Open challenges & future outlook

Resources:

- "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition

- Memriq AI: https://memriq.ai

Unlock how Retrieval-Augmented Generation (RAG) enables AI agents to remember, learn, and personalize over time. In this episode, we explore Chapter 17 of Keith Bourne’s "Unlocking Data with Generative AI and RAG," focusing on implementing agentic memory with the CoALA framework. From episodic and semantic memory distinctions to real-world engineering trade-offs, this discussion is packed with practical insights for AI/ML engineers and infrastructure experts.

In this episode:

- Understand the difference between episodic and semantic memory and their roles in AI agents

- Explore how vector databases like ChromaDB power fast, scalable memory retrieval

- Dive into the architecture and code walkthrough using CoALA, LangChain, LangGraph, and OpenAI APIs

- Discuss engineering challenges including validation, latency, and system complexity

- Hear from author Keith Bourne on the foundational importance of agentic memory

- Review real-world applications and open problems shaping the future of memory-augmented AI

Key tools and technologies mentioned:

- CoALA framework

- LangChain & LangGraph

- ChromaDB vector database

- OpenAI API (embeddings and LLMs)

- python-dotenv

- Pydantic models

Timestamps:

0:00 - Introduction & Episode Overview

2:30 - The Concept of Agentic Memory: Episodic vs Semantic

6:00 - Vector Databases and Retrieval-Augmented Generation (RAG)

9:30 - Coding Agentic Memory: Frameworks and Workflow

13:00 - Engineering Trade-offs and Validation Challenges

16:00 - Real-World Applications and Use Cases

18:30 - Open Problems and Future Directions

20:00 - Closing Thoughts and Resources

Resources:

- "Unlocking Data with Generative AI and RAG" by Keith Bourne - Search for 'Keith Bourne' on Amazon and grab the 2nd edition

- Visit Memriq AI at https://Memriq.ai for more AI engineering deep dives and resources