Artificial Intelligence (AI) has revolutionized numerous industries, enabling organizations to accomplish tasks and solve complex problems with unprecedented efficiency. In particular, large language models (LLMs) have emerged as powerful tools, demonstrating exceptional language-processing capabilities and fueling a surge in their adoption across a wide range of applications. From chatbots and language translation to content generation and data analysis, LLMs are being adopted by companies of all sizes and across all industries. As organizations eagerly embrace the potential of LLMs, the need to understand their behavior in production and use that understanding to improve development with them has become apparent. While the initial excitement surrounding LLMs often centers on accessing their remarkable capabilities with only a small up-front investment, it is crucial to acknowledge the significant problems that can arise after their initial implementation into a product. By introducing open-ended inputs in a product, organizations expose themselves to user behavior they’ve likely never seen before (and cannot possibly predict). LLMs are nondeterministic, meaning that the same inputs don’t always yield the same outputs, yet end users generally expect a degree of predictability in outputs. Organizations that lack good tools and data to understand systems in production may find themselves ill-prepared to tackle the challenges posed by a feature that uses LLMs