LLMs Have Ushered in a New Era of the Software Development Lifecycle (SDLC)

Jay Thakrar

⸱

Head of Product and Strategy

Aug 7, 2024

Introduction

Organizations across various sectors are proactively investing in the application of AI for both internal and external use cases. For example, by offering AI-related services to its clients, McKinsey & Company made over $16 billion in revenue in 2023 , while . Additionally, according to a recent generated over $2 billion in GenAI sales year-to-date in 2024 survey, 87% of organizations are “ Bain & Company . Moreover, the rapid growth and adoption of model development companies, such as (GPT), (Claude), already developing, piloting, or have deployed generative AI in some capacity (Llama), (Gemini), makes it abundantly evident that organizations are eager to integrate LLMs’ capabilities into their use cases.

Our Thesis

These trends bolster our conviction in our thesis → LLMs are market transformative technologies that are here to stay . LLMs will play a meaningful role in the way software is developed in the future . Given the non-deterministic nature of LLMs, a new set of tooling and capabilities are required to build software that is reliable, scalable, and trusted — which are table stakes for any organization.

However, despite the significant financial investments and media hype surrounding AI adoption, developers still lack confidence in deploying LLM-powered applications in production for both internal and external use cases. This is largely due to concerns regarding trust, unpredictability, and quality of insights. To elaborate, product builders of LLM-powered applications have two major challenges:

Trust : How can developers trust an LLM to perform within their application in an accurate, high quality, secure, and reliable manner?
Visibility : How can developers capture key insights into how an LLM or the orchestration framework is performing within their application (i.e. such as text generations, usage, costs, latency), and troubleshoot any incidents immediately?

In order to solve these challenges, the software development lifecycle (SDLC) must be modernized .

The SDLC Waterfall: How Is It Changing?

Let’s explore how the SDLC is evolving. The traditional, non-LLM SDLC generally includes the following stages:

Additionally, with the shift in software development needs (i.e. more complex projects and more dynamic customer requirements) coupled with technological advancements (i.e. better version control systems, automated testing tools, and continuous integration tools), the non-LLM SDLC workflow has become more iterative and agile in nature, however, the “stages” within the workflow largely remain the same.

The Modernized SDLC Waterfall

However, when developer build applications using AI, they need to leverage purpose-built tools to manage the non-deterministic nature of LLMs. This challenge is further exacerbated by the fact that developers are leveraging on average 3–4 models simultaneously, with varying capabilities, for various use cases within their organizations.

As a result, developers need to observe and evaluate how LLMs perform within their applications, and can only do so by integrating a new workflow into the existing SDLC lifecycle that many developers are familiar with.

The modernized SDLC using LLMs includes the following workflow:

Step 1: Compare various LLMs via model playgrounds

Developers are experimenting with multiple models and respective model versions simultaneously to determine which models make most sense for their use cases.
As a result, model playgrounds (which is a tool to quickly compare various LLMs’ performance, costs, and accuracy) is instrumental to leverage early on in the planning, analyzing, and design stages of determining which LLM(s) to further invest development time and efforts in when building your application.

Step 2: Prompt Management

Once developers determine which LLM(s) they intend to leverage for their use case, they need to provide instructions as to how the developer expects the LLM(s) to respond within their application. This step is commonly referred to prompt management.
Keep in mind that all models are not created equal, and as a result, different models (i.e. ‘s Gemini vs. ‘s Claude) and even different model versions of the same model (i.e. ‘s GPT-4o vs. GPT4o mini) can produce varying responses to the same prompts. As a result, efficient prompt management, prompt-tuning, and prompt-versioning are imperative in integrating LLMs within your application.

Step 3: Collect Traces and Logs (i.e. Data) for Analysis

Once you have determined which LLM(s) to integrate into your application, and have created and tested prompts for each LLM, you can ship your application to testing or production environments, and immediately start capturing traces and logs (i.e. data) for analysis. Note: Traces are requests / operations made through a system.

Step 4: Conduct Evaluations and Manage Datasets

The goal of conducting evaluations is to determine if the LLM(s) is responding in an accurate, high quality, secure, and reliable manner.
Given that the traces capture high cardinality data, including the inputs and the responses of each request / operation to the LLM(s), product builders can conduct annotations (also commonly referred to as manual evaluations using human-in-the-loop [HITL]) or automatic evaluations by selecting a more powerful LLM to serve as a judge.
Once evaluations are conducted, product builders can leverage positive outputs (i.e. positive responses from an LLM) to create a golden dataset for future prompt management when new models and / or model versions are released.
Keep in mind that conducting evaluations is very much an open-ended problem today that most organizations struggle with. Moreover, despite the emergence of highly capable open-source LLMs (i.e. ‘s Llama 3.1 and ‘s Claude 3.5 Sonnet) in addition to closed-source models (i.e. ‘s GPT 4o), organizations are struggling to explore switching over to other LLMs primarily due to the inability to measure accuracy in an objective, efficient and trusted way . Hence, tools like are valuable when comparing and evaluating various LLMs’ performance within your applications.

Langtrace is purpose built to help LLM application developers seamlessly integrate the modern SDLC into their existing workflows. Additionally, integrating into your program is lightweight — it only requires . By implementing the modernized SDLC workflow end-to-end across an LLM-powered application development lifecycle, product builders are able to establish a two lines of code tight feedback loop with supporting data, enabling them to build scalable, reliable, trusted applications . In summary, tools like are going to be paramount in both the pre-production and post-production of LLM-powered applications.

Langtrace Platform Demo

Watch the walkthrough video of the Langtrace platform to see the latest capabilities of Langtrace’s managed-service.

How to Get Started with Langtrace

is an open-source and OpenTelemetry-based observability and evaluations SDK and platform designed to improve your LLM applications. Developers can use to compare LLMs, iterate on prompts, set up automatic tracing and logging of their LLM stack, curate datasets, and conduct evaluations. The onboarding process is simple and non-intrusive, it only requires to get started. Visit the two lines of code to get started today. Langtrace website

If you have any questions or would love to chat all things related to LLM application development, observability, and evaluations, please reach out. Thanks for reading!