Today, December 11, 2024, I had the honor of participating in a panel at the AI Summit New York, discussing LLMs Application Solution Lifecycle: Development, validation and implementation

The panel addressed the complexities involved in the development, validation, and deployment of LLM and RAG applications. One of the key topics was the importance of structuring data effectively before indexing, focusing on encoding, chunking, and embedding. We explored how aligning prompts with embedded document structures plays a critical role in enhancing model performance and ensuring more accurate and efficient retrieval.

Regarding model selection, accessibility and quality were highlighted as crucial factors. We emphasized the need to look beyond standard benchmarks by considering the full spectrum of factors, including ease of procurement, licensing, results from custom evaluations, and insights shared by experts across social media platforms. A balanced cost strategy was also discussed, advocating for the use of pay-per-use models initially, with a transition to specialized but cheaper models as use cases scale, ensuring sustained high performance without unnecessary overhead.

A good portion of the discussion focused on fine-tuning and prompt engineering. We highlighted the value of prompt engineering as a preliminary step before pursuing fine-tuning, continuous iteration on prompts and the establishment of a golden evaluation dataset. This dataset serves as a benchmark to evaluate both new models and alternated prompts. Evaluation should closely mirror the real-world to ensure that assessment was meaningful. In the context of fine-tuning, we highlighted the importance of avoiding overfitting and catastrophic forgetting, to retain essential knowledge and generalize effectively across applications.

We also touched on monitoring of AI applications, translating ML Ops to GenAI Ops.

The human element in AI successful development remained a consistent thread throughout the conversation, with a strong focus on collaboration between technical teams, subject matter experts, and legal stakeholders to mitigate risk. Ethical considerations were also addressed, reinforcing the importance of respecting copyrights, licenses, and model authenticity. Additionally, we emphasized the need for user education to ensure responsible and informed use of AI technologies.