Code and notes from my recent talk: Exploring the Future of AI: Introduction to using LLMs using Python

Topics: Large context prompts with LLMs vs. RAGs using embeddings vector stores. How to avoid LLM hallucination. 3 code demos.

started an informal code demo and group conversation Meetup group (link) and today I gave a fifteen minute code demo followed by a conversation with the attendees. Here is a GitHub repo with the code examples: https://github.com/mark-watson/talk2_LLM_Python_intro

Here are my talk notes:

Exploring the Future of AI: Introduction to using LLMs using Python

Riff on ‘AI grounding’ and how LLMs help: LLMs, trained on vast amounts of text, excel at recognizing patterns and providing contextually relevant responses. They mimic grounded understanding by referencing large datasets that encompass a variety of real-world scenarios. For example, they can infer meanings from complex contexts by drawing on their training data. When LLMs are integrated with other modalities, such as vision or audio (e.g., vision-language models), the grounding improves. These models can associate text with images or sounds, making the connections more robust and closer to a human-like understanding of concepts.

Tradeoffs between using large context LLMs: where a large body of text is added to a query prompt, to the alternative approach of breaking multiple documents into many separate chunks of text, calculating an embedding vector for each chunk, and then storing the chunks and their associated embedding vectors ins a vector data store.

Long-Context LLMs: designed to support processing large blocks of text, often an entire book, within a single prompt. These models can accommodate extended sequences of text, enabling them to consider more context at once. This is particularly useful for tasks that require maintaining continuity over long narratives or documents. However, long-context LLMs have limitations, such as performance degradation when the context becomes too long, which can lead to reduced accuracy in generating or retrieving relevant information. These models are also computationally expensive, as handling extensive sequences demands significant resources.

On the other hand, vector stores (or vector databases) work by converting text or other unstructured data into high-dimensional vectors using embeddings. These vectors are stored and can be retrieved based on their similarity to a query vector, allowing for efficient semantic search across vast datasets. This approach provides a form of “long-term memory” for LLMs, enabling them to access and retrieve relevant information from large collections of documents without needing to process the entire context at once. Vector stores are particularly useful in retrieval-augmented generation (RAG) systems, where they help the model to find and focus on the most relevant information, improving both efficiency and accuracy  .

In essence, while long-context LLMs attempt to handle extensive information within the model’s processing window, vector stores offer an external memory solution that complements LLMs by efficiently managing and retrieving relevant information from larger datasets.

What about LLM hallucinations?

Long context windows and retrieval-augmented generation (RAG) data stores significantly reduce LLM hallucinations by improving the model's access to relevant and accurate information during the generation process.

1. Long Context Windows: When LLMs are equipped with long context windows, they can process and retain more information within a single session. This allows the model to maintain continuity and consistency over extended text, reducing the chances of fabricating information that doesn't align with the given context or user query. By having access to more surrounding context, the model can generate more coherent and accurate responses that are anchored in the actual input data.

2. RAG Embedding Vector Data Stores: In a RAG setup, an LLM is paired with a vector store that holds a vast amount of pre-processed, structured information. When a query is posed, the model retrieves relevant documents or data snippets from this store, which then informs the generation process. This retrieval step grounds the model's output in factual data, effectively reducing the likelihood of hallucinations. Since the model can rely on precise and contextually relevant information, it is less prone to generating plausible-sounding but incorrect or nonsensical content.

Together, these approaches enhance the reliability of LLM outputs. Long context windows allow the model to consider more of the input in a single pass, while RAG ensures that the model has access to verified information, leading to fewer instances of hallucination and more trustworthy results.