Lecture-11: RAG for Beginners: Text Chunking Explained | Split Documents for Better AI Retrieval

Screenshot from the tutorial

Chunking for AI Retrieval: A Step-by-Step Guide to Document Splitting

In the realm of Natural Language Processing (NLP) and AI retrieval systems, chunking is a fundamental technique that enhances the efficiency of information retrieval. This blog post serves as a comprehensive guide to understanding document chunking, its significance, and how to implement it using JavaScript. Let's dive into the world of document splitting and learn how to leverage it for better AI performance.

What is Chunking?

Chunking is the process of dividing large documents into smaller, manageable pieces (or chunks). This technique is crucial when working with lengthy texts such as articles, movie plots, support conversations, and more. By breaking down these documents, we can ensure more accurate information retrieval while maintaining context.

Why is Chunking Necessary?

There are two primary reasons for implementing chunking in AI retrieval systems:

1. Embedding Model Limitations: Most embedding models have a limit on the amount of text they can process at once. If a long document is embedded as a single vector, it may result in a loss of semantic accuracy. When a user queries the system, it becomes challenging to pinpoint the specific part of the document that answers their question.

2. Context Preservation: When splitting text, it’s essential to maintain a small overlap between chunks. This overlap prevents the loss of context, especially if a sentence is cut off at the boundary of two chunks.

Implementing Chunking in JavaScript

To illustrate the chunking process, we will create a reusable utility in JavaScript. We'll develop a function that takes in a long piece of text, along with parameters for chunk size and overlap.

Step 1: Creating the Chunking Utility

First, create a file named chunking.js and implement the following method called splitText:

// chunking.js

function splitText(text, chunkSize = 100, overlap = 20) {
    // Check if the text is short enough to return as a single chunk
    if (text.length <= chunkSize) {
        return [text];
    }

    const chunks = [];
    let start = 0;

    while (start < text.length) {
        // Determine the end of the chunk
        let end = start + chunkSize;

        // Adjust the end to avoid cutting mid-sentence
        if (end < text.length) {
            const lastPeriod = text.lastIndexOf('.', end);
            const lastSpace = text.lastIndexOf(' ', end);
            end = lastPeriod > -1 ? lastPeriod : lastSpace;
        }

        // Extract the chunk
        const chunk = text.substring(start, end);
        chunks.push(chunk);

        // Move the start position forward by chunkSize - overlap
        start = end - overlap;
    }

    return chunks;
}

export { splitText };

Explanation of the Code

• The splitText function takes three parameters: text, chunkSize, and overlap.
• If the input text is shorter than or equal to the specified chunk size, the function returns the text as a single chunk.
• A while loop iterates through the text to create chunks. Instead of cutting at a fixed position, it looks for the last period or space near the end of the chunk to avoid splitting sentences.
• After extracting a chunk, the starting point is updated, considering the overlap to ensure some text is shared between consecutive chunks.
• Finally, the function returns an array of clean chunks.

Step 2: Demonstrating the Chunking Process

Next, create another file named chunkingDemo.js to demonstrate how to use the splitText function:

// chunkingDemo.js
import { splitText } from './chunking.js';

const longText = "Inception is a 2010 science fiction film about a professional thief who steals information by infiltrating the subconscious of his targets. The story follows Dom Cobb, who is offered a chance to have his criminal history erased as payment for the implantation of another person's idea into a target's mind. This process, known as inception, involves navigating multiple layers of dreams.";

const chunkSize = 100; // Define chunk size
const overlap = 20; // Define overlap size

const chunks = splitText(longText, chunkSize, overlap);
console.log("Generated Chunks:", chunks);

Running the Demo

When you run the chunkingDemo.js file, it will output the created chunks, demonstrating how the text has been split while maintaining overlaps:

Generated Chunks: [
  "Inception is a 2010 science fiction film about a professional thief who steals information by ",
  "a professional thief who steals information by infiltrating the subconscious of his targets. The story follows Dom Cobb, who is offered a chance to have his criminal history erased as payment for the ",
  "payment for the implantation of another person's idea into a target's mind. This process, known as inception, involves navigating multiple layers of dreams."
]

Conclusion

Chunking is an integral part of preparing data for AI retrieval systems. By implementing a chunking utility, we can effectively enhance the accuracy and relevance of information retrieval. In the next lesson, we will explore how to combine chunking with embedding and store the results in a database like Supabase for a complete retrieval-augmented generation (RAG) system.

If you found this tutorial helpful, don't forget to like and subscribe for more insights into AI and NLP techniques!

Another view from the tutorial

---

Connect with SkillBakery Studios

Explore more tutorials, tools, and resources:

• skillbakery.com
• Facebook
• X (Twitter)
• YouTube Channel

Posted by SkillBakery Studios