From the course: Level up LLM applications development with LangChain and OpenAI

Unlock the full course today

Join today to access over 24,300 courses taught by industry experts.

Load and split documents into chunks

Load and split documents into chunks

From the course: Level up LLM applications development with LangChain and OpenAI

Start my 1-month free trial Buy for my team

Load and split documents into chunks

“

- [Instructor] Now let's go over how to create the Q&A chat bots. So we're going to start with the first step, which is to load and split the documents. So that's going to be during the indexing stage. So first we want to load the data by using a document loader. Next we use a text splitter to split the document into smaller chunks. So we're going to take care of these two first steps, which is to load the documents, then split the documents into smaller chunks. So let's go back to our projects and we're going to find here this function. And first we're going to use this class, TextLoader, which is provided by LangChain in order to load the data source. And so we want to use this text file as a data source, which is a list of frequently asked questions. So we're going to load these documents by using TextLoader. So let's start with that. So we're going to do that right here by creating a raw_text. And so we're going to load the documents by specifying here the destination. And because…

Contents