Knowledge Builder Error > maximum context length is 8192 tokens

Hi @keytrackde

Currently, all three embedding models (text-embedding-ada-002, text-embedding-3-small, and text-embedding-3-large) have a maximum input size of 8191 tokens.

This means data exceeding this token limit cannot be embedded directly.

If you only have a few large posts, here are a couple of workarounds:

Split the Content: Break large posts into smaller sections and embed them manually using the Knowledge Builder text field. You can copy and paste the content in two or three parts.
Use a PDF Converter: Convert your large posts into PDFs using an online tool, then upload them with our plugin’s PDF upload feature. The plugin will embed the content page by page.

From our testing, we’ve found that text-embedding-3-small is faster and cheaper but less accurate compared to text-embedding-ada-002 and text-embedding-3-large. For better accuracy, we’ve migrated all our posts to text-embedding-ada-002.

Let me know if you have any questions or need further help.`

Considering The maximum input size of 8191 tokens is limited to 10% or less if the site language is not Latin, is there anyway to use AI to summarize context (keeping the same info, style of the writer, or number of paragraphs) before embedding?

I think a good prompt (maybe a customized one, by the user) should solve the problem completely.

Please add this suggestion to the next version. It is very needed up here.