My brother and I have been working on DataBridge: an open-source and multimodal database. After experimenting with various AI models, we realized that they were particularly bad at answering questions which required retrieving over images and other multimodal data.
That is, if I uploaded a 10-20 page PDF to ChatGPT, and ask it to get me a result from a particular diagram in the PDF, it would fail and hallucinate instead. I faced the same issue with Claude, but not with Gemini.
Turns out, the issue was with how these systems ingest documents. Seems like both Claude and GPT embed larger PDFs by parsing them into text, and then adding the entire thing to the context of the chat. While this works for text-heavy documents, it fails for queries/documents relating to diagrams, graphs, or infographics.
Something that can help solve this is directly embedding the document as a list of images, and performing retrieval over that - getting the closest images to the query, and feeding the LLM exactly those images. This helps reduce the amount of tokens an LLM consumes while also increasing the visual reasoning ability of the model.
We've implemented a one-line solution that does exactly this with DataBridge. You can check out the specifics in the attached blog, or get started with it through our quick start guide: https://databridge.mintlify.app/getting-started
My brother and I have been working on DataBridge: an open-source and multimodal database. After experimenting with various AI models, we realized that they were particularly bad at answering questions which required retrieving over images and other multimodal data.
That is, if I uploaded a 10-20 page PDF to ChatGPT, and ask it to get me a result from a particular diagram in the PDF, it would fail and hallucinate instead. I faced the same issue with Claude, but not with Gemini.
Turns out, the issue was with how these systems ingest documents. Seems like both Claude and GPT embed larger PDFs by parsing them into text, and then adding the entire thing to the context of the chat. While this works for text-heavy documents, it fails for queries/documents relating to diagrams, graphs, or infographics.
Something that can help solve this is directly embedding the document as a list of images, and performing retrieval over that - getting the closest images to the query, and feeding the LLM exactly those images. This helps reduce the amount of tokens an LLM consumes while also increasing the visual reasoning ability of the model.
We've implemented a one-line solution that does exactly this with DataBridge. You can check out the specifics in the attached blog, or get started with it through our quick start guide: https://databridge.mintlify.app/getting-started
Would love to hear your feedback!