For many RAG applications and AI agents, ingesting web content and converting to Markdown is the starting point for the unstructured data pipeline.

There are many approaches for HTML to Markdown conversion, and web crawling, but creating high-quality Markdown output can be challenging.

At Graphlit, we support web scraping (single page) and web crawling (via sitemap), and converting to Markdown output.

However, the raw Markdown output can often not be optimal for using as RAG context, because it contains the page header, or navigation buttons, or extraneous text that is repeated on every page.

Via Graphlit preparation workflows using the MODEL_DOCUMENT service type, we now support LLM-enhanced conversion from HTML to Markdown.

By taking a screenshot of the original web page, and providing the unfiltered Markdown conversion to an LLM, we prompt the LLM to clean, reformat, and optimize the Markdown output.

Getting Started

You can compare Graphlit for yourself with this Google Colab notebook.

In this example, we are scraping a page from Anthropic's documentation.

Anthropic Sonnet 3.7

Using Graphlit, and Anthropic Sonnet 3.7, we can provide the highest-quality web to Markdown conversion available today.

OpenAI GPT-4o

Graphlit supports any vision-enabled LLM, and here we can compare to using OpenAI GPT-4o.

Firecrawl

Here we compare to Firecrawl, one of the leading web crawling APIs, and you can see how they don't filter the Markdown output and the quality doesn't compare to Graphlit with Sonnet 3.7.

Jina Reader

Here we compare to the Jina Reader API, and you can see that Graphlit extracted Markdown which is not shown in the Jina output.

Cost will obviously be a factor in the solution you choose, and with Graphlit, you can bring your own LLM API keys to keep cost to a minimum.

SUMMARY

Please email any questions on this article or the Graphlit Platform to questions@graphlit.com.

For more information, you can read our Graphlit Documentation, visit our marketing site, or join our Discord community.