AgenticWorkflowSearchEngine-MCP

A web search function that minimizes reliance on scraping as much as possible. It aims to ensure transparency and reduce the load on site operations.

Using Playwright, it performs a series of processes: input a keyword in the search box → search → visit the site → temporarily save to memory using the browser's standard HTML download function → remove unnecessary information and save to an MD file → convert MD to JSON.

It functions as an MCP server, but can also be used as a CLI tool (command: aw-se).

A summarization feature using local LLM via Ollama is also available. In that case, set the argument to enable Ollama usage. (This will take more time.)

By passing not just the search keyword but also the search intent, the local LLM will summarize by omitting unnecessary information.

Additionally, for academic papers and similar content, information retrieval is enabled through official APIs from arXiv and PubMed. In such cases, switch the mode. There are two types: normal web search mode and academic mode.

If the linked destination is a PDF, exception handling is implemented to parse text from the PDF and convert it to MD.

If you want to dig deeper, enabling the secondary search feature allows the local LLM to generate keywords and perform the same process again.

All search results are organized in the "artifacts/" folder, with a folder created for each search keyword containing all related files. Additionally, for reverse lookup purposes, a CSV file is created to pin down information indicating "this information is located at line 〇 in the file." This eliminates the need to unnecessarily have the AI read the entire text, preventing context pollution for the AI.

※ It is packaged as an npm package. Downloading the source code itself is straightforward, but please research how to register it as an MCP server on your own. The source code itself has been verified to work. Additionally, when operating as an MCP server, it is by design that the ollama viewer operates and headless mode is turned off.