AgenticWorkflowSearchEngine-MCP

A web search function that minimizes reliance on scraping as much as possible. The purpose is to ensure transparency and reduce the load on site operations.

Using Playwright, we perform the following series of steps: input a word in the search box → search → visit the site → temporarily save in memory using the browser's standard HTML download function → remove unnecessary information and save to an MD file → convert MD to JSON.

This is an MCP server, but it can also be used as a CLI tool (command: aw-se).
A summarization feature using local LLM via Ollama is also available. In that case, set the argument to enable Ollama usage. (This will take more time.)
By passing not just the search word but also the search intent, the local LLM will summarize while omitting unnecessary information.

Additionally, for academic papers and similar content, we enable information retrieval using official APIs from arXiv and PubMed. In that case, switch the mode. There are two types: normal web search mode and academic mode.
Also, when the link destination is a PDF, we have exception handling that parses text from the PDF and converts it to MD.
If you want to dig deeper, you can enable the secondary search feature, which allows the local LLM to generate new search words and perform the same process again.

All search results are designed to be stored in a folder created with the search word as the name within the "artifacts/" folder, with everything contained within. Additionally, for reverse lookup purposes, we use a CSV file to leave pinned information stating "this information is located at line 〇 in the file." This eliminates the need to unnecessarily have the AI read the entire text, preventing context pollution in the AI.

※ This is packaged as an npm package. Downloading the source code itself is simple, but please research how to register it as an MCP server on your own. The source code itself has been verified to work. Also, when operating as an MCP server, it is by design that the ollama viewer operates and headless mode is turned off.