The Engine

A deep dive into the orchestration, execution, and synthesis core of Lens.

Execution Loop

The Lens engine operates on a non-blocking, multi-threaded event loop. When a query is received, the engine performs the following internal steps:

1. Resolution Phase

The engine resolves the request against the active lens.config.yaml. It identifies which collectors are enabled and ready (auth check).

2. Concurrent Dispatch

Collectors are dispatched into a thread pool. The engine manages individual collector timeouts and retries according to the global engine.timeout setting.

3. ResearchGraph Assembly

As collectors return data, the engine normalizes the results into a ResearchGraph—a directed acyclic graph (DAG) where nodes are research items and edges are citations or topical links.

Normalization Logic

Raw data from heterogeneous sources (JSON, XML, PDF) is converted into a standard internal schema. This is critical for downstream pipeline accuracy.

python

# Internal ResearchItem Schema
{
  "id": "uuid",
  "source": "arxiv",
  "title": "...",
  "content": "...",
  "metadata": {
    "authors": [],
    "citations": []
  },
  "provenance_score": 0.95
}

Synthesis Drivers

The engine supports multiple synthesis "drivers" for summarizing the ResearchGraph. Drivers can be purely algorithmic (e.g., TF-IDF clustering) or LLM-driven (e.g., GPT-4o, Claude 3.5).

Local Synthesis Run small models locally for zero-latency, private research.

Remote Synthesis Utilize powerful cloud LLMs for deep technical reasoning.

Overview