qSight

A worker-based system that finds financial news, loads JavaScript-heavy pages, extracts market insights with Google GenAI, and stores structured results.

View repository · Architecture overview

System architecture

flowchart LR
    D[Operator dashboard] --> S[aiohttp coordinator]
    S --> Q[Task queues]
    W[Playwright workers] -->|Get and submit tasks| S
    W --> N[News feeds and websites]
    W --> G[Google GenAI]
    S --> DB[(Turso or SQLite)]
    P[Optional AI processor] --> DB

Request and data flow

1. The coordinator creates task queues for RSS feeds, news websites, and AI processing.
2. Workers request jobs, open pages in Firefox, block heavy assets, and return links or article text.
3. Relevant articles are processed for timestamps, financial facts, summaries, tickers, and market impact.
4. Results and worker metrics return to the coordinator and are saved to Turso or local SQLite.

Engineering highlights and trade-offs

• Easy worker scaling: Workers pull tasks over HTTP, so more workers can be added without changing the coordinator.
• Failure recovery: Timed-out tasks are retried, and workers restart unhealthy browsers with increasing delays.
• Structured AI output: Pydantic schemas keep model responses consistent while several extraction steps run in parallel.
• Flexible storage: One data layer supports both cloud-hosted Turso and local SQLite. JSON fields stay flexible but are harder to query in SQL.
• Scaling limit: Queues and worker state live in one coordinator process. Checkpoint code exists, but restart recovery is not fully connected.