Distributed System Topology

System Topology & Service Boundaries

MetroMind separates concerns aggressively. HTTP requests, background processing, semantic embedding, and vector storage exist in isolated domains to prevent long-running tasks from degrading the user experience.

Microservices Ingestion & Query Pipeline

Decoupled event flow from client upload to Milvus vector search

Asynchronous Flow
SERVICE CATALOG

Isolated Operational Services

SVC_01 Go

API Gateway

Acts as the single entry point for the frontend. It strictly handles HTTP routing, payload validation, and interacts directly with the Auth Service to validate JWTs before accepting any documents for upload or queries for search. It immediately acknowledges uploads, publishing an event to RabbitMQ, ensuring a non-blocking user experience.

SVC_02 RabbitMQ

Event Bus

Provides asynchronous decoupling. If 500 documents are uploaded simultaneously, the Gateway does not wait for OCR. RabbitMQ queues the tasks. Worker nodes consume these tasks at their own processing capacity. This ensures the system absorbs traffic spikes without memory exhaustion.

SVC_03 Python / Milvus

Intel Core & Vector Index

Python worker nodes execute OCR on binary blobs. Extracted text is normalized, chunked into overlapping windows, and passed to a local embedding model. The resulting dense vectors are indexed into a Milvus Vector Database, appended with metadata (Department ID, Access Level) for hybrid search capabilities.

SVC_04 Postgres

Audit & RBAC Control

Search queries are intercepted here. A user query is vectorized and sent to Milvus, but a mandatory pre-filter is applied using the user Department ID from their validated JWT. Every query, and the resulting documents accessed, are logged to a PostgreSQL audit table for compliance.

VERIFIED EXECUTION EVIDENCE

Pipeline Execution & Latency Trace

Live event trace and CLI execution evidence illustrating asynchronous RabbitMQ OCR ingestion, semantic chunking, and sub-40ms Milvus vector indexing.

metromind-ingest-daemon
ASYNC TRACE
$ metromind ingest --file contract_v3.pdf[+] Ingestion Gateway: Received payload contract_v3.pdf (4.2MB)[+] Auth Service: Validated JWT signature (Role: Ops_Lead)[+] RabbitMQ: Published to exchange `document.process` [RoutingKey: ocr.start][+] Python OCR Worker: Extracted 12,408 words (Confidence: 98.4%)[+] Embed Service: Generated 48 vector chunks via all-MiniLM-L6-v2[+] Vector DB: Indexed in Milvus successfully (latency: 38ms)[AUDIT] Log Service: Ingestion complete. TraceID: 9f8a-4b2c