Engineering Journal

Technical logs from production-minded AI systems.

Backend architecture, retrieval systems, validation layers, deployment tradeoffs, and the engineering decisions behind buyer-facing AI demos.

Archive

Engineering notes, not content filler.

Scope

Posts focus on service boundaries, data flow, model behavior, cost control, and production constraints.

Featured Engineering Log

Workflow SystemsEngineering Log April 08, 2026

Supervisor Agents: Enforcing Deterministic State Across Multi-Agent Workflows

Multi-agent workflows fail when model calls mutate shared state directly. A supervisor layer keeps state transitions, tool access, retries, and audit logs deterministic enough to debug.

Read Log

Archive

Data PipelinesEngineering LogFebruary 26, 2026

The 10M Token Trap: Why Infinite Context Windows Do Not Replace Data Engineering

Large context windows do not remove the need for source selection, freshness checks, permission boundaries, token budgets, and audit logs. They make weak data engineering harder to see.

Read Log
Data PipelinesEngineering LogJanuary 22, 2026

Highground: Building a Backend Pipeline for Market Intelligence

Highground turns market intelligence into an async backend pipeline: URL ingestion, scraping, DOM cleanup, Redis caching, model routing, WebSocket progress, streamed strategic insight, route-specific limits, and export boundaries.

Read Log
Workflow SystemsEngineering LogDecember 09, 2025

EdTech Infrastructure: Moving AI Lesson Generation Out of the Chat Box

Arcadia moves lesson generation out of a blank chat prompt and into a controlled LMS-style workflow with typed teacher inputs, Pydantic validation, task-specific model routing, streamed output, buffering, output cleanup, and LMS-ready data models.

Read Log
Retrieval SystemsEngineering LogNovember 12, 2025

Grounded Legal AI: Building a Research Backend Around Live Case Law

Blackletter keeps legal synthesis grounded by querying live CourtListener records first, cleaning and trimming opinion text, then sending bounded source context into the model with citation mapping and cost controls.

Read Log
SecurityEngineering LogOctober 28, 2025

Risk & Compliance AI: Turning Contract Review Into Structured Audit Data

The verification engine turns contract review into structured audit data first, then builds the memo from normalized findings. The model extracts risk fields, while the backend owns parsing, score normalization, report structure, budget controls, and demo-safe input limits.

Read Log
Streaming SystemsEngineering LogSeptember 25, 2025

Energy Intelligence: Turning Live Telemetry Into Server-Side Aggregates

Sentinel treats smart home telemetry as a backend time-series problem first. The database stores raw readings, the server computes verified aggregates, and the AI assistant explains bounded summaries instead of inventing energy math.

Read Log
Retrieval SystemsEngineering LogSeptember 9, 2025

Medical Research Synthesis: Keeping PubMed Retrieval Separate From LLM Reasoning

MedIntel keeps PubMed retrieval, article normalization, bounded synthesis, and frontend evidence display as separate layers. The model writes from retrieved source objects instead of pretending to know the medical literature from memory.

Read Log
Workflow SystemsEngineering LogAugust 21, 2025

Enterprise Sales AI: Separating Deal Strategy from Proposal Drafting

The proposal engine separates internal deal strategy from client-facing drafting. Messy sales notes become a structured strategy object first, then a streamed proposal document, with cleanup, validation, post-processing, and an internal logic view.

Read Log
Workflow SystemsEngineering LogJuly 24, 2025

MSME Brand Infrastructure: Structuring Positioning Before Copywriting

The brand workflow separates positioning logic from copy generation. MSME inputs become a structured strategy object first, then a buyer-facing memo, with streaming output, cached recovery, and inspectable strategic logic.

Read Log
Workflow SystemsEngineering LogMay 22, 2025

Agentic Inventory: Separating Stock Records From AI Decisions

Stockpile keeps inventory records inside a deterministic FastAPI system of record while routing AI reasoning through a guarded Next.js gateway. The model can analyze and propose, but stock mutations still pass through typed backend rules.

Read Log
Workflow SystemsEngineering LogMarch 20, 2025

Financial Intelligence: Keeping AI Away From the Ledger

Clearworth keeps the financial ledger under deterministic backend control while using AI for language, routing, and structured action proposals. The model can explain and propose, but money records still move through typed schemas, guarded mutations, and normal database rules.

Read Log
Streaming SystemsEngineering LogAugust 12, 2024

Real-Time Multimodal Pipelines for Audio and Video Streams

A live accessibility prototype started lagging after two minutes because the backend treated video like repeated static uploads. The fix was not a bigger model. It was buffering, keyframe selection, stale-work dropping, VAD, and bounded inference.

Read Log
AI SystemsEngineering LogMay 20, 2024

Small Language Models at the Edge

Three narrow healthcare classification paths moved off GPT-4 because the task was bounded, repetitive, and sensitive. The better architecture was local small-model inference inside the client VPC with validation, rules, audit metadata, and explicit fallbacks.

Read Log
Workflow SystemsEngineering LogFebruary 15, 2024

Orchestrating Multi-Agent Workflows Without Letting Them Run the System

A multi-agent research workflow burned through paid compute because the agents owned the loop. The fix was a controller-owned state machine with typed outputs, bounded retries, cost tracking, and audit logs.

Read Log
AI SystemsEngineering LogDecember 15, 2023

Designing a Multimodal Routing Layer Before Vision Access

A JPEG broke the old text-only chat assumption before any vision model was involved. The fix was a multimodal request envelope, file validation, route classification, and provider-agnostic capability routing.

Read Log
Retrieval SystemsEngineering LogOctober 12, 2023

The Embedding Trap: Why Default Models Need Domain Evals

A legal retrieval pipeline looked healthy until vector search returned the wrong kind of contract. The fix was not prompt tuning. It was a domain-specific retrieval eval, embedding versioning, and a safer migration path.

Read Log
InfrastructureEngineering LogJuly 28, 2023

Queueing and Quantized Llama 2 on Local Hardware

A raw Llama 2 70B serving attempt failed before the first token. The useful version was a quantized local model behind a thin API wrapper, single-worker queue, job records, and validation runs.

Read Log
Retrieval SystemsEngineering LogMay 12, 2023

Vector Storage Costs in Document Retrieval Systems

A legal retrieval prototype worked quickly with Pinecone, but the cost and metadata shape did not fit the workload. Moving the storage boundary to Weaviate simplified retrieval, source traceability, debugging, and cost control.

Read Log
Retrieval SystemsEngineering LogMarch 20, 2023

Building Retrieval-Augmented Generation Pipelines from Scratch

A contract retrieval pipeline produced a clean-looking answer from a broken half-clause. The fix was better ingestion: PDF cleanup, boundary-aware chunking, overlap, source metadata, and structured prompt assembly.

Read Log
AI SystemsEngineering LogNovember 30, 2022

The ChatGPT Shift: Conversational State as a Backend Problem

A chat app hit the context limit because the frontend kept sending the full conversation back to the model. The fix was treating conversation history as backend-owned structured state with sessions, turns, summaries, archives, and bounded prompt assembly.

Read Log
InfrastructureEngineering LogAugust 25, 2022

Queue-Isolated Stable Diffusion on Local GPU Hardware

Stable Diffusion could run locally, but serving it behind an API broke when normal web concurrency reached the GPU. The fix was separating request intake from GPU execution with FastAPI, Redis, Celery, object storage, and single-worker queue isolation.

Read Log
AI SystemsEngineering LogJune 14, 2022

Server-Side Session Memory for LLM Support Interfaces

An LLM support interface forgot user context because the model API was stateless. The fix was server-side session memory with Redis, durable PostgreSQL audit history, backend-owned prompt assembly, and clear fallback behavior.

Read Log
InfrastructureEngineering LogApril 18, 2022

Async Job Architecture for Image Generation APIs

An image generation request exceeded the gateway timeout before the result was ready. The fix was moving generation behind an async job boundary with a queue, worker, job table, object storage, and WebSocket completion events.

Read Log
InfrastructureEngineering LogDecember 02, 2021

Streaming Architecture for Sensitive Text Generation Workloads

A text generation endpoint hit a gateway timeout while the instance kept burning compute. The fix was a Server-Sent Events lifecycle with validation, controlled streaming, disconnect cancellation, frontend buffering, and safe operational logging.

Read Log
AI SystemsEngineering LogNovember 18, 2021

Defending Backend Systems Against Probabilistic Model Outputs

A legal extraction prototype broke when the model returned helpful prose instead of JSON. The fix was a defensive backend wrapper with raw response capture, parsing cleanup, strict schemas, source evidence checks, capped retries, a dead letter queue, and human review.

Read Log
InfrastructureEngineering LogSeptember 08, 2021

CI/CD for APIs Handling Sensitive Data

A sensitive-data API needed fewer deployment surprises. The fix was a reproducible CI/CD path with locked dependencies, validation tests, Docker image boot checks, mock sensitive payloads, and release gates before staging.

Read Log
Data PipelinesEngineering LogFebruary 24, 2021

Using a Model to Audit Label Quality at Scale

A document classifier stalled at 0.74 F1 because the training labels were wrong. The fix was a separate label-audit workflow with out-of-fold predictions, confidence disagreement filters, a review queue, correction records, and reproducible cleaned exports.

Read Log
Data PipelinesEngineering LogDecember 10, 2020

Centralizing Feature Logic for Real-Time Inference

A recommendation model looked strong offline and collapsed in production because training and serving computed the same feature differently. The fix was centralizing feature logic, serving versioned features through Redis, storing offline history in PostgreSQL, and enforcing freshness checks before inference.

Read Log
InfrastructureEngineering LogSeptember 28, 2020

Dynamic Batching for Heavy PyTorch Workloads

A BERT classification API had a GPU but still timed out because it fed the model one request at a time. The fix was moving from a custom Python batching queue to TorchServe dynamic batching with per-item validation, request mapping, and serving metrics.

Read Log
AI SystemsEngineering LogJune 28, 2020

The Generative Trap: Defending Against Probabilistic Outputs

A GPT-2 extraction prototype returned polite prose instead of JSON and crashed the database write path. The fix was a defensive model wrapper with raw output capture, parsing cleanup, schema validation, evidence checks, capped retries, a dead letter queue, and human review.

Read Log
Data PipelinesEngineering LogMay 14, 2020

Defensive Ingestion for COVID-19 Data Streams

A real-time COVID-19 forecasting pipeline stalled because external hospital APIs sent messy payloads. The fix was defensive ingestion with Celery workers, strict Pydantic validation, PostgreSQL writes for clean records, a Redis dead letter queue, offline recovery, and source-level freshness monitoring.

Read Log
InfrastructureEngineering LogMarch 11, 2020

Cold Starts and Kubernetes: Managing Latency in Burst Traffic

A model-backed API dropped requests during a burst even while Kubernetes was scaling. The fix was making model readiness explicit, blocking traffic from cold pods, adding ingress rate limits, and treating autoscaling as delayed capacity instead of instant rescue.

Read Log
Data PipelinesEngineering LogNovember 05, 2019

Handling Feature Drift in Production Fraud Pipelines

A checkout fraud model stayed online while quietly blocking good customers. The fix was a separate drift monitoring loop, model/version traceability, decision logs, staged rollout, and rollback paths around the live request flow.

Read Log
InfrastructureEngineering LogAugust 30, 2019

Surviving Spot Interruptions: Decoupling PyTorch Inference on AWS

A PyTorch inference API was tied directly to an interruptible GPU instance. The fix was splitting stable request handling from Spot-backed compute with SQS, job IDs, retryable workers, result storage, and a status endpoint.

Read Log
InfrastructureEngineering LogApril 22, 2019

Moving Past Flask: FastAPI, Async Requests, and Backend Workload Boundaries

A Flask model endpoint backed up under load because database I/O and PyTorch inference were mixed inside one synchronous request path. The fix was moving to FastAPI with async coordination, explicit Pydantic contracts, and a clear boundary around synchronous inference.

Read Log
SecurityEngineering LogFebruary 18, 2019

Securing the Data Behind the Dashboard: Flask, SQL, and Backend Boundaries

A dashboard looked like frontend work until the API started returning more data than the charts needed. The fix was making Flask the data boundary with scoped endpoints, aggregated responses, server-side filters, parameterized SQL, and no browser-side security assumptions.

Read Log
InfrastructureEngineering LogDecember 14, 2018

A Backend Mindset Beyond Jupyter: Serving XGBoost with Flask and Gunicorn

A fraud model worked in a notebook, but the real work started when it had to sit behind checkout. The useful backend pieces were validation, model startup loading, Gunicorn serving, traceable database records, score storage, and decision logs.

Read Log