AppsZone re-architected delivery around AI-native workflows in 2023. This article explains how we run discovery, RAG implementation, and production ship loops in parallel — without skipping threat models, eval harnesses, or stakeholder demos.
Why the old waterfall breaks
Traditional agency flows — discovery deck, then design, then build — collapse when LLM features need rapid iteration on retrieval quality. A three-month spec doc is stale before the first embedding index is warm. Stakeholders ask for a chatbot on Monday; by Friday the corpus has changed, the embedding model has a new version, and the “final” requirements doc references APIs that were deprecated two sprints ago.
We run a four-act framework (Discover → Architect → Ship → Scale) but overlap acts when AI is in scope. Discovery workshops produce user stories AND a retrieval baseline in the same timebox. Design and engineering pair on chunk boundaries while product still validates problem framing.
Act 1 — Discover with retrieval in the room
We start every AI engagement with three artifacts:
- Problem map — Who acts on the output? What decision does bad retrieval break?
- Corpus audit— What documents, APIs, or databases feed the model? What's the PII risk?
- Golden questions — 20–40 real queries stakeholders expect answered correctly.
These feed directly into chunking experiments. No wireframes until golden-set accuracy hits an agreed floor — typically 85%+ on held-out questions. That number is a contract with the business: below the floor, we do not ship UI — we ship better retrieval.
Act 2 — Architect the RAG pipeline
Our default stack pairs structured ingestion with hybrid search:
- PDFs and HTML → normalized markdown chunks with metadata (source, date, tenant).
- Embeddings in pgvector or Pinecone depending on scale and ops maturity.
- Re-ranker pass for top-k before LLM synthesis.
- Output schema enforced with Zod — no free-form JSON in production paths.
Threat modeling covers prompt injection via retrieved docs, tenant isolation in multi-tenant CRM builds, and audit logging for regulated clients. We treat retrieved content as untrusted input — same discipline as user-supplied form data.
Act 3 — Ship loops that stay honest
Sprints are one week. Every Friday: live demo on staging with real corpus slices; eval report with precision@k, faithfulness, cost per query; security checklist sign-off for new data sources.
We deploy behind feature flags. LLM routes get circuit breakers — if latency or error rate spikes, fallback to cached summaries or a human queue. Demos include failure cases on purpose: empty corpus, PII detected, model timeout. Hiding failures in rehearsals creates production surprises.
Act 4 — Scale without silent drift
Production RAG rots when source documents change without re-indexing. We wire webhook-driven re-ingestion on CMS publish events, nightly eval against the golden set with Slack alerts on regression, and cost dashboards per tenant for multi-tenant SaaS.
Multi-tenant systems get per-tenant golden subsets so one client's corpus change does not mask another's regression. Runbooks document rollback: feature flag off, previous index alias, cached response mode. On-call engineers should not need to read LangChain docs at 3 a.m.
What we learned the hard way
“The most expensive mistake is polishing a chat UI before your retrieval layer is measurable.”
Teams that skip eval harnesses rebuild twice — once for demo day, once for production. Bake observability into week one. We have seen six-figure rewrites when a chat shell shipped before chunk strategy was validated against real PDFs with tables, footnotes, and scanned appendices.
When not to use RAG
Not every feature needs retrieval. Classification, extraction, and structured transforms often work better with fine-tuned small models or rule-augmented prompts. We scope AI automation by ROI — agents where they compound, not where they impress in a pitch deck.
The goal is not maximal AI surface area. It is reliable outcomes at a cost the business will still pay for after the launch party. Saying no to RAG early often saves the relationship long term.
