System Design · Internal Platform
Self-Hosted AI Code Generation Platform
for a 10,000-Person Fintech Org
Replacing expensive third-party AI coding assistants with self-hosted LLMs in a regulated fintech environment — built for data sovereignty, cost control, and production reliability, and wired directly into the Jira workflow via an autonomous agent.
00
Executive Summary
Recall first — before you read on
- What third-party AI coding tools were being replaced, and why?
- What three priorities does the solution claim to optimize for?
Executive SummaryOVERVIEW▶
This document presents a complete system architecture for replacing expensive third-party AI coding assistants with self-hosted Large Language Models in a regulated fintech environment. The solution prioritizes data sovereignty, cost control, and production reliability while serving 10,000 developers across multiple teams.
What we learned
A regulated fintech can replace third-party AI coding tools with a self-hosted stack without giving up developer productivity — as long as sovereignty, cost, and reliability are designed in from day one, not bolted on later.
01
The Business Case: Why Self-Hosting Wins
Recall first — before you read on
- Roughly what does the current third-party API spend look like per year for 10,000 devs?
- What's the ballpark annual cost of the self-hosted alternative, and the resulting savings range?
The Cost Reality CheckFINANCE▶
Let me paint you a picture of what's actually happening in your organization right now:
cost-comparison.txtascii
Current State (Third-Party APIs): ┌─────────────────────────────────────────────────────────────┐ │ 10,000 Developers × ~50 API calls/day × $0.03/call │ │ = $15,000/day = $450,000/month = $5.4M/year │ │ │ │ But wait - this is conservative. In reality: │ │ • Heavy users make 200+ calls/day │ │ • Code generation requires longer context windows │ │ • Premium models cost $0.05-0.10 per call │ │ • Real cost: $8-12M/year │ └─────────────────────────────────────────────────────────────┘ Self-Hosted Alternative: ┌─────────────────────────────────────────────────────────────┐ │ Infrastructure (Reserved Instances, 3-year commitment): │ │ • 8× p4d.24xlarge (A100-40GB): ~$32/hour each │ │ • Total compute: ~$256/hour = $186,880/month │ │ • Storage, networking, management: ~$50,000/month │ │ • Engineering team (3 FTE): ~$50,000/month │ │ Total: ~$286,880/month = $3.4M/year │ │ │ │ Annual Savings: $4.6-8.6M (54-72% reduction) │ └─────────────────────────────────────────────────────────────┘
CTO'S REAL QUESTIONIs cost the only driver? No. Here's what keeps me awake at night —
What we learned
At 10,000-developer scale, per-call API pricing scales linearly with usage while self-hosted infrastructure scales in steps (GPU nodes) — so the larger and heavier the usage, the more self-hosting's economics win.
Recall first — before you read on
- Besides cost, what's the other major driver for self-hosting?
- What does code leaving the VPC expose the bank to that self-hosting avoids?
The Control ImperativeGOVERNANCE▶
data-sovereignty-matrix.txtascii
Data Sovereignty Matrix: ┌────────────────────────────────────────────────────────────┐ │ │ Third-Party API │ Self-Hosted │ ├────────────────────┼─────────────────┼─────────────────────┤ │ Code leaves VPC │ YES ⚠️ │ NO ✅ │ │ Provider sees code │ YES ⚠️ │ NO ✅ │ │ Audit trail │ Limited │ Complete │ │ Model customization│ Impossible │ Full control │ │ Latency SLA │ Provider's │ Your infrastructure │ │ Regulatory (SOC2) │ Complex │ Direct control │ └────────────────────────────────────────────────────────────┘
THOUGHT-PROVOKING QUESTIONIf your competitor gets acquired by your AI provider, what happens to all the proprietary code patterns that have been sent to their API?
What we learned
Cost savings alone don't justify the migration risk — the real unlock is that code never leaves the VPC, giving complete audit trails and regulatory control that a third-party API can never offer.
02
LLM Selection: The Evaluation War Room
Recall first — before you read on
- Which 8 candidate models were evaluated, and across how many dimensions?
- Which model scored highest on raw code-gen quality — and was it the one finally chosen?
The Candidate Pool8 MODELS · 6 DIMENSIONS▶
We evaluated 8 models across 6 dimensions. Here's the brutal truth:
evaluation-matrix.txtscale 1–10
Evaluation Matrix (Scale 1-10, higher = better): ┌──────────────────────┬──────────┬──────────┬──────────┬──────────┬──────────┐ │ Model │ Code Gen │ Latency │ Memory │ Fine-tune│ License │ ├──────────────────────┼──────────┼──────────┼──────────┼──────────┼──────────┤ │ Code Llama 34B │ 8.2 │ 7.5 │ 8.0 │ 9.0 │ Open │ │ DeepSeek Coder 33B │ 8.5 │ 7.8 │ 8.2 │ 8.5 │ Open │ │ Mistral 7B │ 6.5 │ 9.5 │ 9.5 │ 8.0 │ Apache │ │ StarCoder2 15B │ 7.8 │ 8.0 │ 8.5 │ 7.5 │ Open │ │ Llama 2 70B │ 8.8 │ 5.0 │ 5.0 │ 9.0 │ Custom │ │ GPT-NeoX 20B │ 6.0 │ 7.0 │ 7.0 │ 7.0 │ Apache │ │ Phi-2 (fine-tuned) │ 7.2 │ 9.0 │ 9.8 │ 6.0 │ MIT │ │ WizardCoder 34B │ 8.7 │ 7.0 │ 7.5 │ 8.0 │ Open │ └──────────────────────┴──────────┴──────────┴──────────┴──────────┴──────────┘
What we learned
Benchmark scores on paper (Code Gen, Latency, Memory, Fine-tune, License) rarely tell the full story — no single model wins on every axis, which is exactly why a multi-model strategy made more sense than picking one 'best' model.
Recall first — before you read on
- What three-model strategy was chosen — a primary, a fast path, and a specialized path?
- Why was Llama 2 70B rejected despite strong benchmark scores?
- How did the internal 500-PR benchmark differ from public benchmarks?
The Final Decision: A Multi-Model StrategyDECISION▶
model-decision-tree.txtascii
WHY WE CHOSE WHAT WE CHOSE: Primary Workhorse: DeepSeek Coder 33B ├── Best code generation quality-to-latency ratio ├── Open license with no commercial restrictions ├── Excellent at understanding financial domain code patterns └── Run on A100-40GB with comfortable headroom Fallback/Fast Path: Mistral 7B (Fine-tuned) ├── Lightning fast for simple completions ├── Can run on T4 GPUs at fraction of cost ├── Perfect for IDE autocomplete scenarios └── 95% of simple queries don't need 33B parameters Specialized Path: WizardCoder 34B ├── Used for complex refactoring tasks ├── Better at test generation └── Only invoked by explicit user request WHY OTHERS WERE REJECTED: Llama 2 70B: Too large, too slow ├── 2.5s inference time on A100 (unacceptable for IDE use) ├── Requires 2× A100s for comfortable deployment └── Meta's custom license created legal review delays GPT-NeoX: Outdated architecture ├── RoPE embeddings missing, context handling poor └── Community support declining StarCoder2: Good but not great ├── Filled-attention bugs in initial release └── Finetuning documentation sparse Phi-2: Too small for complex tasks ├── Failed on multi-file refactoring └── Cannot handle our 16K context requirements
Internal Benchmark — Did we test on OUR code? ▶
Yes. We built an internal benchmark of 500 real PRs from the last year:
internal-benchmark.txtPass@1
Internal Benchmark Results (Pass@1): ┌────────────────────────────────────────────────────────┐ │ Task Category │ DS-Coder │ WizCoder │ Mistral │ ├────────────────────────┼──────────┼──────────┼─────────┤ │ CRUD API Generation │ 92% │ 89% │ 78% │ │ SQL Query Building │ 88% │ 85% │ 82% │ │ Unit Test Writing │ 76% │ 82% │ 65% │ │ Refactoring (Multi-file)│ 71% │ 74% │ 42% │ │ Regex/Pattern Match │ 94% │ 91% │ 88% │ │ Security Review │ 68% │ 65% │ 55% │ └────────────────────────────────────────────────────────┘
What we learned
The best model for a task isn't always the biggest one: routing 95% of simple completions to a small fine-tuned model while reserving the 33B model for real generation work is what actually made the economics and latency work.
03
AWS Infrastructure: The Physical Reality
Recall first — before you read on
- Which AWS network layers sit in front of the wrapper service — public vs private vs GPU compute subnets?
- Where do SageMaker endpoints for DeepSeek and Mistral sit in the VPC?
Topology OverviewVPC / NETWORK▶
vpc-topology.txtascii
INTERNET
│
▼
[AWS WAF + Shield]
│
▼
[Route 53 DNS]──────[CloudFront CDN]
│ │
▼ ▼
┌────────────────────────────────────┐
│ VPC (10.0.0.0/16) │
│ │
│ ┌──────────────────────────────┐ │
│ │ Public Subnets (DMZ) │ │
│ │ │ │
│ │ [Application Load Balancer] │ │
│ │ ┌──────────┐ ┌──────────┐ │ │
│ │ │ ALB-1a │ │ ALB-1b │ │ │
│ │ └──────────┘ └──────────┘ │ │
│ └──────────────┬───────────────┘ │
│ │ │
│ ┌──────────────▼───────────────┐ │
│ │ Private App Subnets │ │
│ │ │ │
│ │ ┌────────────────────────┐ │ │
│ │ │ IDFC-Coder Wrapper │ │ │
│ │ │ (ECS Fargate) │ │ │
│ │ │ ┌──────┐ ┌──────┐ │ │ │
│ │ │ │Task-1│ │Task-2│ │ │ │
│ │ │ └──────┘ └──────┘ │ │ │
│ │ └────────┬───────────────┘ │ │
│ └───────────┼──────────────────┘ │
│ │ │
│ ┌───────────▼──────────────────┐ │
│ │ GPU Compute Subnets │ │
│ │ │ │
│ │ ┌────────────────────────┐ │ │
│ │ │ SageMaker Endpoints │ │ │
│ │ │ │ │ │
│ │ │ Endpoint-1 (DS-33B): │ │ │
│ │ │ ┌──────┐ ┌──────┐ │ │ │
│ │ │ │ml.p4d│ │ml.p4d│ │ │ │
│ │ │ │ .24xl│ │ .24xl│ │ │ │
│ │ │ └──────┘ └──────┘ │ │ │
│ │ │ │ │ │
│ │ │ Endpoint-2 (Mistral): │ │ │
│ │ │ ┌──────┐ ┌──────┐ │ │ │
│ │ │ │ml.g5 │ │ml.g5 │ │ │ │
│ │ │ │ .12xl│ │ .12xl│ │ │ │
│ │ │ └──────┘ └──────┘ │ │ │
│ │ └────────────────────────┘ │ │
│ └──────────────────────────────┘ │
│ │
│ ┌──────────────────────────────┐ │
│ │ Data/State Subnets │ │
│ │ ┌─────────┐ ┌────────────┐ │ │
│ │ │ElastiCache│ │ DynamoDB │ │ │
│ │ │ (Redis) │ │ (Cache) │ │ │
│ │ └─────────┘ └────────────┘ │ │
│ │ ┌─────────┐ ┌────────────┐ │ │
│ │ │ SQS │ │ S3 Model │ │ │
│ │ │ (Queues)│ │ Registry │ │ │
│ │ └─────────┘ └────────────┘ │ │
│ └──────────────────────────────┘ │
└────────────────────────────────────┘
What we learned
Isolating the wrapper service, GPU compute, and data layers into separate subnets isn't just security hygiene — it's what lets each layer scale, fail, and be audited independently.
Recall first — before you read on
- Between A100-40GB and H100-80GB, which was chosen and why?
- What was the target p95 latency and monthly compute budget constraint?
The GPU Math: Why This MattersCAPACITY PLANNING▶
gpu-instance-decision.txtascii
INSTANCE SELECTION DECISION TREE: Our constraints: ├── 10,000 developers, peak concurrency ~500 ├── Target latency: p95 < 800ms ├── Budget: $250K/month for compute └── Must fit models in single GPU (no tensor parallelism) OPTION 1: A100-40GB (p4d.24xlarge) ├── 8× A100 GPUs, 40GB each ├── Can host 1 instance of DeepSeek-33B per GPU ├── With quantization: 2 instances per GPU ├── Per-instance cost: $32.77/hr (1-year reserved) ├── Capacity per node: 8-16 concurrent requests ├── 8 nodes × 8 GPUs × 1.5 (avg utilization) = 96 concurrent └── COST: 8 × $32.77 × 730 = $191,380/month OPTION 2: H100-80GB (p5.48xlarge) ├── 8× H100 GPUs, 80GB each ├── 2-3× faster inference than A100 ├── Per-instance cost: $98.32/hr (1-year reserved) ├── Capacity per node: 24-32 concurrent requests ├── 3 nodes × 8 GPUs × 2 = 48 concurrent (but 2× faster) └── COST: 3 × $98.32 × 730 = $215,200/month WINNER: A100s ├── Better cost/capacity ratio ├── Sufficient performance for our latency targets └── More flexible for fine-tuning workloads
THE REAL QUESTIONWhat happens during Monday morning peak when 40% of developers hit the system simultaneously?
What we learned
The faster, more expensive H100 wasn't the right call here — once you model real concurrency needs against latency targets, the 'better' chip can still be the worse business decision.
Recall first — before you read on
- What's the min/max instance range for the DeepSeek endpoint, and how does it change by time of day?
- How does the batching queue improve throughput without extra GPU memory?
Auto-Scaling ConfigurationSAGEMAKER▶
autoscaling-policy.txtascii
SAGEMAKER AUTO-SCALING POLICY: ┌────────────────────────────────────────────────────┐ │ DeepSeek Endpoint (Primary): │ │ │ │ Target Tracking: │ │ ├── Metric: SageMakerVariantInvocationsPerInstance│ │ ├── Target: 8 requests per instance │ │ ├── Scale-out: When > 10 for 2 minutes │ │ └── Scale-in: When < 3 for 10 minutes │ │ │ │ Min Instances: 4 (off-peak) │ │ Max Instances: 16 (peak) │ │ Cool-down: 300 seconds │ │ │ │ Scheduled Scaling: │ │ ├── Mon-Fri 8:00-10:00: Min 12 instances │ │ ├── Mon-Fri 10:00-17:00: Min 8 instances │ │ └── Weekends/2AM-6AM: Min 2 instances │ └────────────────────────────────────────────────────┘ Batch Optimization Queue: ┌────────────────────────────────────────────────────┐ │ │ │ Client sends: "Write a function to..." │ │ │ │ │ ▼ │ │ [SQS Queue accumulates for 50ms window] │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────┐ │ │ │ Batch of 4 requests formed: │ │ │ │ [req1, req2, req3, req4] │ │ │ └─────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ [Single forward pass through model] │ │ │ │ │ ▼ │ │ [Responses split and returned individually] │ │ │ │ Result: 4× throughput, same GPU memory │ └────────────────────────────────────────────────────┘
What we learned
Static capacity planning fails at this scale — scheduled scaling (matching known daily/weekly patterns) combined with reactive target-tracking is what kept cost and latency both in check.
04
The IDFC-Coder Wrapper: The Intelligence Layer
Recall first — before you read on
- What are the three stages a request passes through before hitting the model — auth/rate-limit, prompt engineering, caching?
- What two-tier caching strategy is used, and what's the combined hit-rate benefit?
Architecture Deep DiveWRAPPER SERVICE▶
wrapper-architecture.txtascii
┌─────────────────────────────────────────────────────────────┐
│ IDFC-Coder Wrapper │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ Request Orchestrator │ │
│ │ ┌────────────┐ ┌────────────┐ ┌────────────┐ │ │
│ │ │ Auth/ │ │ Rate │ │ Team │ │ │
│ │ │ RBAC │──│ Limiter │──│ Quota Check│ │ │
│ │ └────────────┘ └────────────┘ └────────────┘ │ │
│ └──────────────────────┬───────────────────────────────┘ │
│ │ │
│ ┌──────────────────────▼───────────────────────────────┐ │
│ │ Prompt Engineering Engine │ │
│ │ │ │
│ │ Template: │ │
│ │ ┌─────────────────────────────────────────────┐ │ │
│ │ │ System: You are a fintech code assistant. │ │ │
│ │ │ You write secure, compliant Java/Python. │ │ │
│ │ │ │ │ │
│ │ │ Context from codebase: │ │ │
│ │ │ {injected_relevant_files} │ │ │
│ │ │ │ │ │
│ │ │ Jira ticket context: │ │ │
│ │ │ {ticket_description_and_acceptance_criteria} │ │ │
│ │ │ │ │ │
│ │ │ User request: {user_prompt} │ │ │
│ │ │ │ │ │
│ │ │ Respond with code only, no explanations. │ │ │
│ │ └─────────────────────────────────────────────┘ │ │
│ └──────────────────────┬───────────────────────────────┘ │
│ │ │
│ ┌──────────────────────▼───────────────────────────────┐ │
│ │ Caching Layer │ │
│ │ │ │
│ │ Cache Key: MD5(prompt_hash + context_hash) │ │
│ │ │ │
│ │ L1: ElastiCache Redis (in-memory, TTL: 1 hour) │ │
│ │ └── Hit rate: ~40% for repeated patterns │ │
│ │ │ │
│ │ L2: DynamoDB (persistent, TTL: 24 hours) │ │
│ │ └── Hit rate: +15% for cross-instance sharing │ │
│ │ │ │
│ │ Cache Invalidation: On new model version deployment │ │
│ └──────────────────────┬───────────────────────────────┘ │
│ │ │
│ ┌──────────────────────▼───────────────────────────────┐ │
│ │ Queue & Token Management │ │
│ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ Token Counter│ │ Cost Tracker │ │ │
│ │ │ │ │ │ │ │
│ │ │ Input: 450 │ │ Team: Alpha │ │ │
│ │ │ Output: 320 │ │ Daily: 2.3M │ │ │
│ │ │ Total: 770 │ │ Limit: 5M │ │ │
│ │ │ Est. Cost: │ │ Remaining: │ │ │
│ │ │ $0.0008 │ │ 2.7M tokens │ │ │
│ │ └──────────────┘ └──────────────┘ │ │
│ └──────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
What we learned
Every request the wrapper handles goes through auth, quota, prompt engineering, and caching before it ever reaches the model — the 'intelligence layer' is as much about governance as it is about prompting.
Recall first — before you read on
- What's the effective cost per developer request after caching, and how does that compare to a commercial API?
- How are team token budgets enforced, and what happens at 80% and 100% usage?
The Token EconomyCOST MODEL▶
This is where most self-hosted deployments fail. Let me explain why:
cost-per-inference.txtascii
COST PER INFERENCE BREAKDOWN: DeepSeek Coder 33B on A100 (with batching): ┌────────────────────────────────────────────────────┐ │ Fixed costs (per hour): │ │ ├── GPU instance: $32.77 │ │ ├── Overhead (network, storage): $2.50 │ │ └── Total: $35.27/hour │ │ │ │ Throughput at batch size 4: │ │ ├── 400 requests per GPU per hour │ │ └── Cost per request: $0.088 │ │ │ │ With caching (40% hit rate): │ │ ├── Effective cost: $0.053 per request │ │ └── Compare to Claude API: $0.03-0.10 │ │ │ │ BUT! For our use case: │ │ ├── Average tokens: 800 per request │ │ ├── Cached requests: near-zero cost │ │ └── Blended cost: $0.025 per developer request │ └────────────────────────────────────────────────────┘ TOKEN BUDGETING SYSTEM: ┌────────────────────────────────────────────────────┐ │ Team Token Allocation (Monthly): │ │ │ │ Team Alpha (Core Platform, 200 devs): │ │ ├── Budget: 50M tokens/month │ │ ├── Rate: 10K tokens/dev/day │ │ └── Alerts at 80%, blocks at 100% │ │ │ │ Team Beta (Mobile, 150 devs): │ │ ├── Budget: 37.5M tokens/month │ │ ├── Rate: 10K tokens/dev/day │ │ └── Can request increase via Jira ticket │ │ │ │ Enforcement via API Gateway + wrapper layer │ └────────────────────────────────────────────────────┘
HARD QUESTIONWhat happens when a VP of Engineering demands unlimited access? How do you say no with data?
What we learned
Caching isn't optional at this scale — a 40-55% effective hit rate is what turns a marginally-competitive per-request cost into a clearly winning one, and hard token budgets are what make 'no' to unlimited access defensible with data.
05
Jira Integration: The Autonomous Agent
Recall first — before you read on
- How often does the Jira agent poll, and what JQL filter does it use?
- What are the three possible output actions once the agent generates code for a ticket?
System Design — Agent Orchestration FlowEVENTBRIDGE · 30 MIN▶
jira-agent-flow.txtascii
┌─────────────────────────────────────────────────────────┐
│ Jira Agent Controller │
│ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Scheduler (EventBridge) │ │
│ │ ┌────────────────────────────────────────────┐ │ │
│ │ │ Every 30 minutes: │ │ │
│ │ │ cron(0/30 * * * ? *) │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────────┐ │ │ │
│ │ │ │ State Check │ │ │ │
│ │ │ │ (DynamoDB) │ │ │ │
│ │ │ └──────┬──────┘ │ │ │
│ │ │ │ │ │ │
│ │ │ Last poll: 2024-01-15 14:00:00 UTC │ │ │
│ │ │ Cursor: ticket-45678 │ │ │
│ │ └────────────────────────────────────────────┘ │ │
│ └──────────────────────────────────────────────────┘ │
│ │ │
│ ┌──────────────────────▼───────────────────────────┐ │
│ │ Jira Connector (Rate Limited) │ │
│ │ │ │
│ │ API CALL STRATEGY: │ │
│ │ ┌─────────────────────────────────────────┐ │ │
│ │ │ JQL Query: │ │ │
│ │ │ project = "IDFC" AND │ │ │
│ │ │ status in ("In Development", "Ready") │ │ │
│ │ │ AND updated >= -30m │ │ │
│ │ │ ORDER BY updated DESC │ │ │
│ │ └─────────────────────────────────────────┘ │ │
│ │ │ │
│ │ Rate Limiting: │ │
│ │ ┌─────────────────────────────────────────┐ │ │
│ │ │ • Jira Cloud: 10 requests/sec │ │ │
│ │ │ • Our self-hosted: 50 requests/sec │ │ │
│ │ │ • Token bucket algorithm: │ │ │
│ │ │ - Burst: 30 │ │ │
│ │ │ - Sustained: 10/sec │ │ │
│ │ │ • Exponential backoff: 1s, 2s, 4s, 8s │ │ │
│ │ └─────────────────────────────────────────┘ │ │
│ └──────────────────────┬───────────────────────────┘ │
│ │ │
│ ┌──────────────────────▼───────────────────────────┐ │
│ │ Prompt Transformer │ │
│ │ │ │
│ │ Ticket → Prompt Mapping: │ │
│ │ ┌─────────────────────────────────────────┐ │ │
│ │ │ { │ │ │
│ │ │ "ticket_id": "IDFC-1234", │ │ │
│ │ │ "title": "Add fraud detection rule", │ │ │
│ │ │ "description": "...", │ │ │
│ │ │ "acceptance_criteria": [ │ │ │
│ │ │ "Must flag transactions > $10K", │ │ │
│ │ │ "Must not exceed 100ms latency" │ │ │
│ │ │ ] │ │ │
│ │ │ } │ │ │
│ │ │ │ │ │
│ │ │ Transforms to: │ │ │
│ │ │ "Write a Java method that implements..." │ │ │
│ │ └─────────────────────────────────────────┘ │ │
│ └──────────────────────┬───────────────────────────┘ │
│ │ │
│ ┌──────────────────────▼───────────────────────────┐ │
│ │ Result Publisher │ │
│ │ │ │
│ │ Output Actions: │ │
│ │ ┌─────────────────────────────────────────┐ │ │
│ │ │ Option A: Comment on ticket │ │ │
│ │ │ { │ │ │
│ │ │ "body": "🤖 AI Suggestion:\n```java..." │ │ │
│ │ │ } │ │ │
│ │ │ │ │ │
│ │ │ Option B: Create subtask │ │ │
│ │ │ { │ │ │
│ │ │ "summary": "Implement: ...", │ │ │
│ │ │ "description": "AI-generated code..." │ │ │
│ │ │ } │ │ │
│ │ │ │ │ │
│ │ │ Option C: Create pull request │ │ │
│ │ │ (via Bitbucket/GitHub API) │ │ │
│ │ └─────────────────────────────────────────┘ │ │
│ └──────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
What we learned
Turning Jira tickets into code isn't a single API call — it's a full pipeline of polling, rate-limited fetching, prompt transformation, and a deliberate choice of how to publish the result (comment, subtask, or PR).
Recall first — before you read on
- Why can't the agent trust Jira's search index alone, and what's the dual-verification fix?
- What's the fallback when the Jira rate limit is exhausted?
Handling Jira's QuirksCONSISTENCY▶
eventual-consistency.txtascii
EVENTUAL CONSISTENCY PATTERN: ┌────────────────────────────────────────────────────┐ │ Problem: Jira search index may be stale │ │ │ │ Solution: Dual verification │ │ │ │ ┌────────────────────────────────────────────┐ │ │ │ 1. Get tickets from search (fast, maybe │ │ │ │ stale) │ │ │ │ 2. For each candidate, direct GET ticket │ │ │ │ (slow, always current) │ │ │ │ 3. Compare versions: │ │ │ │ if search.version < direct.version: │ │ │ │ Use direct version │ │ │ │ If already processed (idempotency key): │ │ │ │ Skip │ │ │ └────────────────────────────────────────────┘ │ │ │ │ Rate Limit Exhaustion Handling: │ │ ┌────────────────────────────────────────────┐ │ │ │ • Pre-emptive: Track X-RateLimit-Remaining │ │ │ │ • Reactive: Queue overflow → DLQ │ │ │ │ • Backpressure: Skip poll if previous not │ │ │ │ complete │ │ │ │ • Priority queue: Urgent tickets first │ │ │ └────────────────────────────────────────────┘ │ └────────────────────────────────────────────────────┘
CONTROVERSIAL QUESTIONIs automatically generating code from Jira tickets actually helping developers, or are we just creating more review burden?
What we learned
Third-party SaaS APIs are eventually consistent by default — building a dual-verification (search + direct GET) pattern was necessary just to avoid acting on stale data.
06
Production Concerns: The Boring Stuff That Saves You
Recall first — before you read on
- What failure threshold trips the circuit breaker from CLOSED to OPEN?
- What's the retry backoff sequence before a request lands in the dead-letter queue?
Resilience PatternsCIRCUIT BREAKER▶
circuit-breaker-state-machine.txtascii
CIRCUIT BREAKER CONFIGURATION: ┌────────────────────────────────────────────────────┐ │ State Machine: │ │ │ │ ┌──────────┐ │ │ │ CLOSED │ (Normal operation) │ │ └────┬─────┘ │ │ │ │ │ failures >= 5 in 60s window │ │ │ │ │ ▼ │ │ ┌──────────┐ │ │ │ OPEN │ (Fail fast, return cache) │ │ └────┬─────┘ │ │ │ │ │ 30 second timeout │ │ │ │ │ ▼ │ │ ┌──────────┐ │ │ │ HALF-OPEN│ (Test with 1 request) │ │ └────┬─────┘ │ │ │ │ │ Success → CLOSED │ │ Failure → OPEN (back to timeout) │ └────────────────────────────────────────────────────┘ RETRY STRATEGY WITH JITTER: ┌────────────────────────────────────────────────────┐ │ Attempt 1: Immediate │ │ Attempt 2: 1s + random(0-200ms) │ │ Attempt 3: 2s + random(0-400ms) │ │ Attempt 4: 4s + random(0-800ms) │ │ Attempt 5: 8s + random(0-1600ms) [MAX] │ │ │ │ Then: Dead Letter Queue │ │ └── SQS → Lambda → Alert → Slack │ └────────────────────────────────────────────────────┘
What we learned
Circuit breakers and jittered retries aren't just best practice boilerplate — they're what stands between a slow GPU endpoint and a full outage, by failing fast and falling back to cache instead of queuing forever.
Recall first — before you read on
- What canary traffic split is used for a new model version, and over what monitoring window?
- Name two conditions that trigger an automatic rollback.
Model Deployment StrategyCANARY▶
canary-deployment.txtascii
CANARY DEPLOYMENT FOR MODEL UPDATES: ┌────────────────────────────────────────────────────┐ │ Current Model: deepseek-v1.2 (tag: prod) │ │ New Model: deepseek-v1.3 (tag: canary) │ │ │ │ Traffic Splitting (SageMaker): │ │ │ │ Endpoint: idfc-coder-prod │ │ ├── Variant-A (v1.2): 90% weight │ │ └── Variant-B (v1.3): 10% weight │ │ │ │ Monitoring Period: 4 hours │ │ │ │ Automatic Rollback IF: │ │ ├── p95 latency > 1200ms (baseline: 800ms) │ │ ├── Error rate > 5% (baseline: 1%) │ │ ├── Token output length < 50 (hallucination risk) │ │ └── User rejection rate > 20% │ │ │ │ Gradual Promotion: │ │ ├── Hour 4: 25% traffic │ │ ├── Hour 8: 50% traffic │ │ ├── Hour 12: 100% traffic │ │ └── After 24h stable: decommission v1.2 │ └────────────────────────────────────────────────────┘
What we learned
Model updates need the same production rigor as code deploys — canary traffic splitting with automatic rollback thresholds (latency, error rate, even output length as a hallucination signal) catches regressions before they reach everyone.
Recall first — before you read on
- What's the split between business metrics and technical metrics on the dashboard?
- In the X-Ray trace example, which stage consumed the most time?
Observability StackCLOUDWATCH / X-RAY▶
observability-stack.txtascii
MONITORING ARCHITECTURE: ┌────────────────────────────────────────────────────┐ │ CloudWatch Dashboard │ │ │ │ ┌──────────────────┐ ┌──────────────────┐ │ │ │ Business Metrics │ │ Technical Metrics │ │ │ ├──────────────────┤ ├──────────────────┤ │ │ │ • Requests/team │ │ • p50/p95/p99 lat │ │ │ │ • Tokens consumed │ │ • Queue depth │ │ │ │ • Cache hit rate │ │ • GPU utilization │ │ │ │ • Cost per team │ │ • Error rates │ │ │ │ • Code accepted% │ │ • Circuit breaks │ │ │ └──────────────────┘ └──────────────────┘ │ │ │ │ ┌────────────────────────────────────────────┐ │ │ │ X-Ray Tracing │ │ │ │ │ │ │ │ Trace: request-abc123 │ │ │ │ ├── API Gateway: 12ms │ │ │ │ ├── Auth check: 8ms │ │ │ │ ├── Cache lookup: 3ms [MISS] │ │ │ │ ├── SQS Queue time: 45ms │ │ │ │ ├── LLM Inference: 340ms │ │ │ │ ├── Response transform: 5ms │ │ │ │ └── TOTAL: 413ms │ │ │ └────────────────────────────────────────────┘ │ └────────────────────────────────────────────────────┘
What we learned
Business metrics (cost, adoption, code-accepted%) and technical metrics (latency, GPU utilization) have to sit on the same dashboard — otherwise engineering optimizes for speed while the business case quietly erodes.
Recall first — before you read on
- Soft or hard multi-tenancy — which was chosen, and what's the main cost trade-off with the other option?
- What's the biggest risk of the soft multi-tenancy model?
Multi-Tenancy IsolationSOFT vs HARD▶
tenant-isolation.txtascii
TENANT ISOLATION MODEL: ┌────────────────────────────────────────────────────┐ │ Option A: Soft Multi-tenancy (We chose this) │ │ │ │ • Shared inference endpoints │ │ • Queue-level isolation (per-team SQS queues) │ │ • Token quotas enforced at wrapper layer │ │ • Separate DynamoDB tables per team for caching │ │ • CloudWatch metrics tagged with TeamID │ │ │ │ Pros: Cost effective, simple operations │ │ Cons: Noisy neighbor possible │ │ │ │ Option B: Hard Multi-tenancy (Rejected) │ │ │ │ • Separate SageMaker endpoints per team │ │ • Full VPC isolation │ │ • Pros: True isolation │ │ • Cons: 50 teams × 2 GPUs = $1.6M/month minimum │ └────────────────────────────────────────────────────┘
STRATEGIC QUESTIONIn fintech, regulators might want hard multi-tenancy. How do we prepare for that without over-engineering now?
What we learned
Soft multi-tenancy was the pragmatic choice, not the ideal one — the team consciously accepted 'noisy neighbor' risk to avoid a 10x cost blowout, while leaving a path open to harden isolation later if regulators demand it.
07
Iterative Challenges: The War Stories
Recall first — before you read on
- What was the p95 latency before optimization, and what were the main root causes?
- Which single change — batching, streaming, quantization, or ECS — do you think mattered most, and what was the final p95?
Challenge 1 — The Latency NightmareP1 INCIDENT▶
latency-phase1.txtascii
PHASE 1 - Unoptimized (Week 1): ┌────────────────────────────────────────────────────┐ │ Single request flow: │ │ Client → API Gateway → Lambda → SageMaker → Client │ │ │ │ Results: │ │ ├── p50: 1200ms │ │ ├── p95: 3500ms │ │ └── User feedback: "Slower than Copilot!" │ └────────────────────────────────────────────────────┘
Root cause analysis — why so slow?
root-cause.txtascii
Why so slow? ├── No batching: One forward pass per request ├── Cold starts: Lambda initialization 200-400ms ├── No streaming: Waiting for full response ├── Model too big: 33B parameters, lots of compute └── Network latency: Cross-AZ calls
latency-phase2.txtascii
PHASE 2 - Optimized (Week 3): ┌────────────────────────────────────────────────────┐ │ Optimized flow: │ │ Client → ALB → ECS (warm) → Batched → Streaming │ │ │ │ Changes made: │ │ ├── Switched to ECS Fargate (no cold starts) │ │ ├── Dynamic batching (50ms accumulation window) │ │ ├── Token streaming via Server-Sent Events │ │ ├── Model quantization: FP32 → INT8 (2× speed) │ │ ├── KV-cache optimization for repeated prefixes │ │ └── Cross-AZ placement groups for SageMaker │ │ │ │ Results: │ │ ├── p50: 340ms (3.5× improvement) │ │ ├── p95: 720ms (4.8× improvement) │ │ └── Throughput: 800 req/sec (from 200) │ └────────────────────────────────────────────────────┘
What we learned
The first version's 3.5s p95 wasn't a model problem — it was an architecture problem (cold starts, no batching, no streaming). Fixing the plumbing delivered a bigger win than any model swap would have.
Recall first — before you read on
- Why did the naive FP16 memory math not even fit on a 40GB A100?
- What three techniques finally brought memory usage under budget?
Challenge 2 — The OOM CrisisOUTAGE▶
What happened: Tuesday 14:00, production outage — SageMaker returning 500s. Root cause: DeepSeek-33B OOM on the 40GB A100 because the context window grew to 16K tokens.
memory-analysis.txtascii
MEMORY ANALYSIS:
┌────────────────────────────────────────────────────┐
│ GPU Memory Budget (40GB): │
│ │
│ Model weights (FP16): 33B × 2 bytes = 66GB │
│ │
│ Wait, that doesn't fit! │
│ │
│ Let's recalculate with quantization: │
│ │
│ Model weights (INT8): 33B × 1 byte = 33GB │
│ KV Cache (batch_size=4, seq_len=4096): 8GB │
│ Activations: 4GB │
│ CUDA overhead: 2GB │
│ ───────────────────────────────────── │
│ Total: 47GB ❌ STILL DOESN'T FIT! │
└────────────────────────────────────────────────────┘
the-fix.txtascii
THE FIX:
┌────────────────────────────────────────────────────┐
│ Solution 1: Flash Attention 2 │
│ ├── Reduces memory by 5-10× for attention │
│ └── KV Cache: 8GB → 1.5GB │
│ │
│ Solution 2: Gradient checkpointing (even for inf) │
│ └── Activations: 4GB → 1GB │
│ │
│ Solution 3: Batch size reduction │
│ └── From 4 to 2 during peak │
│ │
│ New Memory Profile: │
│ ├── Weights: 33GB │
│ ├── KV Cache: 1.5GB │
│ ├── Activations: 1GB │
│ ├── Overhead: 2GB │
│ └── Total: 37.5GB ✅ FITS! │
│ │
│ Also: Added automatic context window adjustment │
│ ├── Monitor: GPU memory utilization │
│ ├── If > 85%: Reduce max_tokens by 25% │
│ └── Alert: If sustained > 90% for 5 min │
└────────────────────────────────────────────────────┘
What we learned
Back-of-envelope memory math (FP16 weights alone) is dangerously misleading — real GPU budgets have to account for KV cache, activations, and CUDA overhead together, and quantization plus Flash Attention were what finally made the numbers fit.
Recall first — before you read on
- What are the three types of hallucination encountered — think APIs, business logic, and security?
- What does the model do instead of guessing when it's unsure about a policy or API?
Challenge 3 — Hallucination MitigationQUALITY▶
Hallucination types encountered:
- Non-existent APIs — model invents calls like
BankAPI.getInstance().getV2Client(), an internal API not present in training data. - Incorrect business logic — model suggests a $10,000 fraud threshold when policy requires $5,000.
- Security anti-patterns — string-concatenated SQL, a firing offense in fintech code.
Solution — Grounded Prompt Engineering ▶
grounded-system-prompt.txtascii
Enhanced System Prompt:
You are IDFC-First Bank's internal code assistant.
CONSTRAINTS (enforced):
1. Only use APIs from our SDK:
com.idfc.banking.v3.*
com.idfc.fraud.detection.*
com.idfc.compliance.*
2. All database queries MUST use parameterized
statements (PreparedStatement only)
3. All monetary values MUST use BigDecimal,
never float/double
4. All external calls MUST have:
- Timeout (max 30s)
- Circuit breaker
- Retry with backoff
5. If you're unsure about an API or policy,
respond with: "NEEDS_CLARIFICATION: [question]"
Do not guess.
CONTEXT INJECTION:
{relevant_code_from_codebase}
{relevant_confluence_docs}
{jira_acceptance_criteria}
Post-processing Validation ▶
post-processing.txtascii
Before returning to user: 1. Static Analysis (SonarQube rules): ├── No SQL injection patterns ├── No hardcoded credentials └── Checked exceptions handled 2. API Validation: ├── Verify all method calls exist in SDK └── If not found: replace with comment 3. Policy Check: ├── Fraud thresholds from config service └── Inject correct values
What we learned
Prompting alone can't be trusted to stop hallucinated APIs or wrong compliance thresholds — it took explicit constraints, a 'don't guess, ask' instruction, and a post-processing validation layer to make outputs safe to ship.
Recall first — before you read on
- What are the four adoption phases, from pioneers to laggards?
- How was the objection 'it doesn't understand our codebase' addressed?
Challenge 4 — The 10,000 User RolloutADOPTION▶
phased-adoption.txtascii
PHASED ADOPTION STRATEGY: ┌────────────────────────────────────────────────────┐ │ Phase 1: Pioneers (Month 1, 100 users) │ │ ├── Selected power users from Core Platform team │ │ ├── Direct Slack channel with platform team │ │ ├── Weekly feedback sessions │ │ └── Goal: Iron out UX issues, gather testimonials │ │ │ │ Phase 2: Early Adopters (Month 2, 1000 users) │ │ ├── Add 2-3 more teams │ │ ├── Introduce IDE plugin (VS Code, IntelliJ) │ │ ├── Gamification: Leaderboard of tokens saved │ │ └── Goal: Prove productivity improvement │ │ │ │ Phase 3: Majority (Month 3-4, 5000 users) │ │ ├── Self-service onboarding portal │ │ ├── Required: 30-min training video │ │ ├── Auto-enrollment for new hires │ │ └── Goal: Make it default, not optional │ │ │ │ Phase 4: Laggards (Month 5-6, 10000 users) │ │ ├── Mandatory for certain ticket types │ │ ├── "AI-first" workflow: Must review AI suggestion │ │ │ before writing from scratch │ │ └── Goal: Complete transformation │ └────────────────────────────────────────────────────┘
adoption-friction.txtascii
ADOPTION FRICTION POINTS & SOLUTIONS: ┌────────────────────────────────────────────────────┐ │ "It's slower than Copilot" │ │ → Show latency metrics, it's actually faster for │ │ our codebase because of context injection │ │ │ │ "The code is wrong" │ │ → Add "Report bad suggestion" button │ │ → Track rejection rate, feed back to fine-tuning │ │ → Public dashboard showing improvement over time │ │ │ │ "I don't trust AI" │ │ → Start with test generation (low risk) │ │ → Show examples of bugs caught by AI suggestions │ │ → Peer pressure: "Your team's top performer uses it"│ │ │ │ "It doesn't understand our codebase" │ │ → RAG pipeline: Index all repos, retrieve context │ │ → Fine-tune on our private code (with permissions) │ └────────────────────────────────────────────────────┘
What we learned
Technology readiness and organizational readiness are two different problems — the phased pioneer → early adopter → majority → laggard rollout mattered as much as any latency or accuracy fix.
08
Concrete Metrics: The Scorecard
Six-month post-launch results.
54.4%
Year-1 cost reduction
$4.28M
Year-1 savings
720ms
p95 latency, simple gen
1,200 rps
Peak throughput
68%
Acceptance rate, Month 6
+42 NPS
Developer sentiment
Recall first — before you read on
- What was the actual realized savings percentage compared to the original projection?
Cost AnalysisFINANCE▶
cost-analysis.txtascii
COST ANALYSIS: ┌────────────────────────────────────────────────────────┐ │ Previous Year (Copilot Enterprise + Claude API): │ │ ├── GitHub Copilot: 10,000 seats × $39/month │ │ │ = $4,680,000/year │ │ ├── Anthropic API (heavy users): $3,200,000/year │ │ └── TOTAL: $7,880,000/year │ │ │ │ Self-Hosted Solution (Year 1): │ │ ├── Compute (A100 reserved): $2,296,560 │ │ ├── Storage & Networking: $300,000 │ │ ├── Engineering Team (4 FTE): $800,000 │ │ ├── Training & Onboarding: $200,000 │ │ └── TOTAL: $3,596,560 │ │ │ │ SAVINGS: $4,283,440 (54.4% reduction) │ │ │ │ Year 2 Projection (with optimization): │ │ ├── Better batching, higher utilization: -15% compute │ │ ├── More caching, smaller models: -10% │ │ └── Projected: $2,900,000 (63.2% savings) │ └────────────────────────────────────────────────────────┘
What we learned
Real-world savings rarely match the initial projection exactly — tracking actuals against the original business case is what keeps the initiative honest and fundable.
Recall first — before you read on
- How much did p95 latency improve from the unoptimized to the optimized state?
Performance MetricsLATENCY / THROUGHPUT▶
performance-metrics.txtascii
Latency (DeepSeek Coder 33B, INT8): Simple completions (< 200 tokens): ├── p50: 280ms ├── p95: 620ms ├── p99: 980ms └── SLA: p95 < 800ms ✅ Complex generations (200-1000 tokens): ├── p50: 450ms ├── p95: 890ms ├── p99: 1400ms └── Streaming: First token in < 100ms Throughput: ├── Peak: 1,200 requests/second ├── Average: 450 requests/second ├── Daily volume: ~15M requests └── GPU utilization: 73% average
What we learned
The multi-week latency optimization effort translated directly into a metric leadership actually cares about — this is the proof point that turns an engineering win into a business one.
Recall first — before you read on
- What direction did bugs-per-release and security vulnerabilities move after rollout?
Quality MetricsCORRECTNESS▶
quality-metrics.txtascii
Internal Benchmark (vs. Human Baseline): Code Correctness (passes tests first try): ├── Simple CRUD: 91% (human: 95%) ├── Complex business logic: 72% (human: 88%) ├── SQL queries: 88% (human: 93%) └── Unit tests: 76% (human: 90%) Developer Acceptance Rate: ├── Month 1: 34% (rejected or heavily modified) ├── Month 3: 52% ├── Month 6: 68% └── Target: 80% by Month 12 Time Savings: ├── Boilerplate code: 85% faster ├── API integrations: 60% faster ├── Bug fixes: 45% faster └── Average across all tasks: 55% faster
What we learned
Fewer bugs and fewer vulnerabilities post-rollout suggest the guardrails (grounded prompting, post-processing validation) were doing real work, not just adding friction.
Recall first — before you read on
- Roughly how did user adoption progress from Phase 1 pioneers to full rollout?
Adoption CurveGROWTH▶
adoption-curve.txtascii
Monthly Active Users: Month 1: 87 (87% of Phase 1) Month 2: 845 (84.5% of Phase 2) Month 3: 3,200 (64% of Phase 3 target) Month 4: 4,890 (97.8% of Phase 3 target) Month 5: 7,200 (72% of total) Month 6: 8,900 (89% of total) Daily Active Users: ~5,200 (52%) Average sessions per user: 3.4 per day Average requests per user: 47 per day Net Promoter Score: +42 (considered "Great")
What we learned
Adoption doesn't happen by mandate alone — the curve reflects the deliberate phased strategy, where trust was earned with power users before it was ever required of everyone.
Recall first — before you read on
- What's one concrete velocity or business metric that improved post-adoption?
Business ImpactVELOCITY▶
business-impact.txtascii
Development Velocity: Story Points Completed per Sprint: ├── Before: 1,200 points (avg across teams) ├── After 3 months: 1,450 points (+20.8%) └── After 6 months: 1,680 points (+40%) Time-to-Market: ├── Feature delivery: 22 days → 15 days (-31.8%) └── Hot fixes: 4.2 hours → 2.1 hours (-50%) Code Quality: ├── Bugs per release: 45 → 38 (-15.6%) ├── Security vulnerabilities: 12/month → 8/month └── Code review comments: +23% (AI catches patterns)
What we learned
The ultimate test of any platform investment is whether it shows up in delivery velocity, not just in usage dashboards.
09
CTO's Final Thoughts
Recall first — before you read on
- What was the deliberate choice about which team to onboard first, and why?
What We Got RightRETROSPECTIVE▶
what-we-got-right.txtascii
1. STARTED WITH THE HARDEST TEAM └── Core Platform team was most demanding, most skeptical └── If we could satisfy them, others would follow 2. BUILT FOR FAILURE └── Every component has a fallback └── Circuit breakers everywhere └── "What happens when this breaks at 3 AM?" 3. MEASURED OBSESSIVELY └── You can't improve what you don't measure └── Every decision backed by data └── Killed features that didn't show impact 4. TREATED IT AS PRODUCT, NOT TOOL └── UX design, not just API └── Developer advocacy team └── Regular town halls for feedback
What we learned
Starting with the most skeptical, most demanding team was a deliberate stress-test — if the platform could win them over, broader rollout became a much easier sell.
Recall first — before you read on
- What's the biggest regret around initial model size and scope?
- What GPU capacity risk should have been planned for from day one?
What We'd Do DifferentlyLESSONS▶
what-wed-do-differently.txtascii
1. START SMALLER └── Begin with 7B model for autocomplete only └── Add complexity gradually └── We overbuilt the initial system 2. INVEST IN FINE-TUNING EARLIER └── Generic models hallucinate on proprietary APIs └── Fine-tuning on our codebase would have saved months └── Now running weekly fine-tuning jobs 3. PLAN FOR GPU SHORTAGES └── A100 availability was constrained └── Should have reserved instances from day 1 └── Consider multi-cloud (GCP TPUs as backup) 4. OVER-COMMUNICATE COSTS └── Developers don't see GPU bills └── Should show "this request cost $0.003" └── Creates natural conservation behavior
What we learned
Overbuilding early and underestimating GPU supply constraints were the costliest mistakes — starting smaller and reserving capacity earlier would have saved months.
Recall first — before you read on
- Are we making developers better, or just faster — what's the concern behind that question?
- What safeguards were added to counter AI-generated 'code debt'?
The Real Question Nobody AsksREFLECTION▶
"ARE WE ACTUALLY MAKING DEVELOPERS BETTER, OR JUST FASTER?"
This is the uncomfortable truth: AI code generation can create "code debt" at unprecedented speed.
We're now investing in:
- AI-generated code review tools (fighting fire with fire)
- Mandatory test coverage thresholds for AI code
- "Explain the code" sessions in PR reviews
- Tracking long-term maintenance burden of AI-generated code
The goal isn't to replace developers — it's to remove the boring parts so they can focus on architecture, security, and innovation. But we must be vigilant that we're not creating a generation of developers who can't code without AI assistance.
What we learned
Speed isn't the same as skill — the team had to deliberately invest in code review discipline and maintainability tracking to make sure AI-assisted velocity didn't quietly become AI-generated technical debt.
10
Appendix: Key Configuration Files
Recall first — before you read on
- What instance type and initial instance count does the production endpoint config use?
SageMaker Deployment ConfigYAML▶
deepseek-coder-endpoint.yamlyaml
# deepseek-coder-endpoint.yaml
EndpointConfigName: idfc-coder-v1-3
ProductionVariants:
- VariantName: primary
ModelName: deepseek-coder-33b-instruct-int8
InstanceType: ml.p4d.24xlarge
InitialInstanceCount: 4
InitialVariantWeight: 1.0
# Auto-scaling
AutoScalingPolicy:
TargetTrackingScalingPolicyConfiguration:
TargetValue: 8.0 # Requests per instance
ScaleInCooldown: 600
ScaleOutCooldown: 120
CustomizedMetricSpecification:
MetricName: ApproximateBacklogSizePerInstance
Namespace: AWS/SageMaker
Statistic: Average
What we learned
Auto-scaling policy lives in config, not just documentation — the endpoint config is the actual source of truth for how the system behaves under load.
Recall first — before you read on
- What are the default vs fast model names, and what's the Redis cache TTL?
Wrapper Service ConfigurationPYTHON▶
idfc_coder_config.pypython
# idfc_coder_config.py
class IDFCCoderConfig:
# Model selection
DEFAULT_MODEL = "deepseek-coder-33b"
FAST_MODEL = "mistral-7b-finetuned"
# Batching
MAX_BATCH_SIZE = 4
BATCHING_WINDOW_MS = 50
# Token limits
MAX_INPUT_TOKENS = 8000
MAX_OUTPUT_TOKENS = 2000
COST_PER_1K_TOKENS = 0.002 # USD
# Team quotas (monthly tokens)
TEAM_QUOTAS = {
"core-platform": 50_000_000,
"mobile": 37_500_000,
"web": 30_000_000,
"data-engineering": 25_000_000,
}
# Cache
REDIS_TTL_SECONDS = 3600 # 1 hour
DYNAMODB_TTL_HOURS = 24
# Circuit breaker
CIRCUIT_BREAKER_THRESHOLD = 5 # failures
CIRCUIT_BREAKER_TIMEOUT = 30 # seconds
# Jira
JIRA_POLL_INTERVAL_MINUTES = 30
JIRA_MAX_TICKETS_PER_POLL = 50
What we learned
Team quotas, cache TTLs, and circuit-breaker thresholds are all just numbers in a config file — but they're the numbers that encode every governance and cost decision made earlier in this document.
This architecture represents 18 months of evolution, countless production incidents, and a fundamental shift in how our organization writes software.
SELF-HOSTING AI ISN'T JUST ABOUT COST — IT'S ABOUT SOVEREIGNTY OVER YOUR ENGINEERING FUTURE.