Key takeaways
- LLM features are probabilistic — design, test and monitor them differently from regular code.
- Retrieval, tool calling and guardrails matter more than prompt tricks.
- Build an evaluation set early and run it on every change.
- Plan for cost, latency and security from the first sprint.
Customers now expect software to understand natural language, summarise information and take actions on their behalf. Adding large language models (LLMs) to your web or mobile app can unlock that — but it also changes how you design, build, test and run software. Here is what teams need to know about building AI-powered apps.
Common AI features in modern apps
- Natural-language search across products, documents or records.
- Assistants that answer questions and complete tasks inside the app.
- Summaries of long content, conversations or reports.
- Drafting and rewriting — emails, descriptions, proposals.
- Data extraction from uploads such as invoices or forms.
- Recommendations and personalised experiences.
Architecture: the main building blocks
| Layer | Purpose |
|---|---|
| Model gateway | One place to call models, manage keys, switch providers, log usage and apply limits |
| Prompt and policy layer | Versioned instructions, output formats and safety rules |
| Retrieval (RAG) | Supplies relevant, permitted data — see RAG explained |
| Tools / functions | Let the model call your APIs safely with validated inputs |
| Evaluation and monitoring | Quality tests, feedback, cost and latency dashboards |
What changes for engineering teams
Testing becomes evaluation
The same input can produce different outputs. Build a set of realistic test cases with expected qualities (correct facts, right format, safe tone) and score every release against it.
Structured outputs
Ask models for JSON that matches a schema and validate it before your app uses it.
Cost and latency
Choose the smallest model that meets quality needs, cache repeated work, stream responses and set usage limits per user.
Security
Treat user input and retrieved documents as untrusted. Protect against prompt injection, never let the model call sensitive tools without checks, and enforce permissions in your code, not in the prompt.
UX patterns that build trust
- Show sources and let users open them.
- Make AI suggestions easy to edit, accept or reject.
- Explain limitations and offer a non-AI path.
- Collect feedback in context to improve quality.
From prototype to production
- Prototype the feature with real user scenarios.
- Create the evaluation set and agree on quality thresholds.
- Harden: guardrails, permissions, logging, rate limits.
- Release to a small group, monitor and iterate.
- Scale with dashboards for quality, cost and usage.
Build AI into your product the right way
Our AI-Powered Software & Application Engineering team designs, builds and runs production-grade AI features.
Frequently asked questions
Which LLM should we use for our app?
It depends on quality needs, latency, cost, data policies and languages. A model gateway lets you test several and switch later without rewriting the app.
Can AI features be added to an existing app?
Yes. Most AI features are added as new services connected through APIs, so existing web and mobile apps can be extended step by step.
How do we stop the AI from giving wrong answers?
Ground answers in retrieved data, require sources, validate outputs, run evaluations on every change and let users report problems easily.
What is prompt injection?
It is an attack where text in user input or documents tries to override the model’s instructions. Defences include input handling, restricted tools and permission checks in code.
