Generative AI consulting covers far more than writing prompts: retrieval architecture, data pipelines, evaluation harnesses, fine-tuning decisions, security, and integration with business systems. Prompt engineering is the visible, easy part. Perceptive Analytics treats it as the starting point of an engagement, not the deliverable.
Introduction
Ask most people what a generative AI consultant does, and the answer is some version of “writes better prompts.” That’s the part anyone can watch happen. It’s also a small fraction of the actual work.
Menlo Ventures’ 2025 survey of nearly 500 U.S. enterprise decision-makers, The State of Generative AI in the Enterprise, found that prompt design remains the dominant customization technique in production, followed by retrieval-augmented generation, while more advanced approaches such as fine-tuning, tool calling, and context engineering are still niche, used mostly by frontier teams. The same report found that only 16% of enterprise deployments and 27% of startup deployments actually qualify as true agents; most of what gets called “agentic AI” is still a fixed-sequence workflow wrapped around a single model call.
That gap between what’s easy to demo and what’s actually running in production is where generative AI consulting earns its fee. Perceptive Analytics spends most of an engagement on the parts that never show up in a slide: retrieval quality, evaluation, and the plumbing that keeps a system grounded in real data.
What Does Generative AI Consulting Actually Cover?
A generative AI consulting engagement typically spans model selection, retrieval architecture, data preparation, evaluation, security, and integration with the systems a business already runs on. Prompt engineering shows up early, usually in the first few weeks, and then becomes a much smaller share of the work as the project moves toward production.
Perceptive Analytics structures its generative AI consulting work around this reality: a working prototype is the easy 20%, and the other 80% is retrieval quality, data pipelines, evaluation, and the production hardening covered in more detail below.
Why “Prompt Engineering” Became Shorthand for the Whole Discipline
Prompt engineering is visible, teachable in an afternoon, and produces something that looks impressive almost immediately. That combination made it the public face of generative AI work, even though it’s the smallest piece of what a production system needs.
The Menlo Ventures data backs this up directly: prompt design is still the most common technique enterprises actually use, which explains why so many generative AI projects plateau exactly where prompting stops being enough. The moment a use case needs current company data, tool use, or reliable multi-step reasoning, prompting alone runs out of runway fast.
The Technical Stack Behind Production-Grade Generative AI
Layer | What It Actually Involves |
Model selection and orchestration | Choosing and routing between models based on cost, latency, and task complexity |
Retrieval and data pipeline (RAG) | Ingesting, chunking, embedding, and indexing company data so answers are grounded in it |
Context engineering | Deciding what information actually reaches the model at each step, and in what form |
Evaluation and monitoring | Testing accuracy and hallucination rates against real queries, then tracking it in production |
Fine-tuning (when it’s actually needed) | Adapting a model’s behavior for a narrow, high-volume task, used selectively, not by default |
Governance and security | Access controls, audit trails, and data handling requirements for the systems involved |
Integration | Connecting the system to the CRM, ERP, helpdesk, or database it actually needs to read and write to |
Prompt engineering touches the first and third rows of that table. The other five rows are where a generative AI consulting engagement actually spends its time.
RAG vs. Fine-Tuning vs. Prompt Engineering: When Each Actually Applies
These three techniques get used interchangeably in casual conversation, and they solve different problems. Microsoft’s Azure Architecture Center guidance on RAG frames retrieval as the right tool when a system needs to reason over data the model wasn’t trained on, which is most enterprise use cases.
Technique | Best For | Not Suited For |
|---|---|---|
Prompt engineering | Shaping tone, format, and instructions for a capable general model | Answering questions that require current or proprietary data |
RAG | Grounding answers in company documents, policies, or records that change over time | Tasks needing a fundamentally different behavior pattern than the base model has |
Fine-tuning | Narrow, high-volume tasks where consistent behavior matters more than fresh data | Frequently changing information; fine-tuned data goes stale the same way documentation does |
Most production generative AI systems use prompt engineering and RAG together, with fine-tuning reserved for the specific cases where it earns its added cost and maintenance burden.
Why Most Generative AI Projects Stall at the Prototype Stage
A prototype built with prompt engineering alone can look production-ready in a demo and fall apart within a week of real use. The pattern is consistent enough that it shows up across every major study on the subject.
- Retrieval wasn’t built or tested at scale. It worked against 50 clean documents and breaks against the actual 10,000-document knowledge base.
- No evaluation harness exists. Nobody can say, with a number, whether the system got more or less accurate after the last change.
- Context management was never designed. The prompt grows unmanageably as more instructions get bolted on, and the model’s performance degrades along with it.
- Governance was assumed, not built. Access controls and audit logging get discovered as missing during the security review that blocks launch.
Perceptive Analytics treats each of these as a specific, checkable item during a generative AI engagement, not an afterthought discovered during a production incident.
What a Generative AI Consulting Engagement Actually Looks Like
A typical engagement moves through architecture review, data and retrieval design, evaluation setup, integration, and production deployment, in that order:
- Architecture review: what exists already, and what’s missing against the stack above
- Data and retrieval design: chunking, embedding, and indexing strategy for the actual knowledge base
- Evaluation harness: a defined, repeatable way to measure accuracy before anything ships
- Context and prompt design: the part most people picture, positioned correctly as one step among several
- Integration: connecting to the real systems the use case depends on
- Production deployment and monitoring: the governance, security, and observability layer that keeps the system reliable after launch
Build, Buy, or Partner for Generative AI Work
Large global consulting and technology firms, Accenture, Deloitte, Cognizant, TCS, Infosys, McKinsey, and PwC, run generative AI programs at enterprise scale across many business units. For a single production system, that scale is frequently more overhead than the project calls for. Our guide on how to choose an AI consulting partner for strategy and automation covers this decision in more depth.
Requirement | Large Consulting/SI Firm | Perceptive Analytics |
Enterprise-wide GenAI transformation program | Strong fit | Not the primary use case |
Single production RAG or agent system | Can be suitable, often over-scoped | Strong fit |
Senior engineer stays hands-on through deployment | Depends on engagement structure | Senior-led by design |
Existing prototype that needs retrieval and evaluation work | Suitable | Strong fit, and a common starting point |
Mid-market budget and timeline | Frequently a mismatch | Built around mid-market scope |
Multi-country governance rollout | Strong fit | Better suited to focused engagements |
Key Takeaways
- Generative AI consulting covers model selection, retrieval, evaluation, governance, and integration, with prompt engineering as a small, early piece of the work.
- Menlo Ventures’ 2025 enterprise survey found prompt design still dominant in production, while RAG, fine-tuning, and context engineering remain comparatively niche.
- Only 16% of enterprise deployments qualify as true agents; most production systems are simpler, fixed-sequence workflows.
- RAG and fine-tuning solve different problems: RAG for current, changing data; fine-tuning for narrow, stable, high-volume tasks.
- Most generative AI prototypes stall in production because retrieval, evaluation, and governance were never built for real scale.
- Perceptive Analytics scopes generative AI engagements around the full stack, not just the prompt layer, and often starts by hardening an existing prototype rather than rebuilding from scratch.
Conclusion
Prompt engineering is the part of generative AI work that’s easy to see and easy to talk about. It’s also the smallest part of what actually makes a system reliable enough to run in production.
If your generative AI prototype looks great in a demo and has stalled since, the missing piece is almost never a better prompt. Perceptive Analytics builds the retrieval, evaluation, and governance layers that turn a working demo into a system your team can actually trust.
Frequently Asked Questions About Generative AI Consulting
Is generative AI consulting the same as prompt engineering consulting?
No. Prompt engineering is one technique used early in a generative AI engagement. The larger share of the work, according to Menlo Ventures’ 2025 enterprise survey, involves retrieval, evaluation, and the production infrastructure that most prompt-only prototypes never get built.
What's the difference between RAG and fine-tuning?
RAG retrieves relevant information from your own data at the moment a question is asked, so it stays current as your data changes. Fine-tuning adapts a model’s underlying behavior on a fixed dataset, which is better suited to narrow, stable tasks than to information that updates regularly.
Do we need an AI agent, or is a simpler system enough?
Most use cases don’t need a true agent. Menlo Ventures found only 16% of enterprise deployments qualify as genuine agents, with the rest built on simpler, fixed-sequence workflows that are easier to build, test, and govern. Start simple, and move to agent architecture only when the use case genuinely requires multi-step, adaptive reasoning.
Why did our generative AI prototype work in testing but fail in production?
Almost always because retrieval, evaluation, or governance was never built for production scale. A prototype tested against a clean sample of documents behaves very differently once it’s pointed at the real, messy knowledge base.
How long does a generative AI consulting engagement take?
It depends on the complexity of the retrieval and integration work involved, not on the prompt design. Ask for a timeline based on your specific data sources and systems rather than a generic figure.
Do we need to fine-tune a model to get good results?
Usually not. Most production use cases get grounded, accurate answers from a strong base model plus well-designed retrieval. Fine-tuning is worth the added maintenance cost only for narrow, high-volume tasks where consistent behavior matters more than access to current information.
Can Perceptive Analytics take over a generative AI prototype another team already built?
Yes, and it’s one of the most common starting points for an engagement. Perceptive Analytics typically starts with an architecture review of what’s already working before recommending what needs to be rebuilt versus hardened.