Questions to Ask an AI Marketing Agency Before Hiring

Every founder we speak to at Absolutely AI has met with three or four AI marketing agencies before us, and every one of them tells the same story: the pitches sound identical, the case studies are thin, and nobody can explain what happens when the underlying model changes next quarter. The old agency vetting playbook (references, portfolios, retainers) still matters, but it misses the layer that determines whether your investment survives contact with reality.

Why the Standard Agency Vetting Questions No Longer Work

Asking an AI agency for a portfolio and a client list tells you what they made last year. It does not tell you which models produced it, who owns the prompts, whether a human read the output before it went live, or whether the agency can still deliver the same work when GPT-5 sunsets and Midjourney rewrites its terms. The gap between an AI studio and a traditional studio is real, but the gap between a serious AI operator and a Canva reseller with an API key is wider.

Good vetting now sits on four pillars: model stack, data and IP posture, human-in-the-loop discipline, and measurable proof in the new search environment. If an agency cannot answer confidently in each area, the retainer will feel cheap for six weeks and expensive for the next twelve months.

Questions About Their AI Stack

Ask which specific models sit behind each deliverable. A capable agency should name them without hesitation: GPT-5 or Claude for copy, Midjourney or a fine-tuned diffusion model for stills, Runway or Sora for motion, Gemini for research, and something opinionated for voice. If the answer is a vague reference to "our proprietary AI," that usually means a thin wrapper around one commercial API with a custom system prompt.

• Which models do you use for copy, image, video and research, by name?
• Which parts of the stack are genuinely proprietary versus commercial APIs?
• What is your migration plan when a model deprecates or a provider changes pricing mid-campaign?
• How do you version-control prompts and fine-tunes so output stays consistent quarter to quarter?

Model-deprecation risk is the question almost nobody asks and every agency dreads. Providers sunset endpoints, rewrite content policies, and shift pricing without warning. An agency that has lived through two or three of these cycles will have a documented answer. One that has not will improvise on your budget, which is a topic worth reading up on in our breakdown of the Australian AI content agency landscape.

Questions About Data and IP Ownership

The most expensive clause in an AI retainer is usually the one that is missing. When the engagement ends, who owns the custom prompts, the brand-tuned model weights, the reference libraries and the generated assets? Default agency contracts often assign IP to the agency and license it back, which sounds fine until you try to leave and discover your brand voice lives on their server.

1. Who owns the prompts, fine-tunes and system instructions built for our brand?
2. Who owns the generated assets on termination, and in what format are they delivered?
3. Is any of our brief data, imagery or performance data used to train third-party models?
4. What is your SOC 2, ISO 27001 and GDPR posture, and can you show the report?
5. Where is our data stored, and which sub-processors touch it?

If the agency handles regulated content (health, finance, legal, government), push harder. Ask specifically whether prompts and outputs are logged with the model provider, whether zero-retention endpoints are used, and whether staff access is scoped or open. Vague answers here are a compliance liability you will inherit.

Questions About Human-in-the-Loop

Pure automation produces content that reads like it was written by pure automation. The agencies delivering work you would actually publish always have a defined ratio of AI draft to human edit, and a named editor accountable for what leaves the building. Ask for that ratio explicitly and ask who the editor is.

• What percentage of a typical deliverable is AI-generated versus human-written or human-edited?
• Who is the named strategist and editor on our account, and are they full-time or contract?
• Is there a pooled account manager between us and the creative team, or direct access?
• Walk me through the last piece you killed in QA and why.

That final question is the tell. An agency that has never killed a piece in QA either has no QA or is not being honest about it. This is one of the clearest differences we cover in AI content agency versus freelancer, and it is usually the deciding factor for brands that have been burned before.

Questions About Proof and Case Studies

Vanity metrics are cheap in an AI-native workflow. Anyone can generate a thousand blog posts and screenshot the impression graph. The useful question is what happened underneath: the starting point, the specific intervention, and the impact on qualified leads or revenue.

• Show me one case study with the starting point, the change you made, and the revenue or pipeline outcome.
• What is your best GEO win? Which brand, which query, which AI engine, how did you measure the citation?
• Can we speak to a client who is currently under retainer, not just a testimonial?
• Ask them to generate a sample against our brief on the second call, live.

The live-sample request separates the operators from the deck-builders instantly. A confident agency will run a real workflow in front of you. A prompt-jockey will find a reason to postpone.

Questions About Pricing and Scope

AI retainers are often priced like traditional agency retainers with an AI discount, which suits the agency and confuses the client. Ask how pricing is actually structured, what triggers overages, and which tool costs pass through. Our guidance on AI content retainer pricing covers the common models in more detail.

• Is this retainer, project-based, or performance-based? What is the mix?
• Which tool subscriptions are included in the fee and which are passed through at cost?
• What triggers an overage, and how is it approved before work continues?
• What is the notice period, and what happens to in-flight work on termination?
• Is there a minimum term, and why?

Pricing at Absolutely AI is quoted per scope after a short brief review; we describe it as a project or retainer engagement depending on cadence. Any agency quoting a headline rate before understanding volume, channels and turnaround is either guessing or padding, and both cost you later.

Questions About GEO and the SEO Transition

Generative Engine Optimisation is where most AI marketing agencies are still bluffing. AI Overviews, ChatGPT search, Perplexity, Gemini and Claude are reshaping how buyers discover brands, and traditional SEO metrics no longer capture the full picture. Ask how the agency actually measures citation frequency across those engines and how they influence it.

• How do you optimise for AI Overviews and generative answer engines specifically?
• What entity, schema and knowledge-graph work do you do, and can you show an example?
• How do you measure citation frequency on ChatGPT, Perplexity, Gemini and Claude?
• How do you attribute AI referral traffic in GA4, and what tools do you use?
• What is your view on the trade-off between traditional rankings and AI citations?

A good answer names specific monitoring tools, describes a schema and entity workflow, and shows an example of a query where the brand is being cited by a named engine. A weak answer is a paragraph about "AI-friendly content" and no measurement layer.

Questions About Compliance and Brand Safety

Hallucinations are a feature of generative models, not a bug that will be patched. The question is what the agency does about it before content ships. Ask for the written policy, not a reassurance.

• What is your hallucination detection and fact-checking process, step by step?
• How do you disclose AI involvement in published content, and does that align with our brand policy?
• What industry-specific guardrails do you apply for regulated categories?
• Who signs off on final output before it reaches our channels?
• What is your process if a piece of published AI content is factually wrong?

Red Flags to Walk Away On

Certain phrases and behaviours reliably predict a bad engagement. If you hear two or more of these in a first meeting, save yourself the second one.

• "We use AI" with no named tools, models or workflow.
• "Proprietary AI" with no technical detail on what is actually proprietary.
• No named editor and no human review step in the workflow diagram.
• Refusal to introduce you to a current client, only past ones.
• Guaranteed results in 30 days, or specific ranking or traffic promises.
• Ownership of prompts, fine-tunes or generated assets remains with the agency after termination.
• No answer for what happens when a core model deprecates.
• The sample work in the pitch deck is visibly generic AI output with no brand calibration.

The 15-Question Checklist

Copy this into your next agency meeting. If you get confident, specific answers to twelve of the fifteen, you are talking to a serious operator.

1. Which specific models do you use for copy, image, video and research?
2. What is your plan when one of those models deprecates or changes terms?
3. Who owns the prompts and fine-tunes built for our brand?
4. Who owns the generated assets on termination?
5. Is our data used to train any third-party model?
6. What is your SOC 2, ISO 27001 and GDPR posture?
7. What is the ratio of AI draft to human edit on a typical deliverable?
8. Who is the named editor accountable for our account?
9. Show me one case study with starting point, intervention and revenue impact.
10. What is your best GEO win, on which engine, and how did you measure it?
11. Can we speak to a current retainer client?
12. Will you generate a live sample against our brief on the next call?
13. How do you measure AI citation frequency and AI referral traffic?
14. What is your hallucination and fact-checking process?
15. What is the notice period and what happens to in-flight work?

Frequently Asked Questions

How long should an AI marketing agency vetting process take?

Two to three weeks is realistic. That covers a discovery call, a technical deep-dive on the questions above, a live sample against your brief, and a reference call with a current client. Anything faster usually skips the reference or the sample, which are the two most predictive steps.

Should I hire an AI-native agency or a traditional agency adding AI?

It depends on the mix of judgement and volume in your brief. AI-native agencies tend to move faster and cost less per unit of output but can be weaker on brand strategy. Traditional agencies with a genuine AI practice are the opposite. Ask both types the same 15 questions and compare the answers directly.

What is a reasonable retainer size for an AI marketing agency?

Retainer sizing depends on channels, volume and turnaround, not on the fact that AI is involved. Ask for a scope-based quote after a brief review rather than a headline monthly figure, and make sure tool pass-through costs and overage triggers are written into the agreement.

How do I know if the agency actually uses AI or just claims to?

Ask them to run a workflow in front of you. Not a demo video, a real generation against your brief. Operators will do this on the second call. If they resist, the workflow is not repeatable enough to show a prospect, which means it is not repeatable enough to run your account either.

What is GEO and why does it matter more than SEO now?

Generative Engine Optimisation is the practice of getting your brand cited inside AI answers on ChatGPT, Perplexity, Gemini, Claude and Google's AI Overviews. It matters because buyers increasingly ask AI engines the questions they used to type into search, and if you are not cited in the answer, you are not in the consideration set. It does not replace SEO, it sits alongside it.

Who should own the prompts and fine-tunes built for my brand?

You should, on termination at minimum. Prompts and fine-tunes encode your brand voice, tone and content patterns. Leaving them on an agency's server means the next agency has to rebuild that IP from scratch, which is expensive and slow. Get this written into the agreement before signing.

How do I measure AI citation frequency across engines?

Use a combination of dedicated monitoring tools (Profound, Otterly, Peec AI and similar), manual query sampling on a fixed cadence, and GA4 referral segmentation for traffic tagged from AI engines. A capable agency will already have a stack in place and will show you the dashboard, not describe it.

What should a first-week deliverable look like from a serious AI agency?

An audit of your existing content and brand voice, a technical GEO baseline showing your current citation footprint, a documented prompt and workflow spec for the first live piece, and a named editor introduction. If week one is just a kickoff deck, the engagement will drift.

The Short Version

The AI marketing category is going to get more crowded, not less, and the gap between the operators and the opportunists is going to widen. The questions above are the fastest way to tell which side of that line an agency sits on before you commit budget. If you want to see how we answer them ourselves, the team at Absolutely AI is happy to run the interrogation live on a call, sample brief included.