When I click “run” on ordinary software, I can point to every rule in the code and explain what happens next. After 22 years in digital marketing, that predictability is something I’ve come to expect.
But with modern AI, we’re in uncharted territory. Even the creators admit they can’t trace every step. These systems aren’t hand-built – they’re grown. Engineers set the conditions, the model absorbs oceans of data, and a vast tangle of digital “neurons” organises itself in ways nobody fully maps. It’s more like tending a wild garden than wiring a dishwasher.
That uncomfortable truth makes me look at AI with a growing level of unease and drives the industry push for interpretability – tools that would let us look inside the black box.
The Unpredictable Output Problem
Here’s something I’ve experienced firsthand that might sound familiar: Ask an AI the same question twice, and you’ll often get two different answers. Why? Because our prompts aren’t specific enough.
Think about it – if I ask one of my team members to “create a report on our Google Ads performance,” I might get widely different results depending on who I ask and what mood they’re in that day. Without specifics like “show me last month’s conversion rates compared to the previous three months, focusing on our top 5 campaigns,” there’s room for interpretation.
AI works the same way, except it has an even broader range of possible responses. We’re dealing with a fundamentally random environment that won’t give consistent outputs unless properly constrained.
This means if we want reasonably reliable results, we need to invest time in prompt engineering – essentially bolting down the AI environment to deliver exactly what we need, each and every time. For businesses integrating AI into their workflows, this isn’t optional – it’s essential.
What the insiders are saying
I’ve been following this conversation closely, and it’s eye-opening what the industry leaders themselves are admitting:
“Generative AI systems are grown more than they are built… When they do something, we have no idea, at a precise level, why they make the choices they do.” — Dario Amodei, CEO Anthropic, The Urgency of Interpretability, April 2025
“How do we ensure we can stay in charge, control them, interpret what they’re doing, and put the right guardrails in place? That is an extremely difficult challenge.” — Demis Hassabis, CEO Google DeepMind, TIME100 interview, May 2025
Why this matters for NZ businesses
After talking with our clients about this, I see four main risks and practical steps you can take today:
| What could go wrong | What you can do today |
|---|---|
| Random surprises – A chatbot hallucinates facts or injects bias you never coded. | Treat every AI output as a draft, not gospel. Keep human eyes on customer-facing or legal decisions. |
| Inconsistent outputs – The same prompt produces different results each time, making processes unreliable. | Develop detailed prompt templates with specific instructions, examples, and constraints for business-critical tasks. |
| Regulatory blind spots – NZ Privacy Act & Algorithm Charter lean on explainability. For instance, if you can’t show why the model said “no” to a loan, you may face scrutiny. | Ask vendors for audit logs and external safety reports. If they shrug, shop around. |
| Tech lock-in – Black-box tools make it hard to switch providers or prove compliance later. | Prefer AI products that publish a road-map to interpretability (many now highlight this in FAQs). |
The road ahead – in plain language
Having watched marketing technology evolve over two decades, here’s my take on the timeline:
- Next 18 months: Labs like Anthropic and OpenAI will ship “thinking-out-loud” modes that reveal a simplified chain of thought. Nice for debugging, but still only showing us part of the picture.
- By 2027 (best case): Researchers hope for an “MRI for AI” – automated scans that flag dangerous circuits (deception, power-seeking) before a model ships.
- Policy tail-wind: The EU AI Act already mandates explainability for high-risk uses; Australia is consulting; Wellington is watching both moves closely. Expect similar clauses to land in big-client contracts here in New Zealand.
Take-away for Kiwi owners & marketers
AI can feel like magic, but it’s really maths that even the magicians don’t fully read yet.
My advice? Enjoy the productivity boost, but:
- Keep the human double-check
- Log what you feed the model and what comes back
- Develop standardised prompts for repetitive business tasks
- Create a prompt library for your organisation’s common needs
- Quiz your suppliers on how they plan to open the black box
That healthy scepticism, combined with diligent prompt engineering, forms your safety net while the scientists race to build better x-ray goggles. And in business, as in marketing, sometimes the strongest position is admitting what we don’t know – and then methodically controlling what we can.
