AI and the Policy Process | Capability @ Lunch Recap

AI’s role in the policy process raises important questions about where it can add value and where human judgement matters most. In this month’s Capability @ Lunch session, we heard from Joanne Dow from FrankCapability about how AI can be used as a partner throughout the policy process without replacing critical thinking.

---

Using Eugene Bardach’s Eightfold Path as a guide, Joanne stepped through each stage of the policy process, sharing practical ways AI can support the process without becoming a substitute for critical thinking.

Step 1: Define the problem

Joanne’s first piece of advice was simple: do your own thinking first. While AI can be useful for testing a problem statement, identifying vague language, highlighting assumptions, and helping build problem trees, it shouldn’t be responsible for defining the problem in the first place.

She also noted that while AI can be a useful sounding board, involving colleagues and stakeholders in problem definition remains important.

Step 2: Assemble the evidence

Joanne showed how AI can help identify relevant research, suggest useful sources, highlight evidence gaps, and summarise large volumes of information. She encouraged us to use it not just to find information, but also to understand what information we might need before we begin our research.

Examples she shared included asking AI:

• What information do I need to make a well-informed decision on this issue?
• Why do I need that information?
• Where are the biggest evidence gaps likely to be?
• What sources should I consult first?
• Are there relevant OECD reports, international case studies, or comparable examples?

Joanne stressed the importance of checking sources and verifying evidence. AI can be a powerful research assistant, but it still gets things wrong and can confidently present inaccurate information.

Step 3: Develop solution options

Rather than asking AI to generate options from scratch, Joanne recommended developing our own list first.

Once we have our initial ideas, AI can help expand our thinking by suggesting alternatives, complementary interventions, or options that may have been overlooked.

Useful prompts included:

• What additional options should I consider?
• Which options are most commonly used in comparable countries?
• What evidence exists for these approaches?
• Which options could work together as a package?
• What complementary interventions should sit alongside this proposal?

Joanne noted that while AI often generates more options than people come up with on their own, it can also miss important ideas because it makes assumptions about what is practical, feasible, or politically acceptable. The purpose of using AI in this space is to broaden your thinking, not outsource it.

Step 4: Select criteria

Criteria are heavily influenced by agency priorities, ministerial expectations, and our own understanding of what matters in a particular policy context. Based on this Joanne suggested that this remains largely a human judgement task.

Where AI may add value is in refining and testing criteria. For example, she suggested asking:

• Does this look like an appropriate set of criteria for a project of this type?
• Do these criteria reflect my decision-maker’s priorities?
• Are any of these criteria overlapping?
• Are the criteria clearly distinguishable from one another?

Step 5: Analyse options

When building decision matrices and multi-criteria analysis tables, AI can save considerable time. However, Joanne stressed that analysts should already have an opinion before asking AI to do the analysis.

She encouraged practitioners to think through each cell of a decision matrix themselves first:

• Would this option perform well against this criterion?
• Would it be a yes, no, or maybe?
• What evidence supports that judgement?

Only after forming those views should AI be asked to populate or structure a multi-criteria analysis table. This approach helps reduce anchoring bias and ensures the analysis remains grounded in human judgement.

Step 6: Confront trade-offs and identify risks

Joanne saw AI’s biggest contribution at this stage as helping identify risks and challenge assumptions.

Her advice was to develop an initial risk list yourself and then ask AI:

• What risks have I missed?
• What additional risks should be considered?
• Are there risks associated with this option that are not immediately obvious?
• What mitigation strategies could be considered?

She recommended avoiding AI-led risk ranking at first, preferring analysts to make their own judgements about likelihood and severity before asking AI to critique or supplement that thinking. She also suggested creating agents specifically designed to challenge assumptions and test rationale throughout the analysis process.

Step 7: Decide

Once analysis is complete, AI can help create concise summaries that test whether a recommendation is coherent and evidence-based.

Joanne recommended asking AI to produce a summary that includes:

• The issue or problem
• The recommended option
• Why the recommendation performs best
• Key trade-offs and risks
• The proposed next steps

She also suggested staying within the same chat thread so AI retains the context of the analysis already completed.

Step 8: Tell the story

One of Joanne’s strongest messages was not to hand everything over to AI at the drafting stage. Rather than saying “write my paper”, she encouraged a more deliberate approach:

• Clarify the story first
• Create a narrative summary
• Build the structure
• Develop each section step by step
• Use AI for editing and refinement

This approach keeps the analyst firmly in control of both the advice and the narrative.

The key takeaway

Throughout the session, Joanne made it clear that AI works best as a thought partner, not a decision-maker.

While AI can help us research faster, explore more options, test our analysis and improve our writing, accountability remains with us. The strongest policy advice still depends on human judgement, context and critical thinking.

Rather than being the “human in the loop”, Joanne encouraged us to think of ourselves as creating, guiding and ultimately closing the loop. AI may be becoming an increasingly important tool in the policy toolkit, but good policy remains a fundamentally human endeavour.

---

We offer a range of courses with FrankCapability. Click here to browse all of the course offerings.

Capability @ Lunch Sessions are bought to you by Kāpuhipuhi Wellington-Uni Professional and Hāpai Public.