Research questions

1. Interpretability

   Can you find a feature in a small open model that tracks a simple concept?

   Interpretability tries to understand what is happening inside a model. Finding and testing a single feature, such as one that tracks a language, a topic or a kind of error, is a manageable first project that teaches the core tools.

   Start with: Neuronpedia, ARENA interpretability exercises.

2. Interpretability

   Do truthfulness probes in an open model generalise across topics?

   Researchers have trained simple probes that seem to detect whether a model “believes” a statement is true. Testing whether a probe trained on one topic still works on another tells us how much we can rely on them.

   Start with: The Geometry of Truth (Marks and Tegmark), ARENA interpretability exercises.

3. Evaluations

   How often do models give in when a user pushes back, and does model size matter?

   Models sometimes change a correct answer just because a user disagrees, which is called sycophancy. Measuring it across several open models is a clear, repeatable experiment that needs little more than an API and a spreadsheet.

   Start with: Towards Understanding Sycophancy in Language Models, Inspect evaluation framework.

4. Evaluations

   Do models behave differently when a prompt suggests they are being tested?

   If models act differently when they think they are being evaluated, safety tests could give a misleading picture. A small study comparing behaviour on test-like and natural prompts is a useful contribution.

   Start with: Large Language Models Often Know When They Are Being Evaluated, Inspect evaluation framework.

5. Chain of thought

   If a model’s reasoning is shortened or reworded, does its answer change?

   Reading a model’s step-by-step reasoning is one of our best ways to monitor it, but only if that reasoning actually drives the answer. Editing the reasoning and watching what happens is a direct test of this.

   Start with: Measuring Faithfulness in Chain-of-Thought Reasoning, Language Models Don’t Always Say What They Think.

6. Governance

   What do frontier AI companies’ safety frameworks commit to, and where do they differ?

   Several leading AI companies have published frameworks describing when they would pause or add safeguards. A clear side-by-side comparison helps policymakers and researchers see where commitments are strong and where they are vague.

   Start with: METR research on frontier AI safety policies, 80,000 Hours: AI governance.

7. Governance

   How do the UK, EU and US define frontier or general-purpose AI, and what does each definition leave out?

   Laws only cover what they define. Comparing definitions across countries shows which systems fall through the gaps, and needs careful reading rather than technical skills.

   Start with: EU AI Act explorer, UK AI Security Institute.

8. Compute governance

   What would it take to track where large AI training runs happen?

   Training the most capable models needs huge amounts of computing power, which makes compute one of the few things governments can measure. Reviewing proposals for tracking it, and their practical problems, is a strong starter policy project.

   Start with: Computing Power and the Governance of AI (Sastry et al.), BlueDot Impact AI governance course.

Got a question you think belongs here? Mention it when you sign up and we’ll consider adding it.