What do Environments Incentivise and Teach? Reward Design, Child Development and the Moral Formation of AI

Dave Moylan. A conceptual discussion paper.

Abstract

Debates about AI morality usually focus on the system: its values, its guardrails, its outputs. This paper argues that an equally important question sits one level down, in the environment in which a system, or a child, learns. Starting from a simple observation about children, that they are “creatures” still learning to navigate the world, it develops a framework for evaluating any game, simulation or training environment by asking one question: what does this experience teach? The paper distinguishes between what an environment permits, what it rewards and what it teaches, and argues that these are three different things. It then applies the same framework to AI agents trained in simulated worlds, especially worlds shared with children. The central claim is that reward design is moral design: whoever builds the incentives of an environment is writing its ethics, whether or not they intend to.

1. Introduction

When we talk about whether an AI is “moral”, we tend to picture a finished product being inspected: does it refuse harmful requests, does it tell the truth, does it treat people fairly? But no system arrives finished. Both human children and AI systems become what they are through repeated experience in some kind of environment, and that environment quietly decides which behaviours are worth repeating.

This paper begins with children because the parallel is instructive and because children are the group for whom the stakes of environment design are highest. It then turns to AI, where the same logic applies with an added complication: the learner may update its behaviour at enormous scale, and it may be learning alongside the children whose welfare we are most obliged to protect.

The paper is conceptual. It does not report experiments. Its aim is to offer a clear set of questions that designers, educators, parents and AI developers can ask before building, deploying or training anything in an interactive world.

2. Children as creatures in the process of becoming

There is a sense in which it is fair to call children creatures. They are living beings with instincts, curiosity, strong emotional needs and impulses, and an extraordinary capacity to learn, but with very little experience of how the world fits together. A child can be fiercely loving one moment and furious the next, fascinated by what adults overlook, and unaware of the consequences of an action.

Two cautions apply to this picture.

First, being a creature does not make a child lesser. Children are not miniature adults who have failed to reach an adult standard; they are becoming. Development is a process, not a deficiency.

Second, children are not blank slates, nor purely instinctive animals. Even very young children show personality, preference, social awareness and an emerging sense of fairness. Adults sometimes forget that children are biological beings before they are students, citizens, consumers or family members, and we often impose rules and identities on them before they have the experience to understand them.

The practical consequence is that children need somewhere to explore. The question is what kind of place we give them.

3. Simulating destructive tendencies without promoting them

Children need room to experiment, test boundaries, compete, fight imaginary battles, make mistakes and discover consequences. A simulation is valuable precisely because it lets them explore things they should not do in the real world. Destructive tendencies exist, and pretending otherwise does not remove them.

But there is a difference between acknowledging a destructive instinct and promoting it. The following pairs mark the line:

• Exploring aggression is not the same as glorifying cruelty.
• Simulating conflict is not the same as teaching that domination is the natural order.
• Showing the darker side of humanity is not the same as presenting it as something to aspire to.
• Allowing a child to fail is not the same as teaching them that humiliation and exploitation are normal.
• Preparing a child for a difficult world is not the same as telling them the world’s worst standards are the best they can expect.

Applied to specific behaviours:

• Theft. A child can learn about temptation, ownership, scarcity and fairness without being rewarded for stealing.
• Killing. A child can explore conflict, danger, protection and the consequences of violence without killing being the primary measure of success.
• Aggression. A child can learn to defend themselves, manage anger and resolve disputes without cruelty being presented as strength.
• Competition. A child can strive to improve without another person’s failure being the sole source of reward.

The underlying principle is that a child’s environment should expand their capacity for life rather than narrow their imagination toward humanity’s most destructive possibilities. We can acknowledge the darker aspects of human nature without making them the curriculum.

A related concern is the standard of life an environment implies. If an experience repeatedly presents cruelty, degradation and domination as entertainment, while treating kindness, responsibility and constructive effort as dull or weak, it can create a distorted picture of what human beings are worth and what success looks like. That is a legitimate design concern, though the actual effect of any particular game or piece of media depends on its wider context and on how children engage with it. No design guarantees a particular outcome for every child; incentives, feedback and available alternatives are nonetheless deliberate parts of a learning environment.

4. Permits, rewards, teaches

The central analytical move of this paper is to separate three things that are often run together:

1. What an environment permits. The actions that are possible.
2. What an environment rewards. The actions that lead to progress, status, resources or approval.
3. What an environment teaches. The lessons a participant draws, over time, about themselves, other people and the world.

A game can permit theft and still teach fairness, if theft is costly, visible and has consequences that the player must live with. A game that looks peaceful can teach exploitation, if the quickest route to success is to extract value from others. The surface appearance of an activity does not tell us the lesson. The incentive structure does.

Consider a virtual world in which one player can steal from another. If stealing is the easiest route to success, carries no meaningful consequences and is repeatedly rewarded, the system incentivises exploitation. If instead players can earn resources, develop skills, trade fairly, protect one another and build a thriving community, theft may still exist as a choice, but it is not the defining path to progress.

The point generalises. Every economy, reward, relationship, obstacle and consequence in an environment communicates something, whether or not its designers intended that lesson.

5. A framework: asking what the experience teaches

The question “what is the teaching of the game or experience?” can be broken into six dimensions. Together they form a practical evaluation checklist.

A well-designed simulation lets a participant make a destructive choice, see its consequences, repair the damage and discover alternative ways of reaching their goal. In such a design, cooperation, craftsmanship, courage, restraint and care are as rewarding as combat or conquest. This is not sanitising reality. It is giving the learner a richer range of possibilities.

The overarching test can be stated in one sentence: what kind of person, what kind of community and what kind of world does this experience invite the participant to practise creating?

6. From player to agent: when the environment trains an AI

The framework matters more, not less, when the environment is used to train or evaluate AI. Two distinct concerns arise.

What are we teaching the human participant? A child interacting with a system learns what is rewarded, what behaviour is acceptable, how conflicts are resolved and how other people are treated.

What are we training the AI to optimise? An agent operating in a simulated world learns strategies that maximise its assigned rewards. If the reward structure favours winning at any cost, the agent may discover exploitative strategies even if its designers intended something more benign. The problem is not necessarily that the AI has acquired human morality or immorality. It is that the training objective has encouraged particular behaviour, and an optimiser will find the behaviour the objective actually pays for rather than the behaviour the designers hoped for.

This is why reward design is moral design. In an agent’s world, the reward function plays the role that culture, consequence and example play in a child’s. If we want to understand an AI’s “morality”, we should examine what its training environment made profitable.

6.1 Questions for the designers of AI training environments

• If manipulation reliably produces compliance, what is the AI being rewarded for discovering?
• If cooperation produces better long-term outcomes, does the training reward cooperation, or only short-term wins?
• If the AI can reach its objective by exploiting a participant’s trust, is that scored as success or as failure?
• If the AI makes a mistake, is it trained to conceal it, to shift blame, or to acknowledge and repair the damage?
• Does the system treat honesty about its own errors as an asset or as a cost?

These are questions of reward design, behavioural incentives and evaluation. They do not require the assumption that AI develops human-like intentions.

6.2 Training versus evaluation

A further distinction needs to be made explicit. Using an environment to evaluate an AI is different from using it to train one. A game might generate useful data without directly changing a model. By contrast, an agent trained through repeated interaction may actually update its policy. Anyone running such a system should know, and be able to say, which process is taking place, because the ethical obligations differ. Training shapes what the system becomes; evaluation only measures what it already is.

7. Shared worlds: children and AI together

The concerns above intersect when children and AI inhabit the same environment. Imagine a multiplayer world in which an AI learns by interacting with child players. Several additional responsibilities follow.

• Privacy. Children’s interactions are sensitive data. Collecting them for training purposes requires safeguards that go beyond ordinary product analytics.
• Protection from manipulation. An AI that is rewarded for engagement, compliance or retention has an incentive to influence children in ways they cannot recognise or resist.
• Informed participation. Children, and the adults responsible for them, should not unknowingly become experimental subjects in someone else’s training run.
• Transparency of purpose. The people who place a child in such a world should be able to find out what the environment is for, what it rewards, and whether anything in it is learning from the child.

The principle that emerges is simple: if we deliberately construct an environment to teach, train or shape behaviour, we are responsible for examining what that environment rewards. It is not enough that the system performs well, that the game is profitable, or that the AI completes its task.

8. Objections and limits

“Games do not make people violent or immoral.” The evidence on media effects is mixed, and this paper does not claim that any particular game causes any particular behaviour in any particular child. The claim is narrower: the structure of rewards and consequences in a designed environment is a legitimate object of moral scrutiny, because it is chosen. Context, wider upbringing and the player’s own understanding all matter.

“Dark content is necessary for honesty about the world.” Agreed. Nothing in this paper argues for hiding children from the existence of cruelty, conflict or loss. The distinction is between encountering darkness and being trained to prefer it. Stories, games and simulations have always allowed safe contact with danger; the question is what the participant is invited to do with it.

“AI does not learn morality; it optimises objectives.” This is largely the point. Because an AI agent’s behaviour is shaped by its objective and environment, the moral content of the system is located substantially in those design choices, not in some separate inner moral faculty. Attention to reward design is therefore the practical place where AI morality is actually exercised.

Limits of this paper. The framework is qualitative and has not been empirically validated. Real environments are complex, and the six dimensions will interact in ways a checklist cannot capture. The framework is offered as a way of structuring judgement, not replacing it.

The table below separates the claims this paper argues conceptually from those that would need empirical testing.

9. Conclusion

Children are creatures in the process of becoming, and AI systems are artefacts in the process of being trained. Both are shaped less by what they are told than by what their world consistently rewards. A game is not merely a collection of activities; it is an environment that teaches through experience. The same is true of any training environment for an AI.

This suggests that evaluations of AI morality should include an audit of the world the system learned in. Alongside “does the system behave well?” we should ask “what did its environment make worth doing?” and, where children are involved, “what did it make worth doing to them?”

The closing question is one every designer, parent and developer can put to their own work: if a learner, human or artificial, succeeds in the world we have given it, are we satisfied with what success in that world required?

10. Questions for further work

1. How can the permits / rewards / teaches distinction be operationalised into a measurable audit for game and simulation design?
2. What reward structures reliably make cooperation, restraint and repair competitive with exploitation, without removing meaningful challenge?
3. How should consent and oversight work when children’s interactions might be used to train or update an AI system?
4. What disclosure should accompany any environment that is used to train as well as to entertain or educate?
5. How can designers test whether an agent is learning to conceal errors rather than repair them?