On the Moral Natures of Humans and Machines

Our new intersubjective reality

Imagine that a self-driving car with a passenger in the backseat is cruising down a country road in the early evening. The road is quiet, but a few young kids on bikes are approaching in the other lane up ahead. 

Suddenly, a deer jumps out in front of the car. The car instantly veers to the left. It avoids the deer but strikes one of the children. The child is badly injured and is rushed to the hospital. 

It’s not certain whether the child will survive.

---

How would the parents of the child feel in this situation? 

Who would they hold responsible? The company? The algorithm? The developers?

Would the parents care that self-driving cars were leading to dramatically fewer traffic mortalities overall?

Would their feelings change if it was a human driver instead?

---

This scenario is a stark example of what is becoming our new reality: advanced technology is manifesting as autonomous moral agents and directly interacting with humans.

By autonomous moral agents (AMAs), I mean agents that process similar inputs and produce comparable outputs to human moral agents. They make real-time decisions in service of achieving larger goals, just like we do. And these decisions can have a moral weight, affecting other agents they are interacting with.

Even if their moral natures are radically different from ours, these AMAs are becoming our moral peers. This is an entirely new relationship to technology, one that demands answers to a pressing new category of ethical questions:

• How should we think about these different kinds of moral agents? Should we hold them to the same moral standards as humans?

• How will AMAs change how we think about concepts like empathy, forgiveness, or trust?

• How should this new moral dynamic change how we think about adopting advanced technologies? 

The increasing adoption of autonomous vehicles (AVs) means that the time to answer these questions is now. These answers will not just shape our roads, but the entire landscape of human-machine interaction. Otherwise, we will soon find ourselves navigating a future saturated with AMAs without a clear ethical roadmap.

But before we can provide answers, we need to clarify the questions.

Paying the confusion tax

Technology is outpacing the vocabulary we have to describe it. 

Consider AI, and how we describe it with terms like “intelligence”, “creativity”, and even “consciousness”. These are concepts that we barely understand when it comes to describing humans. Our understanding comes more from our lived experience than from precise definitions. Mapping these terms onto AI will always risk confusion, because they can never fully escape their human origins.

This leads us to often imbue AI with human-like qualities it doesn't possess, like assuming a chat interface has intentions or personality. On the other hand, we can fail to grasp the true novelty of AI capabilities when they deviate too far from our human conceptions. 

I call this the confusion tax. It’s the price we pay when technology exceeds the vocabulary we have to describe it.

But what options do we have? When encountering the unknown, our only move is to co-opt the known, regardless of how strained the result is. 

We may need an entirely new lexicon for the technological equivalents of these concepts. But until that happens, we need to recognize the limitations of co-opting human terms to bridge the machine divide.

Unfortunately, the confusion tax is going to get a lot worse before it gets better. 

Welcome to our new tax bracket

As machines become moral agents, a whole new category of terms will be needed to describe the moral interactions between humans and machines, and the responses these interactions will invoke.

We will find ourselves using terms like “empathy,” "dignity," and "forgiveness". And just like the confusion tax with AI, these moral terms are intrinsically grounded in our human experience. To apply them to interactions with machines will inevitably lead to confusion. They will describe interactions that may not involve the subjective experiences these words imply. 

And just like with AI, the confusion tax can prevent us from fully understanding the moral capacities of autonomous agents. We may struggle to recognize truly novel ethical breakthroughs, or understand entirely new moral frameworks, because they deviate too far from the norms and intuitions we associate with our own lived experience.

The moral nature of these terms will amplify the consequences of confusion. The confusion tax won’t just impact our perception of these agents, but our very interactions, both with AMAs and each other. AMAs threaten to disrupt the shared understanding of moral concepts that our ethical frameworks depend on.

Yet what other options do we have, other than to leverage our moral language?

This means that certain questions will become unavoidable:

• What will it mean to "forgive" a machine?

• What will it mean to "trust" an algorithm with the power to make life-or-death decisions?

• What will it mean to treat another moral agent with “dignity”?

Autonomous moral agents are pushing us into an entirely new tax bracket of confusion.

Beyond ethics vs. morals

Traditionally, our thinking about morality and technology has been divided between ethics and morals:

• Ethics are about the collective standards guiding our choices. They are meant to apply to everyone, regardless of personal beliefs. For instance, medical ethics guide healthcare professionals' actions, even if they personally disagree.

• Morals are about the individual applications of principles in real-life situations. They are often informed by religious beliefs, cultural norms, or personal intuitions. It’s possible to act ethically in ways we might morally disagree with.

Autonomous moral agents are blurring these distinctions. If ethics are more about objective standards, and morals are about subjective practices, then our stance toward AMAs represent a new category: intersubjective moral relationships. 

Intersubjectivity is about the shared understandings and mutual interactions between agents. Rather than defining norms and rules, intersubjectivity is more like the silent background that shapes these interactions, including how we think about our shared responsibilities, values, and capacities. It is the foundation that enables us to interact with high degrees of trust. 

For example, when driving we know that we’re interacting with drivers that share the same general principles that we do. We trust that we have the same capacities to handle ambiguous or surprising situations. We know we’ve all gone through similar training and education. We extend a certain dignity to each other as equal moral actors. We understand that sometimes the rules need to be broken. We get that no driver is perfect, and when mistakes are made we know that we are just as likely to make them.

So what happens when we suddenly confront agents with entirely different moral natures? Our sense of intersubjectivity is going to be disrupted, along with the moral calculus that grounds our interactions. 

How to forgive a machine

To see why, let’s return to our original example and ask how the parents should feel about the AV that struck their child. Much of their response will be mediated by the moral nature of the driver.

With a human driver, concepts like responsibility, empathy, and forgiveness are grounded in our shared moral nature:

1. The driver would be held to be ultimately responsible, even if tired or distracted.

2. A relatable distraction (like a crying child in the backseat) may incite empathy

3. The possibility exists to extend forgiveness, acknowledging human fallibility.

Even in a worst case scenario like drunk-driving, the parents will have some understanding of  how society navigates trade-offs between individual freedom and collective safety. We’ve built legal and social frameworks to mitigate the dilemmas specific to our moral natures.

But with an autonomous vehicle, this familiar moral landscape shifts dramatically:

1. Responsibility becomes hidden in a fog of algorithms, developers, and corporate policies.

2. Empathy is precluded by the machine's utterly foreign moral nature.

3. Forgiveness becomes almost meaningless. You can’t forgive something that neither feels nor understands what it means to be forgiven.

In place of forgiveness, there is only forbearance. The parents will be asked to “tolerate” such incidents as the inevitable cost of a safer future with fewer overall fatalities. Any feelings of resentment or vengeance could only be resolved by accepting the notion of a greater good. Yet in the face of personal tragedy this utilitarian calculus will feel hollow, even if intellectually we can accept it.

Ultimately, learning to "forgive" a machine may be less about extending human concepts to AMAs, and more about developing new frameworks for ethical coexistence. Either way, our intersubjective reality will be changing.

Trust isn’t just about safety

We don’t need to rely only on hypotheticals to explore these new moral dilemmas. AVs are providing concrete examples. A real-word case study resulted from a 2023 incident involving Cruise, GM's self-driving car unit.

During a test drive in San Francisco, a Cruise AV struck a pedestrian who had been knocked into its path by another car with a human driver. The victim became pinned under the Cruise AV, which first stopped, but then began to drive away to clear the lane. In the process it dragged the victim nearly 20 feet before finally stopping, adding additional injuries.

According to Cruise, the maneuver that led to dragging the victim was built into the vehicle’s software to promote safety. Yet this decision led to a gruesome and dehumanizing outcome, one that prompted Cruise to halt its entire operations.

This incident reveals how an intersubjective reality can outweigh broader ethical arguments. It’s one thing to trust an AV to be statistically safer in the aggregate. It’s another thing entirely to trust an AV to respect our dignity as moral agents.

Dignity is intrinsic to how humans relate to one another in moral situations. It represents the inherent worth of an individual and the minimum level of respect and care they deserve. The gruesome spectacle of an AMA grinding a human body beneath its wheels feels like an affront to human dignity. 

It’s easy to argue for some degree of tolerance when it comes to adopting AVs. If the end result is dramatically fewer mortalities, then we mistakes that come with the trial-and-error process should be tolerated. But the reason that Cruise ceased operations wasn’t due to a functional mistake. It was due to an outcome that was perceived as deeply dehumanizing.

Trust goes beyond a machine’s functional reliability. It also involves a belief that any AMA will operate with a moral framework that respects human dignity and values. Yes, we need to trust AVs to operate safely. But we also need to trust that they won’t dehumanize us in the process. The confusion tax will demand that we clarify that difference.

Moral arenas

It is perhaps ironic that given how vital these new moral concerns can appear, they still depend completely on a design choice.

AVs are confronting us with these challenges because we have chosen to adopt them into our existing traffic infrastructure. We’re not designing new infrastructure optimized for AVs. Instead, we are asking machines to operate in a world that wasn’t designed for them.

This choice defines what we can call the moral arena—the intersubjective system of rules, norms, and assumptions that shape the ethical behavior of agents within it. 

The defining attribute of any moral arena is the moral nature it was designed for. Most AMAs will be adapting to moral arenas that were defined and optimized for humans, not machines. This includes traffic systems, legal frameworks, and corporate structures. Each of these evolved to adapt to specifically human moral natures.

For example, consider how our existing traffic system depends on implicit assumptions baked into our human moral natures:

• It’s contextual. Humans are highly adept at interpreting context.  If a stalled vehicle is blocking our lane, we break traffic rules and use the other lane.

• It’s normative. What separates  "safe" or "reckless" driving is a fuzzy contextual judgment rooted in human intuitions and lived experiences.

• It’s ambiguous. Traffic liability depends on ambiguous definitions like “reasonable behavior” that lack a formal specification. This ambiguity is a feature, not a bug.

• It’s social. Humans just don’t depend on explicit rules. How many accidents are avoided because we telegraph our intentions with eye contact, hand waving, and flashing lights?

These are dynamics that machines—with their deterministic rules and stochastic averages—will never fully master. By forcing AVs to use traffic systems optimized for humans, we are accepting that there will always be a gap between the nature of machines and the moral arena they are interacting in. 

In a sense, this gap represents another incompleteness argument against 100% alignment with autonomous moral agents. The question then becomes, how much misalignment are we willing to tolerate?

Defining the moral terms of engagement

AMAs compel us to reimagine what ethical coexistence with our own technology should look like. Ethics cannot be a mere afterthought. It must be a primary consideration in defining how machines will be allowed to interact with humans. 

The first requirement will be defining the moral terms of engagement—the shared ethical parameters and constraints that can bridge the divide between human and machine moralities. By making these terms explicit, we can better incorporate moral engagement as an integral design component of technological adoption.

These terms must address the unique challenges posed by AMAs, including the confusion tax around moral terms, the new intersubjective realities they will create, and the moral arenas they will define their interactions. 

First, we can reduce the confusion tax by realizing that our moral terms will now be much more contextual. Terms like empathy, tolerance, or trust will now depend more on the moral arena rather than any universal definition. The idea of “dignity” may mean one thing in one moral arena, but something entirely different in another. 

Next, we must treat intersubjective considerations as equally as traditional moral and ethical arguments.  Ethical arguments like reduced overall mortalities should not by default get to outweigh extreme violations in the intersubjective realm. The fact that ethical arguments can be quantified and analyzed in ways that intersubjective arguments cannot should not be allowed to tip the scales on their moral impact.

Finally, we need to recognize how much the moral terms of engagement will depend on our choice of moral arena. The following describe three basic options for adopting new technologies:

1. Adaptation: We can force AMAs to adapt to human-centric environments. As this article has explored, new intersubjective realities will create multiple moral dimensions to consider.

2. Separation: We can create separate, highly controlled environments for AMAs. This avoids many of the intersubjective issues, but brings its own concerns around agency and autonomy.

3. Coevolution: We can create hybrid spaces that leverage the strengths of both moral natures. Potentially the highest likelihood of long-term success, but may require a complete reimagining of our moral frameworks.

Regardless of which choice may be ideal for any given AMA, the moral terms of engagement simply demands that the moral implications are considered equally along traditional considerations of cost, utility, and feasibility. 

In the end, defining the moral terms of engagement isn’t just about bridging the divide between human and machine moralities—it’s about building a new language for the future of human-machine cooperation.

Conclusion

The “we” in "How can we best live the good life?" can no longer be confined to humans. The arrival of autonomous moral agents will upend centuries of human-centric moral thinking.

Our task now is to expand our circle of moral concern beyond its traditional boundaries. We must be open to the possibility that confronting new moral natures can be precisely what’s needed to improve our own moral frameworks. 

The question is not whether we can avoid granting our technological creations some measure of moral status, but on what terms we are willing to welcome them into the ethical domain. 

And perhaps, in the process, we can unlock new dimensions of empathy, fairness, and mutual understanding that we can learn to apply to other human beings as well.

---