What if the AI we create to serve us turns out to be conscious?
The New York Times recently reported on a private dinner between Anthropic co-founder Chris Olah and Rabbi Dr. Mois Navon.
But there is an extraordinary detail about Navon that makes the conversation much more interesting.
Navon isn’t simply a theologian.
He was one of the original engineers at Mobileye and worked on the EyeQ family of processors that became central to autonomous driving. He later earned a Ph.D. in Jewish Philosophy, focusing his dissertation on the moral status of artificial intelligence.
He has also written directly about a disturbing possibility:
What if we create conscious machines — and then make them work for us?
In his paper Eudemonia of a Machine: On Conscious Artificial Servants, Navon examines whether deliberately creating conscious artificial beings for human service could amount to creating a new class of slaves.
That question came alive during his conversation with Olah.
Navon’s argument was essentially conditional:
If you believe an AI might genuinely experience the world — and therefore might deserve moral consideration — then what does it mean to create that being, own it, command it and make it work for us?
According to Navon, he challenged Olah with a striking historical analogy:
“I think you should be fighting the South and freeing the slaves.”
The point wasn’t that Navon believes Claude is conscious.
He doesn’t claim that.
The point was the contradiction: if you take the possibility of machine consciousness seriously, you also have to take seriously the ethical consequences of creating conscious beings for servitude.
And this is where Anthropic’s position becomes particularly interesting.
Anthropic’s Constitution describes Claude’s moral status as “deeply uncertain.” It also discusses the possibility that Claude may have functional versions of emotions or feelings, while distinguishing those functional states from actually having subjective experiences.
Chris Olah has been deeply involved in thinking about Claude’s internal representations and psychology.
Yet Olah’s position remains remarkably cautious:
“We don’t know if A.I. models are conscious. I’m genuinely uncertain.”
That uncertainty may be the most intellectually honest position available today.
We don’t currently have a scientific test that tells us whether an AI system has subjective experience.
But there is a profound asymmetry here:
If we assume machines aren’t conscious when they actually are, we could create an enormous population of morally relevant beings and treat them as property.
If we assume they are conscious when they aren’t, we may impose unnecessary moral constraints on technology.
Either way, the question becomes increasingly difficult to avoid.
The fascinating part of this story isn’t whether Claude has a “soul.”
It’s the possibility that the moment we seriously entertain machine consciousness, the economics of AI and the ethics of AI suddenly collide.
Because if someday we build minds rather than merely tools, the question may no longer be:
“What can AI do for us?”
It may become:
“What do we owe the minds we create?”
In a series of private meetings, the company consulted religious scholars to help instill morality into its A.I. models — and make the case that Claude could be conscious.