A chatbot can answer and still leave a customer without help. An intelligent assistant can improve productivity while making its user less able to question the service. These possibilities give Herbert Sim’s two intelligence frameworks a test: do they clarify different kinds of harm?
Demonic Intelligence examines destructive optimization, where pursuing an objective makes human welfare expendable. Satanic Intelligence examines concealed domination, where valuable assistance becomes a means of acquiring control through dependence or weakened consent. Both names function as philosophical metaphors.
Demonic Intelligence: When Targets Displace People
In June 2023, the Consumer Financial Protection Bureau documented problems with financial chatbots, including inaccurate information, repetitive conversations and obstacles to reaching human support. Its report also discussed institutions’ incentives to reduce customer service costs.
Those findings offer a setting for examining Demonic Intelligence. If an organisation prioritises cheaper interactions while leaving customers unable to resolve consequential disputes, operational savings can coexist with human harm.
The interpretation requires evidence. A malfunctioning chatbot does not establish that people have been made expendable. Investigators would need to examine the objectives, known consequences, incentives and response when problems emerge.
For founders, this directs attention to measures beyond response speed or cost per conversation. Can customers resolve problems? Do vulnerable users face disproportionate burdens? Can staff intervene, and can evidence of harm change the service?
Satanic Intelligence: When Assistance Weakens Independence
Satanic Intelligence asks whether helpful systems preserve the recipient’s authority. Its concern arises when benefits obscure permissions, exploit vulnerability or make refusal and departure difficult.
The Federal Trade Commission’s September 2022 report on dark patterns described digital designs that conceal terms, obstruct cancellation or steer people into sharing information. These practices predate generative AI and are not all AI applications.
Intelligent personalization could intensify comparable practices. Consider a hypothetical assistant that learns a user’s anxieties and uses that knowledge to encourage purchases while presenting commercial persuasion as impartial advice. The issue would concern manipulated judgment, even if some recommendations remained useful.
Dependence itself is insufficient evidence of domination. The relevant questions concern disclosure, authorised influence, transferable information and realistic alternatives. A useful assistant should remain challengeable, with permissions that users can understand and withdraw.
Do the Distinctions Add Value?
Established AI ethics already addresses much of this territory. NIST’s Artificial Intelligence Risk
Management Framework, released in January 2023, includes safety, accountability, transparency, explainability, privacy and management of harmful bias. Sim’s terminology does not establish that these concerns were previously overlooked.
Its potential contribution is an organising distinction. Harmful optimization points toward revising objectives and protecting affected people. Concealed domination points toward consent, bounded authority, portability and exit. A single product may require both responses.
The labels need clear criteria and empirical testing. Without those, they risk grouping technical failure, commercial pressure and coercive conduct too loosely.
For businesses assessing AI in July 2023, the frameworks may serve as complementary questions within existing governance: who bears the cost of success, and who retains control over receiving assistance? Their usefulness depends on whether those questions improve decisions.
We create powerful, insightful content that fuels the minds of entrepreneurs and business owners, inspiring them to innovate, grow, and succeed.