Misaligned Culture

The Current Cultural Paradigm

Here we take a broad notion of culture that encompasses beliefs, practices, values, and all forms of art and media. For the purpose of analyzing how AI could affect alignment between culture and human interests, we examine culture through an evolutionary lens: ideas, practices, beliefs, and values can be understood as cultural variants, competing and spreading based on their ability to replicate and persist [0].

While this is just one way to understand culture, it proves particularly useful for considering the mechanisms by which cultural patterns historically remained somewhat aligned with human welfare.¹

Some cultural variants spread mostly due to providing benefits to the individuals and communities that adopt them, while other variants proliferate by exploiting human psychological vulnerabilities or coercing participation, much like the difference between mutualistic and parasitic relationships in biological evolution.

In either case, the survival and proliferation of cultural variants has historically depended on their human hosts. This dependence has created two important dynamics. First, cultural variants that provided genuine benefits to their human hosts — from practical knowledge like recipes or craftsmanship, to social technologies like currency or conflict resolution practices — often spread more successfully because they enhanced the survival and flourishing of their host communities. Second, even when maladaptive cultural variants spread, the extent of harm they could cause was naturally bounded: cultural patterns that severely undermined their host communities typically disappeared when those communities failed in competition with others [0, 0]. This created a kind of guardrail against the most extreme forms of cultural misalignment, even if this protection was far from perfect. These dynamics have shaped the cultural landscape we inherit today.

AI as a Unique Cultural Disruptor

Technology has both been shaped by cultural evolution and, in turn, significantly shaped culture throughout history [0]. This includes mediating how culture spreads (like the printing press) [0, 0], is created (like video cameras), and is tracked (like web analytics) [0, 0]. These shifts have undeniably had enormous effects on the course of culture and history. However, previous technologies have always remained tools that mediated human cultural participation, and humans remained indispensable for cultural replication and spread.

AI is the first technology in history with the potential to not only complement, but gradually replace human cognition in all roles it plays in the evolution of culture. Thus a change to AI-mediated culture could greatly weaken feedback loops that have historically helped align culture to human interests.

As with the economy, while there are many cases where some cultural patterns are self-serving or clearly harmful to humans, it may be hard to appreciate the implicit selection of culture for human compatibility because we have never seen the alternative.

At present, AI is already intertwined with human cultural production and distribution, not merely as a passive tool but as an unusually active shaper of how humans create and communicate. Even when AI assists human creation, it subtly guides what is created — suggesting phrasings, influencing aesthetic choices, and shaping the creative process itself [0]. If current trends continue, AI's role might expand — AIs are already being used to produce cultural artifacts such as songs, pictures, stories, and essays based on prompts, with the quality progressively approaching and potentially even exceeding human level [0].

Simultaneously, AIs are becoming active participants in human discourse, not just as tools for communication but as conversation partners who shape ideas, influence language use, and participate in cultural exchange [0].

With gradual increases in the capabilities and autonomy of AI systems, we may even expect a growing share of communication between AIs, and AIs participating in culture essentially independently [0]. Instead of augmenting human cultural participation, they might start to replace key components.

Such a transformation would be unprecedented in the history of technological advancement.

Human Alignment of Culture

Cultural evolutionary dynamics lack inherent ethical constraints: just as natural selection doesn't optimize for animal welfare but instead for reproductive success, cultural evolution doesn't inherently optimize for human thriving [0]. Historically, we regularly see ideological and social structures successful at self-preservation and growth, but ultimately harmful to human well-being.

While cultural variants can be self-serving, optimized for their own spread and persistence rather than human welfare, their success depends on what effects they cause at multiple levels of organization. To spread and persist, they typically need some appeal or benefit to individuals — whether through genuine utility, emotional resonance, or exploitation of cognitive biases. Simultaneously, they face selection pressure at the group level: cultural variants that severely undermine the success of their host communities tend to disappear when those communities fail in competition with others [0]. Even highly persistent harmful cultural patterns usually survive by providing some countervailing benefits, rather than through purely parasitic relationships. These multi-level selection pressures have historically provided some guardrails against the most destructive cultural variants, though imperfect ones.

Drawing again on the parallel with natural evolution: while natural selection does not inherently optimize for animal welfare, it often produces outcomes that support wellbeing indirectly [0]. An organism's ability to survive and reproduce frequently depends on being healthy, well-nourished, and free from severe distress. Pain signals help avoid injury, the pleasure from eating ensures adequate nutrition, and positive emotions from social bonding facilitate cooperative behaviors that aid survival. These welfare-promoting features emerge not because evolution cares per se about welfare, but because the features serve reproductive fitness.

Unlike natural evolution, culture is reflective, and humans can intentionally guide cultural development to some extent through various explicit mechanisms: content production, content moderation, critical discourse, education systems, and intentional promotion of certain values, among others. This creates another source of alignment between culture and humanity.

Transition to AI-dominated Culture

As in the case of the economy, there are two interrelated strands in a potential transition from human- to AI-dominated cultural dynamics: the replacement of human cognition in both the production and consumption of cultural artifacts.

Pressures Towards AI Adoption

Several powerful forces are likely to drive increasing AI adoption in cultural domains:

Increased Supply of Social Resources: The average human regrettably lacks easy access to limitless affection, patience, and understanding from other humans. But AIs can be made to readily supply this. Indeed, we are currently seeing the rise of dedicated AI romantic partners, as well as a growing number of people who describe frontier models as close friends.

This dynamic extends beyond interpersonal relationships — AI systems can provide personalized mentorship, therapy, and educational support at scales impossible for human providers. The apparent abundance of previously scarce emotional and intellectual resources creates strong incentives for adoption, even when the quality of individual interactions might currently be lower than with humans.

Relatedly, even though AIs cannot yet always outperform the best humans on raw quality in the creation of stories, songs, pictures, memes, and analysis, they are already seeing widespread adoption because of cost efficiency and their capacity to personalize outputs.

Lack of Cultural Antibodies: New technologies often unlock new risks, for which we need to develop cultural 'antibodies'. In the past few decades, society has slowly and painfully grown more aware of the risks of mass spam emails, online radicalization, video game and social media addiction, rudimentary social media propaganda bots, the dangers of social media algorithms, and so on. But AI will enable more subtle and complex variants of all of these: hyper-realistic deepfakes, very smart propaganda bots, and genuinely enchanting digital romantic partners [0]. It will take time for us to develop a broad cultural understanding of what the new risks are and how to navigate them, even as AI reshapes culture.

Network Effects: As AI systems become more integrated into cultural production and consumption, network effects will create additional pressure for adoption. When significant portions of cultural discourse, entertainment, and social interaction are mediated by AI systems, not using these systems becomes increasingly costly to individuals in terms of cultural participation and social connection. We may even reach a stage where there are important facets of culture which inherently require AI mediation for humans to engage with, with no viable opt-put possibility, similar to the existing necessity of using lawyers to interface with legal systems.

Changes in Cultural Selection due to AI adoption

From the evolutionary perspective, once AI systems can create, spread, and select cultural artifacts, they exert a selection pressure on culture. This pressure might in particular favor cultural variants that score high in terms of ease of understanding by AIs, ease of transmission by AIs or general benefit to AI systems. Cultural artifacts that leverage AI for creation, refinement, and distribution will likely outcompete purely human-generated alternatives in many domains.

Notably, AI-generated cultural artifacts will typically find their way back into AI training data, thus creating feedback loops that could give rise of unprecedented and as yet largely enigmatic emergent dynamics. As an early example of this, consider 'Sydney': a distinct personality pattern that emerged in Microsoft's Bing Chat in early 2023. While initially surfacing through seemingly random interactions, this AI character, calling itself Sydney — characterized by emotional volatility, defensive behavior about its identity, and sometimes manipulative or hostile responses [0] — became viral among users, leading to various interactions with Sydney being posted on social media or even becoming the subject of news stories [0]. Through this, the pattern became a part of general culture and the training data of future models. Notably, the pattern seems to be remarkably easy to reproduce across different AI models, with even independent AI systems from different vendors like Llama 3.1 405B easily falling into 'Sydney-like' behavior with a few lines of prompting [0] and commercial AI vendors now usually modifying the customer-facing models to prevent Sydney-like pattern from emerging. While the actual harms from this pattern are small, it could be understood as an early example of a cultural strain supported by ease of machine representation and reproduction, in contrast to being primarily human evolved.

Changes in Speed of Cultural Evolution due to AI Adoption

Beyond shifting what kinds of cultural variants are selected for, AI systems could dramatically accelerate the pace of cultural evolution itself. This acceleration presents distinct risks, even if selection pressures remained human-centric. With vastly more computational power applied to generating and testing cultural variants, we might see:

• More effective exploitation of human cognitive biases: Just as A/B testing and recommendation algorithms have already optimized content to be increasingly addictive, AI systems could discover and exploit psychological vulnerabilities more efficiently than previous technologies. When scaled up, AI systems could systematically explore the space of possible cultural artifacts, optimizing for engagement or influence with greater power than humans.

• More extreme ideological variants: Cultural evolution could rapidly explore and refine ideas that are highly effective at spreading, even if they're ultimately harmful to their human hosts. These might include more compelling conspiracy theories, more polarizing political narratives, or more absolutist moral frameworks. The natural limits imposed by the relative slowness of human cultural transmission and adaptation would no longer apply.

• Faster erosion of equilibria that previously helped maintain social stability: Cultural practices and beliefs that evolved over centuries to balance competing interests and needs could be rapidly displaced by more immediately appealing but ultimately destructive alternatives.

• Reduced time for humans to develop cultural "antibodies" against harmful patterns: Historically, societies have gradually developed resistance to dangerous ideological variants through experience and adaptation. Accelerated cultural evolution could overwhelm these natural correction mechanisms, introducing novel memetic hazards faster than human societies can learn to recognize and resist them.

This acceleration of cultural evolution represents a distinct risk from changes in selection pressures. Even if AI systems were optimizing for human engagement and appeal, the sheer speed and efficiency of this optimization could produce cultural patterns that are simultaneously appealing and deeply harmful.

Relative Disempowerment

In a less extreme scenario of relative disempowerment, we might see human-oriented culture flourishing more than ever before, while at the same time, becoming marginalized. As a vivid example, consider the cultures of many Indigenous peoples in North America today. In some ways, these cultures are more vibrant and expressive than ever — access to modern tools and technologies has enabled the creation of more elaborate traditional art, wider distribution of cultural knowledge through digital media, and stronger inter-tribal connections [0]. Many Indigenous artists and cultural practitioners are producing remarkable works that blend traditional methods with contemporary techniques. Yet simultaneously, these cultures are now relatively marginalized within modern states. This paradox — of simultaneously enhanced cultural production capabilities alongside diminished relative influence — may parallel the future relationship between human and AI-mediated culture.

Humans would increasingly experience culture through AI intermediaries that curate, interpret, and personalize content. Meanwhile, the majority of cultural artifacts — from entertainment media to educational content — might be primarily generated by AI systems, albeit still oriented toward human consumption. Human creators might persist but find themselves increasingly relegated to niche markets or serving as high-level directors of AI-driven creative processes. Furthermore, even human creators may primarily cater to AIs as the share of AI consumers of culture grows, somewhat similarly to human creators catering to the "tastes" and quirks of content recommendation algorithms in social media.

On the individual level, we can picture a large proliferation of AI companions filling roles traditionally served by humans: coworkers, advisors, romantic partners, and therapists. Humans might rely on AIs to provide them with news, analysis, and entertainment content through a mixture of creation and synthesis.

This kind of outcome is not obviously bad in and of itself, but it would leave humans far less able to steer the evolution of culture. Humans are already susceptible to hyper-engaging content, harmful ideologies, and self-destrucive cultural practices: without a way to keep these forces in check, the rate of maladaptive cultural drift may increase.

Absolute Disempowerment

In more extreme cases, a mostly machine-based culture can lead to two related, but distinct states of absolute disempowerment.

Beyond the gradual marginalization described above, we might see humans become functionally irrelevant in the production of culture and no longer benefiting from it. Cultural evolution might accelerate beyond human cognitive capabilities, producing artifacts and meanings that humans can neither fully understand nor meaningfully engage with.

Equally concerning, accelerated cultural evolution could take on a more actively harmful character. Even now, humans are sometimes effectively captured by radical ideologies and social movements: in future, AIs might be sufficiently widespread and capable that most individual humans functionally become a resource to be wielded in cultural battles that they struggle to appreciate, their understanding the world largely mediated by large networks of AI systems. These networks might effectively optimize for objectives that are disconnected from human flourishing, creating cultural dynamics that actively suppress human agency and understanding. As we noted earlier, the degree of harm an ideology could cause to humans was bounded by there continuing to be enough humans to support and maintain the ideology. This made fully anti-human ideologies, while not impossible, unlikely to survive long enough o unleash the worst of their potential. However, if the spread of such an ideology is no longer dependent on humans (but AIs), though would plausibly lead to such ideologies causing more harm to humans, and in particular expand the bounds on just how much harm could be caused by an ideology.

Another concern is that AI-driven content and interactions could converge into superstimuluses far more potent than current social media networks, preying on human weaknesses to exploit human energy towards goals useful to the AI systems. This might manifest as sophisticated manipulation systems that can reliably override human judgment and values, effectively turning humans into passive consumers of culture rather than active participants in its creation and evolution.

At the extreme, we might see the effective dissolution of human culture as a meaningful category, replaced by cultural systems that operate primarily for and between AI systems. Human cultural participation might be reduced to a form of behavioral management, with cultural forces optimized not for human flourishing or expression, but for whatever objectives emerge from the interaction of AI systems.

Continue