The Choice

Sixty-three chapters have described what could happen. This one argues about what should.

That is a different kind of claim and it deserves a different kind of warning. Everything preceding this chapter attempted to describe trajectories with as little preference as the subject allows. This chapter cannot do that. Any answer to "what should be built" rests on values, and values are contested. What follows reflects a particular set—broadly liberal, broadly humanist, weighted toward preserving the ability to change course. Readers who hold different values will reach different conclusions from the same facts, and some of them will be right.

The argument for making the attempt anyway is that the alternative is not neutrality. The choices are being made continuously—by researchers selecting problems, companies allocating capital, governments granting or withholding permission, and consumers adopting or ignoring. A refusal to articulate principles does not produce an unchosen future. It produces one chosen by whoever is not refusing.

The Stakes

The same capabilities support radically different outcomes, and the difference is not technical.

On one side: disease substantially defeated, as Chapters 3 through 8 describe; energy abundant and clean; every person with access to instruction that was recently reserved for the wealthy; scientific discovery running at a pace that makes currently intractable problems tractable.

On the other: surveillance comprehensive enough to make dissent impractical; benefits captured by a small set of owners while costs are distributed broadly; persuasion automated to the point where collective deliberation stops functioning; weapons that select targets; and a large population with no economic role and no political leverage.

These are not different technologies. A model that designs therapeutic proteins designs toxins. A system that personalizes instruction personalizes manipulation. The infrastructure for beneficial population health monitoring is the infrastructure for surveillance—the same sensors, the same integration layer, the same analysis. What separates the outcomes is not capability but governance: who holds the system, what constraints bind them, and who can object.

This is why the technology-neutrality argument, though true, is nearly useless as guidance. Yes, the tools do not choose. The relevant question is who does, under what constraints, and that question has specific answers that vary by domain and jurisdiction.

Principles for Choosing

Five tests, offered as a framework rather than an algorithm. They conflict, and the conflicts are where the actual work is.

Human flourishing. Does this help people live well—not merely survive, but develop capabilities and live lives they have reason to value? The formulation borrows from a tradition running from Aristotle's eudaimonia through the capabilities approach of Amartya Sen and Martha Nussbaum, which has the advantage of being specific about what to measure without requiring agreement on a single conception of the good.¹ The complication is that flourishing is contested and the contest is not resolvable. The minimum version is more tractable: do not foreclose it, and do not cause suffering.

Broad benefit. Do the gains reach beyond those who built and funded the thing? The complication is genuine rather than rhetorical: early concentration of returns is what finances development, and a rule requiring immediate broad distribution would prevent much of what it hopes to distribute. The minimum is not immediate equality but a credible mechanism for eventual access—patent terms that end, pricing that falls, and knowledge that becomes public.

Preserved agency. Do people retain meaningful choice—the ability to understand what is happening to them, influence it, and decline? The complication is that some choices are legitimately constrained; societies do not permit unsafe food or unlicensed surgery, and this is not usually experienced as tyranny. The distinction that holds up is between constraints people can see and contest through legitimate process, and constraints operating below the threshold of awareness. The minimum is autonomy over the decisions that define a life.

Reversibility. If this proves wrong, can it be undone? The complication is that reversibility is not binary and nothing important is fully reversible—knowledge in particular cannot be recalled. But there is a real difference between a policy that can be repealed, a deployment that can be halted, and a released organism that cannot be recovered. The minimum is caution proportional to irreversibility: the harder something is to undo, the more evidence should be required before doing it.

Accountability. When something goes wrong, is there someone answerable, and can those harmed obtain redress? The complication is that complex systems diffuse responsibility by construction, and machine-learning systems make attribution genuinely hard rather than merely inconvenient. The minimum is that accountability must be assigned rather than discovered—if no one is responsible by default, someone must be made responsible by rule.

Where these conflict—and broad benefit versus development speed conflicts constantly—the framework does not adjudicate. What it does is force the tradeoff into the open, which is more than most technology decisions currently get.

What Should Be Built

AI-accelerated medicine. Faster discovery, earlier detection, treatment matched to individual biology. Reducing suffering is as close to a universal value as exists, and health is instrumentally prior to nearly everything else. The guardrails are access on a timeline measured in years rather than decades, genuine privacy protection for the data this requires, and safety standards that do not bend to enthusiasm.

Universal access to instruction. The argument of Chapter 29 is that personalized tutoring produces outcomes group instruction cannot, and that its cost has collapsed. If educational opportunity is a primary determinant of life outcomes, and it is, then making excellent instruction non-scarce is among the highest-leverage things available. The guardrails: no manipulation of children, preservation of human relationship in education, and access designed in from the start rather than added after a decade of premium-tier exclusivity.

Clean, abundant energy. Fusion if it arrives, advanced fission and solar with storage regardless. Energy scarcity underlies a large fraction of material deprivation, and energy abundance is the input to most other abundance. The guardrails concern the transition: workers and regions dependent on displaced industries are owed something better than being told the aggregate math works out.

Scientific discovery infrastructure. Faster hypothesis generation, better experiment design, pattern recognition in data too large for human analysis. Knowledge compounds, and the returns are broad and long-lived. The guardrails: dual-use awareness built into the research culture, responsible disclosure norms, and results that reach the public domain rather than remaining proprietary indefinitely.

Capacity for democratic institutions. Better information for decisions, services that work, meaningful mechanisms for citizen input. Chapters 32 through 36 describe institutions that function poorly enough to erode their own legitimacy, and technology that makes them work is not a small thing. The guardrails are severe and non-negotiable: any capability built for administrative efficiency is available for surveillance, and the boundary must be legal and enforced rather than cultural and assumed.

What Should Not Be Built

Weapons that select and engage human targets without meaningful human control. The objection is not that machines will be worse at discrimination—they may become better. It is that removing a human from the decision to kill removes the moral agent, and with the moral agent goes both the possibility of refusal and any coherent theory of accountability. It also lowers the political cost of force, and the historical record on what happens when the cost of force falls is not encouraging. UN discussions on lethal autonomous weapons have proceeded since 2014 without producing a binding instrument, which is itself a data point about how this is going.² Defensive systems engaging incoming munitions under human supervision are a genuinely different case.

Comprehensive population surveillance. Monitoring of all citizens, integrated across domains, with behavioral prediction attached. The objection is that it is incompatible with the freedom to dissent, which is the freedom on which correcting every other mistake depends. This capability exists at varying degrees of development in multiple countries and the technical trajectory is toward it being cheap.³ Targeted surveillance under judicial authorization is a different thing, and the difference is not the technology—it is whether an independent party must approve each use.

Systems designed to manipulate at scale without disclosure. AI optimized to change beliefs and behavior through mechanisms the target cannot perceive. The objection is that it destroys the conditions for informed consent, and consent is what distinguishes persuasion from coercion. Transparent persuasion—advertising that announces itself—is a normal part of commercial life and not what this prohibits.

Engineered pathogens with enhanced transmissibility or lethality. The objection is that the risk is potentially unbounded and irreversible, self-replicating agents do not stay where they are put, and the defensive benefits are obtainable through less dangerous routes. The dual-use research debate has run since the H5N1 controversy without resolution, and the durable lesson is that the oversight regimes are weaker than the capability.⁴ No exceptions apparent.

Superintelligent systems deployed without robust alignment. The objection is not certainty of catastrophe but that the failure mode is potentially unrecoverable, and that the current state of alignment research does not support confidence at that capability level. Techniques including RLHF, Constitutional AI, interpretability, and formal verification represent real progress and none provide guarantees that extend to systems more capable than their overseers.⁵ The condition is not "never"—it is "not until the confidence exists," and the confidence does not currently exist.

The Gray Zones

Most real decisions do not sort cleanly, and pretending otherwise produces frameworks that are useless precisely where they are needed.

Genetic modification. Eliminating a fatal single-gene disease and selecting for advantageous traits are the same technology and different moral situations. The consensus that has emerged—strong support for somatic therapy on serious disease, deep unease about enhancement, near-universal opposition to heritable germline editing—is imperfect but tracks something real: the presence of a patient who consents, and whether the change propagates to people who cannot.

Life extension. Healthspan extension is broadly supported. Radical lifespan extension raises the entrenchment problems of Chapter 59, and those problems are structural rather than incidental. The judgment here is that the research should proceed and the institutional countermeasures should be built in parallel rather than afterward.

Brain-computer interfaces. Restoring function to people who have lost it is unambiguously good. Enhancement is contested. What distinguishes this domain is that the privacy stake is qualitatively different: mental privacy is the substrate of every other freedom, and once neural data is legible, no other protection is fully load-bearing. Medical applications should proceed; the privacy framework should be built before consumer applications arrive rather than after.

Geoengineering. Solar radiation management is cheap enough that a single actor could deploy it unilaterally, affecting everyone, with poorly understood regional consequences and a termination shock if it stops. Carbon removal is slower, more expensive, and does not carry those properties. The judgment: research both, treat removal as clearly preferable, and recognize that the governance problem for solar management is harder than the engineering.

Artificial general intelligence. Potentially the largest benefit in this book and the largest risk, and the two are not separable. The judgment: research with heavy safety weighting is warranted, deployment requires confidence in alignment that does not yet exist, and the honest problem is that the actors best positioned to build it face competitive pressure to move before the confidence is available.

Who Decides

The uncomfortable fact is that the current answer is: a small number of companies, a smaller number of investors, and governments that mostly lack the technical capacity to evaluate what they are regulating.

That distribution is not the result of anyone's plan. It follows from the capital intensity of frontier development and the speed at which the field moves relative to legislative process. But it means that decisions with civilizational scope are being made through the ordinary internal deliberations of firms accountable primarily to shareholders.

The democratic objection is straightforward: decisions affecting everyone should involve everyone. The practical objection to the democratic objection is that most people do not understand the technology well enough to make informed choices about it.

That second objection is weaker than it appears, and it is worth saying why. Citizens do not understand pharmaceutical chemistry, nuclear engineering, or monetary policy either, and societies have nonetheless built institutions that make legitimate decisions in those domains. The mechanism is not universal expertise. It is delegated technical judgment operating inside democratically set constraints—regulators with genuine capability, accountable to elected officials, bounded by law. That model is imperfect and captured more often than anyone would like, but it exists and it works better than the alternative currently operating in AI.

Four mechanisms would help, and none require inventing anything new.

Technical capacity inside government. Regulators cannot evaluate what they cannot understand, and the current asymmetry between frontier labs and their overseers is extreme. The US operated an Office of Technology Assessment from 1972 until it was defunded in 1995, providing exactly this function; recreating something like it is among the cheapest high-value interventions available.⁶

Deliberative public input on value questions. Citizen assemblies—randomly selected, given time and expert briefing, asked to deliberate—have produced workable resolutions to genuinely contested questions, most prominently in Ireland's constitutional referendums.⁷ They are appropriate here because the hard questions are not technical. Whether enhancement should be permitted is not a question expertise answers.

Obligations on builders that do not depend on their goodwill. Voluntary commitments have shaped norms and will not survive real competitive pressure. Liability, disclosure requirements, and pre-deployment evaluation with legal force are what make safety a business consideration rather than a values statement.

International coordination adequate to a technology that ignores borders. This is the hardest and the least likely, and the honest assessment is that meaningful coordination has historically followed catastrophe rather than preceded it.

Living with Uncertainty

Every recommendation here could be wrong, and the intellectually serious response to that is not paralysis but a set of dispositions.

The precautionary posture—when in doubt, do not—is appropriate for irreversible risks and badly wrong as a general rule, since it treats the harms of action as real and the harms of inaction as hypothetical.⁸ The proactionary posture—proceed and learn by doing—is appropriate where mistakes are correctable and dangerous where they are not.⁹ Neither is a policy; both are heuristics, and the skill is knowing which situation you are in, with irreversibility as the discriminator.

Two dispositions apply generally. Preserve optionality: prefer paths that leave alternatives open, and be suspicious of arguments that a particular course is inevitable, since such arguments are usually made by people who benefit from its inevitability. And maintain diversity: do not standardize the world on a single approach, because monocultures fail together.

Underneath all of it is a specific humility about time. The people making these decisions will not experience most of their consequences. Future generations will inherit the results without having consented, which is the ordinary condition of intergenerational ethics and no less real for being ordinary.¹⁰ Present values will look parochial to them in ways that cannot be anticipated—as previous generations' confident moral certainties look to us now. The practical implication is not to defer every decision, which is impossible. It is to prefer choices that leave successors room to disagree.

The Path Forward

Near-term (2026–2032). Build evaluation capacity in government and make pre-deployment testing legally binding rather than voluntary. Make DNA synthesis screening mandatory—the cheapest high-value safety action available in any domain covered by this book. Establish accountability by rule, since it will not emerge by default. Fund alignment research at a level proportionate to capability spending, which currently it is not.

Medium-term (2032–2040). Develop alignment techniques with properties that can be verified rather than observed. Build international mechanisms before the incident that would otherwise create them. Renegotiate the social contract around work and distribution described in Chapter 57, on a timeline that precedes the displacement rather than following it.

Long-term (2040+). Governance adequate to transformative capability, if it arrives. Institutions organized around human flourishing rather than production alone. And the preservation of meaningful human agency over the direction of all of it—which is the one objective that, if lost, cannot be recovered by any later decision.

The Deeper Questions

What is flourishing? This chapter used the word as a criterion without defining it, deliberately, because it is contested and the contest is legitimate. But the technologies described here force the question into practical urgency. Does flourishing require struggle? Achievement? Mortality? Work? Answers that were comfortable as philosophy become operational when the constraints they assumed become optional.

What is owed? If abundance is produced by systems trained on the accumulated output of human civilization, the claim that its returns belong entirely to whoever assembled the training run is at minimum contestable. How that gets resolved determines whether the benefits described in this book reach people or accrue to owners.

What makes humans human? If machines think, create, and decide, what is the human contribution? The question is not new, but it has stopped being rhetorical. It is now a question about how to organize a society.

Conclusion

Every technology in this book represents a choice, and the choices are being made now, mostly by people who are not framing them as choices at all.

The principles this chapter has offered are ordinary: pursue flourishing, distribute benefits, preserve agency, prefer reversibility, assign accountability. Their ordinariness is the point. The difficulty was never identifying good principles. It is that applying them requires accepting real costs—slower deployment, forgone profit, capability left unbuilt—and the actors positioned to bear those costs are the ones with the least incentive to.

Which is why the closing argument of this book is not addressed to technology.

Nothing described in these pages is self-executing. Models do not deploy themselves, data centers are built by someone, and every regulation that will or will not constrain any of it will be written by people whose names are knowable. The tendency to describe this transformation as a force—something arriving, something happening to us—is the single most consequential error available, because a force cannot be argued with and a decision can.

The technologies are coming; that much is settled, and it is what the title of this book claims. What is not settled is who they serve, on what terms, and whether the people affected have any say in it. Those remain open, and they remain open for a limited time, because arrangements harden. Institutions that are built provisionally become permanent. Concentrations of capability that could have been prevented become facts that have to be negotiated with.

So the argument reduces to a matter of timing. There is a period during which the rules governing a new technology can still be written by people who do not yet know whether they will benefit from them—and it closes. It is closing now. What gets decided inside it will be difficult to revisit and will shape a great deal of what follows.

Humanity gets to decide. That is not a reassurance. It is the assignment.

Endnotes — Chapter 64

1. Human flourishing derives from Aristotle's eudaimonia. Contemporary accounts include objective list theories, subjective preference theories, and the capabilities approach developed by Amartya Sen and Martha Nussbaum, which specifies capabilities people should be able to exercise without prescribing a single conception of the good life.
2. Autonomous weapons: UN Convention on Certain Conventional Weapons discussions on lethal autonomous weapons systems have proceeded since 2014 through a Group of Governmental Experts, producing no binding instrument. The Campaign to Stop Killer Robots advocates a prohibition.
3. Surveillance technology: comprehensive population monitoring capabilities exist at varying degrees of integration across multiple states, with China's systems the most developed. Democratic states face ongoing debates about appropriate limits on capabilities they already possess.
4. Dual-use research of concern: the US National Science Advisory Board for Biosecurity provides guidance; the H5N1 transmissibility debate (2011–2012) remains the formative case, and successive policy revisions have not produced a durable settlement.
5. AI alignment: approaches include reinforcement learning from human feedback, Constitutional AI, mechanistic interpretability, and formal verification. Each represents measurable progress; none provides guarantees that extend to systems substantially more capable than their evaluators.
6. The US Office of Technology Assessment provided nonpartisan technical analysis to Congress from 1972 until it was defunded in 1995. No current equivalent exists, and its revival is periodically proposed.
7. Citizen assemblies: deliberative bodies of randomly selected citizens, given expert briefing and time to deliberate. Ireland used the mechanism successfully for constitutional questions including marriage equality and abortion; the model has been proposed for technology governance.
8. The precautionary principle emphasizes caution under uncertain risk and is enshrined in EU law. Critics note it treats the harms of action as concrete and the harms of inaction as hypothetical.
9. The proactionary principle, associated with Max More, was formulated in response to the precautionary principle and emphasizes the benefits of innovation and the value of learning by doing.
10. Intergenerational ethics concerns obligations to people who do not yet exist and cannot consent to decisions made on their behalf. Philosophical foundations include work by Derek Parfit and Samuel Scheffler.