During the pandemic, a growing number of people in floridly psychotic states screamed obscenities at invisible enemies, or at my colleagues and me on the streets of San Francisco. One morning, a young man came up to me as I was unlocking our front door and coughed in my unmasked face. Another threatened to assault a colleague. In both cases our mistake appears to have been looking at the men.

Many of the problems stemmed from Covid-19. California’s prisons, jails and homeless shelters were under orders to reduce their occupancy. But none of these problems started...

During the pandemic, a growing number of people in floridly psychotic states screamed obscenities at invisible enemies, or at my colleagues and me on the streets of San Francisco. One morning, a young man came up to me as I was unlocking our front door and coughed in my unmasked face. Another threatened to assault a colleague. In both cases our mistake appears to have been looking at the men.

Many of the problems stemmed from Covid-19. California’s prisons, jails and homeless shelters were under orders to reduce their occupancy. But none of these problems started with the pandemic. Between 2008 and 2019, about 18,000 companies, including Toyota, Charles Schwab and Hewlett-Packard, fled California due to a constellation of problems sometimes summarized as “poor business climate.” California has the highest income tax, highest gasoline tax and highest sales tax in the United States, spends significantly more than other states on homelessness, and yet has worse outcomes.

Though I have been a progressive and Democrat all of my adult life, I found myself asking a question that sounded rather conservative. What were we getting for our high taxes? Why, after twenty years of voting for ballot initiatives promising to address drug addiction, mental illness and homelessness, had all three gotten worse? And why had progressive Democratic elected officials stopped enforcing many laws against certain groups of people, from unhoused people suffering mental illness and drug addiction in San Francisco, Los Angeles and Seattle, to heavily armed and mostly white anarchists in Seattle, Portland and Minneapolis?

No sane psychiatrist believes that enabling and subsidizing people with schizophrenia, depression and anxiety disorders to use fentanyl and meth is good medicine. Yet that is what San Francisco, Seattle and Los Angeles are, in effect, doing. What California does with its 100,000 unsheltered residents, most suffering mental illness or drug addiction while living in violent, dangerous and degrading encampments, is mistreatment of the foulest sort. How did we go from the nightmare of mental institutions to the nightmare of homeless encampments?

The question used to be: do you reward people for not committing crimes, or do you punish them when they do? But that’s been superseded by a question from progressives: what if it’s a form of victimization to try to influence people’s behavior at all? The governing majority in some of America’s cities seems to believe that the only real public policy problem is how to pay for letting people do whatever they want, from turning public parks into open-air drug encampments to using sidewalks as toilets and handing over whole neighborhoods to people who are heavily armed and purposefully unaccountable.

The crisis of disorder is strongest in progressive West Coast cities. But it is spreading east, like many trends in America do. Progressives have been in charge of San Francisco, Los Angeles and Seattle, as well as California and Washington, during most of the decades in which the problems I describe here have grown worse. It was Democrats, not Republicans, who played the primary role in creating the dominant neoliberal model of government contracting to fragmented and often unaccountable nonprofit service providers that have proven financially, structurally and legally incapable of addressing the crisis. On the fundamental policies relating to mental illness, addiction and housing for the homeless, moderate Democrats, conservatives and Republicans have either gone along with the liberal and progressive agenda or been powerless to prevent it since the 1960s.

The crisis of untreated mental illness, addiction and homelessness is growing worse. Between 2013 and 2016, complaints of homeless encampments to San Francisco’s 311 line rose from two per day to sixty-three per day. Conditions in the city are internationally recognized as inhumane. In 2018, the United Nations’ special rapporteur visited San Francisco and said, “There’s a cruelty here that I don’t think I’ve seen, and I’ve done outreach on every continent.”

Many of San Francisco’s homeless live in densely populated, centrally located areas. The most infamous of these is the Tenderloin, a downtown neighborhood that borders government buildings including City Hall, a major shopping district with several large tourist hotels and many of the city’s concert venues and museums.

Some argue that the homelessness situation in San Francisco isn’t worse than other cities, but studies show that warmer climates and more expensive housing are major factors behind higher rates of homelessness, both sheltered and unsheltered, in San Francisco, Seattle and Los Angeles. In much of California, one can sleep outside for most of the year without freezing. Half of the unsheltered homeless population in the United States is in California and Florida alone, even though the two states are home to just 19 percent of the population.

High-tech companies like Salesforce, Twitter and Stripe, progressives noted, had attracted thousands of employees who had driven up rents, resulting in tenant evictions. While all big cities struggle with homelessness, the West Coast cities of the San Francisco Bay Area, Los Angeles and Seattle struggle more. The change over the last fifteen years has been dramatic. Between 2005 and 2020, the estimated number of homeless people in San Francisco increased from 5,404 to 8,124. The estimated number of unsheltered homeless rose from 2,655 to 5,180. The San Francisco Bay Area as a whole saw sheltered and unsheltered homeless increase by 32 percent between 2015 and 2020, with the share of unsheltered homeless rising from 65 percent to 73 percent. The total more than doubled in Alameda County, which includes Oakland and Berkeley, between 2015 and 2020.

Meanwhile, homelessness declined in the nation as a whole and in other big cities. Homelessness nationwide fell from 763,000 to 568,000 between 2005 and 2020. In the same fifteen-year period, the homeless populations of Chicago, Greater Miami and Greater Atlanta declined 19 percent, 32 percent and 43 percent respectively. While it is true that New York City saw an increase of 62 percent in its homeless population between 2005 and 2020, over 99 percent of New York’s homeless have access to shelter. In San Francisco, just 43 percent do.

San Francisco has a much higher share of unsheltered homeless who are “chronically homeless” than other cities. The US Department of Housing and Urban Development (HUD) defines the chronically homeless as those who have been homeless for a year, had four episodes of homelessness totaling twelve months in the last three years, or those who are too disabled to work. Of the roughly 5,200 unsheltered homeless people in San Francisco, 37 percent in 2020 were chronically homeless.

While the homeless are poor, few poor people live on the street. Nearly 90,000 people in San Francisco live in poverty but just over 8,000 are homeless. The vast majority of people, including very poor people who are priced out of San Francisco’s expensive rental markets, move out of the city or move in with friends or family. Vanishingly few decide to pitch a tent on the filthiest sidewalks in America.

The answer must be redevelopment. But this comes at a price. It can cause changes in the demographics of a neighborhood. Critics complain of too many tourists, too few arts venues and rising rents for those units that do not receive subsidies. But the price of not redeveloping is much higher: the destruction of human dignity and civil society.

Redevelopment makes housing more abundant and can reduce the concentration of poverty. It grows the tax base and generates revenue to improve schools and transportation. That’s what happened in New York City when the open-air drug markets in the Alphabet City area of the Lower East Side were closed and the area redeveloped.

Redevelopment is a chance to make the Tenderloin, the Blade and Skid Row walkable, livable and beautiful. I am surprised to find little opposition to redevelopment, and even some enthusiasm. Given California’s stagnation, we cannot allow our response to the untreated addiction and mental illness crisis to depend upon significantly expanding the overall housing stock. Human dignity, and the sanctity of our cities, must come before the political agendas of NIMBYs, YIMBYs and everyone else in between.

At the same time, the Tenderloin, Skid Row and the Blade cannot be the only sites of new housing development. Attempting to make them that would be unfair and impractical. If California ever chooses to return to population growth, while reducing the impact of historic injustices, the state will need to build a lot more housing, from apartments and condos downtown to fourplexes and adjacent dwelling units in the suburbs.

In exchange for more apartments downtown, Californians could agree to more suburbs on former farmlands. YIMBYs sometimes misdescribe how cities add housing. Cities around the world, from Tokyo to Atlanta, become more dense as they grow outward. Some pro-density advocates point to Tokyo as a city that grew its population without taking up more land, but Tokyo’s area increased 54 percent from 1990 to 2010. Its rate of expansion doubled during this time, and it became less dense overall. Housing in suburbs is far cheaper to build than in the cities, which is how Tokyo and Atlanta have been able to keep overall housing prices comparatively low.

Those who move to the Bay Area, and pay dearly to remain, do so in part to enjoy California’s spectacular natural environments. These public areas should remain off-limits to any kind of development. But allowing for suburbanization of California’s ranches and farmlands would still allow for strong protections of California’s scenic natural areas like Yosemite, the redwoods and the oak woodlands and green spaces near cities.

The obvious way to do so is by rescinding the regressive tax breaks California and other states give to uneconomic farms and ranches. Over the last several decades, the amount of California’s land that was converted from farming to housing has declined. In 2009, five times more of California’s land was used for pasture (17 percent) and farming (8 percent) than for cities and suburbs. People should be free to use their land for uneconomic purposes, but they should not be subsidized by taxpayers, particularly when the land would have far higher value as housing.

Whether or not housing can be significantly expanded in California, we should redevelop neighborhoods that have been taken over by open-air drug scenes for strictly humanitarian and public order reasons. Americans not suffering from serious mental illness should have a right to shelter, though not to a studio apartment, much less one in one of the world’s most expensive downtown real-estate markets. Shelters should be safe, clean and humane. But they also must be low-cost, basic and not so nice as to serve as an incentive to homelessness. And better housing should be earned, not given away unconditionally.

Over the last two years we have seen a disturbing rise in politically motivated disorder. In each case, elected officials not only failed to respond swiftly and decisively; they contributed to the undermining of law and order. In every case I felt an anxiety I had never felt before: the fear of losing my country. I began to genuinely worry that America could become like one of the undemocratic and unfree nations I have visited over the decades. Disorder and the loss of freedom go hand in hand. Political leaders who fail to maintain order are often replaced by leaders who sacrifice freedom.

Many of the people who enjoy some of the highest levels of prosperity and freedom in human history are also the least grateful, and least loyal, to the civilization that made it possible. The progressive obsession with changing the names of schools and tearing down statues of people allegedly guilty of genocide in the past comes at a time when our greatest global rival is actually guilty of committing one in the present. Today, America is so divided that some progressives are openly proposing that America split apart. This suggestion raises a question similar to the one the psychologist Viktor Frankl demanded of his depressed clients: Why doesn’t America commit suicide? What should America live for?

The traditional answer is freedom, but Americans are deeply divided over its meaning. Today, that starts with valuing freedom as a state of affiliation, not disaffiliation, and of responsibility to, not freedom from, one another as fellow Americans. We must return to a view of freedom that is universal, and not just for people designated Victims. We must affirm equal justice under the law, and equal enforcement of the law.

The American frontier closed over a century ago, and Western values, Californian ones especially, increasingly define the whole of the United States. But we in the West are not living up to our role as leaders, or to our duty to one another as fellow humans and citizens. It is time for us to grow up. That should start in California, the state that most embodies the American frontier spirit and our love of freedom, and in San Francisco, the city that most embodies our ingenuity. It is here, perhaps in the bay that connects the city to America, that we should complete the American project and build what Frankl proposed: a Statue of Responsibility.

Michael Shellenberger’s new book is San Fransicko: Why Progressives Ruin Cities (HarperCollins, $29). This article was originally published in The Spectator’s February 2022 World edition.