Mistakes #4: breaking Chesterton’s fence in the presence of bull

(Mistakes #1, #2, #3)

Lots of prominent activities don’t immediately make pragmatic sense. I mean, they make sense in the sense that you want to do them, but not in the sense that you can give an explicit account of why you want to do them.

For instance, visiting family in holidays. For one thing, why do we even have holidays? For a few days, and not on other days? And why eat turkeys or champagne then, and not eat them the rest of the time? For another thing, why do we have families? And why see your family exactly that week? For those who like their families, is that the most convenient week? Or do people really need to coordinate in holding family dinner at the same time as people not in their families? For those who don’t like their families, why go to so much trouble to see people whose only special feature is that you were already forced to spend too much time with them decades ago? Also, why guess what someone else wants you to buy for them, while they spend the same amount on a guess for you? And why does everything have to be decorated in colors and sparklinesses that nobody during more sober times finds aesthetically pleasing?

As a young person, learning about the world, you can respond to this sort of thing in at least two ways. One option is to go along with the things. Perhaps you have great trust in society. Perhaps you don’t notice. Perhaps you have little curiosity or passion for improving the world. Perhaps you have heard of Chesterton’s fence.

Another option is to politely disregard any of the things that don’t make apparent sense, and redesign your life according to reason. Or rudely disregard any of the things that don’t make sense, if you don’t see the use in politeness! If you see no reason for eating dinner before dessert, or exchanging gifts, or doing well in school, then you just don’t do those things.

I have a soft spot for the social innovator, who sees the people needlessly toiling for senseless or forgotten goals and is willing to face social censure to make a better world. However I think young, intelligent people often make mistakes this way. It’s not clear to me that these mistakes can be avoided without giving up a valuable attitude, but my guess is that some can.

There are at least three closely related mistakes: breaking down Chesterton’s fence because you don’t know why it’s there; breaking down Chesterton’s fence because the owner doesn’t know why it is there; and breaking down Chesterton’s fence because the owner thinks it is there for reasons you know to be nonsense.

Chesterton’s fence

Chesterton’s fence is from a famous principle, which basically says ‘don’t take down fences unless you know why they were put up’. Or relatedly, don’t try to reform society while you don’t understand the reasons for its present behaviors. Most fences are put up not by crazy people, but by people who had some sensical motive. Even if you can’t see a bull, the fact that there is a fence there is suggestive.

Suppose you don’t understand why people go out for coffee instead of caramels. And suppose that you have an important date to organize which is likely to succeed if you don’t mess it up. One thing you might say is, ‘well, people go out for coffee a LOT, but I don’t see why coffee is especially good for dates, and caramels are cheaper and more delicious’. Then you might opt for the caramel date. This would be knocking down Chesterton’s fence.

When younger, I didn’t intellectually understand why people would shave their legs, or maintain their garden, or try at sports at school, or listen to their feelings if they weren’t obviously well-aligned, or get a job, or keep in much touch with their family once they had left home, or live in a building, or drink alcohol. So I either didn’t do these things, or assumed I would not later on, and so did not prepare for them. Even though I knew they were popular activities among humans. They aren’t obviously good activities, even now, but the point is that I didn’t really consider whether people have them for a reason that I didn’t understand. I quickly disregarded them because I didn’t understand them. And in fact I think I was often wrong to think they were worthless.

The basic mistake with Chesterton’s fence is either not taking the existence of a thing as evidence that someone has a reason for it, or inferring too little from that about whether it is good on your values.

Chesterton’s fence after contacting the caretaker

A more subtle mistake involves inquiring after people’s reasons for the inexplicable fences they protect, finding that can report none, then hastily trashing the fences. For instance, you ask your mother why it is important for you to have table manners, and she says ‘it just is’, so you stop doing it.

The problem with this is that people do lots of relatively useful things without having much explicit understanding of what they are doing or why. People pick most behaviors up from other people.

Sometimes someone once understood why the behavior was good, and intentionally constructed it for others to use. Like CBT or the Alexander Technique. Sometimes the thing wasn’t spread because someone understood it, but still it was experimentally checked, at least informally. Like drinking lemon and honey when you are sick, or making eye contact the right amount. Other times perhaps nobody ever understood or had the opportunity to check well for efficacy, but still social forces preferentially select for things that work, at least to some extent, for some goals. Arguably like religion, or being idealistic as a young person, or following your curiosity.

Even if never in history had anyone ever explicitly understood why table manners were important, their prevalence is decent evidence that you shouldn’t immediately abandon them.

Chesterton’s fence with a stupid sign

After realizing this potential mistake, there is another mistake you can make however. That is where you ask people the reason for a thing, and they give you a reason, and it’s a bad one—and then you’re allowed to break down the fence, right? If you know it was put there by some crazy person to keep the flying pigs at bay?

For instance, suppose you ask your math teacher why math is important, and she says that as an adult you will need to be able to add and subtract numbers when you are shopping or if you have a job in administration or science. And you know that people can have calculators, so adding skills are not really important. And you don’t care much about grocery price comparisons anyway, since your value of time is too high. And basically none of what you are doing is adding or subtracting anyway! It’s all trigonometry and calculus. Then you might reasonably infer that math is of no use to you. If this person who appears to be very invested in math—whose whole life is about encouraging you to do math—can’t come up with anything better than that, then surely you are safe to ignore math. (Loosely based on a true story).

The problem here is that often instead of doing reasonable things while being clueless about why (as discussed in the last section), people come up with reasons for the things they are doing, which are unrelated to the process that caused them to be doing the thing.

Perhaps the builder of the fence did it because something scares his dogs when they go up the hill. Whatever it is, he doesn’t want it coming down to his house. He has a correct intuition that a fence there would make him feel safer. He comes up with a story that the problem is flying pigs, which are fortunately respectful of fences. This is pretty much irrelevant. In this case the fence does indicate something you should watch out for.

The math teacher is teaching math because other forces recognized the value of math, recognized that she was good at both high school math and at teaching, and paid her enough to teach it. She doesn’t know about the details of how this came about, or why it was considered a good idea. However sometimes students ask her why they should do math. She ponders this, and reasonably thinks about when math comes up in her own life. She comes up with calculating things. She might also think of ‘being a math teacher’—an equally thrilling inducement to the ambitious student. Though the math teacher is in charge of the fence, she doesn’t automatically know why it is there. It feels like she should, which may compel her to search for reasons, but the reasons have had somewhere between seconds and hours of thought applied to them, while the behavior itself underwent a more thorough optimization process.


This makes things hard, because when are you allowed to just write off things as not valuable? If someone looks you straight in the eye and says ‘I’m clashing these pots together because aliens’, do you have to reserve substantial credence that they are doing it for a good reason?

I think the main point is that in these cases you need to have actual beliefs about what is going on, and decide based on the apparent situation. You don’t get to just blanket disregard things because you don’t understand them, because its proponents don’t understand them, or because its proponents claim to do them for bad reasons. But you can update somewhat toward thinking the things are useless, and perhaps a lot.

An important part of figuring out when you should disregard inexplicable behaviors is understanding where the behavior came from. If your pot-clashing friend feels compelled to clash pots because it makes him feel less anxious about the aliens, then probably it is worth it for him, for entirely non-alien reasons. There is even some chance that it would also be useful in relieving your own anxieties about other things. If your friend clashes pots because he read online that this is the anti-alien-club sanctioned way to fend off aliens, then there are fewer models on which clashing pots is advantageous for either of you.

Sometimes understanding where behaviors come from will tell you that a behavior is well-honed, but not for goals you have, so this understanding allows you to break down fences you might have otherwise respected. For instance, if you are a teetotaler, and your best guess is that the excitement about festival X is closely related to its cheap and delicious booze, then you can infer that the festival is an unusually poor fit for you, even if nobody can explain well to you why they love it.


In general, I propose acting like a person who takes seriously the possibility that there is a bull in the paddock, rather than someone who is obliged to do a checking ritual before they are allowed to gleefully smash down the fence. Then follow the usual epistemological procedures for determining what statements are true and what actions are good.

Misunderstandings of not understanding

There are probably lots of attitudes to a thing we might call ‘understanding’ it. Here are examples of two:

“I understand combustion”

“I understand her feelings”

In the first statement, the person has a good explicit model of combustion, and the range of ways it can happen, and what causes it, and that sort of thing. In the second, her feelings feel natural and comprehensible to the speaker. T might have an intuitive sense for how her feelings will evolve, but they probably don’t have an explicit model of the psychology, biology or game theoretic reasons behind her feelings.

These different instances of ‘understanding’ probably vary on a few different dimensions that could be separated, but this is going to be a short blog post.

One thing that separates these kinds of understanding is their objects. People rarely say “I understand combustion” to mean that it feels intuitively natural to them, in the absence of any explicit model. And they rarely say “I understand her feelings” to mean that can answer questions about her alien neural chemistry.

There are some things where both kinds of understanding sometimes apply though. For instance “I don’t understand art” can mean “when I look at art, it doesn’t feel especially natural or good to me” or it can mean “I do not know what caused humans to like art so much”. The seconds speaker may love art.

I think failure to distinguish these kinds of understanding causes misunderstanding. For instance, suppose you say you don’t understand why people go out to bars. This could mean you don’t enjoy doing so personally, or that you don’t see why bars in particular evolved to be a big thing while public swimming pools full of tiny balls did not. If the two aren’t really distinguished in people’s minds, then they will suppose you are saying both, even once your elaboration strongly implies the latter.

Perhaps this is a reasonable shorthand, because in practice the only people who wonder why humans gather at bars in particular are those who aren’t enjoying bars enough. But I think it confuses things (for instance, impressions of how fun I am).

A particularly costly kind of misunderstanding is when people themselves assume they are saying both, because they think there is just one thing. Then for instance they notice that they don’t know why art exists, and don’t think to independently check whether they like art. Or they don’t understand why sports is in the curriculum, so they assume that they don’t enjoy it. Or ‘tradition’ seems an incomprehensible justification for a behavior, so they don’t pay attention to whether they personally get anything out of traditions.

I don’t really know how much any of this is a thing—I feel like I have seen quite a few examples of it, but I don’t remember many. What do you think?

(Related: Meta-error: I like therefore I am)

Mistakes I’ve made part 3: Poor sacrificial accounting

There was a time when I routinely refused car travel, in favor of my more sustainable bicycle. Not always, but I had a high bar—if it was bucketing down with rain and I had no plastic pants, this was not sufficient excuse for instance. I would sometimes decline a ride even when others were purportedly driving somewhere anyway, to avoid encouraging them to be ‘driving anyway’ more often. I did enjoy cycling, on a good day, but many days weren’t good. People at school laughed at me on my bike, so sometimes I walked instead because I couldn’t stand them looking at me. Refusing cars was often a sacrifice.

My concern was the climate. My basic model of ‘sustainability’ was this: humanity has a bunch of resources, and when they run out we all die. If we use them slow enough, they can mostly last forever, but if we use them faster, we will die soon. Much like a bank account that earns lots of interest, or can be spent down in an afternoon. Space in the atmosphere for greenhouse gasses was a key resource. Driving was known to be unsustainable: it added carbon dioxide to the atmosphere faster than could be supported. This model has a few problems I think, but it is debatable, and this blog post is not about them. So lets suppose for now that I was right, and every car trip took humanity a notch closer to annihilation.

Given that driving was unsustainable, I wanted people to stop driving, so we wouldn’t all die. Naturally, it followed that I should not drive (I thought). If the cost of a billion people driving was too great for the benefits of a billion people driving, then the cost of one of those people driving was probably too great for the benefit of that person driving. Furthermore, I seemed a particularly easy person to convince, from my perspective. Furthermore, the benefit of me getting to school sooner and with dry pants was obviously unimaginably minuscule compared to any decrement in humanity’s ability to survive in the long term.

This is all very wrong. Why? Let’s say cycling cost me ten minutes per day (it took more like half an hour per day, but driving also takes time, and cycling had some benefits, such as exercise). If I had spent this 5h per month on writing about the problem, I think I could have convinced more than one person ever to habitually ride a bike. If I had just earned money and given it to an organization specializing in this, they probably could have done better. This is an empirical question, so maybe not, but the problem was that I didn’t even look for such things.

One way to think about this is to say that I was wrong in thinking myself an especially cheap person to change. I might be easy to communicate with and readily convinced (from my perspective). I might be able to get on a bike right now, whereas I had not much idea how to convince anyone else. But I am an extremely expensive person to use the time of (from my perspective), because I am one of the few people who is on board with my schemes and willing to do what I tell myself, and so my own time is one of the few resources that can be spent on the plan of forwarding my values. And given this, there are ways to leverage ten minutes of my time into more than an extra bike ride’s worth of value.

Relatedly, I didn’t compare between options further afield. Even if unsustainable transport was set to destroy the world, and personal action was the cheapest way to get change, it does not follow that I should cycle. For instance, maybe it is even more important to change people’s unsustainable purchasing behavior, and I should have spent that time purchasing things better. Or earning money, and spending it on forwarding sustainability in other ways. Or thinking about how robots might destroy the world more imminently.

More fundamentally, my error was comparing the costs of my sacrifice to what was gained by it, instead of comparing my gains to what I might have had instead. This was both the wrong comparison, and naturally blinded me to the better purchases I might have made. If the question you ask is whether cycling everywhere is worth a tiny fraction of the future, then there is no reason to check how valuable reading about geoengineering is.

So this is what I would tell my younger self. Whether your personal sacrifices are money, time or frustration, you should usually be comparing the benefits of making one personal sacrifice to the benefits of making another. You should not be comparing the personal cost of making a personal sacrifice to the benefits of making that sacrifice. Because what matters is ‘opportunity cost’—the value of the next best option that you are foregoing. And rarely is the next best option to ramp up your leisure budget. Usually the next best option is making a sacrifice for something else overwhelmingly valuable; something measured in units of humanity’s future, or in senseless torment averted. This is true whether you are willing to sacrifice everything for the greater good, or only a tiny bit. Either way, you should spend the sacrifices you make on the best things sacrifices can buy.

Mistakes I’ve made, part 2: Father Christmas

I believed in Father Christmas for longer than I believed in God. This is mostly surprising because it is unusual. There was both more evidence for Father Christmas, and more motivation to propel cognition.

I stopped ‘believing’ in Father Christmas in grade seven, when my drama teacher cavalierly mentioned his unreality. It wasn’t shocking—more a feeling of something I knew somewhere in my heart being made official. It was a bit embarrassing.

How could I neglect to explicitly infer the lack of a Father Christmas until then?

I don’t think my epistemic failures here were embarrassing for a human, given the proliferation of supernatural objects of belief, and the fact that Father Christmas alone among them actually leaves concrete stuff that you wish for.

My guess is that most people do better, not by noticing their confusion, but by interacting more with other children. Noticing confusion would also alert people to the implausibility of other supernatural things (unless perhaps it is the very evidence for Father Christmas which makes him confusing). And contact with subject matter experts is known to be a good prophylactic against being wrong.

But what if I wanted to instruct my past self on how to independently notice that Father Christmas was implausible? What should have been clues, to someone fairly ignorant about the world?

Father Christmas purportedly travels very fast, and delivers more presents than might fit in a sleigh. Also his sleigh can fly. From a child’s perspective though, many things travel very fast. Especially ones that fly. Not usually ones that are propelled by reindeer, but it wouldn’t be shocking if the reindeer were not the main engine, or if the illustrations portrayed the vehicle somewhat whimsically. Having a lot of presents could also be explained by being able to move things fast. Father Christmas himself would also have to move very fast, compared to a human. Though it also shouldn’t be surprising if much of his effort were outsourced; we also say that Steve Jobs made the iPhone. Nonetheless, one could notice that these speeds were suspiciously high.

This would require some understanding of contemporary technology, since such speeds are not obviously inconceivable for any technology. This is complicated by the world being big and varied. For instance, these days I sometimes hear that people elsewhere in the world are doing things well beyond the technological capability of people and companies I usually interact with. For instance, creating exotic physical particles or going to Mars. Yet I don’t usually suspect it at all. Should I? For better or worse, I believe other groups have technology that is much better than that in commercial use, and am not very surprised by it.

Perhaps I should have noticed that there was a more plausible explanation? The real explanation, as it turns out, is that there is a global conspiracy by one demographic group against another. The ‘very large scale conspiracy’ hypothesis is perhaps best known for its role as a reductio against theories that requires its truth. So I’m not sure that this alternative should have jumped out at me. Maybe I should give such theories more credence, given Father Christmas. My guess is mostly not, but that global conspiracies can grow from smaller conspiracies which become tradition and then spread.

Father Christmas has other surprising purported characteristics, such as an abundant appetite and access to all buildings. But presumably he doesn’t eat all the cookies at the time, and we used to leave the door open for him.

I’m not actually convinced that I should have done better, as an ignorant child independently assessing Father Christmas. Had I looked into the technology purportedly being used, then I should have done better. However I’m not sure Father Christmas should have stood out as the most worthy target of deeper investigation, even if I had been sensible enough to investigate the most worthy targets of deeper investigation. So, if I made mistakes, probably I’m still making them! Do you have better hints for cheaply being appropriately confused?

Mistakes I’ve made, part 1: greedy altruism

I have made many mistakes. Unfortunately, for some of my older, larger mistakes, it becomes hard to remember or imagine why I would have made them. The alternative position comes to seem literally inconceivable. At the same time, I forget that I ever conceived of it. This is tragic, because for anything realized some way into my life, there are surely many people who don’t (yet?) share my view. Also because any insight into why I might have believed inconceivable things in the past might prevent me from believing such in the future.

So it seems a valuable exercise to recall and dissect some of my errors before they evaporate from my memory. Here’s one:

For almost all of my teenage years, I believed that small amounts of money could be used to save lives in the developing world (an error, but one for another time), but consequently collected small amounts of money where possible, rather than optimizing for long-run ability to earn money (which would also not necessarily be the best thing, but is obviously superior to greedily earning small sums). e.g. I would do chores for money instead of practicing useful skills.


One obvious possibility is that it didn’t occur to me. That would seem surprising, and I don’t think it’s true, but it might be. Also something intermediate seems plausible—like, I was not fully and abstractly aware of a trade-off between greedily accruing small amounts of money and investing in better opportunities, though it did occur to me that I could practice math instead of doing a chore and that this might help.

If we suppose I was at least somewhat aware of this option, I think I probably didn’t see how any alternative activities would genuinely improve my prospects. Especially at the five minute level. If I didn’t do this chore, it didn’t seem like I would really find something else to do, right here and now, that would cause me to earn twenty cents in the long run.

I suspect one mistake was failing to add these five minutes up, and say that if I always didn’t do chores, I would have a fair bit more time, and even if the first twenty increments weren’t that great, given the options in my house, it should be possible to look further afield and find ways to make a difference.

In my past self’s defense, I can’t remember what I actually did with my free time, and I doubt it was that sensible, so I may have been right that the isolated intervention of buying more time for myself was not great. In my past self’s further defense, I think my ‘free’ time was often poorly spent because I had to take care of my younger brothers most of the time, which was pretty distracting. However I don’t think any of this is sufficient defense at all.

Another error might have been failing to have a model of myself as something that could accrue any kind of improvements. I think I saw most of my value as residing in my intelligence, which I supposed was fixed. I was fairly oblivious of the differences between me and e.g. a competent middle-aged person. However it’s hard to see how this could be so, in conjunction with having high hopes for the future, and achieving little at the time. I saw living at home as a huge impediment to anything, which would naturally end at some point. I think I may also have implicitly supposed that becoming an adult would actually bring a host of improvements, such as knowing how to usefully improve the world. I doubt I checked whether these beliefs were quantitatively plausible and consistent.

I suspect I also focussed too much on the comparison of someone’s life and me making effort now—I can save a bit of a life by doing this annoying thing; what could be better than that?!—instead of focusing on the comparison between this way of saving a bit of a life and other ways. This is an issue of psychological focus rather than a wrong belief. It seems like an especially easy mistake to make if other people fail to notice the comparison between your effort and other people’s lives, and you are kind of indignant about it.

I think relatedly, I believed that earning some money would really help, whereas everything else felt nebulous. Could you really let a person die, so you could do something more fun that might help at some point on some messy, half-thought out model? It perhaps felt easy to cheat.

How could I avoid such problems, if I were talking to my teenage self, or hoping to not make analogous errors now? Some ideas, more welcome:

  1. Explicitly state my models of how my actions are reasonable, which would hopefully reveal gaps and incoherencies
  2. Describe my explicit models to other people, who are hopefully more in touch with common sense (e.g. you can often learn things, which will affect your earnings)
  3. Ask what will happen if I follow my current behavior forever, and/or when I intend to actually stop it.
  4. If I’m unreasonable in one way, try to fix that more before giving up and directing resources away from myself.
  5. Note that I am a physical system, not a disembodied spirit of pure intellect.
  6. Try to be clear on counterfactuals, rather than what is being directly traded. e.g. if you give money to charity A, the counterfactual is often giving to charity B, rather than making less effort. So the comparison between your effort and charity A is irrelevant.
  7. Adopt growth mindset