AI Impacts

I’ve been working on a thing with Paul Christiano that might interest some of you: the AI Impacts project. The basic idea is to apply the evidence and arguments that are kicking around in the world and various disconnected discussions respectively to the big questions regarding a future with AI. For instance, these questions:

  • What should we believe about timelines for AI development?
  • How rapid is the development of AI likely to be near human-level?
  • How much advance notice should we expect to have of disruptive change?
  • What are the likely economic impacts of human-level AI?
  • Which paths to AI should be considered plausible or likely?
  • Will human-level AI tend to pursue particular goals, and if so what kinds of goals?
  • Can we say anything meaningful about the impact of contemporary choices on long-term outcomes?
For example we have recently investigated technology’s general proclivity to abrupt progress, surveyed existing AI surveys, and examined the evidence from chess and other applications regarding how much smarter Einstein is than an intellectually disabled person, among other things.
Some more on our motives and strategy, from our about page:

Today, public discussion on these issues appears to be highly fragmented and of limited credibility. More credible and clearly communicated views on these issues might help improve estimates of the social returns to AI investment, identify neglected research areas, improve policy, or productively channel public interest in AI. The goal of the project is to clearly present and organize the considerations which inform contemporary views on these and related issues, to identify and explore disagreements, and to assemble whatever empirical evidence is relevant. The project is provisionally organized as a collection of posts concerning particular issues or bodies of evidence, describing what is known and attempting to synthesize a reasonable view in light of available evidence. These posts are intended to be continuously revised in light of outstanding disagreements and to make explicit reference to those disagreements.

In the medium run we’d like to provide a good reference on issues relating to the consequences of AI, as well as to improve the state of understanding of these topics. At present, the site addresses only a small fraction of questions one might be interested in, so only suitable for particularly risk-tolerant or topic-neutral reference consumers. However if you are interested in hearing about (and discussing) such research as it unfolds, you may enjoy our blog. If you take a look and have thoughts, we would love to hear them, either in the comments here or in our feedback form. Cross-posted from Less-Wrong.

When should an Effective Altruist be vegetarian?

I have lately noticed several people wondering why more Effective Altruists are not vegetarians. I am personally not a vegetarian because I don’t think it is an effective way to be altruistic.

As far as I can tell the fact that many EAs are not vegetarians is surprising to some because they think ‘animals are probably morally relevant’ basically implies ‘we shouldn’t eat animals’. To my ear, this sounds about as absurd as if Givewell’s explanation of their recommendation of SCI stopped after ‘the developing world exists, or at least has a high probability of doing so’.

(By the way, I do get to a calculation at the bottom, after some speculation about why the calculation I think is appropriate is unlike what I take others’ implicit calculations to be. Feel free to just scroll down and look at it).

I think this fairly large difference between my and many vegetarians’ guesses at the value of vegetarianism arises because they think the relevant question is whether the suffering to the animal is worse than the pleasure to themselves at eating the animal. This question sounds superficially plausibly relevant, but I think on closer consideration you will agree that it is the wrong question.

The real question is not whether the cost to you is small, but whether you could do more good for the same small cost.

Similarly, when deciding whether to donate $5 to a random charity, the question is whether you could do more good by donating the money to the most effective charity you know of. Going vegetarian because it relieves the animals more than it hurts you is the equivalent of donating to a random developing world charity because it relieves the suffering of an impoverished child more than foregoing $5 increases your suffering.

Trading with inconvenience and displeasure

My imaginary vegetarian debate partner objects to this on grounds that vegetarianism is different from donating to ineffective charities, because to be a vegetarian you are spending effort and enjoying your life less rather than spending money, and you can’t really reallocate that inconvenience and displeasure to, say, preventing artificial intelligence disaster or feeding the hungry, if don’t use it on reading food labels and eating tofu. If I were to go ahead and eat the sausage instead – the concern goes – probably I would just go on with the rest of my life exactly the same, and a bunch of farm animals somewhere would be the worse for it, and I scarcely better.

I agree that if the meat eating decision were separated from everything else in this way, then the decision really would be about your welfare vs. the animal’s welfare, and you should probably eat the tofu.

However whether you can trade being vegetarian for more effective sacrifices is largely a question of whether you choose to do so. And if vegetarianism is not the most effective way to inconvenience yourself, then it is clear that you should choose to do so. If you eat meat now in exchange for suffering some more effective annoyance at another time, you and the world can be better off.

Imagine an EA friend says to you that she gives substantial money to whatever random charity has put a tin in whatever shop she is in, because it’s better than the donuts and new dresses she would buy otherwise. She doesn’t see how not giving the money to the random charity would really cause her to give it to a better charity – empirically she would spend it on luxuries. What do you say to this?

If she were my friend, I might point out that the money isn’t meant to magically move somewhere better – she may have to consciously direct it there. She might need to write down how much she was going to give to the random charity, then look at the note later for instance. Or she might do well to decide once and for all how much to give to charity and how much to spend on herself, and then stick to that. As an aside, I might also feel that she was using the term ‘Effective Altruist’ kind of broadly.

I see vegetarianism for the sake of not managing to trade inconveniences as quite similar. And in both cases you risk spending your life doing suboptimal things every time a suboptimal altruistic opportunity has a chance to steal resources from what would be your personal purse. This seems like something that your personal and altruistic values should cooperate in avoiding.

It is likely too expensive to keep track of an elaborate trading system, but you should at least be able to make reasonable long term arrangements. For instance, if instead of eating vegetarian you ate a bit frugally and saved and donated a few dollars per meal, you would probably do more good (see calculations lower in this post). So if frugal eating were similarly annoying, it would be better. Eating frugally is inconvenient in very similar ways to vegetarianism, so is a particularly plausible trade if you are skeptical that such trades can be made. I claim you could make very different trades though, for instance foregoing the pleasure of an extra five minute’s break and working instead sometimes. Or you could decide once and for all how much annoyance to have, and then choose most worthwhile bits of annoyance, or put a dollar value on your own time and suffering and try to be consistent.

Nebulous life-worsening costs of vegetarianism

There is a separate psychological question which is often mixed up with the above issue. That is, whether making your life marginally less gratifying and more annoying in small ways will make you sufficiently less productive to undermine the good done by your sacrifice. This is not about whether you will do something a bit costly another time for the sake of altruism, but whether just spending your attention and happiness on vegetarianism will harm your other efforts to do good, and cause more harm than good.

I find this plausible in many cases, but I expect it to vary a lot by person. My mother seems to think it’s basically free to eat supplements, whereas to me every additional daily routine seems to encumber my life and require me to spend disproportionately more time thinking about unimportant things. Some people find it hard to concentrate when unhappy, others don’t. Some people struggle to feed themselves adequately at all, while others actively enjoy preparing food.

There are offsetting positives from vegetarianism which also vary across people. For instance there is the pleasure of self-sacrifice, the joy of being part of a proud and moralizing minority, and the absence of the horror of eating other beings. There are also perhaps health benefits, which probably don’t vary that much by people, but people do vary in how big they think the health benefits are.

Another  way you might accidentally lose more value than you save is in spending little bits of time which are hard to measure or notice. For instance, vegetarianism means spending a bit more time searching for vegetarian alternatives, researching nutrition, buying supplements, writing emails back to people who invite you to dinner explaining your dietary restrictions, etc. The value of different people’s time varies a lot, as does the extent to which an additional vegetarianism routine would tend to eat their time.

On a less psychological note, the potential drop in IQ (~5 points?!) from missing out on creatine is a particularly terrible example of vegetarianism making people less productive. Now that we know about creatine and can supplement it, creatine itself is not such an issue. An issue does remain though: is this an unlikely one-off failure, or should we worry about more such deficiency? (this goes for any kind of unusual diet, not just meat-free ones).

How much is avoiding meat worth?

Here is my own calculation of how much it costs to do the same amount of good as replacing one meat meal with one vegetarian meal. If you would be willing to pay this much extra to eat meat for one meal, then you should eat meat. If not, then you should abstain. For instance, if eating meat does $10 worth of harm, you should eat meat whenever you would hypothetically pay an extra $10 for the privilege.

This is a tentative calculation. I will probably update it if people offer substantially better numbers.

All quantities are in terms of social harm.

Eating 1 non-vegetarian meal

< eating 1 chickeny meal (I am told chickens are particularly bad animals to eat, due to their poor living conditions and large animal:meal ratio. The relatively small size of their brains might offset this, but I will conservatively give all animals the moral weight of humans in this calculation.)

< eating 200 calories of chicken (a McDonalds crispy chicken sandwich probably contains a bit over 100 calories of chicken (based on its listed protein content); a Chipotle chicken burrito contains around 180 calories of chicken)

= causing ~0.25 chicken lives (1 chicken is equivalent in price to 800 calories of chicken breast i.e. eating an additional 800 calories of chicken breast conservatively results in one additional chicken. Calculations from data here and here.)

< -$0.08 given to the Humane League (ACE estimates the Humane League spares 3.4 animal lives per dollar). However since the humane league basically convinces other people to be vegetarians, this may be hypocritical or otherwise dubious.

< causing 12.5 days of chicken life (broiler chickens are slaughtered at between 35-49 days of age)

= causing 12.5 days of chicken suffering (I’m being generous)

< -$0.50 subsidizing free range eggs,  (This is a somewhat random example of the cost of more systematic efforts to improve animal welfare, rather than necessarily the best. The cost here is the cost of buying free range eggs and selling them as non-free range eggs. It costs about 2.6 2004 Euro cents [= US 4c in 2014] to pay for an egg to be free range instead of produced in a battery. This corresponds to a bit over one day of chicken life. I’m assuming here that the life of a battery egg-laying chicken is not substantially better than that of a meat chicken, and that free range chickens have lives that are at least neutral. If they are positive, the figure becomes even more favorable to the free range eggs).

< losing 12.5 days of high quality human life (assuming saving one year of human life is at least as good as stopping one year of an animal suffering, which you may disagree with.)

= -$1.94-5.49 spent on GiveWell’s top charities (This was GiveWell’s estimate for AMF if we assume saving a life corresponds to saving 52 years – roughly the life expectancy of children in Malawi. GiveWell doesn’t recommend AMF at the moment, but they recommend charities they considered comparable to AMF when AMF had this value.

GiveWell employees’ median estimate for the cost of ‘saving a life’ through donating to SCI is $5936 [see spreadsheet here]. If we suppose a life  is 37 DALYs, as they assume in the spreadsheet, then 12.5 days is worth 5936*12.5/37*365.25 = $5.49. Elie produced two estimates that were generous to cash and to deworming separately, and gave the highest and lowest estimates for the cost-effectiveness of deworming, of the group. They imply a range of $1.40-$45.98 to do as much good via SCI as eating vegetarian for a meal).

Given this calculation, we get a few cents to a couple of dollars as the cost of doing similar amounts of good to averting a meat meal via other means. We are not finished yet though – there were many factors I didn’t take into account in the calculation, because I wanted to separate relatively straightforward facts for which I have good evidence from guesses. Here are other considerations I can think of, which reduce the relative value of averting meat eating:

  1. Chicken brains are fairly small, suggesting their internal experience is less than that of humans. More generally, in the spectrum of entities between humans and microbes, chickens are at least some of the way to microbes. And you wouldn’t pay much to save a microbe.
  2. Eating a chicken only reduces the number of chicken produced by some fraction. According to Peter Hurford, an extra 0.3 chickens are produced if you demand 1 chicken. I didn’t include this in the above calculation because I am not sure of the time scale of the relevant elasticities (if they are short-run elasticities, they might underestimate the effect of vegetarianism).
  3. Vegetable production may also have negative effects on animals.
  4. Givewell estimates have been rigorously checked relative to other things, and evaluations tend to get worse as you check them. For instance, you might forget to include any of the things in this list in your evaluation of vegetarianism. Probably there are more things I forgot. That is, if you looked into vegetarianism with the same detail as SCI, it would become more pessimistic, and so cheaper to do as much good with SCI.
  5. It is not at all obvious that meat animal lives are not worth living on average. Relatedly, animals generally want to be alive, which we might want to give some weight to.
  6. Animal welfare in general appears to have negligible predictable effect on the future (very debatably), and there are probably things which can have huge impact on the future. This would make animal altruism worse compared to present-day human interventions, and much worse compared to interventions directed at affecting the far future, such as averting existential risk.

My own quick guesses at factors by which the relative value of avoiding meat should be multiplied, to account for these considerations:

  1. Moral value of small animals: 0.05
  2. Raised price reduces others’ consumption: 0.5
  3. Vegetables harm animals too: 0.9
  4. Rigorous estimates look worse: 0.9
  5. Animal lives might be worth living: 0.2
  6. Animals don’t affect the future: 0.1 relative to human poverty charities

Thus given my estimates, we scale down the above figures by 0.05*0.5*0.9*0.9*0.2*0.1 =0.0004. This gives us $0.0008-$0.002 to do as much good as eating a vegetarian meal by spending on GiveWell’s top charities. Without the factor for the future (which doesn’t apply to these other animal charities), we only multiply the cost of eating a meat meal by 0.004. This gives us a price of $0.0003 with the Humane League, or $0.002 on improving chicken welfare in other ways. These are not price differences that will change my meal choices very often! I think I would often be willing to pay at least a couple of extra dollars to eat meat, setting aside animal suffering. So if I were to avoid eating meat, then assuming I keep fixed how much of my budget I spend on myself and how much I spend on altruism, I would be trading a couple of dollars of value for less than one thousandth of that.

I encourage you to estimate your own numbers for the above factors, and to recalculate the overall price according to your beliefs. If you would happily pay this much (in my case, less than $0.002) to eat meat on many occasions, you probably shouldn’t be a vegetarian. You are better off paying that cost elsewhere. If you would rarely be willing to pay the calculated price, you should perhaps consider being a vegetarian, though note that the calculation was conservative in favor of vegetarianism, so you might want to run it again more carefully. Note that in judging what you would be willing to pay to eat meat, you should take into account everything except the direct cost to animals.

There are many common reasons you might not be willing to eat meat, given these calculations, e.g.:

  • You don’t enjoy eating meat
  • You think meat is pretty unhealthy
  • You belong to a social cluster of vegetarians, and don’t like conflict
  • You think convincing enough others to be vegetarians is the most cost-effective way to make the world better, and being a vegetarian is a great way to have heaps of conversations about vegetarianism, which you believe makes people feel better about vegetarians overall, to the extent that they are frequently compelled to become vegetarians.
  • ‘For signaling’ is another common explanation I have heard, which I think is meant to be similar to the above, though I’m not actually sure of the details.
  • You aren’t able to treat costs like these as fungible (as discussed above)
  • You are completely indifferent to what you eat (in that case, you would probably do better eating as cheaply as possible, but maybe everything is the same price)
  •  You consider the act-omission distinction morally relevant
  • You are very skeptical of the ability to affect anything, and in particular have substantially greater confidence in the market – to farm some fraction of a pig fewer in expectation if you abstain from pork for long enough – than in nonprofits and complicated schemes. (Though in that case, consider buying free-range eggs and selling them as cage eggs).
  • You think the suffering of animals is of extreme importance compared to the suffering of humans or loss of human lives, and don’t trust the figures I have given for improving the lives of egg-laying chickens, and don’t want to be a hypocrite. Actually, you still probably shouldn’t here – the egg-laying chicken number is just an example of a plausible alternative way to help animals. You should really check quite a few of these before settling.

However I think for wannabe effective altruists with the usual array of characteristics, vegetarianism is likely to be quite ineffective.

Seán Ó hÉigeartaigh on FHI and CSER

This is the last part of the Cause Prioritization Shallow, all parts of which are available here. Previously in this series, conversations with Owen Cotton-BarrattPaul ChristianoPaul PenleyGordon Irlam, and Alexander Berger, and Robert Wiblin.

Nick Beckstead interviewed Seán Ó hÉigeartaigh on the Future of Humanity Institute (FHI) and the Center for the Study of Existential Risk (CSER). The notes are here.

sean

Robert Wiblin on the Copenhagen Consensus Center

This post summarizes a conversation which was part of the Cause Prioritization Shallow, all parts of which are available here. Previously in this series, conversations with Owen Cotton-BarrattPaul ChristianoPaul PenleyGordon Irlam, and Alexander Berger.

LOGO CHANGES

Participants

Katja Grace: Research assistant, Machine Intelligence Research Institute

Robert Wiblin: Executive Director, Center for Effective Altruism

Summary

Robert talked extensively with the Copenhagen Consensus Center (CCC) while investigating them as a potential Giving What We Can recommendation for funding[1]. This is Katja’s summary of relevant things Robert learned, from a conversation on the 16th of January 2014, supplemented with facts from the CCC website.

Activities and impact

At a high level, CCC’s main activity is prioritizing a broad variety of altruistic interventions based on cost-effectiveness. They do this by commissioning top economists to research and write about good spending opportunities, using a cost-benefit analysis framework. They do secondary research, assembling existing academic evidence into actionable priorities for governments and philanthropists. An important feature of this work is that they squeeze the analysis of a wide range of topics into the same framework, so one can make reasonable comparisons, given a lot of assumptions.

CCC also devotes substantial efforts to encouraging people to use this decision-support, and in general to prioritize based on good data analysis. In the Millenium Development Goals project for instance, money is probably divided between one third on research and two thirds on dissemination of that research.

CCC usually has around one main project at a time, and as one finishes they dovetail into another. They have around 4-5 core staff, and bring in extra contractors for a lot of the work. The annual budget is $1-2M, and the cost of core staff is probably only around a couple of hundred thousand dollars annually.

The value from CCC’s work does not usually come from finding unsuspected good interventions. It is rather from linking together evidence to make a strong cases for activities that are already believed to be good among experts, but which aren’t widely supported. CCC has for instance highlighted the high value of health interventions relating to nutrition and contagious disease. The notion that these are very good interventions is not unusual among development people, but most of the money is spent elsewhere, so there is a lot of value in making such cases. That CCC usually reaches such plausible conclusions suggests their research method is sensible. Their view on climate change is an exception to this trend; it is quite unusual.

CCC have provided a number of documents on the impact of their work[2]. They have numerous examples of people listening to them and doing things. They can also point to media coverage and a modest number of cases where they said to do something, and talked about it and soon afterwards the person did something like that. It is hard to establish causation, but this is suggestive evidence.

Very few people do anything similar to CCC. Cause prioritization is rare. Doing comparison work at all is a somewhat unique selling point, as is asking for quantitative estimates on things that are not often quantified. Talking about why climate change is not the best cause is also a niche activity.

Contributing to CCC

When Robert spoke to them, CCC was looking for funding for their post-2015 (Millenium Development Goals) project[3]. Their website suggests they still are, along with an American Prosperity Consensus 2014 project and a Global Consensus 2016 project. If the post-2015 is not completely funded there will be less outreach than hoped. They will engage less with the media, and won’t be able to afford some events with officials, where they intend to describe the research and try to persuade them.

Their other recent work includes a book on how much problems have cost the world[4]. Much of the data in it had never been published before, since economic models are seldom run “backwards” – into history.

CCC is currently looking for two summer interns for their back-office in Budapest, Hungary. The desired profile for these positions includes graduate education in an area relevant to CCC’s work (in particular, relating to research project management and outreach) and an interest in and aptitude for digital media (including social media, search, video, web sites). Interns will be assigned tasks and mini-projects within the post-2015 project and the general outreach program, and report to the post-2015 engagement manager and/or the post-2015 project manager. Good mutual match could lead to a permanent position.

[1] Their report is Smart Development Goals: A promising opportunity to influence aid spending via post-MDGs? Giving What We Can.

[2] See Smart Development Goals: A promising opportunity to influence aid spending via post-MDGs? Giving What We Can, p11-12

[3] http://www.copenhagenconsensus.com//post-2015-consensus

[4] http://www.copenhagenconsensus.com/scorecard-humanity

Alexander Berger on GiveWell Labs

This post summarizes a conversation which was part of the Cause Prioritization Shallow, all parts of which are available here. Previously in this series, conversations with Owen Cotton-BarrattPaul ChristianoPaul Penley, and Gordon Irlam.

Participants

Alexander Berger: Senior Research Analyst, GiveWell

Katja Grace: Research Assistant, Machine Intelligence Research Institute

Notes

Since this conversation, GiveWell Labs has become the Open Philanthropy Project.

This is a summary made by Katja of points made by Alexander Berger during a conversation about GiveWell Labs and cause prioritization research on March 5th 2014.

OPP720_2_1-2_copy_for_web

GiveWell Labs

Focus

GiveWell Labs is trying to answer the same basic question as GiveWell: “what’s the best way to spend money?” However GiveWell Labs is answering this question for larger amounts of money, which is less straightforward. Causes are a more useful unit than charities for very large donors. So instead of trying to figure out which charity one should give to this year, they are asking which program areas a foundation should work on.

Givewell’s relationship with Good Ventures is a substantial reason for focussing on the needs of big donors, and GiveWell Labs research has been done in partnership with Good Ventures. The long term vision is to have ongoing research into which areas that should be covered are not, while providing support for a wide range of foundations working on problems they have previously identified as important.

Approach to research

GiveWell Labs primarily aggregates information, rather than producing primary research. It also puts a small amount of effort into publicizing its research.

Their research process focuses on answering these questions:

  • how important is the issue?
  • how tractable is it?
  • how crowded is it?

They attempt to answer the questions at increasing levels of depth for a variety of areas. It is not certain that these are key criteria for determining returns through a program, but they seems correct intuitively.

Most research is done through speaking to experts (rather than e.g. reading research papers). The ‘importance’ question is the only one likely to have academic research on it.

The learning process

GiveWell Labs is prioritizing learning value and diversification at the moment, and not aiming to make decisions about cause priorities once and for all. Alexander would guess that the impact of GiveWell Labs’ current efforts is divided roughly equally between immediate useful research output and the value of trying this project and seeing how it goes.

In the time it has existed, GiveWell Labs has learned a lot. A big question at the moment is how much confidence to have in a cause before making the choice to dive into deeper research on it.

Spending money

Starting to spend money is probably a big part of diving deeper. Spending money is useful for learning more about an area for two reasons. Firstly, it makes you more credible. Secondly, it encourages people to make proposals. People don’t tend to have proposals readily formulated. They respond to the perception of concrete available funding. This means you will get a better sense of the opportunities if you are willing to spend money.

Transferability of learning

Alexander doesn’t know whether methodological insights discovered in one cause prioritization effort are likely to be helpful to others. One relevant factor is that people at GiveWell Labs have priors about what’s likely to be successful that are partly based on what they have learned before starting the process. But if you didn’t share the starting priors, you might not end up with the same current beliefs. This might be true regarding explicit expected value calculations, and how to weigh robustness or reliability against a high upside, in particular. If you don’t share the same prior, the lessons learned may not be very communicable.

Funding cause prioritization

Adding resources to GiveWell

An outside funder trying to donate to GiveWell Labs couldn’t change the distribution from GiveWell’s conventional research to GiveWell Labs. It would also be hard to change the total work done by donating. Donating would mainly change the amount of time GiveWell would spend raising other funds, and the amount to which they depend on Good Ventures.

Other cause prioritization efforts

Projects like Katja’s cause prioritization shallow investigation are unlikely to be done by Givewell.

Katja’s Structured Case on AI project is also unlikely to overlap based on GiveWell’s current plans. If Alexander were working on something like this, he would typically initially try to effectively aggregate the views of credible people, rather than initially forming object level views. For instance, he would like to know what would happen if Eliezer could sit down with highly credentialed AI researchers and try to convince them of his view. The AI Structured Case on the other hand is more directed at detailing object level arguments.

Cause prioritization work can become fairly abstract. Givewell Labs tries to keep it grounded in looking for concrete funding opportunities. Others may have comparative advantages in more philosophical investigations, which avoids overlapping, but is also less likely to be informative to GiveWell Labs. GiveWell is unlikely to focus on prediction markets, though it’s not out of the question.

General considerations for funding such research

If others were going to do more concrete work, it is a hard question whether it would be better at this point for them to overlap with GiveWell Labs to provide a check, or avoid overlapping to provide broader coverage.

Answering high level questions such as ‘how good is economic growth?’ doesn’t seem very decision relevant in most cases. This is largely because these issues are hard to pin down, rather than because they are unlikely to make a large difference to evaluations if we could pin them down, though Alexander is also doubtful that they would make a large difference. For instance, Alexander doesn’t expect indirect effects of interventions to be large relative to immediate effects, while Holden Karnofsky (co-executive director of GiveWell) does, but their views on this do not seem to play a big role in their disagreements over what GiveWell Labs should prioritize.

When deciding what to do on cause prioritization, it is important to keep in mind how it will affect anything, such as who will pay attention, and what decisions they will change as a result.

Similar projects

Nick Beckstead and Carl Shulman do similar work in their own time.

Alexander’s understanding is that Copenhagen Consensus Center is doing something a bit different, especially around modeling cost effectiveness estimates. They also seem to be less focussed on influencing specific decisions.

Alexander is not aware of any obvious further people one should talk to that Katja has not thought of.