Belief aggregation

Belief aggregation,^[1] also called risk aggregation,^[2] opinion aggregation^[3] or probabilistic opinion pooling,^[4] is a process in which different probability distributions, produced by different experts, are combined to yield a single probability distribution.

Background

Expert opinions are often uncertain. Rather than saying e.g. "it will rain tomorrow", a weather expert may say "it will rain with probability 70% and be sunny with probability 30%". Such a statement is called a belief. Different experts may have different beliefs; for example, a different weather expert may say "it will rain with probability 60% and be sunny with probability 40%". In other words, each expert has a subjeciive probability distribution over a given set of outcomes.

A belief aggregation rule is a function that takes as input two or more probability distributions over the same set of outcomes, and returns a single probability distribution over the same space.

Applications

Documented applications of belief aggregation include:

• Prediction of volcanic activity.^[5][6]
• Predicting the likelihood of abrupt climate change.^[7]
• Predicting future polar bear population.^[8]

During COVID-19, the European Academy of Neurology developed an ad-hoc three-round voting method to aggregate expert opinions and reach a consensus.^[9]

Common rules

Common belief aggregation rules include:

• Linear aggregation (also called average voting rule) - selecting the weighted or unweighted arithmetic mean of the experts' reports.
• Geometric aggregation - selecting the weighted or unweighted geometric mean of the reports.
• Multiplicative aggregation - selecting the product of probabilities.

Dietrich and List^[4] present axiomatic characterizations of each class of rules. They argue that that linear aggregation can be justified “procedurally” but not “epistemically”, while the other two rules can be justified epistemically. Geometric aggregation is justified when the experts' beliefs are based on the same information, and multiplicative aggregation is justified when the experts' beliefs are based on private information.

Properties of belief aggregation rules

A belief aggregation rule should arguably satisfy some desirable properties, or axioms:

• Zero preservation^[3] means that, if all experts agree that an event has zero probability, then the same should hold in the aggregated distribution. An equivalent axiom is consensus preservation^[10] or certainty preservation^[1], which means that, if all experts agree that an event has probability 1, then the same should hold in the aggregated distribution. This is a basic axiom that is satisfied by linear, geometric and multiplicative aggregation, as well as many others.
• Plausibility preservation means that, if all experts agree that an event has a positive probability, then the same should hold in the aggregated distribution. This axiom is satisfied by linear aggregation.
• Proportionality means that, if each expert assigns probability 1 to a single outcome, then the aggregated distribution is the average (or the weighted average) of the expert beliefs. This axiom is satisfied by linear aggregation.
• Diversity is stronger than proportionality. It means that the support of the aggregated distribution contains the supports of all experts' beliefs. In other words, if any event has a positive probability to at least one expert, that it has a positive probability to society. This axiom is satisfied by linear aggregation.

Truthful aggregation rules with money

Most literature on belief aggregation assumes that the experts report their beliefs honestly, as their main goal is to help the decision-maker get to the truth. In practice, experts may have strategic incentives. For example, the FDA uses advisory committees, and there have been controversies due to conflicts of interests within these committees.^[11] Therefore, a truthful mechanism for belief aggregation could be useful.

In some settings, it is possible to pay the experts a certain sum of money, depending both on their expressed belief and on the realized outcome. Careful design of the payment function (often called a "scoring rule") can lead to a truthful mechanism. Various truthful scoring rules exist.^{[12][13][14][15]}

Truthful aggregation rules without money

In some settings, monetary transfers are not possible. For example, the realized outcome may happen in the far future, or a wrong decision may be catastrophic.

To develop truthful mechanisms, one must make assumptions about the experts' preferences over the set of accepted probability-distributions. If the space of possible preferences is too rich, then strong impossibility results imply that the only truthful mechanism is the dictatorship mechanism (see Gibbard–Satterthwaite theorem).

Single-peaked preferences

A useful domain restriction is that the experts have single-peaked preferences. An aggregation rule is called one-dimensional strategyproof (1D-SP) if whenever all experts have single-peaked preferences, and submit their peaks to the aggregation rule, no expert can impose a strictly better aggregated distribution by reporting a false peak. An equivalent property is called uncompromisingness:^[16] it says that, if the belief of expert i is smaller than the aggregate distribution, and i changes his report, then the aggregate distribution will be weakly larger; and vice-versa.

Moulin^[17] proved a characterization of all 1D-SP rules, as well as the following two characterizations:

• A rule is anonymous and 1D-SP for all single-peaked preferences iff it is equivalent to a median voting rule with at most n+1 "phantoms".
• A rule is anonymous, 1D-SP and Pareto-efficient for all single-peaked preferences iff it is equivalent to a median voting rule with at most n-1 phantoms.

Jennings, Laraki, Puppe and Varloot^[18] present new characterizations of strategyproof mechanisms with single-peaked preferences.

Single-peaked preferences of the pdf

A further restriction of the single-peaked domain is that agents have single-peaked preferences with L1 metric on the probability density function. That is: for each agent i, there is an "ideal" probability distribution p_i, and his utility from a selected probability distribution p* is minus the L1 distance between p_i and p*. An aggregation rule is called L1-metric-strategyproof (L1-metric-SP) if whenever all experts have single-peaked preferences with L1 metric, and submit their peaks to the aggregation rule, no expert can impose a strictly better aggregated distribution by reporting a false peak. Several L1-metric-SP aggregation rules were suggested in the context of budget-proposal aggregation:

• Goel, Krishnaswamy and Sakshuwong^[19] proved the existence of a Pareto optimal aggregation rule that is L1-metric-SP;
• Freeman, Pennock, Peters and Vaughan^[20] presented a rule called moving phantoms, which is L1-metric-SP and satisfies a fairness property (but it is not Pareto-optimal). They also presented a family of L1-metric-SP rules based on the median rule.

However, such preferences may not be a good fit for belief aggregation, as they are neutral - they do not distinguish between different outcomes. For example, suppose there are three outcomes, and the expert's belief p_i assigns 100% to outcome 1. Then, the L1 metric between p_i and "100% outcome 2" is 2, and the L1 metric between p_i and "100% outcome 3" is 2 too. The same is true for any neutral metric. This makes sense when 1,2,3 are budget items. However, if these outcomes describe the potential strength of an earthquake in the Richter scale, then the distance between p_i to "100% outcome 2" should be much smaller than the distance to "100% outcome 3".

Single-peaked preferences on the cdf

Varloot and Laraki^[1] study a different preference domain, in which the outcomes are linearly ordered, and the preferences are single-peaked in the space of cumulative distribution function (cdf). That is: each agent i has an ideal cumulative distribution function c_i, and his utility depends negatively on the distance between c_i and the accepted distribution c*. They define a new concept called level-strategyproofness (Level-SP), which is relevant when society's decision is based on the question of whether the probability of some event is above or below a given threshold. Level-SP provably implies strategyproofness for a rich class of cdf-single-peaked preferences. They characterize two new aggregation rules:

• The order-cumulative rules are the only aggregation rules that satisfy Level-SP, anonymity, certainty-preservation and plasubility-preservation. A special case of this family is the middlemost cumulative, which is an order-cumulative based on the median.
  • However, these rules are not diverse, for example: if three experts report "99% outcome 1" and one expert reports "99% outcome 2", then every order-cumulative rule will choose either "99% outcome 1" of "99% outcome 2"; however, an outcome such as "75% outcome 1 and 25% outcome 2" is more reasonable.
• The proportional-cumulative rule is the only aggregation rule that satisfies Level-SP and proportionality. It also handles profiles with dominations (where the cdf of each agent i is either entirely above or entirely below the cdf of any other agent j) in a natural way. However, it violates plausibility-preservation.

Other results include:

• There is no aggregation rule that satisfies diversity, Level-SP and unanimity.
• When there are at least 4 outcomes, the only rules that satisfy Level-SP, L1-metric-SP and certainty-preservation are dictatorships (there are rules that satisfy Level-SP and L1-metric-SP, but not certainty-preservation; with 3 outcomes, every level-SP rule is also L1-metric-SP).
• Most results can be extended to assign different weights to different experts (representing their level of expertise).
• A new voting method: majority judgement with uncertainty (MJU). It is a variant of majority judgement which allows voters to express uncertainty about the qualities of each candidate.

Software

ANDURIL^[21] is a MATLAB toolbox for belief aggregation.

See also

• Ensemble forecasting - instead of making a single forecast of the most likely weather, a set (or ensemble) of forecasts is produced, aiming to give an indication of the range of possible future states of the atmosphere.
• Aggregative Contingent Estimation Program - a program of the Office of Incisive Analysis that ran between 2010 and 2015.
• Data assimilation - a mathematical discipline that seeks to optimally combine theory (usually in the form of a numerical model) with observations.
• Scoring rule - can be used to incentivize truthful belief aggregation.
• Sensor fusion - combining sensor data from disparate sources.
• Budget-proposal aggregation - a similar problem in which each expert reports his ideal budget-allocation, and the goal is to aggregate the reports to a common budget-allocation.
• Belief merging - similar to belief aggregation, except that the beliefs are given by logical formulae rather than by probability distributions.

Further reading

Several books on related topics are available.^[22][23][3]

References

1. Varloot, Estelle Marine; Laraki, Rida (2022-07-13). "Level-strategyproof Belief Aggregation Mechanisms". Proceedings of the 23rd ACM Conference on Economics and Computation. EC '22. New York, NY, USA: Association for Computing Machinery. pp. 335–369. doi:10.1145/3490486.3538309. ISBN 978-1-4503-9150-4.
2. Boyer-Kassem, Thomas (January 2019). "Scientific Expertise and Risk Aggregation". Philosophy of Science. 86 (1): 124–144. doi:10.1086/701071. ISSN 0031-8248.
3. Martini, Carlo; Sprenger, Jan (2017-01-01), Boyer-Kassem, Thomas; Mayo-Wilson, Conor; Weisberg, Michael (eds.), "Opinion Aggregation and Individual Expertise", Scientific Collaboration and Collective Knowledge, Oxford University Press USA, pp. 180–201, ISBN 978-0-19-068053-4, retrieved 2023-11-09
4. Dietrich, Franz; List, Christian (March 2014). Probabilistic Opinion Pooling (Report). University Library of Munich, Germany.
5. Aspinall, William P. (2006), "Structured elicitation of expert judgement for probabilistic hazard and risk assessment in volcanic eruptions", Statistics in Volcanology, The Geological Society of London on behalf of The International Association of Volcanology and Chemistry of the Earth's Interior, pp. 15–30, doi:10.1144/iavcei001.2, ISBN 978-1-86239-623-4, retrieved 2023-11-09
6. Christophersen, Annemarie; Deligne, Natalia I.; Hanea, Anca M.; Chardot, Lauriane; Fournier, Nicolas; Aspinall, Willy P. (2018). "Bayesian Network Modeling and Expert Elicitation for Probabilistic Eruption Forecasting: Pilot Study for Whakaari/White Island, New Zealand". Frontiers in Earth Science. 6. doi:10.3389/feart.2018.00211. hdl:10356/85752. ISSN 2296-6463.
7. Arnell, Nigel W.; Tompkins, Emma L.; Adger, W. Neil (December 2005). "Eliciting Information from Experts on the Likelihood of Rapid Climate Change". Risk Analysis. 25 (6): 1419–1431. doi:10.1111/j.1539-6924.2005.00689.x. ISSN 0272-4332.
8. O’Neill, Saffron J.; Osborn, Tim J.; Hulme, Mike; Lorenzoni, Irene; Watkinson, Andrew R. (2008-10-21). "Using expert knowledge to assess uncertainties in future polar bear populations under climate change". Journal of Applied Ecology. 45 (6): 1649–1659. doi:10.1111/j.1365-2664.2008.01552.x. ISSN 0021-8901.
9. von Oertzen, T. J.; Macerollo, A.; Leone, M. A.; Beghi, E.; Crean, M.; Oztuk, S.; Bassetti, C.; Twardzik, A.; Bereczki, D.; Di Liberto, G.; Helbok, R.; Oreja‐ Guevara, C.; Pisani, A.; Sauerbier, A.; Sellner, J. (January 2021). "EAN consensus statement for management of patients with neurological diseases during the COVID‐19 pandemic". European Journal of Neurology. 28 (1): 7–14. doi:10.1111/ene.14521. ISSN 1351-5101. PMC 7675361. PMID 33058321.
10. Dietrich, Franz; List, Christian (2017-04-01). "Probabilistic opinion pooling generalized. Part one: general agendas". Social Choice and Welfare. 48 (4): 747–786. doi:10.1007/s00355-017-1034-z. ISSN 1432-217X.
11. Elster, Jon (2015-06-26). Secrecy and Publicity in Votes and Debates. Cambridge University Press. ISBN 978-1-107-08336-3.
12. Good, I. J. (January 1952). "Rational Decisions". Journal of the Royal Statistical Society, Series B (Methodological). 14 (1): 107–114. doi:10.1111/j.2517-6161.1952.tb00104.x. ISSN 0035-9246.
13. McCarthy, John (September 1956). "Measures of the Value of Information". Proceedings of the National Academy of Sciences. 42 (9): 654–655. doi:10.1073/pnas.42.9.654. ISSN 0027-8424. PMC 534271. PMID 16589926.
14. Winkler, Robert L. (September 1969). "Scoring Rules and the Evaluation of Probability Assessors". Journal of the American Statistical Association. 64 (327): 1073–1078. doi:10.1080/01621459.1969.10501037. ISSN 0162-1459.
15. Friedman, Daniel (April 1983). "Effective Scoring Rules for Probabilistic Forecasts". Management Science. 29 (4): 447–454. doi:10.1287/mnsc.29.4.447. ISSN 0025-1909.
16. Border, Kim C.; Jordan, J. S. (1983). "Straightforward Elections, Unanimity and Phantom Voters". The Review of Economic Studies. 50 (1): 153–170. doi:10.2307/2296962. ISSN 0034-6527. JSTOR 2296962.
17. Moulin, H. (1980-01-01). "On strategy-proofness and single peakedness". Public Choice. 35 (4): 437–455. doi:10.1007/BF00128122. ISSN 1573-7101. S2CID 154508892.
18. Jennings, Andrew B.; Laraki, Rida; Puppe, Clemens; Varloot, Estelle M. (2023-08-28). "New characterizations of strategy-proofness under single-peakedness". Mathematical Programming. arXiv:2102.11686. doi:10.1007/s10107-023-02010-x. ISSN 1436-4646.
19. Goel, Ashish; Krishnaswamy, Anilesh K.; Sakshuwong, Sukolsak; Aitamurto, Tanja (2019-07-29). "Knapsack Voting for Participatory Budgeting". ACM Transactions on Economics and Computation. 7 (2): 8:1–8:27. arXiv:2009.06856. doi:10.1145/3340230. ISSN 2167-8375.
20. Freeman, Rupert; Pennock, David M.; Peters, Dominik; Wortman Vaughan, Jennifer (2019-06-17). "Truthful Aggregation of Budget Proposals". Proceedings of the 2019 ACM Conference on Economics and Computation. EC '19. New York, NY, USA: Association for Computing Machinery. pp. 751–752. arXiv:1905.00457. doi:10.1145/3328526.3329557. ISBN 978-1-4503-6792-9.
21. Leontaris, Georgios; Morales-Nápoles, Oswaldo (2018-01-01). "ANDURIL — A MATLAB toolbox for ANalysis and Decisions with UnceRtaInty: Learning from expert judgments". SoftwareX. 7: 313–317. doi:10.1016/j.softx.2018.07.001. ISSN 2352-7110.
22. Cooke, Roger M. (1991-10-24). Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press. ISBN 978-0-19-536237-4.
23. Armstrong, J. Scott, ed. (2001). "Principles of Forecasting". International Series in Operations Research & Management Science. 30. doi:10.1007/978-0-306-47630-3. ISBN 978-0-7923-7401-5. ISSN 0884-8289.