UAI'08 Workshop: Evaluating and Disseminating Probabilistic Reasoning Systems

Date: July 9, 2008 - Co-located with UAI'08

The workshop will provide a forum for discussing issues arising in empirical evaluation and dissemination of probabilistic reasoning algorithms. It will also provide a framework for a probabilistic reasoning evaluation which will take place a month before the workshop and whose results will be part of the workshop discussion.

MOTIVATION

Over the past two decades a variety of exact and approximate algorithms were developed across several communities (e.g. UAI, NIPS, SAT/CSPs) for answering optimization and likelihood queries over probabilistic graphical models. Since all these tasks are NP-hard, theoretical guarantees are rare and empirical evaluation becomes a central evaluation tool. Yet, the empirical comparison be- tween algorithms requires agreement on representations, benchmarks and evaluation criteria which is challenging, especially in the context of approximation algorithms.

Some communities have already addressed similar challenges through yearly empirical evaluations and competitions (e.g. SAT, CSP and planning) which proved effective, leading to algorithmic advances and to software development and dissemination. We believe that such an effort could benefit probabilistic inference algorithms as well. Probabilistic reasoning presents additional challenges, however, as it tends to be harder, requires heterogenous knowledge representation frameworks, and must deal with the issue of evaluating approximate inference algorithms.

GOALS

Our goal is to establish some standards for evaluating probabilistic reasoning systems based on both exact and approximate algorithms that take the following issues into account:

On the dissemination side, the goal is to reinforce a tradition of building and sharing probabilistic reasoning systems that allows easy access to state-of-the-art inference algorithms by members of the broader scientific and engineering communities. This dissemination is meant to achieve a number of objectives:

FORMAT

The workshop will consist of paper and poster presentations, invited talks, panels, and system demonstrations. An inference evaluation will take place in the month preceding the workshop, with the results presented and discussed during the workshop.

CALL FOR PAPERS

We welcome abstracts describing contributions as well as position papers which will be reviewed and selected for either plenary or poster presentations. Subjects of interest include (but are not limited to) evaluation criteria of probabilistic reasoning algorithms, whether domain specific or domain independent, especially on problems for which exact inference is not feasible; trading off accuracy with computational resources in real-world applications; descriptions of challenging benchmarks, whether real-world or synthetic, and their role in driving empirical evaluations; representations of graphical models (and factors) that are commonplace in certain domains (e.g., speech recognition); system descriptions and demonstrations.

Abstract submissions should not exceed 10 pages and must be in pdf format (plain text is acceptable for short abstracts).

CALL FOR PARTICIPATION IN THE EVALUATION

We encourage participation in the probabilistic inference evaluation which will include both Bayesian and Markov networks and consider three inference tasks: probability of evidence (partition function), most probable explanations (also called MPE, MAP or energy minimization), and node marginals. The evaluation will consider both exact and approximate algorithms, especially any-time algo- rithms that improve their approximations with time. Details of the evaluation can be found at:

CALL FOR BENCHMARKS

We encourage the submission of benchmarks in the form of either Bayesian or Markov networks. The preferred file format is described at:

Other formats may potentially be acceptable, yet the evaluation will assume the format above.

DATES

Submissions should be emailed to pc-chairs@ics.uci.edu by the following deadlines:

ORGANIZERS

Workshop (last edited 2008-09-14 21:21:09 by ArthurChoi)