Results for Marginals Solvers: Exact Track
The following solvers participated in the Exact Marginals evaluation:
 Hugin (Hugin)
 Accepts only Bayesian networks
 Evaluated doubleprecision and singleprecision solvers
 Ace (UCLA)
 Accepts all network types
 SMILE (UPitt)
 Accepts only Bayesian networks
In the following we give three sets of plots depicting two measures:
Cumulative time as a function of number of instances solved (which we call "cumulative score"): compares the cumulative time
latex error! exitcode was 2 (signal 0), transscript follows:
it takes to solve the firstlatex error! exitcode was 2 (signal 0), transscript follows:
fastest instances. (2 sets of plots)Number of instances solved over time: illustrates the number of instances solved,
latex error! exitcode was 2 (signal 0), transscript follows:
, by timelatex error! exitcode was 2 (signal 0), transscript follows:
.
These measures are described in more detail below.
Instances Solved vs Cumulative Time
These plots illustrate simultaneously how many instances a solver can solve, and how quickly it solves them. For each solver, we sort its solved instances by the time it took to solve them. Consider the following, which plots the performance of a single solver.

At a point
latex error! exitcode was 2 (signal 0), transscript follows:gives us the cumulative time that a solver took to solve its first
latex error! exitcode was 2 (signal 0), transscript follows:fastest instances. From lefttoright, we can gauge the relative difficulty of instances for this particular solver, on this particular benchmark. More specifically, we see that many instances are relatively easy (solved quickly), where a fraction of the most difficult instances required the majority of the solver's time.
In these plots, given below, the further right a solver reaches (more instances solved), and closer to the xaxis a solver is (faster), the better it is considered to perform. Note that instances are sorted by time for a solver, independently of the other solvers. That is, easy instances of one solver are not necessarily easy instances for another solver.
Plots are in tabular form. Each row corresponds to a set of benchmarks. The first column corresponds to a linear yaxis plot, whereas the second column corresponds to a log yaxis plot. Differences between solvers are more visible in the log plot.
For convenience, the following plots are equivalent to the above linear yaxis plots, at a larger scale.
Number of instances solved over time
In these plots, given below, we plot for each exact solver, the number of instances solved as a function of time (with a limit of 20 minutes). For example, the following plots performance over all benchmarks.

Note that the above plot does not take into account the vastly differing number of benchmark set sizes as well as the fact that some solvers do not solve Markov networks.