Results for Marginals Solvers: Exact Track

The following solvers participated in the Exact Marginals evaluation:

In the following we give three sets of plots depicting two measures:

These measures are described in more detail below.


Instances Solved vs Cumulative Time

These plots illustrate simultaneously how many instances a solver can solve, and how quickly it solves them. For each solver, we sort its solved instances by the time it took to solve them. Consider the following, which plots the performance of a single solver.

example.png

At a point

latex error! exitcode was 2 (signal 0), transscript follows:

gives us the cumulative time that a solver took to solve its first
latex error! exitcode was 2 (signal 0), transscript follows:

fastest instances. From left-to-right, we can gauge the relative difficulty of instances for this particular solver, on this particular benchmark. More specifically, we see that many instances are relatively easy (solved quickly), where a fraction of the most difficult instances required the majority of the solver's time.

In these plots, given below, the further right a solver reaches (more instances solved), and closer to the x-axis a solver is (faster), the better it is considered to perform. Note that instances are sorted by time for a solver, independently of the other solvers. That is, easy instances of one solver are not necessarily easy instances for another solver.

Plots are in tabular form. Each row corresponds to a set of benchmarks. The first column corresponds to a linear y-axis plot, whereas the second column corresponds to a log y-axis plot. Differences between solvers are more visible in the log plot.

For convenience, the following plots are equivalent to the above linear y-axis plots, at a larger scale.


Number of instances solved over time

In these plots, given below, we plot for each exact solver, the number of instances solved as a function of time (with a limit of 20 minutes). For example, the following plots performance over all benchmarks.

exact-mar.solved.all.png

Note that the above plot does not take into account the vastly differing number of benchmark set sizes as well as the fact that some solvers do not solve Markov networks.

Evaluation/Report/ExactMAR (last edited 2008-09-18 04:04:26 by ArthurChoi)