Comparing Configurations Using the Configuration Performance Metrics Dialog
After you have created multiple configurations and optimized them, you'll need a way to compare their relative performance in an objective way. The Configuration Performance Metrics dialog allows you to view computed optimization errors for any configuration in your project. To invoke it, choose Config, Performance Metrics from the main menu. You're probably familiar with optimization errors from the display in the Optimization Status tab of the Output window when the optimization is running. See the figure below for an example.
Error display during optimization is useful for determining when to stop the optimization and whether the results are satisfactory. But after an optimization is stopped and another is run, or if the project is closed, this error information would ordinarily be lost. The Configuration Performance Metrics dialog allows you to see a variety of optimization error metrics at any time, for any configuration in the project, even before an optimization has been run on it. This dialog is shown below.
The controls of this dialog are described below.
On the left side of the dialog is a list of the configurations in the project. Select the appropriate configuration to see the errors for that configuration.
Frequency Range to Compute Performance Control Group
At the top left of the dialog is a collection of controls grouped within the Frequency Range to Compute Performance group box.
Radio Buttons for Choosing Between Default and Custom Frequency Ranges
The radio buttons labeled Use the optimization frequency range of the configuration and Use a custom frequency range allow you to choose between default and custom frequency ranges over which the response errors are calculated. When Use the optimization frequency range of the configuration is chosen, the optimization frequency range specified on the Criteria property page of the Optimization Options property sheet for the configuration is used to compute the response errors. This range is stored in the configuration itself. When Use a custom frequency range is chosen, the frequency range over which the response errors are computed is determined by the values you specify in the Minimum freq, Hz and Maximum freq, Hz edit controls.
Edit Controls for Specifying a Custom Frequency Range
The Minimum freq, Hz and Maximum freq, Hz edit controls allow you to specify the frequency range over which the response errors are calculated when Use a custom frequency range option is selected. When this option is not selected, and instead the default selection of Use the optimization frequency range of the configuration is in effect, these controls display the default optimization frequency range stored in the configuration, and their contents are read-only. When the Use a custom frequency range option is in effect, editing of the custom frequency range values is enabled, and these controls display the custom frequency range.
When editing of the Minimum freq, Hz and Maximum freq, Hz is in progress, MSO detects this situation and enables the Recalculate button. When you are done editing these values, click Recalculate to signal that editing is complete. This allows MSO to check the validity of the frequency values entered. Upon receiving valid frequency data, it computes the new errors. Whenever the Recalculate button is active and you're done editing, make sure to click it to ensure the error calculations are up to date.
When you toggle between the Use the optimization frequency range of the configuration and Use a custom frequency range options, MSO will automatically recalculate the errors for you. The Recalculate button is only needed when editing of the custom frequency values is in progress.
Why Choose a Custom Frequency Range?
A common application of the Configuration Performance Metrics dialog is to compare the performance of multiple configurations. But sometimes these different configurations arise from trying out different frequency ranges for optimization. Examples of this are found in the tutorial. In such cases, if you try to compare their errors using the default option of Use the optimization frequency range of the configuration, you'd be making the error comparison over different frequency ranges. These would not be fair comparisons. To correct this, specify Use a custom frequency range for all the configurations you're comparing, and set the frequency ranges to be identical for all of them. Then you'll be comparing the errors of the configurations in question over the same frequency range.
Overall Metrics For Configuration Control Group
This group of controls displays five different errors:
- MLP target error, dB RMS
- Seat-to-seat variation, dB RMS
- Final error, dB RMS
- Unweighted target error, dB RMS
- Weighted target error, dB RMS
For the majority of situations, the first three errors are the only ones of interest. These are indicated in the figure below.
These three errors are relevant when the Minimize seat-to-seat variations option is chosen on the Method property page of the Optimization Options property sheet. When in doubt, these are the ones to look at.
The last two errors are relevant when the As flat as possible without additional global EQ option is chosen on the Method property page of the Optimization Options property sheet.
The details of all these errors will be discussed below. These descriptions will make references to the How Optimization Errors are Calculated: the Details page, which has separate sections for Minimize seat-to-seat variations mode and As flat as possible without additional global EQ mode.
MLP target error, dB RMS Edit Control
This error is a measure of how much the MLP response deviates from the target. If the MLP response perfectly matched the target, this error would be zero. A plot of the difference between the MLP response curve and the target curve is computed, and the RMS value of this difference is computed over frequency to yield this error value. Specifics of its computation are described in the Calculating the error of the MLP from the target section of the error calculation details page.
Seat-to-seat variation, dB RMS Edit Control
This error is a measure of how much the responses at all the listening positions deviate from one another. If all the responses matched one another perfectly, this error would be zero regardless of the shape of the responses. Computation of this error is described in detail in the Calculating the average curve and seat-to-seat variation section of the error calculation details page.
Final error, dB RMS Edit Control
When the Minimize seat-to-seat variations option is chosen on the Method property page of the Optimization Options property sheet, this is the error displayed in the Output window when the optimization is running. It combines the MLP target error, dB RMS and the Seat-to-seat variation, dB RMS together into one composite error that the optimizer attempts to minimize. Calculation of this error is described in the Calculating the total error section of the error calculation details page.
The Relationship of the Preceding Three Errors to One Another
The error calculation details page gives details of the error computations by giving a symbolic name to each error, so that a formula can be used to express their relationship to one another in a compact way. The relationship of the three error names above to their counterparts in the formula are specified below.
MLP target error, dB RMS = errorMLP-tgt
Seat-to-seat variation, dB RMS = errorStS
Final error, dB RMS = errortotal
These are related to one another by:
errortotal = sqrt(0.25 * errorMLP-tgt2 + 0.75 * errorStS2)
The formula for the final error can be thought of as a weighted RMS combination of the MLP flatness error and the seat-to-seat variation, with a weight of one-fourth given to the MLP flatness error, and a weight of three-fourths given to the seat-to-seat variation.
Weighted and Unweighted target error, dB RMS Edit Controls
These errors only apply when the As flat as possible without additional global EQ option is chosen on the Method property page of the Optimization Options property sheet. This option is only used for unusual configurations.
For both of these calculations, the flatness error at each listening position is calculated as described in the Details of As Flat as Possible Without Additional Global EQ Mode section of the error calculation details page. The difference between the weighted and unweighted errors lies only in how the errors at each listening position are combined into a single error. This is described in The Distinction Between Unweighted and Weighted Flatness Errors Over Listening Position section of the error calculation details page. When error weights that aren't all equal are provided by the user, both the weighted and unweighted errors will be calculated, and will differ from one another. When no error weights are provided, the weighted error will be equal to the unweighted error.
Errors Pertaining to Individual Listening Positions
The list view titled Metrics For Each Position lists errors associated with each listening position individually. The name of the listening position is given in the Listening position column on the left.
Target error, dB RMS
When the listening position in the left column corresponds to the MLP, the Target error, dB RMS corresponds exactly to the value in the MLP target error, dB RMS edit control, and its calculation is as described above for that item.
When the listening position is not the MLP, Target error, dB RMS refers to the RMS error of that position's frequency response from the target curve. This error is for information purposes and not used in the optimization itself.
Standard Deviation and Mean of Errors From the Average
The two rightmost columns of the list view show statistics of the response at each listening position compared to the average curve. Computation of the average curve is described in the Calculating the average curve and seat-to-seat variation section of the error calculation details page. These errors are computed for each position as follows:
- At each frequency, subtract the average curve's dB SPL value from the dB SPL of the response at that position to form the raw error vs. frequency for that position.
- Compute the average value of the raw error over frequency. This value will appear in the rightmost column, Mean of error from ave, dB.
- At each frequency, subtract this average value from the raw error to form the shifted error (which will have zero mean over frequency after this operation).
- Take the sum of the squares over frequency of the shifted errors, then divide by the number of frequencies, then take the square root to get the standard deviation. This will appear in the Std dev of error from ave, dB column.
While these errors are not used by the optimizer, they might be useful for pointing out problematic listening positions.