The Configuration Performance Metrics Dialog
After you have created some configurations and optimized them, you'll need a way to objectively compare their performance. The Configuration Performance Metrics dialog allows you to view computed optimization errors for any configuration in your project. To invoke it, choose Config, Performance Metrics from the main menu.
The Nature of the Displayed Errors
You're probably familiar with optimization errors from the display in the Optimization Status tab of the Output window when the optimization is running. See the figure below for an example.
Error display during optimization is useful for determining when to stop the optimization and whether the results are satisfactory. But after an optimization is stopped and another is run, or if the project is closed, this error information would ordinarily be lost. The Configuration Performance Metrics dialog allows you to see a variety of optimization error metrics at any time, for any configuration in the project, even before an optimization has been run on it. This dialog is shown below.
The Controls of the Configuration Performance Metrics Dialog
The controls of the Configuration Performance Metrics dialog are described below.
The Configuration List
On the left side of the dialog is a list of the configurations in the project. Select the appropriate configuration to see the errors for that configuration.
Frequency Range to Compute Performance Control Group
At the top left of the dialog is a collection of controls grouped within the Frequency Range to Compute Performance group box.
Radio Buttons for Choosing Between Default and Custom Frequency Ranges
The radio buttons labeled Use the optimization frequency range of the configuration and Use a custom frequency range allow you to choose between default and custom frequency ranges over which the response errors are calculated. When Use the optimization frequency range of the configuration is chosen, the optimization frequency range specified on the Criteria property page of the Optimization Options property sheet for the configuration is used to compute the response errors. This range is stored in the configuration itself. When Use a custom frequency range is chosen, the frequency range over which the response errors are computed is determined by the values you specify in the Minimum freq, Hz and Maximum freq, Hz edit controls.
Edit Controls for Specifying a Custom Frequency Range
The Minimum freq, Hz and Maximum freq, Hz edit controls allow you to specify the frequency range over which the response errors are calculated when Use a custom frequency range option is selected. When this option is not selected, and instead the default selection of Use the optimization frequency range of the configuration is in effect, these controls display the default optimization frequency range stored in the configuration, and their contents are read-only. When the Use a custom frequency range option is in effect, editing of the custom frequency range values is enabled, and these controls display the custom frequency range.
When editing of the Minimum freq, Hz and Maximum freq, Hz is in progress, MSO detects this situation and enables the Recalculate button. When you are done editing these values, click Recalculate to signal that editing is complete. This allows MSO to check the validity of the frequency values entered. Upon receiving valid frequency data, it computes the new errors. Whenever the Recalculate button is active and you're done editing, make sure to click it to ensure the error calculations are up to date.
When you toggle between the Use the optimization frequency range of the configuration and Use a custom frequency range options, MSO will automatically recalculate the errors for you. The Recalculate button is only needed when editing of the custom frequency values is in progress.
Why Choose a Custom Frequency Range?
A common application of the Configuration Performance Metrics dialog is to compare the performance of multiple configurations. But sometimes these different configurations arise from trying out different frequency ranges for optimization. Examples of this are found in the tutorial. In such cases, if you try to compare their errors using the default option of Use the optimization frequency range of the configuration, you'd be making the error comparison over different frequency ranges. These would not be fair comparisons. To correct this, specify Use a custom frequency range for all the configurations you're comparing, and set the frequency ranges to be identical for all of them. Then you'll be comparing the errors of the configurations in question over the same frequency range.
Overall Metrics For Configuration Control Group
This group of controls displays five different errors:
- MLP target error, dB RMS
- Seat-to-seat variation, dB
- Final error, dB RMS
- Unweighted target error, dB RMS
- Weighted target error, dB RMS
For the majority of situations, the first three errors are the only ones of interest. These are indicated in the figure below.
These three errors are relevant when the Minimize seat-to-seat variations option is chosen on the Method property page of the Optimization Options property sheet. When in doubt, these are the ones to look at.
The last two errors are relevant when the As flat as possible without additional global EQ option is chosen on the Method property page of the Optimization Options property sheet.
The details of all these errors will be discussed below. These descriptions will make references to the How Optimization Errors are Calculated: the Details page, which has separate sections for Minimize seat-to-seat variations mode and As flat as possible without additional global EQ mode.
MLP target error, dB RMS Edit Control
This error is a measure of how much the MLP response deviates from the target. If the MLP response perfectly matched the target, this error would be zero. A plot of the difference between the MLP response curve and the target curve is computed, and the RMS value of this difference is computed over frequency to yield this error value. Specifics of its computation are described in the Calculating the error of the MLP from the target section of the error calculation details page.
Seat-to-seat variation, dB Edit Control
This error is a measure of how much the responses at all the listening positions deviate from one another. If all the responses matched one another perfectly, this error would be zero regardless of the shape of the responses. Computation of this error is described in detail in the Calculating the average curve and seat-to-seat variation section of the error calculation details page.
Final error, dB RMS Edit Control
When the Minimize seat-to-seat variations option is chosen on the Method property page of the Optimization Options property sheet, this is the error displayed in the Output window when the optimization is running. It combines the MLP target error, dB RMS and the Seat-to-seat variation, dB together into one composite error that the optimizer attempts to minimize. Calculation of this error is described in the Calculating the total error section of the error calculation details page.
The Relationship of the Preceding Three Errors to One Another
The error calculation details page gives details of the error computations by giving a symbolic name to each error, so that a formula can be used to express their relationship to one another in a compact way. The relationship of the three error names above to their counterparts in the formula are specified below.
MLP target error, dB RMS = errorMLP-tgt
Seat-to-seat variation, dB = errorStS
Final error, dB RMS = errortotal
These are related to one another by:
errortotal = sqrt(0.25 * errorMLP-tgt2 + 0.75 * errorStS2)
The formula for the final error can be thought of as a weighted RMS combination of the MLP flatness error and the seat-to-seat variation, with a weight of one-fourth given to the MLP flatness error, and a weight of three-fourths given to the seat-to-seat variation.
Weighted and Unweighted target error, dB RMS Edit Controls
These errors only apply when the As flat as possible without additional global EQ option is chosen on the Method property page of the Optimization Options property sheet. This option is only used for unusual configurations.
For both of these calculations, the flatness error at each listening position is calculated as described in the Details of As Flat as Possible Without Additional Global EQ Mode section of the error calculation details page. The difference between the weighted and unweighted errors lies only in how the errors at each listening position are combined into a single error. This is described in The Distinction Between Unweighted and Weighted Flatness Errors Over Listening Position section of the error calculation details page. When error weights that aren't all equal are provided by the user, both the weighted and unweighted errors will be calculated, and will differ from one another. When no error weights are provided, the weighted error will be equal to the unweighted error.
Calculation Method Control Group
The Calculation Method control group allows you to specify how certain calculations are performed. Depending on how you set up your configuration, some controls may not be available.
Allow different SPLs at different listening positions Checkbox
This checkbox is always available, regardless of how you set up the corresponding configuration. It affects how the Seat-to-seat variation, dB is calculated. When Allow different SPLs at different listening positions is checked, the Seat-to-seat variation, dB calculation ignores SPL differences between listening positions that are independent of frequency. When it is unchecked, such frequency-independent level differences are included in the error, usually making the error slightly larger.
In mathematical terms, when Allow different SPLs at different listening positions is checked, the Seat-to-seat variation, dB is a standard deviation calculation. When this option is unchecked, it is an RMS calculation.
The default checkbox state matches the state of the Allow different SPLs at different listening positions checkbox on the Method property page of the Optimization Options property sheet. That checkbox is unchecked by default.
Checking the Allow different SPLs at different listening positions checkbox on the Method property page of the Optimization Options property sheet is appropriate in the case for which there are only front subs, and you wish to optimize the seat-to-seat variation in multiple seating rows. That would allow the rear rows to have an average SPL that's less than the front rows. In all other cases, that checkbox on the Optimization Options property sheet should be left unchecked.
Listening Positions to Include in Calculations Control Group
The radio buttons of this control group are disabled for almost all configurations. If you have a configuration containing one or more plot-only measurement groups, these radio buttons will be enabled. This allows you to specify what role the plot-only measurement groups play in the error calculations of this dialog.
- Only optimized positions: Plot-only measurement groups play no role in the error calculations.
- Both optimized and non-optimized positions: All measurement groups, including plot-only ones, are included in the error calculations.
- Only non-optimized positions: Only the plot-only measurement groups are included in the error calculations.
If you have measured more listening positions than you have optimized, and you are curious about the various response errors at listening positions that aren't explicitly optimized, you can use these checkboxes to examine these errors in the ways described above.
Errors Pertaining to Individual Listening Positions
The list view titled Metrics For Each Position lists errors associated with each listening position individually. The name of the listening position is given in the Listening position column on the left.
Target error, dB RMS
When the listening position in the left column corresponds to the MLP, the Target error, dB RMS corresponds exactly to the value in the MLP target error, dB RMS edit control, and its calculation is as described above for that item.
When the listening position is not the MLP, Target error, dB RMS refers to the RMS error of that position's frequency response from the target curve. This error is for information purposes and not used in the optimization itself.
Total Error and Mean of Errors From the Average
The two rightmost columns of the list view show statistics of the response at each listening position compared to the average curve. Computation of the average curve is described in the Calculating the average curve and seat-to-seat variation section of the error calculation details page. These errors are computed for each position as follows:
- At each frequency, subtract the average curve's dB SPL value from the dB SPL of the response at that position to form the raw error vs. frequency for that position.
- Compute the average value of the raw error over frequency. This value will appear in the rightmost column, Mean of error from ave, dB.
The following steps depend on whether or not the Allow different SPLs at different listening positions option is checked.
- If it is checked, at each frequency subtract the average value computed in step 2 above from the raw error described in step 1 above to form the shifted error (which will have zero mean over frequency after this operation).
- If it is unchecked, skip to step 4 below.
- Take the sum of the squares over frequency of the errors computed in step 3 above, divide this number by the number of frequencies, then take the square root. When the Allow different SPLs at different listening positions option is checked, this number will correspond to the standard deviation of the error from the average curve. When this option is unchecked, it will equal the RMS error. This number will appear in the Total error from average, dB column.
While these errors are not used by the optimizer, they might be useful for pointing out problematic listening positions.
Sorting the Errors in Each Column
The errors in any column can be sorted in increasing or decreasing order by clicking on the header of the list view's corresponding column. Sorting by the measurement group name in the left column is done alphabetically. When plot-only measurement groups are present in the configuration, the list rows containing such groups will be highlighted in a light gray color.
Exporting the Results to a Text File
You can save the results of all the calculations of this dialog to a text file by pressing the Export button.