How We Test Camcorders

First, a brief note about scoring. As you read through a review, you’ll probably notice that we don’t use a simple ‘1 to 10’ type of system. It can confuse some readers, but the reason is simple. Technology changes constantly. If we review a camcorder in January and it’s got the best color performance we’ve ever seen, we could give it a ‘10.’ But what happens when an even better camcorder comes along in October? Is that a ‘10’ as well? Do we go back and downgrade the scores in all the other reviews? That’s just not practical. Instead, we use a relative scoring system on an infinite scale. That way, we can allow for a score of ‘10.5’ when something better comes along. And every couple of years, we reset the clock with a new rubric. You can see how the rubrics are blocked together on the Ratings page… 2009-2010, 2007-2008, and so on.

Each review has over thirty scored sections, but not every section has equal importance. For instance, we think Video Sharpness is more crucial than Playback features, so a weighting system is employed. You can see the weightings for each section on the last page of every review (see Ratings section of Panasonic HDC-TM700, for example). We use the weighted total score for our final, cumulative score. 

DSC Labs 28R ChromaDuMonde Chip Chart
We use it so often, the ChromaDuMonde chart could practically be our logo. This chart, designed by DSC Labs, is used for testing color accuracy in bright light and low light. Because it has resolution trumpets, it can be used as a quick reference for resolution, but we use the chart described below for our final numbers.

DSC Labs CamAlign MultiBurst Test Pattern Chart
This chart is used to test video resolution.

X-Rite ColorChecker Chart
This chart, designed by X-Rite, has long been used as the standard for color testing in photography. We use it to test the color accuracy of still photos taken by camcorders.

Applied Image ISO 12233 Resolution Chart
Applied Image was the provider of this standard chart for testing still photo resolution.

Camcorderinfo.com Moving Still Life Scene
A charming concoction of colorful, moving parts, our custom-built still life scene includes an electric train, a waterfall feature, a wrist watch, a collection of brightly-colored pipe cleaners, and two moving discs – a black & white pattern and a color wheel. Both discs are mounted on Lego Mindstorm NXT motors and programmed to spin at specific speeds.

Final Cut Express
Final Cut Express is the primary software we use to capture video clips, so long as the file formats prove compatible. Because new formats are introduced routinely, no single NLE can handle them all (at least not at the same pace as a busy editorial calendar). In this light, we may use other NLEs when necessary.

DV Rack 2.0 HD
This software is designed as a set of evaluation tools for field use. We use only one function, the waveform monitor, the principle tool for our Low Light Sensitivity test. DV Rack has since been folded into Adobe and marketed as Adobe OnLocation.

Imatest
Imatest is our primary tool for analyzing color accuracy, noise, and still photo resolution. Reviewed.com has worked extensively with Norman Koren, the developer of Imatest, to understand and customize the analysis to meet our goals. Because Imatest is primary a tool for still photos, frame grabs of the video clips are typically run through the software for analysis.

Spectra Professional IV-A Digital Exposure Meter
This is the tool we use to determine light values, measured in lux.

Johnson Hot Shot Laser Level
A tripod-mounted laser, we use this tool to determine the camcorder’s maximum wide angle.

Westinghouse LVM-37w3 HDTV
We use this television as a quick-reference monitor for aligning shots. It’s also the standard monitor for watching the motion test footage and evaluating resolution. However, the TV is not used for determining color accuracy.

Lighting
For bright light testing, we use two Lowell RIFA-Lite softboxes loaded with 750W GE Quartzline Halogen bulbs with a color temperature of 3200 K. In low light, we use two Litepanels LP-Micro LED arrays with a color temperature of 5600 K.

Stabilization
To produce shake, we use a T-LSR-75A linear slide and a T-RS60 rotary stand, produced by the Canadian company Zaber. To track motion we use Adobe After Effects software.

Color (Bright Light)
To determine a camcorder’s color accuracy in bright light, we light the DSC Labs ChromaDuMonde chart at an even 3000 lux. The camcorder then shoots from a fixed position, behind a small opening in a Duvetyn curtain. At minimum, all camcorders are shot in auto mode with a manual white balance. Depending on the complexity of the camcorder, we may shoot the chart in other modes, as well.

The footage is imported to a computer and a series of high-quality bitmap frame grabs are created, typically using Final Cut Pro. The bitmap images are then imported into Imatest, where we run the Multicharts module. Imatest analyzes the 28 color patches on the ChromaDuMonde chart and matches the known color values to what the camcorder recorded. These values are run through a complex series of equations called the CIEDE2000 color error formulas, which calculate the ‘color error’ of the image. The smaller the error, the better the performance.

The chart that we show is called the color error map. It illustrates the difference between the known value and the recorded value for each of the color patches. A longer tail indicates a greater error. Also, consider the direction of the error. Recorded values that move towards the center of the chart are undersaturated, while those moving towards the outer edge are oversaturating. It’s an extremely interesting chart when you compare performance across multiple camcorders, because you begin to see manufacturer preferences in tinkering certain colors.

The final score for this section is based on the average color error under auto mode with a manual white balance.

Noise (Bright Light)
To determine noise in bright light, we light an X-Rite ColorChecker chart at an even 3000 lux. For consumer camcorders, we shoot in auto mode with a manual white balance. It’s rare that a camcorder in this class offers any control over sensitivity, which has the most direct bearing on noise. We may perform additional tests if it does, but the score is based on the auto mode performance.

For professional camcorders, we perform more extensive tests to get an idea of the camcorder’s overall noise characteristics. Using the X-Rite ColorChecker chart, we shoot under every preset gain setting that the camcorder offers (typically in the -6db to +18dB range, with increments every 3dB). Of course, most pro camcorders also include some noise reduction settings. So we go one step farther and test every gain preset in every noise reduction mode. All this data is assembled in one easy to read chart.

In this section of the review, we also publish 100% crops of the ChromaDuMonde chart. We find that this chart is more illustrative than the X-Rite ColorChecker chart of how the human eye perceives noise. However, the ChromaDuMonde is not used in the scoring.

Once all the footage is shot, it’s captured on a computer, bitmap frame grabs are generated, and those bitmaps are imported into Imatest. The noise scores are then calculated by Imatest, broken down into four channels: red, green, blue, and luminance. The smaller the noise percentage, the better the performance.

On consumer camcorders, the final score is based on the noise average of all four channels, according to the performance in auto mode after a manual white balance. On professional camcorders, the score is based on noise values across the entire gain range, including noise reduction modes.

Motion
The quality and characteristics of a camcorder’s rendering of motion is a difficult thing to define. Unlike color, noise, resolution, and so on, there is no standardized metric for quantifying motion performance from a camcorder. (There are methods of quantifying motion on a television, but such methods examine only the quality of the playback, not of the source material.)

We shoot our custom-designed ‘moving still life scene,’ also called the Train Station. It contains a number of simple and complex types of motion, all of which move at fixed speeds. The Train Station is shot at every available frame rate that the camcorder offers.

We then import the video clips and prep them for upload to YouTube. To get the best results, we’ve using QuickTime to convert clips to 1280 × 720 in the .MOV format at an 8000kbps bit rate. Obviously, these are not as high quality as the original, unconverted video, but our traffic would simply get out of hand if we offered uncompressed clips for download. We are considering it for the future.

The final score is based on qualities of trailing, artifacting, smoothness, and high frequency detail.

Video Sharpness
The sharpness that a camcorder actually produces is rarely the same number that the manufacturer advertises. For instance, camcorders that output a 1920 × 1080 picture are not actually capturing one thousand nine-hundred and twenty horizontal lines of information. That’s simply the size of the ‘container’ that the camcorder outputs (also known as the resolution). In fact, there are lots of ways that manufacturers can play with the numbers, emphasizing capabilities of the lens, or the sensor, or something else. The simple fact is, you don’t buy a sensor, and in most cases, you don’t buy a lens. You buy a camcorder – a complete, pre-assembled camcorder, so that’s how we test them.

We light a DSC Labs Multiburst chart at an even 3000 lux. The camcorder is stationed in a fixed position on a tripod. We aim the camcorder, aligning with the chart’s 16:9 guideframes. We then pan the camcorder slowly left and right for about 30 seconds. Then we re-align the camcorder and tilt slowly up and down for about 30 seconds.

After the shooting is complete, we connect the camcorder to our HDTV using the camcorder’s highest quality connection, typically either composite-out, S-video, component-out, or HDMI. We examine the playback footage, looking for the point at which the lines on the chart become indistinguishable.

The reason we test sharpness with the camcorder in motion, rather than a static shot, is simple. When was the last time you shot a video with nothing moving? The inherent nature of video is movement. This may not be the method manufacturers would prefer, but we think it makes the most sense.

The final score for this section is based on the horizontal and vertical sharpness as recorded in auto mode in the 60i frame rate. We may also examine the sharpness in other frame rates, but it does not factor into the score.

Color (Low Light)
We test color accuracy in low light much the same as we do in our bright light color test, with a few obvious changes. The ChromaDuMonde chart is lit at an even 60 lux, roughly equivalent to a single table lamp in a household at night. At 60 lux, you could read a book, but not for extended periods with any comfort.

For consumer camcorders, we manually white balance then test in auto mode with the auto gain control in the ‘on’ position. In most instances, there is no way to even turn it off, so the standard of leaving it on makes sense. If a consumer camcorder offers additional gain controls, we may test them, but they are not factored into the score. Also, we may test in multiple frame rates, if available, but scores are based on the 60i frame rate.

For professional camcorders, we shoot with the gain set to 0dB, 6dB, and 12dB. As with consumer camcorders, we may shoot in alternate frame rates for the sake of exploration, but scores are based on the performance at 60i for purposes of standardization.

The footage is then imported to a computer, bitmap frame grabs are created, and the bitmaps are run through Imatest for analysis. Imatest creates a color error map, which shows the difference between the known value for each of the 28 color patches and what the camcorder actually produced. The smaller the error, the better the performance.

The final score for this section is based on the average color error.

Noise (Low Light)
The noise tests in low light are similar to how we test bright light noise, but less extensive. We light the X-Rite ColorChecker chart at an even 60 lux. Consumer camcorders are shot in auto mode in the 60i frame rate with a manual white balance. If there are alternate frame rates or gain settings, we may experiment with them, but only the 60i is used for scoring. For professional camcorders, we shoot in 60i at gain settings of 0dB, 6dB, and 12dB. As with consumer camcorders, we like to play around with other settings and frame rates and share our findings, but only the 60i performances are factored into the scoring.

After the footage is shot, we import the clips, generate bitmap frame grabs, and run those bitmaps through Imatest for analysis. Imatest reports the noise percentages for four channel: red, green, blue, and luminance. The smaller the noise percentage, the better the performance.

The final score for this section takes the average noise for all four channels. For consumer camcorders, we use the 60i footage in auto mode. For professional camcorders, we use the 60i footage at 0dB, 6dB, and 12dB.

 Sensitivity (Low Light)
The low light sensitivity examines how much light a camcorder needs in order to produce a decent image. We measure that threshold according to the camcorder’s outputted exposure levels. Exposure can be measured in IRE levels, which ranges from 7.5 IRE at the darkest levels (NTSC standards) up to 100 IRE for the brightest exposure.

The test procedure starts with the ChromaDuMonde chart, lit around 60-80 lux. The camcorder is aimed at the chart and connected to a waveform monitor, which provides realtime data about the camcorder’s IRE levels. We then hit the record button and slowly and steadily lower the light, constantly checking the lux levels with a light meter. The light is lowered until the waveform monitor indicates that the camcorder is producing a maximum exposure of 50 IRE – right in the center of the range. At 50 IRE, the picture quality would generally be darker than ideal, but still bright enough to make out colors and fine detail.

In order to account for possible data degradation when testing live (due to poor quality connections, etc.), we import the recorded video to a computer, then run the test again on the original footage.

The final score is calculated by taking the lux level at which the camcorder can produce a maximum exposure of 50 IRE. On consumer camcorders, the score is based on the performance in auto mode at highest quality using the default frame rate setting. On professional camcorders, the score is based on the performance in auto mode and 0dB gain, both at 60i (or 60p if the camcorder is natively progressive). Other frame rates and settings may be tested and discussed, but they are not necessarily factored into scoring.

Wide Angle
This test measures how wide a shot the camcorder’s lens can capture. In order to determine the maximum wide angle, we place the camcorder under a laser level. The level is marked with 360 degree increments. We record footage of the laser going across the entire width of the camcorder’s field of vision. To account for information that may be cut off by the camcorder’s LCD or viewfinder, we import the footage and watch it on a computer monitor.

The final score for this test is simply based on the maximum wide angle, measured in degrees.

Still Color
Because nearly every camcorder has the ability to take still photos, and because manufacturers make big claims about the their quality, we test still quality with equal rigor. Much of our testing is a subset of the more expansive procedures on our sister publication, DigitalCameraInfo.com.

In order to test the accuracy of colors in still photos, we shoot an X-Rite ColorChecker chart at an even, bright light. The camcorder is aligned with the chart and a manual white balance is performed. Because camcorders typically oversaturate if the exposure is left in auto, we shoot across a series of exposures, either adjusting the exposure or aperture settings. We shoot at about seven different settings.

Once all the shots are taken, we run them through Imatest software. Imatest maps the difference between the know values of each color patch and what the camcorder actually produced. The smaller the error, the better the performance.

The final score for this section is based on the average color error, according to the best possible performance that we obtained.

Still Noise
The noise of the still performance is obtained from the same procedures as Still Color, described above. When we run the still photos through Imatest, a separate module analyzes the noise percentages across four channels – red, green, blue, and luminance.

The final score for this section is based on the average across all four channels, according to the best performance of all the shots that we taken.

Still Sharpness
The sharpness of the still photo performance is tested by shooting an ISO 12233 resolution chart, lit at an even and bright light. Like the Still Color and Still Noise tests, we shoot across a series of exposure levels. The photos are then imported into Imatest for analysis. In Imatest, we select two regions of the chart. The slanted vertical line is used to find the horizontal sharpness, and the slanted horizontal line is used for the vertical sharpness.

Imatest measures sharpness in ‘line widths per picture height.’ These numbers differ considerably from the resolution numbers that manufacturers advertise, which sometimes causes confusion among our readers. It’s because manufacturers report the ideal resolution of the sensor, or the frame size, while we’re testing the entire camera system as a whole, which factors in the lens, the focusing systems, the processing, and the JPEG compression.

The final score for this section is based on the horizontal and vertical sharpness scores. Over- or under-sharpening of more than 10% results in a score penalization.

Stabilization
To test image stabilization, we use a T-LSR-75A linear slide and a T-RS60 rotary stand, produced by the Canadian company Zaber. We mount cameras and camcorders on these stands using a standard tripod mount. These devices allow us to apply shake to cameras and camcorders in a precisely controlled way, meaning that we can mimic human hand shake without the unpredictable nature of real humans. We use a custom script to control these devices to produce the required levels of movement to accurately mimic human hand shake. We record footage using the camcorder’s maximum optical zoom at two shake speeds, high and low, and we record with the camcorder’s stabilization feature turned on and off. Next, we use the motion tracking tool on Adobe After Effects to determine how much an individual point on our test image moves during the shake footage. Our score is based on the ability of the camcorder’s stabilization feature to limit shake and motion at each speed. We give more weight to the lower speed, as it mimics the movement of average hand-held shooting.

Our image stabilization testing system (seen with an SLR camera)

Battery Life
The camcorder’s battery life is tested very simply: we hit record and wait for the battery to die. In order to standardize the test, we charge the battery to full beforehand. If the camcorder has an LCD screen, it is open and set to automatic brightness. During the test, no buttons are touched and a completely static subject is recorded.

Battery life is often diminished by the complexity of what is being recorded. (As more information is processed, the camcorder has to work harder and often uses more power.) Thus, our battery tests usually demonstrate the maximum possible life of a single battery charge. If you’re recording more complex scenes, the battery will often die more quickly.

Some camcorders have recording media that fill up before the battery dies. In these instances, we plug in the DC power and remove the battery while the media (i.e., tape, DVD, memory card, etc.) is changed or wiped. Then we remove the DC power, plug the battery back in, and continue the test.

For professional camcorders, we run the test twice. The first test is exactly as described above. The second test involves closing the LCD and using the viewfinder instead. Typically, this should extend the battery life considerably.

The final score for this section is based on the number of minutes the battery lasts while recording.