Processing images by digital rather than optical Fourier methods offers several advantages. The main advantages are that digital methods are quantitative and adaptable. In addition, 3D reconstruction and rotational filtering are impractical or nearly impossible to achieve with optical techniques. It is also not practical to carry out quantitative analysis and data manipulation on an optical bench. For example, image aberrations such as astigmatism and defocusing, or specimen distortions such as crystal lattice imperfections or curvature in filamentous specimens can be corrected quite easily with digital procedures (Table 1.III.B.6, Baker, 1981). Diffraction amplitudes and phases can be measured and modified, for example, to correct for contrast transfer effects (see Table 1.III.C.3.g, Baker, 1981). Another advantage of digital processing is that separate image reconstructions can be averaged together and a measure of their agreement can be quantified (Table 1.III.B.3.b,c, Baker, 1981). Digital processing offers virtually infinite flexibility in data manipulation. For example, in "pseudo-optical filtering", the digital equivalent of optical filtering, filter masks with an infinite variety and combination of hole sizes, shapes, and "transparencies" can be designed.

Computer image processing has replaced the requirement for high-quality, expensive optical systems. Nevertheless, there are certain disadvantages such as the necessity for discrete sampling of the data. This produces aliasing artifacts (transform overlap) which can be reduced, although never totally eliminated, by judicious choice of scanning conditions. DeRosier and Moore (1970) define and discuss the aliasing problem inherent to digital image processing.

The initial costs in time and money in setting up a functioning digital system can be prohibitive unless one is genuinely committed to image processing studies. It is fruitless to develop a digital system just to view specimen diffraction patterns. An optical diffractometer is both inexpensive and operates at the speed of light! In addition, the qualitative results of careful optical filtering studies are comparable to those obtained with digital methods (Aebi, et al., 1973; Misell, 1978) despite the small differences resulting from digital aliasing errors (see next section, III.D.3.a). Optical diffraction is the best way to assess the quality of images since it is fast and inexpensive compared to digital methods. Aebi, et al. (1973), Misell (1978) and the table in the next section compare the advantages and disadvantages of optical and computer Fourier processing methods. Additional applications and selected examples of digital processing are outlined in Table 1.III.B of Baker (1981).

a. Comparison of optical and computer image analysis

Despite the obvious differences, optical and digital Fourier processing of electron micrographs are similar in many ways. The advantages and disadvantages of each of these procedures is summarized in the following table:

OPTICAL	COMPUTER
Original micrograph used	Micrograph digitized and "floated"
Bench required for diffraction can be simple and inexpensive	Requires fast computer for interactive results
Formation of diffraction pattern instantaneous	Careful digitization is slow and computation of diffraction pattern may take 1 or more minutes depending on the CPU power of the computer
Filtering operations require high quality (i.e. expensive) optics	Computers keep getting more powerful and cheaper
Accurate filter masks tedious to make	Only limited by quality of software
Filtered image recorded photographically	Reconstructed images displayed and photographed using computer graphics devices
Quantitative information difficult or nearly impossible to obtain	The essence of computing IS to quantitate
Amplitudes and phases difficult to manipulate	Infinite control over amplitudes and phases
Attenuation of zero-order beam to improve contrast in filtered image (may cause frequency doubling)	Control of contrast simple and straightforward
Imposing idealized, non-translational symmetries virtually impossible	Any symmetries (even incorrect) can be easily imposed
Correction for lattice distortion virtually impossible	Lattice distortions fairly easy to correct by reinterpolating original image onto perfect lattice
Data (diffraction patterns and filtered images) are continuous (i.e. vary smoothly)	Data are discrete (pixels)
Fast for screening and selecting best images for additional analysis	Forget it (maybe in 5 or so years when CCD technology gets cheap enough).
Reconstruction of 3D structure essentially impossible	Procedures rather straightforward with "right" software
Impractical to average data from different micrographs	Easy to average data from different micrographs

b. Digital processing steps

A typical digital processing procedure includes the following steps:

- Image selection - Densitometry - Boxing and floating - Fourier transformation - Indexing 2D lattices (for objects with translational symmetry) - 2D filtering/ 3D reconstruction

After an initial screening by eye (to discard obvious bad images), several (perhaps 50 or more) micrographs are examined by optical diffractionunsuitable method for selecting images of particles with well-preserved symmetry for digital, rotational filtering. Instead, the rotational power spectrum is computed from the digitized image and analyzed (Sec. III.E.2).

Fig.III.78. Schematic view of the principal parts of the micro-densitometer. (From Arndt et al., 1969, p.386)

The micrograph is digitized on a scanning densitometer, a device which converts optical densities on the photographic emulsion to a digital image (a numerical array corresponding to the optical densities at discrete positions in the image). Several types of densitometers are available. The most precise and most expensive are the flat-bed type devices (e.g. Figs.III.78-80) which digitize the micrograph laid flat. In rotating-drum densitometers, the micrograph is fixed to a cylindrical surface (drum) and the micrograph is scanned while it is translated along and rotated about the drum axis. A demonstration of the operation of this type of microdensitometer will be given in class. The use of CCD (charge coupled device) cameras for microdensitometry is gaining popularity as they become more affordable.

Fig.III.79. Photograph of a combined optical diffractometer and microdensitometer. (From Longley, 1980, p.248)

Fig.III.80. Ray diagram: (a) the instrument as a diffractometer; (b) as a microdensitometer. (From Longley, 1980, p.249)

The density value at each point in the digitized image is represented as a pixel with an intensity ranging between 0 and 255 (an eight-bit number) or 4096 (12-bit number) or even higher in some CCD cameras. The information contained in a single 1024 by 1024 digital image (1,048,576 pixels) is quite staggering: it is roughly equivalent to slightly more than the entire contents of the lecture notes (text only) for this course. Note that, at a typical raster step size of 25 µm, the area of the micrograph digitized for a 1024 by 1024 array would be 2.56 cm² or only ~8% of the entire area of a typical 8 x 10 cm micrograph.

Images are scanned at raster settings corresponding to one-third or less of the expected pixel resolution in the image to minimize aliasing artifacts (Table 1.III.C.2.c, Baker, 1981; Fig.III.81). The equivalent step size (pixel resolution) in the biological specimen depends on the magnification of the micrograph or photograph scanned. For example, if the micrograph was recorded at a magnification of 49,000X and scanned at 25 µm intervals, then each pixel corresponds to 0.510 nm at the specimen. Thus, the maximum resolution one can expect to recover from the digitized image is about 1.53 nm (= 3 x 0.510). This assumes the specimen is preserved to at least this resolution and the electron optical conditions allow recovery of this information. The following table identifies the maximum pixel resolution (in nm) recoverable from a digitized image, for images at different magnifications and scanned at different step sizes (those available, for example, on an Optronics Photoscan P1000 rotating-drum type microdensitometer).

Fig.III.81. Effect of sampling interval on recovery of information. In this example, a sampling interval of 32 appears to be just fine enough to recover the shape of the 1D function without loss of information. At coarser sampling intervals (4-16), the subtler features in the data are lost. In practice, one aims to digitize the data at a fine enough interval to be certain that no information is lost. Thus, using the three-times pixel resolution criteria, in this example one ought to sample the data 96 (= 3 x 32) or greater to be certain to recover all the information contained in the data.

Microdensitometers are computer-controlled devices. They measure optical densities in the micrograph on a square grid (i.e. at equal-spaced intervals in x and y directions: Fig.III.82). The change in intensity of a small beam of light after it passes through the micrograph is measured using a photomultiplier which converts the analog signal (beam of light) to a digital signal (intensity value typically between 0 and 255). The digitized data can be displayed directly on a raster graphics TV screen or stored on various media (e.g. magnetic disk or magnetic tape) for subsequent manipulations. Since digitized images quite often contain significantly large amounts of data (pixels), they are normally stored on magnetic tape when not being processed or analyzed. Just a few years back, before the days of high capacity 8mm and 4mm tape storage technology, the large (2400'), 1600 bpi (bits per inch), 9-track magnetic tapes only held about 25 1024 by 1024 images (with each pixel stored as a 16 bit value). The higher density (6250 bpi) magnetic tapes could store four times as much data. For those that can still remember the standard double-sided, double-density floppy diskette commonly used in personal computers, such diskettes could only store 360 Kbytes (here one byte = 16 bits) which is barely sufficient for one 512 by 512 image. Our modern 4mm and 8mm tape backup technology with 2-5GB (GB = gigabyte) capacity, means that up to several thousand 1024² images can be stored!!! However, this seemingly wonderful capacity has a potential downside in that book-keeping of such massive amounts of data can rapidly become a nightmare.

Fig.III.81. Sampling a 2D image on a square lattice. Upper left shows the sampling if it could be performed at discrete points. Lower left shows the usual situation in which the diameter of the illuminating light beam exactly matches the sampling interval. In the upper right panel, the illuminating beam is larger than the raster, which leads to smearing of information because the intensity of the image in neighboring regions gets averaged in with the intensity of the area adjacent to each lattice point. The illuminating beam in the lower right panel is is too small, thus only a small fraction of the area defined by the sampling lattice is illuminated about each point. The ideal situation would occur if the image could be illuminated with a square beam, whose dimensions exactly matched the sampling lattice. This is not accomplished in practice.

The entire digital image or selected (boxed or windowed) areas may be used for subsequent processing steps. If only a portion of the scanned image is needed, the area of interest is boxed in a manner similar to that used to mask micrographs for optical diffraction and filtering experiments. Thus, areas outside the biological specimen (e.g. carbon film) can be selectively removed since these areas mainly contribute noise to the image. Boxing is conveniently performed directly with the digital image displayed on a TV graphics monitor.

Regions of the digital image outside the area of interest are zeroed (equivalent to masking out an area of the micrograph for optical diffraction) and the numerical image is "floated" by subtracting the average image intensity around the perimeter of the boxed area from ALL image intensities within the masked area. Floating suppresses intense diffraction spots generated by edges of the masked area. The characteristic "cross" observed in optical diffraction patterns is caused by the strong diffraction that occurs at the edges of the square or rectangular mask where the contrast is much higher compared to that within the windowed portion of the micrograph. As you encounter pictures of diffraction patterns in the literature, make note of the presence or absence of the strong diffraction peaks near the center of the pattern. The presence of a strong spike or other diffraction at the center of the transform indicates that the pattern was generated optically. Alternatively, if the pattern was generated on a computer and still shows strong diffraction spikes or other such effects, this would signify that the image was not floated properly before Fourier transformation.

The Fourier transform of the numerical array is usually computed by means of Fast-Fourier methods (Table 1.III.C.3.c, Baker, 1981). The usual procedure is to 'pad' the boxed image to produce a larger image whose dimensions are powers of two (e.g. 64², 128², 256², 64x512, etc.). Thus, if the original boxed image was a 55 by 450 pixel array, then this image would be padded AT LEAST out to a 64 by 512 array and then Fourier transformed. Padding just adds pixels with zero intensity to the columns and rows of the boxed image array to make it meet the power of two criteria. This is NOT an essential criteria but it does lead to faster computation of the Fourier transform because most fast-Fourier transform (FFT) computer algorithms are more efficient with images sized this way.

The Fourier transform of an n by m image results in an n by m complex array of numbers. Each complex number is a Structure Factor (Sec.III.C.6.f). Each Structure Factor is stored in computer memory either as a structure factor amplitude and phase or as the real (A-part) and imaginary (B-part) parts of the Structure Factor (Sec.III.C.6.f). Diffraction amplitudes and phases may be displayed in a variety of ways such as on a 2D raster graphics device or as line printer output.

As has already been emphasized, successful application of image analysis procedures largely depends on correct indexing of the diffraction pattern. For well-ordered, 2D crystalline biological specimens, the diffraction pattern consists of a series of discrete spots (Bragg reflections) that lie on a reciprocal lattice. Such patterns are usually fairly easy to index (i.e. define the reciprocal lattice parameters and assign Miller indices to each of the spots). Recall that the process of indexing is usually already performed by inspecting an optical diffraction pattern of the specimen image. The indexing of multilayered or two-sided structures (e.g. biological aggregates with helical symmetry) can be quite tricky, so care must be used before proceeding to the next step (filtration and reconstruction).

Correct indexing of a diffraction pattern is tantamount to deciding which regions of the Fourier transform are attributed to 'noise' and which regions are attributed to 'signal'. Once the decision is made as to what is signal and what is noise, the computed Fourier transform is 'masked' in a manner completely analogous to the process used to mask the diffraction pattern on an optical bench (Sec.III.D.2). Thus, the amplitudes in the computed Fourier transform are zeroed everywhere except at the reciprocal lattice points. In pseudo-optical filtering, the term "points" actually refers to finite regions ("holes" in the "filter mask") centered at the mathematical points of an ideal reciprocal lattice: these are left as is or may be multiplied by a function which weights highest those transform values lying closest to the ideal lattice. The modified ("filtered") diffraction pattern is mathematically back-transformed to reconstruct an averaged, reconstructed image. Complete Fourier averaging (all unit cells are averaged together with equal weight) is accomplished by computing a single Structure Factor amplitude and phase at each of the reciprocal lattice points and reconstructing the structure of a single unit cell by Fourier synthesis (Sec. III.C.6.c, pp.196-198).

Reconstructions may be displayed in a variety of ways. 'Old-timers' (and readers of the original image analysis literature) will recall the use of character over-printing on a line-printer, contour plotting, cathode ray density plotting, film writing, etc. (see Table 1.IV.B of Baker, 1981 for citations of examples). Modern 2D raster graphics devices provide a variety of ways to render such reconstructions in clear, interpretable form.

If the 3D structure of a particle is to be reconstructed, Structure Factor phases and amplitudes must be determined in three dimensions to fill in and generate a complete, 3D Fourier transform. This is accomplished, in the case of a 2D crystal structure, by combining Structure Factor data from several 2D diffraction patterns of independent views of the crystalline specimen. The extent of the 3D transform, and hence ultimate resolution that can be computed, depends both on the number and uniqueness of the specimen images that are included in the data set. Note that, in theory, one could add an 'infinite' number of images to achieve 'infinitely' high resolution, but, in reality, the actual resolution is limited by many other factors (e.g. radiation damage to the specimen, specimen distortions, image drift and astigmatism, defocus level, etc.).

The rationale for collecting and combining information from distinct views differs depending on the type (e.g. helical, spherical, 2D, 3D, etc.) of specimen being studied (Sec. III.E).

c. Hardware/Software

Two major disadvantages of digital processing are the expense and complexity of the required hardware (microdensitometer and computer) and software (programs for carrying out the image processing procedures). Most protein crystallography laboratories are equipped with the needed hardware, and often have programs (for example, Fast-Fourier transform, film scanning, and plotting routines) which are easily adapted for most of the basic image processing tasks. Microdensitometers can cost as much as $50,000 (rotating-drum) to over $150,000 (high precision, flat-bed). Most multi-user image processing can be performed quite adequately now on computer graphics workstations costing as little as $10,000 or less. Rotating-drum microdensitometers are less expensive and generally scan images more quickly than flatbed microdensitometers which are more precise and have greater flexibility (larger choice and range of scanning parameters). Rotating-drum densitometers are obviously unsuitable for scanning glass plates (the image may be copied onto film but this is not generally recommended owing to non-linearities in duplicating contrast levels and relative dimensions). The flatbed densitometer is useful for examining data from unstained specimens studied by low-dose imaging techniques. Here, images recorded at medium-low magnification (10,000-40,000x), containing medium-high resolution details (0.5-2.0 nm), require a small scan raster and aperture (<20 µm). Also, reflections in electron diffraction patterns (from which diffraction amplitudes are determined from specimens studied by low dose) are usually 20 µm or smaller in diameter and must be scanned with high precision on a very fine raster (<5 µm).

Many laboratories engaged in digital processing studies prefer to tailor their own computer software systems since the programs can then be designed to efficiently analyze specimens of particular interest. In this way, results become easier to understand and interpret. If a highly-specialized system is not essential, it might be advantageous to acquire a portable, established system developed by others (e.g. Table 1.III.c.3.a, Baker, 1981). This can save considerable effort (and frustration) in the development and testing of programs. The main disadvantage of "black-box" systems is the danger of incorrect implementation by untrained users.