Sonogram
visible speech
ONLINE HELP

© by Christoph Lauer --- christoph.lauer@gmail.com

Content:

1. Preface:

Sonogram has been originally programmed at the German Research Center for Artificial Intelligence (Deutsches Forschungszentrum für Künstliche Intelligenz DFKI, www.dfki.de) and is now open sourced by the developer Christoph Lauer. Sonogram is a tool to analyze speech and sound signals with methods of:

1.) Fast Fourier Transformation (FFT)
2.) Linear Predictive Coding (LPC)
3.) Wavelet Transformation
4.) Cepstral Analysis
5.) Autokorrelation
6.) Pitch Detection

Sonogram's ultimate goal is to be a little speech recognizer that works internally with hidden makrov models (HMM). But at this time, HMM's and other speech recognition tools are not implemented, only signal analysis tools are implemented. It supports drag 'n' drop features under Windows/Macintosh OSX, and can open nearly any sound file format (also video files!). It has been tested on Windows, Linux,SUN/Solaris and Macintosh OSX. Sonogram use internal JMF (the Java Media Framework). It's no problem to install it. Java3D is not necessary to run Sonogram but without it the 3D Perspectogram feature willl not be available. If you want to play the audio files with Sonogram you must install the OS native version of the Java Media Framework (JMF). For more detailed information about digital signal analysis, download the "Digital Signal Analysis with Sonogram" paper from my homepage. Note: Sonogram is only free for non comercial and research purposes, commercial users should contact me for further informations.

2. Opening Media Files & Supported File Types:

To open a file, simply drop it over the main window, or select it with the File Chooser dialog on the toolbar or menu bar. Sonogram can open both audio and video files. Alternative media files can be opened via Internet connection and known URLs, or with a hotlist entry which can store all successfully opened Files. To open files via URL, select menu item and type the full URL into the dialog.(e.g.. http://www.dfki.de/~clauer/sonogram/demoC.wav). While opening the file, Sonogram checks the sample rate, bitrate, duration... and will convert the signal to 8-bit amplitude resolution and give a message if it can't convert the file, or if the selected file is not a known media file type. Internally, Sonogram uses the JMF to open and play these soundfiles. Sonogram includes some JMF routines to get access to the samples in the file, but if you want to play audio files with Sonogram you need to install JMF. At this time (10/2006), the internal JMF can support the following filetypes:

For more details on supported file types, see Sun's jmf supported media formats page.

3. The Sonogram View & Playing Sounds:

If a file is dropped (this feature works only in Windows and Macintosh OSX), the program begins to read out the file, transform it and generate a sonogram image. Sonogram's software architecture uses double-buffering technology to store the image in its memory; that means pictures are only generated new for one time. Should a new file be opened, or program parameters changed, or window be resized on the screen, the image will refresh itself very quickly. Between the opening and viewing of a file, a progress dialog is displayed onscreen.  On the bottom edge of the main view, a time window with a timeline is shown where you can slide a slider; this slider represents the start position for playing the sound. At the lower left side, two numbers are shown: WN for window number, and WL for window length. Note: Maximum frequency is half your window length, so frequency resolution is sample rate/WL/2. Over both of these selected values, the transformation type is shown (either FFT or LPC). If you move the mouse over the Sonogram view on the lower right side, four values will appear: P: the peak of the spectrum under the cursor, F: the actual frequency located at the cursor, T: the actual time at the cursor location, and A: relative amplitude, in percentage, at the selected point of the cursor.  On the left side. the spectrum with a smoothed curve is displayed.

4. Introduction to Windowing,  Sonogram Generation and Signal Analysis:

"Windowing" in signal analysis means that the signal is split into pieces, which are then singly transformed into the frequency domain. In this way, resolution over time is possible. When a signal with 8000 samples and duration from one second is transformed without windowing,  each frequency in the frequency domain of this signal contains information about complete time from one second. If it is split into 100 pieces, an 80-sample time resolution would be 10ms for each frequency. But, as time resolution is goes up, frequency resolution goes down, because fewer samples are being considered for transformation. "Overlapping" means that transformation jumps not in steps from the length but in steps they are lesser than the length. The Overlap value can be changed in the General Adjustment Dialog as divide time window length, also with one increment length is time window length, with two half timew..... Windowing means also to place functions over this "piece" of signal. So jagged edges are straightened out (over the time domain) and aliasing effects are prevented. The standard transformation for calculating a sonogram is Fast-Fourier-Transformation, but in Sonogram, Linear Predictive Coding method can also be used to transform. Linear Predictive Coding (LPC) has been used in speech recognition for a long time, and calculates an LPC-spectrum in three steps: 1. It apriximates prediction coefficients (Number of Coefficients can be chosen in the General Adjustment Dialog -AD-) for a given amplitude vector (length of ampl.vec. can selected in GAD, and is named previous samples); 2. With these coefficients, LPC will calculate the future-amplitude-time-array; 3. Fourier-transform with this timeline vector gives the LPC-spectrum. For more detailed information about digital signal analysis, download the "Digital Signal Analyse with Sonogram" paper from my homepage.

5. General Adjustment Dialog (GAD):

All Options may be changed in the General Adjustment Dialog. It is shown in various register tabs; the function of each of which is described below:

• General Options: Blue checkboxes change the options for the single curve on the right side. Monochrome single curve colors this curve with no amplitude-dependent colors. Always Normalize your multiplicity curve to its maximum amplitude, so that time values with little amplitude might be shown in better contrast. The Spectrum, while playing the sound, animates the single curve in a nice way. "Smooth out" draws a red line, which will show as a smoothed curve. Orange checkboxes control smooth functions for the time and frequency dimensions. Violet checkboxes are relevant for generic program settings. With Save Hotlist, screen position and save configuration --all options for SonogramConfig.xml-- can be set in order to manage all these options. Open Last File, opens, as its name suggests, the last file at Start. With "save zoom history list", the last file will be opened with its zoom settings. Black boxes are a grid flag (the same as in logarithm frequency) and show intensity, which converts the amplitude time line to an energy timeline. "Search for local peak" searches for the local peak in the current selection of spectrum and highlights it with a red circle. The Green checkbox allows the user to open all files at sample rate of 8kHz, so the maximum frequency in the frequency axis is 4kHz.
• Overlapping: As described in Chapter 4, overlapping is controlled with this slider. The number refers exactly to the number of transformations per window.
• Colors: Five color schemes are selectable and invertible.
• Window Functions: Functions exponents with time windows. Best signal-to-noise is obtained with the Hamming and Gauss functions.
• Window Length: The automatic search for window length can be disabled, and the number of transformation points used for FFT can be user-selected. With window length, time for each window and amount of windows are changed.
• Pitch: Pitch detection simply highlights the peaks for each time point, and so on for each transformed time window. Various settings are available. (E.g. Fog over sonogram picture...)
• Transformation: Either Fast Fourier Transformation or Linear Predictive Coding Transformation can be chosen for calculating a Sonogram. If LPC is chosen, the number of LPC coefficients and prediction buffer sizes must be regulated on the LPC tab. (See chapter 4)
• Linear Predictive Coding: Settings here are for the LPC Dialog and LPC Transformation to generate a Sonogram. As described in Chapter 4, all parameters for LPC are modified here: The number of previous samples to generate coefficients, the number of coefficients to generate, the length of future vectors to transform with FFT and the frequency ratio for the outputted frequency vector. Additionally, amplitude can be shown logarithmically in the LPC-dialog, and the format finder to locate local maximums can be enabled or disabled.
• FFT: For the Fourier dialog, frequency ratio and FFT length are configured here.
• Perspectogram: Rendering style can be switched from a Point Cloud to a Wire Grid to a Filled-in Surface. The Color button changes the background color for the perspectogram, and the controls below it adjust plot dimensions in the x and y planes.
• Logarithm Amplitude: Simply enable or disable this feature and control its intensity with the slider below.
• Logarithm Frequency: For better legibility in the lower values of frequency, change the slider below to stretch the logarithm base. A checkbox to enable or disable the frequency grid is also available in general settings.

6. Zooming:

Zooming in Sonogram is very simple. You simply select an area with the left mouse button and press the Zoom-in button; a new view is then generated for the selected zone. The Zoom-back button zooms back into the complete span, and the Zoom-previous button zooms back to the previous zoom view. If Save Zoom History is selected in the General Adjustment Dialog, the last zoom will be displayed on Startup, and the zoom-previous history will be restored.

7. The 3D Perspectogram:

With the 3D Button, every generated sonogram picture can be transformed into a three-dimensional "mountain-plot". Default, this option generates a grid-surface with the colors selected in the adjoining Dialog. In this Dialog you can select the Render Style for the 3D-plot. "Point" makes a point-cloud, "Lines" is the default grid-plot, and "Fill" draws a sleek exterior. The Axes are colored differently: Green is the frequency-axis, Blue is the timeline-axis, and the amplitude-axis goes up. To use this feature, Java3d must be installed on your machine. (See www.java.sun.com for Win,UNIX,Mac and www.blackdown.org for Linux). If Java3d is not installed on your local or network computer, Sonogram will start up normally, but the 3D feature will not be available. With the slider you can select the point-reduction factor for plotting. (This function is still under construction at the moment so it will not work correctly). With mouse surface position, the zoom and rotation is user-selectable. Logarithm frequency representation is not yet available. One very cool bonus feature is the Universe switch on the perspectogram tab in the adjustment dialog... When this option is selected, a background of 2D-plot is rendered with a planetary nebular picture! The Universe is completely rotatible, movable and zoomable with rendered surface too. You can also use the keyboard (arrow keys and page up/down) to navigate thought the Perspectogram.

8. Cepstrum and Fourier Dialogs:

These features only have a small requirement to run: they run only if the signal is long enough (say, longer than ~0.2 sec.). Please don't forget: to generate these views the program requires a time area, so the mouse point is the starting point for calculating, and plots are built from this starting point on, with the span of window length indicated in the information dialog. You can see that if you go to the end of the Signal. In Fourier View, Sonogram tries to calculate the ten local maximums of the spectrum and shows their frequencies in amplitude order. In the General Adjustment Dialog are two sliders for changing the options for this calculation dialog: 1. Frequency ratio for better viewing under the frequency scale, and 2. Transformation-Window-Buffer-Size which changes the number of FFT points and reduces CPU load. NOTE: For windowed transformation, Heisenbergs's concept of "Unschärferelation" from the quantum mechanics is at work, so as your buffer length increases, the frequency resolution goes up--  and the time spent over that transformation goes up!!! That means that with a greater time buffer length, your time resolution goes down!!! Time Buffer Length is shown in the information dialog. Cepstrum View is calculated with the standard formula for Cepstrum =(InvFFT(log(FFT(timeline array)))) and shows the wavelength for outstanding frequency (Quefrenz). It is useful in speech recognition.

9. Linear Predictive Coding Dialog:

Here, as before, an area of samples is used to generate the LPC-View. Time Window Length refers to the previous samples. Formant Finder searches for the relative maximums in the curve and displays them. In GAD, the frequency ratio and transformation points can be selected.

10. Pitchfinder:

This nifty feature will easily search within each time-stamp for its peak point and highlight them. Custom colors for these points and maximum search frequencies can be selected, and a tracing of these points with a smoothed frequency vector is available.

11. Logarithm Amplitude and Frequency:

Logarithm amplitude is set to "enabled" as default. Amplitude of frequency range and logarithm is controlled with the slider position. You'll want to experiment with each feature here to find out how it works. With a logarithm frequency, the frequency axis can be stretched within the scale. The base for your logarithm is selectable with a slider in GAD. Enabling the Grid view is recommended in order to better discern the separations of your frequencies.

12 . Import and Export to Scalable Vector Graphics:

SVG (http://www.w3.org/TR/SVG/) is currently a W3C proposed recommendation; SVG is an XML language that can be used to specify drawings, including the attachment of animation effects and event-handlers on objects within the display. SVG graphical objects include text areas and standard drawing objects, such as lines and polygons. The most important factor in the development of SVG is to allow faster web downloads for graphics: SVG files are typically much smaller than the bitmaps they replace. Sonogram's SVG export feature uses only simple SVG features, such as drawing rectangles for each spectrum point, and lines for the grid.   Zoom and colors are adjustable in the export dialog. SVG stored spectrums can be re-imported into Sonogram to display them. But without re-opening the original file, not all functions of the program will be available. During re-import, the program will ask you what to do, and if the original file is available. Also export to raw SVG is possible, so Sonogram-specific tags will not be included in the SVG file. With no Sonogram specific tags, re-import into Sonogram is not available; the program will check this and warn you if you try it. With the Graphic Size slider, the multiplication factor for graphic size can be regulated.

13. Exporting to Bitmap:

Export To a bitmap file is achieved by looking under File Menu---> Save Sonogram Picture to BMP File. The file format is the Windows 3.1 bitmap, which not all common graphic programs can open. (Programs such as Gimp, Electric Eye, and MS Paint can open this format, though).

14. Printing Sonograms:

Depending on your operating system, a printing dialog is shown, from which you can configure various parameters.

15. Information Dialog:

This dialog shows all internal parameters of a sonogram. Especially useful for ascertaining values for time window lengths and resolutions.

16. Other Comments:

• In System Information (under the Help menu), the internal parameters of Java, the operating system and Sonogram may be viewed.
• Sonogram typically handles files with a maximum length of five minutes. When files longer than this are opened, Sonogram will analyze them until its memory is exhausted.
• Different operating systems render subordinate program elements in different ways, so the size for your GAD will appear differently on different Operating Systems. After the first start,  you should place your windows to their right places and sizes, so the program will save this data in its configuration file.
• Equal Tempered Scale: An octave is divided into twelve intervals from the chromatic scale. Each of these intervals is further divided into 100 cents. Cents are used to describe small differences in pitch in therms of percentages of a semi tone. The fomula f(N)=Fa*2^(N/12) can be used to dermine the frequency for corresponding to a given groundfrquency Fa. Once N is figured, the integer value closest to it can be looked up in the following table.

17. Known Bugs:

• If a media file's duration is longer than 4-5 minutes, Sonogram's internal memory-structure has problems with constructing very long vectors. To give JavaVirtual Machine more memory, use the -Xmx parameter as parameter. For exapmle call:java -Xmx256mb -jar Sonogram.jar to tell the JavaVM that they can use 256 MB for dynamical allocated arrays and objects. You can increase this value anymore to 512 or 1024 MB.
• If a very large amount of points are created in a 3D Perspectogram, Sonogram may cause a Memory overflow. (see above).
• A further error under LINUX and KDE2 or fvwm2 is that the Dialog Windows may not be not shown in the right size and position. This known Bug is in the Window Manager Code or in the underlaying lesstif code !

October 2006 by Christoph Lauer , and very thanks to David Lincoln Brooks

christoph-lauer@gmail.com - www.christoph-lauer.de