64-bit Platforms

Running Omniscope on 64-bit Platforms

Beginning with Omniscope 2.4, a 64-bit installer is now provided. From the download page you can now select "Windows (64-bit)" under the section entitled "Also available for". If you choose the full installer, a 64-bit private version of Java will also be installed. Both 64 and 32-bit versions of java can run on the same machine, but both 32 and 64-bit versions of Omniscope cannot run on the same machine at present.

File size benefits - 64-bit avoids the 32-bit Windows bottleneck on memory addressing

Devices running 32-bit Windows and Java can only allocate for Omniscope use a little over 1 GB of system memory, limiting Omniscope file size to a little over a million records. Devices running 64-bit Windows and Java are limited only by the amount of RAM physically present. It is now quite easy to purchase PCs at consumer prices with say 4 GB or more of memory, although the PC must have a 64-bit Windows operating system installed to use more than about 3 GB.

Full 64-bit platform functionality limitations - some things do not yet work

If you have 64-bit Windows, and also 64-bit Java and 64-bit Omniscope, the following will be unavailable:

• The DataPlayer view - you will not be able to create and export interactive Flash DataPlayers
• The Web view
• Google Maps within the Map view

If you have 64-bit Windows and you install 32-bit Java and 32-bit Omniscope (use the offline bundled 32-bit Installer for both Java and Omniscope), full functionality will be available, but maximum file size/RAM utilisation will be the same as for 32-bit Omniscope. Note: If you have 64-bit Windows, and you install only 32-bit Java, you cannot install 64-bit Omniscope.

More detail 32 versus 64-bit 

Operating systems and programs both come in two versions; 32-bit (each instruction processed is 32 bits long) and 64-bit (each instruction processed is 64-bits long).  Most processors are 64-bit capable now, and most powerful servers have already been converted to 64-bit because of its superior speed and higher memory-addressing limits. Vista is available for the desktop in both 32-bit and 64-bit, and the trend is to convert 32-bit desktop computers to 64-bit as they are replaced over the next few years. Windows XP, 2003 Server and Vista are all available in 64-bit versions.  The 32-bit version of Omniscope (which can be installed on both 32-bit and 64-bit Windows) suffers from the Windows-imposed memory limit, effectively limiting the typical Omniscope file size to about 1.2 million records because Windows is unable to provide Omniscope with access to more than about 1 GB of installed RAM.  Running 64-bit Omniscope on 64-bit platforms removes this limit (like using longer telephone numbers permits more numbers to be issued) and allows Omniscope to scale up to multi-million record capability on machines with 4 or more GB of installed RAM.

The 64-bit version of Omniscope provides far greater capabilities in terms of data volume than the 32-bit version, but requires a 64-bit processor (almost all now are) and a 64-bit version of Windows (x64 version of XP, 2003 or Vista 64-bit) and also the 64-bit version of Java (installed along with Omniscope 64-bit full install). The 64-bit Omniscope installer is a different installer, but it installs to the same location, and there is no additional licensing implication of running on 64-bit systems. However, existing licenses should be de-activated before, and re-activated after, switching between the 32-bit and 64-bit installation. If you are installing Omniscope on a machine running a 64-bit version of Windows, for example, you will not escape the addressing memory limit in the default 32-bit Java unless you also install and run the 64-bit version of Java bundled as a PVM with 64-bit Omniscope. To switch between 32-bit and 64-bit Omniscope on a 64-bit Windows PC, you must de-activate your license (if you have one), re-install Omniscope using whichever installer you choose, then re-activate your license if your are outside the free trial period.

Customising 64-bit RAM memory allocation

By default, the 64-bit version of Omniscope limits Omniscope to 75% of physical RAM memory.  If you wish to increase or decrease this, read on.

In the Omniscope Program Folder (typically found in Windows at C:\Program Files\Visokio Omniscope ) there is a file called installconfig.properties. Open this file in Notepad (it should look similar to the text below) and look for the line starting #MAX_MEMORY_MB= (highlighted below). By default this line is commented out with a #.  Remove the # in front (and leave no spaces at the beginning of the line), and enter the value in MB (e.g. 3500 for approximately 3.5GB) at the end of the line.  Save the file with this change.  Note: if you change back to the 32-bit version, this line must be commented out again.

...

# This is an optional manually-specified max memory cap for the Java VM, an integer specifying the
# megabytes to allow the JVM. Must be at least 64.
# If unspecified, 75% of physical RAM will be used as the cap.
# MAX_MEMORY_MB=300

...

Back to Scaling Omniscope

---

Knowledge Base Top

Pivot View

Pivot View data truncation

 

The current version of the Pivot View has a memory use profile that prohibits large datasets and large cell counts. To avoid out-of-memory errors, which can be fatal, the Pivot View automatically truncates the number of rows and/or columns shown when necessary.

When this happens, Omniscope dipslays warning in red below the pivot table such as:

Warning: data has been truncated to show only the first 70 columns and 71 rows. Click for details.

The truncation of the X and/or Y values happens irrespective of sorts, so do not assume the smallest values are truncated first.

Reducing / avoiding data truncation

To avoid this behaviour:

• Change the X and/or Y fields in the View Toolbar. Choose Category fields with fewer unique values
• Reduce the amount of data (records) showing. Do this either at source or by filtering.
• Install more RAM memory in your PC.

Disabling data truncation

If you are a power user, you may wish to experiment with disabling this safety check because the imposed limit is conservative. Please save your work before proceeding.

To disable truncation for the duration of your Omniscope session, click the red warning label and choosing Show all values.

To disable truncation permanently, un-tick Tools menu > Advanced tools > Application-wide settings > Misc. > Truncate rows/columns in Pivot view to avoid out-of-memory errors.

You may need to re-open the file.

 

Back to Scaling & Performance

---

Knowledge Base Top

 

Memory diagnosis

Memory diagnosis

 

Omniscope 2.3 introduced new tools to help Visokio diagnose specific memory problems, whether leaks or out of memory errors. A memory leak is when Omniscope's memory footprint increases over time during the same session for no apparent reason, resulting in eventual slow-down (typically over a significant period of time). An 'out of memory' error occurs when an operation is attempted for which there is not enough physical memory in your PC allocated to Omniscope.

To use these tools, you will need Java 1.5.0 b7+ or Java 1.6+. Check your active version of Java in Help > About. Use Java 6 if possible, as the memory tool support is much better. You will also need Omniscope 2.3 or later.

Diagnosing memory leaks

If Omniscope appears to slow down over time, when the data and view complexity is relatively constant, particularly after a long period of use, this is likely to be a memory leak. Please see Memory leak analysis.

Diagnosing out of memory errors

There are some datasets that Omniscope can't handle if there is not enough memory available. However, you may discover a fault where Omniscope attempts to request more memory than it should. If this occurs, you can configure automatic generation of memory dump files, which you can send to Visokio for analysis.

The Memory Diagnosis tool

Open the Memory Diagnosis tool by pressing Ctrl+Shift+Alt+M or choosing Tools > Advanced tools > Diagnose memory use.

If you have Java 5 (1.5) or earlier you will see a Force out-of-memory button which is used to trigger a memory dump. Naturally occuring out-of-memory errors will also trigger this. You will need to have edited your installconfig.properties file to enable "heap dump on out of memory" prior to starting Omniscope. When you are confident a memory leak is apparent, make sure you have saved your work, and click this button. This will cause a memory dump file (*.hprof) to be created in the program folder.

If you have Java 6, you will see a tickbox Dump memory on out-of-memory. Tick this box to enable memory dumping. You will also see the button Dump memory. Use this button to create an ad-hoc memory dump file - you will be prompted to choose for a save location, then Omniscope will freeze for a few seconds as it generates the dump file.

Note that memory dumps can take up a lot of space on disk, so generate memory dumps sparingly and clean up old files.

Sending us memory dumps

Please contact us before attempting to produce or send memory dumps. If you are trying to open 1 million records on a Windows 98 PC with 64 MB of RAM, for example, this will never work. Memory dumps can be large, so please compress using ZIP or another common format. Please be sure to give us the full product version (e.g. "2.3 b231") if you send us a dump file.

 

Back to Scaling & Performance

---

Knowledge Base Top

Leak analysis

Memory leak analysis

A memory leak may be the cause if Omniscope's memory footprint increases over time for no reason. memory leaks are unlikely in production releases of Omniscope, but more likely in daily, alpha or beta releases. If you suspect a memory leak in Omniscope, please follow these steps to verify the leak.

Verifying a memory leak

The Memory Diagnosis tool  is used to observe the patterns in the Omniscope's memory use. Press Ctrl+Shift+Alt+M in Omniscope to open this tool.

Memory management algorithms in the Java VM used by Omniscope can cause apparently random fluctuations - sudden increases and drops, or small continuous changes. We will use the Extreme GC button (which takes about 8 seconds) to ask Omniscope to free up as much non-essential memory as possible, removing this "noise" from the results. We will leave the Memory Diagnosis window open throughout this analysis.

Steps

First you must identify and narrow down the task or group of tasks causing the leak.

Then do the same exact task 3 times (or more) in a row.

After both iterations 2 and 3 (and perhaps further), click the Extreme GC button and note the memory usage. Ignore small changes (1% differences), but anything larger is a potential leak.

Example results:

Not a leak:
- exit and open report page X
- exit and open report page X -> Extreme GC -> 150mb
- exit and open report page X -> Extreme GC -> 150mb +/- approx. 1%

Probably a leak:
- exit and open report page X
- exit and open report page X -> Extreme GC -> 150mb
- exit and open report page X -> Extreme GC -> 160mb

You have to do at least 3 iterations, but may choose to do more. If, for example, you do 5 iterations, and there is a steady increase across several of the iterations, this is an even more compelling proof of a leak.

Note: it is not considered a leak if a task consumes memory which isn't released immediately, but only if the exact same task done repeatedly consumes more and more memory in a memory-confined situation (simulated by Extreme GC). This is why we discount any increase in memory used for iterations 1 to 2.

I have found a memory leak - what next?

Please report from inside Omniscope using Help > Error reporting -> Report a problem.

In the report, describe the task you are repeating in precise detail. Include the results of your memory diagnosis.

You may be subsequently contacted to ask for the IOK file (in confidence), so keep a copy of the file in the correct state needed to reproduce the problem (layout, data, settings, etc.).  We may also ask for a memory dump, although this is unlikely.

 

Back to Omniscope Scaling

---

Knowledge Base Top