Data Refresh Options

 

Visokio applications are designed to be part of data integration, management, reporting, presentation and publishing solutions. Activated editions of Omniscope can connect to and refresh from data files and repositories such as transactional database tables, data warehouses, analytic data structures, etc. via standard protocols such as ODBC and JDBC. Omniscope's direct database connection can import the results of any SQL statement that returns a table. Omniscope also contains a non-SQL wizard that enables anyone to perform merges and joins to assemble and import a flat tabular structure, and these operations are remembered and re-executed on file open or refresh.

Omniscope .IOK/.IOM files can have other larger .IOK/.IOM files as their linked sources and can refresh on opening, as well as auto-refresh from these linked source files.  This means that a set of large, entity-level (e.g. people and all things joined to people, places and all things joined to places, things and all things joined to things) Omniscope files automatically refreshing from the data warehouse can act as 'data marts' with many 'child' .IOK/.IOM files each having the central .IOK/IOM data mart file(s) as their linked source. This architecture maintains a single data source in the database, yet eliminates the 'SQL bottleneck' that arises from the need to write and maintain a proliferation of SQL queries tied to ever-changing reports. Using Omniscope, end users themselves can open/assemble the source files they need for a given report or analysis. Once their file is configured, it will auto-refresh not directly from the database but rather via the automatically-refreshed .IOK/.IOM data mart files on the server. No direct database access or 'retail' query loads, and no SQL, or other types of commands required of the user.

Below we discuss data import and refresh options including importing/refreshing data from delimited or XML-tagged data files or direct connections to relational transactional and analytical databases. Web integration options are discussed here. Omniscope also supports an outside browser option that allows menus of multiple files to be browsed and selected from within Omniscope before any .IOK file is opened.

Data Import Options

Omniscope can import data from files in supported file formats, or via direct connections to reporting view(s)/table(s) in relational databases using standard protocols such as ODBC and JDBC. Connectors can be developed for specific applications when performance or other special requirements do not permit standard protocols to be used.

Importing from Data Files

Omniscope can import data exported from relational database tables and other sources if they have been formatted in a supported file format, such as Comma Separated Value (.CSV) flat files, in the flavour exported by Excel (without 65,000 record limit).

Omniscope can also import XML data files, in the Visokio XML data schema intended for flat file data transmission in XML. If you already have a different XML output schema implemented, it is easy to apply XSL transforms on import.

Refreshed source data files can be provided either as files on a local disk or network share, or as HTTP or FTP URLs to intra-/extranet servers or public websites (the latter requiring any necessary proxy servers to be configured in Omniscope). For more information see Proxy Settings.

Direct Database Connections using ODBC/JDBC

Omniscope can import from any ODBC or JDBC data source. This typically means a database, although it is possible to implement an ODBC or JDBC interface to a spreadsheet, or a proprietary system. It is also possible to develop bespoke Connectors to optimise the update/refresh cycle and deal with other limitations of ODBC and JDBC-based connections. A SQL statement returning a table can be embedded in the Omniscope database connection wizard and will be re-evaluated on opening/refresh.

Data Refresh Options

Omniscope files running in activated editions remember their linked sources, and refresh their data from the linked source in various configurable ways:

Note: free Viewers are not activated and will only open existing Omniscope .IOK files, and will not check linked sources for updates/refresh. Only activated Omniscopes will check linked sources for updates/refresh, either on opening, or via real-time auto-refresh.

1. Frequent data file dump and refresh - activated Omniscopes: Your proprietary application/database produces a reporting data 'dump' file is .CSV (or other supported delimited format), or in .XML, as frequently as necessary. Omniscope files having the 'dumped' data file as a linked source are configured to refresh data from this linked data source file on opening and/ via auto-refresh if necessary.  Note: only saved changes to the source data file will be detected and imported
2. Refresh on demand via the we - activated Omniscopes: You provide a full data dump on demand at a URL via HTTP (for example, by using a Java servlet in a web server dumping .CSV or .XML). Omniscope file is configured to auto-refresh data from the URL according to a frequency.
3. Automation of .IOK 'data mart' file(s) refresh using Enterprise XML Actions: You provide an XML "script" of actions, executed by Omniscope Enterprise, which update central .IOK 'data mart' files from one or more flat files or databases. You either trigger execution of this script from within your proprietary system as and when the data changes, using the Scheduler's  'watch folder' (for immediate execution) or by setting the Scheduler to refresh the 'data mart' .IOK files on the server at short intervals.
4. Client-side Automatic Refresh - Using one of the above options to populate and periodically refresh an .IOK/.IOM source 'data mart' source on a server, the client-side 'child' version of the file can be configured to auto-refresh the data from is parent over a network or the open web according to a client refresh frequency you specify. Omniscope installs with an example of a file that auto-refreshes over the open web from our server to your desktop. Try it from File > Open demo > Live data refresh demo. More detail on Using Auto-Refresh.
5. Proprietary Connector- If necessary, we can work with you to define a Connector API for retrieving just the updated data from your proprietary system at appropriate times. This will be highly specific to the nature of your own systems, the installation environment, the data structure and degree of data "liveness". Once your users' needs are defined, we will develop a Connector specifically for your API that efficiently retrieves just the updated data, only when necessary, from your system. Using custom Connector, rather than standard protocols like ODBC/JDBC, the refresh can get very very close to real time*. This option involves chargeable consultancy and development work.

A given solution architecture can employ multiple options for refreshing Omniscope files on both always-on servers and local or remote client installations. For example, a leading asset manager uses the Enterprise Edition Scheduler to refresh a server-based Omniscope Source file every 30 seconds. About 15 Omniscope Professionals use the Automatic Refresh option to keep an open connection to the Source file, and refresh the data continuously...effectively a real-time update solution. In addition, using the Outside Browser option, users can change from one Omniscope Source file to another (and other web-based applications) by selecting from a menu of files displayed on an internal web page (see below).

Using .IOK files as linked data sources-

Q: How can I refresh .IOK files that users have configured in ways different to the 'data mart' source .IOK file, and personalised in ways unknown to me?

A: If your 'data mart' source .IOK/.IOM files are the linked data source for users' child .IOK/.IOM files, users can customise and re-configure their Omniscope working/reporting files however they wish. Whenever users open their file, only the data (not the tab configurations or query/filter settings will be refreshed, leaving their individually-personalised view configurations, filter settings and tabs settings unchanged.

Presenting File Menus

In addition to opening files accessible to the local machine via the File > Open File dialog, Omniscope can also open displaying a menu of available source/reporting files displayed in the external Outside Browser view. This view is available with no specific .IOK/.IOM file selected, and is intended to provide a web-page based menu of files for users to select from while in Omniscope. This option is not available from the free Viewer. More detail on using the Outside Browser.

Integration with Proprietary Systems

Visokio applications are standalone installed applications. Visokio does not currently provide API type integration options for embedding Omniscope as a visual module within a larger application. To integrate with a proprietary data system, you must consider Omniscope to be standalone clients with various options for refreshing the data in batches (even at 1 second intervals) as discussed above.

---

*Genuinely real-time data, updating continuously, is not directly possible in Omniscope, for usability and architectural reasons. Instead, data updates are received and displayed in batches when the application is idle or the user explicitly requests. This can be done as often as every second, depending on bandwidth, dataset size & update efficiency. For all but the very strictest real-time requirements, this near-real time live approximation is more than 'live' enough.