Tokenized Data

Sometimes, more than one value is associated with certain attributes of a record.  For example, shoes can be available in more than one size, or you may have more than one image associated with with a house (or whatever). Shoe sizes and multiple images of a house are examples of many-to-one relationships. Omniscope handles these kinds of relationships using a data subtype for Text and Category fields (columns) called 'Tokenized'.

When you declare a Text or Category data type to be 'Tokenized,' it means each cell in a field can contain more than one value, i.e. multiple "tokens" separated by a character you decide. The default separator is a comma, but you can choose to use any thing: a space to separate words, or an unusual character such as a  \ backslash if you need to use commas within your values.

The example below (you can download a copy of this example file here) is a shoe store data set.  Each record (row) corresponds to a pair of shoes, and a 'Size' field containing size numbers in stock (38, 39, 40, 41 etc.). The 'Size' field has been declared a Category field since it contains only a few values, and since the size numbers awill be treated as text, not numbers used for calculations. 

Using Edit > Manage fields > Configure > Field options, we have set the data sub-type of the 'Size' Category to be 'Tokenized Category' with the default comma separator  dividing the values within the cells. We then check the data to ensure that each available shoe size is entered in the 'Size' column with a comma separator as shown in the Table View above.

We can quickly check that all values have been entered correctly in the Side Bar query device for 'Size', each size should appear separately. In the example below, the user has set filters to display only shoes available in sizes 39 and 40.

Notice that in the Chart View, the highlighting for a specific pair of shoes available in 39 or 40 (or both) still shows the full range of sizes available. This also affects other views that display breakdowns of Tokenized Categories, such as the Pie and Tile Views.

If there were multiple images for each pair of shoes, we would have done the same thing in the 'Photos'  image references field (column), so that we could relate more than one image to each pair of shoes, and display the multiple images in the Slideshow window.

Back to Configure your Data