Data Handling

The problem with data

Laurens van der Wee


Introduction

Jacob Hart and I have been working on visualisation and interaction for computational musicology. This happens at the cross-section of the IRiMaS and FluCoMa projects, incorporating ideas, code and strategies from both. In the most recent HackSpace session, we spoke mostly about the problem of implementing or choosing a database technology. For context: we work from within Max 8, but may want to open our tools to other languages and software applications, so generic formats and exporting functions are crucial to these efforts.

Initial Experiments


Before looking to build our own technology to solve the problems of data storage and retrieval, we investigated existing solutions.

entrymatcher by Alex Harker:

Although entrymatcher ships with extremely rapid queries (especailly on the bleeding edge compiles), it’s querying language is not suited to our research aims and preferences compositional aims. Also, despite the existence of neat and concise C++ code to build into a Max object it would not be frictionless to port this particular object to other environments.

Max native dict object

Max’s native dict object can restore and store to a .json file which makes this a solid choice for interoperability. However, the interface for querying dict is cumbersome and makes it hard to develop your own interfaces that might support a more bespoke syntax for access. Dictionaries become much more usuable from within javascript in Max, but this adds another layer of abstraction for accessing the data.

Data


What we’re doing: generating data in a multitude of ways and then using this data for a variety of visualisations. Since these are not aligned or we don’t know beforehand what we may need later on, we want a flexible underlying system to deal with data and potential metadata. A database seems the right option. I’ve presented and am still working on a Javascript file that can be required by other javascripts to populate databases, change data in them or read from them. All based on this tutorial that lays down the basics for SQLite in Max:

By interfacing with SQLite through javascript we achieve:

  • simple access methods with a high ceiling for complexity
  • portability: the SQLite database is a file that can be parsed in other languages and environments
  • simple read mechanisms, just point toward the entire database contained in a file

Future


Once thoroughly tested our tools for computational musicology will be released into the public domain, most likely through github. As we are currently in the early development stages we are also interested in fielding any comments or insight into this area of research.