Database Sampling to Support the Development of Data-Intensive Applications

Abstract

A prototype database is a model of a database which exhibits the desired properties, in terms of its schema and/or data values, of an operational database. Database prototyping has been proposed as a technique to support the database design process in particular, and the whole data-intensive application development process in general (e.g. requirements elicitation, software testing, experimentation with design alternatives). Existing work on this area has been widely ignored in practice mainly on the grounds that its benefits, i.e. an increase in the quality of the resulting software systems, do not justify the costs of developing and using a satisfactory prototype of the database under construction. Increasingly more software development projects consist of extending or enhancing existing systems, as opposed to developing new ones. Legacy information systems migration and WEB-enabling existing systems are examples where operational data can be expected to be available at development time. In these types of projects, using the entire operational database may not be cost-effective and a carefully selected subset may be more appropriate. The availability of operational data and a database schema can significantly reduce the effort required to build an appropriate prototype database to support the project at hand. The benefits of using such a prototype, with domain-relevant data and semantics, can also be expected to be higher than in those cases where software is developed from scratch and thus an operational database may not be available from which to build a prototype database. This thesis investigates how a prototype database can be constructed from an existing database. A prototype database which is populated with data from an operational database is referred to as a Sample Database; when it is populated using synthetic data values, it is called a Test Database. The context in which prototype databases in general, and Sample

Databases in particular, can be used is analysed. Existing database prototyping approaches are reviewed and a framework to evaluate them is developed. The thesis studies the process of extracting a sample from a database, giving special consideration to the semantic content of the resulting sample. Semantic information commonly used in practice, and how it can be included in the sampling process, is investigated. A formal framework is also developed as a more abstract study of database sampling. A prototype of a database sampling tool, CoDaST, was implemented to test the concepts developed in this thesis.