The On-Line Data Archives Program

Material Prepared for CRC Annual Report, 1998

Program Leader: Ken Hawick, (khawick@cs.adelaide.edu.au)
Business Development Officer: John O'Callaghan (john.ocallaghan@cmis.csiro.au)
Senior Research Advisor: Francis Vaughan (francis@cs.adelaide.edu.au)

Annual Report, 1998

Introduction

The On-Line Data Archives program is concerned with the delivery of decision support data products derived from active archives of large data sets. Active archives available on wide area broadband networks allow end users to browse data catalogues and schedule the creation of decision support information on-demand.

End users include those in the agribusiness, defence, emergency services and mineral exploration business sectors. The OLDA program is developing prototype demonstrations of a distributed high performance computing infrastructure - known as DISCWorld - which provides the middleware to glue together computing systems and data archives distributed around Australia.

The OLDA program is structured around core research activities and a postgraduate education program with collaborative demonstration development projects leveraging off the research. OLDA has two main development teams based in Adelaide and in Canberra.

Objectives

Development of training material to help industry use high performance and distributed computing systems.
Consulting services in high-performance computing, and use of object-oriented software environments such as Java and CORBA in developing networked applications for distributed data management and processing.
Development of software demonstrators for collaborators and clients.
Maintaining a challenging education program for postgraduate and honours students to ensure adequate training of the next generation of advanced computational systems experts.
Development and continued research into state-of-the-art metacomputing middleware issues.

Highlights

Delivery of a distributed image archive system for the management of photo- reconnaissance data to the Australian Defence Science and Technology Organisation (DSTO).
Completion of designs and prototypes for the DISCWorld core architecture for managing distributed archives scaleably.
Demonstration of a distributed data management system for agribusiness products to Agrecon Pty Ltd

Planned Outcomes

Development of a data delivery and visualisation system for the Australian Bureau of Meteorology.
Integration of a bushfire simulation system with on-line data delivery infrastructure to provide access to near real-time data sources such as weather and satellite updates, as well as user access to the decision support products generated, for emergency services operation and planning.
Development of a middleware system providing a standards-based interface to querying and processing services for distributed on-line archives of earth observation data.
Development of a pilot demonstrator using the general middleware system to create an application that will automatically identify fires, fire scars and high fire risk areas based on analysis of satellite imagery.

Projects

Distributed High Performance Computing Infrastructure (DHPC-I)

This project, led by Ken Hawick, forms the research core of the OLDA program and is developing the DISCWorld high performance computing infrastructure upon which the demonstrator projects under OLDA are based. DISCWorld addresses the problems of scheduling network and computing resources, code and data in a wide area broadband heterogeneous distributed computing environment. Other research aspects include the secure and authenticated transmission of objects; optimising data transmission efficiency through smart caching and compression algorithms; embedding parallel and advanced computing resources in a manner transparent to the user; and managing the naming and discovery of resources. The project is also developing algorithms and software to manage hierarchical storage systems such as disk arrays and tape silos, and new methods for providing user access through adaptive graphical client programs on the World Wide Web.

CROP

CROP is led by Stuart Hungerford and has developed a data processing and delivery system for agribusiness data products such as wheat crop forecasts derived from satellite imagery. The demonstrator developed is implemented using Java sockets and embedded Unix processes running data processing jobs. This demonstrator was developed primarily for Agrecon Pty Ltd, but will be enhanced to allow broader uses in the environmental and agricultural business areas.

Distributed Geographic Information Systems (DGIS)

This project is led by Paul Coddington and is developing applications demonstrators and a general service-based infrastructure for querying and high performance processing of distributed archives of satellite data and other geospatial data. The project has developed an image archive repository management system, based on a standardised interface using CORBA, for managing geospatial imagery such as photo-reconnaissance data. This work was carried out under contract for DSTO. The project is also developing demonstration systems for the ingest and processing of near real time data in support of bushfire emergency response and planning. The complementary systems developed by the CROP and DGIS projects will be integrated to develop a general system for supporting applications utilising distributed archives of geospatial and earth observation data.

Meteorological Data Dissemination and Visualisation (MDDV)

This project was recently set up in collaboration with the Australian Bureau of Meteorology (BoM) to investigate distributed computing technology for delivering data products from the BoM's headquarters in Melbourne to its regional forecast offices such as the one in Adelaide. The project is targetting delivery of satellite data as well as the simulated data fields output from the Bureau's weather forecasting models run daily on its supercomputing facilities. The project will prototype use of a broadband network for delivery of this data around Australia. Currently we have investigated the applicability of a number of 3D data visualisation programs, including recent products such as Java3D, for use with meteorological data.

Webmaster@DHPC