The On-Line Data Archives Program

Material Prepared for CRC Annual Report, 1997

Overview

The On-Line Data Archives flagship program of the Advanced Computational Systems (ACSys) and Research Data Network (RDN) Cooperative Research Centres, and involves interactive access to large data sets via on-line processing services. The program applies expertise in multi-processor architectures for data archives, distributed computing and earth observation data processing. An early demonstration of these ideas was given at International CEOS meeting (November, Canberra) showing access to earth observation data at Asia-Pacific Earth Observation centres. Support for the program comes from HPC centres at Adelaide, ANU and CSIRO/Bureau of Meteorology and other links with COSSA and other CRCs.

Objectives

Enable on-demand access to large distributed archives of earth observation data.
Demonstrate high-performance activities to time-critical applications using environmental, meteorological and other large data-sets.

Introduction

With the restructuring of the ACSys activities, the earlier RDN/ACSys project on Distributed High-Performance Computing has been incorporated into a Flagship R&D Program targeting On-Line Data Archives (OLDA).

As a result, the activities in the RDN project have increased their focus on applications which have a time-critical requirement to the delivery of value-added data products and services. Current focus is on Earth observation applications - for example, emergency services involving environmental data and meteorological forecasts.

The Program has continued to produce demonstrable prototypes as a way of designing and evaluating distributed high-performance computing infrastructure to support these applications.

The research team at the University of Adelaide has been strengthened and now has four researchers and six postgraduate students. Although not an original participant, in this RDN project, CSIRO is now contributing to a specific project on forecasting wheat crop yields. This project also feeds user-requirements analysis into the infrastructure design work.

Achievements

Infrastructure

Considerable effort has been spent on preparing large data sets for the demonstrations. These include:

GMS5 - Japanese Geostationary Meteorological Satellite data sets (daily from June 1995).
Rainman - Meteorological Station Rainfall data sets (historical data sets from 1920s to present).
High-resolution elevation models for selected areas (Adelaide, Canberra).

These are archived on a prototype hierarchical storage environment making use of RAID and tape-silo technology at both Adelaide and Canberra.

Aquisition of new data sets in support of collaborator interests are planned for the forthcoming year. These include medical imaging datasets; meteorological forecast simulation output sets; additional polar-orbiting satellite data sets.

Research Investigations

Distributed High-Performance Computing Infrastructure

The design and development of the DHPC infrastructure has been considered during the past year as a separate project. Investigations have been conducted on a number of technologies (such as Java, RMI, CORBA and Nexus) to investigate the computing, mass storage and visualisation resources in the infrastructure.

The "high-performance" component of the infrastructure has been regarded as a "cloud" of high performance, highly-interconnected resources, providing services which can be accessed by outside clients through the Internet and the World-Wide Web.

This infrastructure has provided the framework for considering a range of applications of earth observation data: crop yield forecasting, emergency services and offshore weather. Many of the techniques for processing these data were produced by the ADVISE and ObjectMap projects in ACSys. Many have also been drawn from our international collaborator organisations such as the Edinburgh Parallel Computing Centre in Scotland and the Centre for Research on Parallel Computation at Syracuse University, New York, Caltech, University of Tennessee, Argonne National Laboratory and Rice University in the USA.

Research experiments have been conducted and prototype software components developed in the areas of computer server-server interaction over wide areas; satellite image compression; scheduling on-demand processing of imagery; high-speed network delivery for value added Earth observation products; metadata management and image catalogue browsing.

Distributed Geographic Information Systems

Several demonstrations of processing and accessing geographic data have been developed in the past year.

Landcare management with the Soil and Land Management CRC:
- prediction of rainfall (and frost) at higher resolutions using (GMS) satellite data.
- demonstratable interactive techniques to allow browsing and data selection from a large data repository.
- incorporated "on-demand" techniques for rectification and derived data computation.
- production of the ERIC satellite imagery archive browser and processing scheduler was completed in May 1997.
- A high performance data interpolation framework to merge grid data and non-gridded data is under development for the SLM CRC.
Bushfire simulation with the Geographic Information Systems Centre of Adelaide (GISCA):
- demonstratable integrated simulation system involving a firefront model using fuel load and elevation models.
- allows multiple forecast simulations to be run simultaneously to generate statistical pools of convergence.
- an enhanced computational model and near-time intraction control framework is a planned deliverable for Dec 1997.
On-line processing of airborne data with Airborne Research Australia (Flinders University):
- demonstratable on-line systems to post-process air reconnaissance data;
- allows investigation of high-precision rectified satellite (GMS and NOAA data).
On-line data storage and post-processing of model simulation data (Physics Department of Adelaide University)

Wheat Crop Yield Forecasts (CROP)

In May 1997, a specific project to provide wheat crop yields for user-selected regions of Australia was initiated with Agrecon, a small Australian company providing value-added services to the agricultural industry.

The CROP project builds on an earlier demonstration (AEON) developed by CSIRO to browse and select remote sensing data from Government agencies. The OATS satellite image catalogue browser developed by the DHPC project and demonstrated at ATG in May 97 is also input to the CROP project.

The project aims to design and demonstrate a new business model to deliver on-line remote sensing products and services to end-users using the Internet. The initial demonstration of the model is scheduled for December 1997.

Research Planned for 1997/98

Offshore weather: Bureau of Meteorology, Sydney Vislab
An applications program interface (API) is currently under development for the DHPC Infrastructure. This will provide a common target for applications development in the CROP, DGIS and Offshore Weather projects as well as for the Infrastructure development effort itself.
We plan new areas of on-line data archives are planned - including medical imaging and simulation model output analysis. We plan to develop user interfaces to navigate and schedule processing.

Program Leader: Ken Hawick, (khawick@cs.adelaide.edu.au)
Business Development Officer: John O'Callaghan (john.ocallaghan@cmis.csiro.au)
Senior Research Advisor: Francis Vaughan (francis@cs.adelaide.edu.au)

Webmaster@DHPC