The Distributed and High-Performance Computing Group

Material Prepared for the University of Adelaide Faculty of Engineering Report, 1997

The Distributed and High Performance Computing Group, the Advanced Computational Systems CRC and Research Data Networks CRC

The Distributed and High-Performance Computing Group in the Department of Computer Science is led by Dr Ken Hawick and carries out research and development work in high performance middleware. The group is funded under the Research Data Networks CRC but managed as part of the Sdvanced Computational Systems CRC. The Group at Adelaide operate and manage the On Line Data Archives (OLDA) program within ACSys. The OLDA program is concerned with the delivery of decision support data products derived from active archives of large data sets. Active archives available on wide area broadband networks allow end users to browse data catalogues and schedule the creation of decision support information on-demand.

End users include those in the agribusiness, defence, emergency services and mineral exploration business sectors. The OLDA program is developing prototype demonstrations of a distributed high performance computing infrastructure - known as DISCWorld - which provides the middleware to glue together computing systems and data archives distributed around Australia.

The OLDA program is structured around core research activities and a postgraduate education program with collaborative demonstration development projects leveraging off the research. OLDA has two main development teams based in Adelaide and in Canberra.

In addition to carrying out collaborative projects with CRC partners, the group at Adelaide also work with local computational scientsists in departments around the University. The group worked with Prof A.Thomas and Dr A.Williams in the Centre for Subatomic Structure of Matter (CSSM) group in applying high performance clusters of workstations for their computational physics calculations. A joint workshop was held between CSSM and the DHPC group in October 1998.

The group are also working with Dr M.Buntine in the chemistry department, and are investigating how a Beowulf cluster of personal computers could be used to aid Dr Buntine's group's computational chemistry requirements. The DHPC group built a small PC cluster to trial some ideas needed in making a grant application for funds to build a production system.

The DHPC group works very closely with the South Australian Centre for Parallel Compiuting (SACPC) and has help SACPC users to program the Connection machine and Power Challenge Supercomputers.

Drs Hawick and Coddington ran a couse in distributed and high-performance computing for honours level students last year. This was well attended by students at Adelaide as well as from Flinders University and the University of South Australia.

Some Recent Resaerch Highlights include:

Completion of the Experimental Broadband network ATM infrastructure between Adelaide and Canberra. The DHPC group successfully ran several distributed computing applications over this network and used it on a day to day basis for video conference collaboration between the two teams at Adelaide and Canberra. This network will be replaced in 1998 by a higher bandwidth link running at 155Mbps, funded by the Research Data Networks CRC.
Presentation of research papers at the International Supercomputing Conference SC97 in San Jose and High Perfornace Computing and Network HPCN97 in Vienna.
Invitation to contribute an article on the DHPC Group's DISCWorld metacomputing software in a special issue of the International Journal of Future Genteration Computer Systems.
Invited visits by Dr Ken Hawick and graduate students Craig Patten and Heath James to Digital Research laboratories and StorageTek's research plant in the USA and the Edinburgh Parallel Computing Centre in Scotland.

OLDA Program Objectives

Development of training material to help industry use high performance and distributed computing systems.
Consulting services in high-performance computing, and use of object-oriented software environments such as Java and CORBA in developing networked applications for distributed data management and processing.
Development of software demonstrators for collaborators and clients.
Maintaining a challenging education program for postgraduate and honours students to ensure adequate training of the next generation of advanced computational systems experts.
Development and continued research into state-of-the-art metacomputing middleware issues.

Program Highlights

Delivery of a distributed image archive system for the management of photo- reconnaissance data to the Australian Defence Science and Technology Organisation (DSTO).
Completion of designs and prototypes for the DISCWorld core architecture for managing distributed archives scaleably.
Demonstration of a distributed data management system for agribusiness products to Agrecon Pty Ltd

Planned Outcomes

Development of a data delivery and visualisation system for the Australian Bureau of Meteorology.
Integration of a bushfire simulation system with on-line data delivery infrastructure to provide access to near real-time data sources such as weather and satellite updates, as well as user access to the decision support products generated, for emergency services operation and planning.
Development of a middleware system providing a standards-based interface to querying and processing services for distributed on-line archives of earth observation data.
Development of a pilot demonstrator using the general middleware system to create an application that will automatically identify fires, fire scars and high fire risk areas based on analysis of satellite imagery.

Projects

Distributed High Performance Computing Infrastructure (DHPC-I)

This project, led by Ken Hawick, forms the research core of the OLDA program and is developing the DISCWorld high performance computing infrastructure upon which the demonstrator projects under OLDA are based. DISCWorld addresses the problems of scheduling network and computing resources, code and data in a wide area broadband heterogeneous distributed computing environment. Other research aspects include the secure and authenticated transmission of objects; optimising data transmission efficiency through smart caching and compression algorithms; embedding parallel and advanced computing resources in a manner transparent to the user; and managing the naming and discovery of resources. The project is also developing algorithms and software to manage hierarchical storage systems such as disk arrays and tape silos, and new methods for providing user access through adaptive graphical client programs on the World Wide Web.

CROP

CROP is led by Stuart Hungerford and has developed a data processing and delivery system for agribusiness data products such as wheat crop forecasts derived from satellite imagery. The demonstrator developed is implemented using Java sockets and embedded Unix processes running data processing jobs. This demonstrator was developed primarily for Agrecon Pty Ltd, but will be enhanced to allow broader uses in the environmental and agricultural business areas. The CROP project completed in August 1998 and has fed valuable input to the DGIS Project.

Distributed Geographic Information Systems (DGIS)

This project is led by Paul Coddington and is developing applications demonstrators and a general service-based infrastructure for querying and high performance processing of distributed archives of satellite data and other geospatial data. The project has developed an image archive repository management system, based on a standardised interface using CORBA, for managing geospatial imagery such as photo-reconnaissance data. This work was carried out under contract for DSTO. The project is also developing demonstration systems for the ingest and processing of near real time data in support of bushfire emergency response and planning. The complementary systems developed by the CROP and DGIS projects will be integrated to develop a general system for supporting applications utilising distributed archives of geospatial and earth observation data.

Meteorological Data Dissemination and Visualisation (MDDV)

This project was recently set up in collaboration with the Australian Bureau of Meteorology (BoM) to investigate distributed computing technology for delivering data products from the BoM's headquarters in Melbourne to its regional forecast offices such as the one in Adelaide. The project is targetting delivery of satellite data as well as the simulated data fields output from the Bureau's weather forecasting models run daily on its supercomputing facilities. The project will prototype use of a broadband network for delivery of this data around Australia. Currently we have investigated the applicability of a number of 3D data visualisation programs, including recent products such as Java3D, for use with meteorological data.

Personnel

The DHPC Group presently includes: Ken Hawick (Project leader), Paul Coddington, Kevin Maciunas, Francis Vaughan, Andrew Wendelborn and seven postgraduate students. Dr Fred Brown recently joined the group.

For more information, contact Ken Hawick.

Webmaster@DHPC