|
First Workshop on Real, Large Distributed Systems — Abstract
DISC: A System for Distributed Data Intensive Scientific Computing
George Kola, Tevfik Kosar, Jaime Frey, and Miron Livny, Computer Sciences Department, University of Wisconsin-Madison;
Robert Brunner, Department of Astronomy and NCSA, University of Illinois at Urbana-Champaign;
Michael Remijan, NCSA, University of Illinois at Urbana-Champaign
Abstract
The increasing computation and data requirements of scientific applications have necessitated the use of distributed resources owned by collaborating parties. While existing distributed systems work well for computation that requires limited data movement, they fail in unexpected ways when the computation accesses, creates, and moves large amounts of data over wide-area networks. In this work, we analyzed the problems with existing systems and used the result of this analysis to design our own system. Realizing that it takes a long while for a new system to stabilize, we tried our best to reuse existing components. We added new components only when we could not get by with adding features to existing ones. We used our system to successfully process three terabytes of DPOSS image data in under a week by using idle CPUs in desktops and commodity clusters in the UW-Madison Computer Science Department and Starlight.
- View the full text of this paper in HTML and PDF.
- If you need the latest Adobe Acrobat Reader, you can download it from Adobe's site.
|
Need help? Use our Contacts page.
