Analyzing System Logs: A New View of What's Important

Sivan Sabato, Elad Yom-Tov, Aviad Tsherniak
IBM Haifa Labs, Haifa University Campus, Haifa 31905, Israel
Saharon Rosset
IBM T.J. Watson Research Center , Yorktown Heights, NY 10598

Abstract System logs, such as the Windows Event log or the Linux system log, are an important resource for computer system management. We present a method for ranking system log messages by their estimated value to users, and generating a log view that displays the most important messages. The ranking process uses a dataset of system logs from many computer systems to score messages. For better scoring, unsupervised clustering is used to identify sets of systems that behave similarly. We propose a new feature construction scheme that measures the difference in the ranking of messages by frequency, and show that it leads to better clustering results. The expected distribution of messages in a given system is estimated using the resulting clusters, and log messages are scored using this estimation. We show experimental results from tests on xSeries servers. A tool based on the described methods is being used to aid support personnel in the IBM xSeries support center.