Simple and Elegant

I was recently reminded of the power of graphical analysis to illustrate trends and represent data in a way that makes the data easy to understand and almost obvious to anyone looking at the graphic. A coworker of mine was working on a rather pesky case for a customer who had uploaded an 80MB text dump of his logs. The problem was of an unknown origin with no error messages, but happened at a known interval of time. How do you begin to analyze this data? Look for trends of course. ;)

What he did, using good ol' sed and awk, was process the log and generate trending data which he then graphed in Excel. He ended up with very simple graphs showing the statistical information he wanted.

This reminded me of the works of Edward R. Tufte. Of course, the type of information and graphics generated by my coworker weren't near those expressed by Tufte, but it reminded me how a simple and elegant expression of large quantities of data can be very powerful in our comprehension of that data.

Data representation was an issue I was particularly concerned with when I did application development and interface design. At a previous employer I was part of the ground station software team. It was our responsibility to create interfaces which would expose the myriad of telemetry data to the satellite operations team. We needed to represent that data in a clear and concise way to prevent confusion or misinterpretation on the part of the operator. If we didn't, we could increase the potential for a very expensive mistake to happen. We had a tight feedback loop with the ops team, and any changes we made to the interface had to be vetted and approved. In the end, the interface was very utilitarian, but got the job done.