The growing complexity of current parallel system makes performance analysis more and more important while simulation more and more infeasible, which leads to the solution of integrating performance monitor into system. The challenge is to design an effective and flexible performance monitor, while keep the perturbation to the system minimized.
We built a software-based performance monitor on Shrimp-Myri system. The performance monitor is integrated into Myrinet's MCP program and features global synchronized clock among all nodes, multi-dimension histogram, and histogram triggered trace. We have found that besides flexibility coming with it's software-based feature, the performance impact and perturbation is tolerable.