collectdViewer is a system for browser-based, high-frequency visualization of server performance data acquired by collectd. The system combines the use of WebSockets and the new AMQP plugin forcollectd 5.0.0 to achieve a data display update rate in the browser of two times per second. This article and the accompanying video describe the operation of the system and demonstrate how the system can be used to monitor server performance under different loading conditions at very small time scales.
Collectd is a statistics collection daemon used for server performance monitoring. Typically, this data is stored in RRD files and periodically converted by a back-end application such as drraw into image files which can then be viewed in a browser.
To allow users to visualize collectd data at high frequency and in real-time, data from collectd can now be pushed to and presented in a browser as soon as it is acquired. This approach is feasible by combining the use of
the WebSocket interface to push data continuously at high rates to a browser.
Bergmans Mechatronics LLC has developed a browser-based, high-frequency collectd data visualization system named collectdViewer which employs both WebSocket and the AMQP plugin as a demonstration of the capabilities of such a system.
With reference to the system schematic shown in Figure 1, collectdViewer operates as follows. A collectd daemon runs on a target server instance. For demonstration purposes, this is an Amazon EC2 instance, but could be any collectd-compatible platform. Three "read" plugins are currently active in this demonstration system: the cpu, interface and memory plugins. The AMQP "read/write" plugin is also active, but in this case, it is only being used as a "write" plugin.
Figure 1. collectdViewer System Schematic (Click to enlarge)
A screenshot of the browser display (Figure 2) shows the three time-history plots of data from the active collectd "read" plugins.
CPU Plot This is a plot of cpu usage in jiffies per second for each of the eight cpu plugin type instances. This is an accumulative plot meaning that the height of any data point is determined by the value of that point plus the value of the parameters plotted below it. The overall height of this plot always remains approximately constant at about 100 jiffies per second.
Memory Plot This plot presents memory usage of the EC2 instance. This plot is also an accumulative plot. The top of this plot always remains constant at 1.7 GB regardless of how the instance memory is used.
Interface Plot This final plot shows the data transferred through the eth0 plugin instance of the interface plugin. Several user-selectable views of this data are available: kilobytes per second, packets per second, and errors per second.
A pull-down menu in the lower-left corner of the browser window allows the user to adjust the time scale of the plot from 10 seconds to 5 minutes.
Figure 2. collectdViewer screenshot showing system response to several load tests. Time scale of each plot is 60 seconds. (Click to enlarge)
The plots in Figure 2 also illustrate how collectdViewer enables observation of the response of the EC2 instance to different types of simple load tests, at very small time scales. These load tests are:
A C-based CPU load program that consists of an infinite while loop. This program can be seen to consume about 40 jiffies per second of user cpu and 60 jiffies per second of steal cpu while it was operated over a period of approximately 10 seconds.
A PHP-based memory consumption program that consumes memory at a rate of about 120 MB/sec over approximately 15 seconds.
The use of the wget command to retrieve a 20 MB file from a separate server in 3.8 seconds. wget reported an average data transfer rate of 5.76 MB/sec for this operation. This value is consistent with the average of the plot of the received data rate (red "RX" line) during the wget operation.
collectdViewer combines the use of WebSockets and the new collectd AMQP plugin to enable high frequency server performance monitoring in the browser. The capability of the system to display collectd data at high update rates in the browser could make it a useful tool for server monitoring applications.
Future work with the system could include:
development of a commercial, on-line, browser-based service for monitoring real-world applications
providing support for other collectd plugins
customizations for specific needs such as
support for monitoring multiple servers
data filtering and analysis
user interface improvements, including display pause and manual scroll
Please leave any comments you might have about the system below or feel free to contact Bergmans Mechatronics if you'd like to discuss any of the above items for future work.
Bergmans Mechatronics LLC, founded by John Bergmans in 2003, performs WebSocket application research and development and training. In addition, BML develops custom data acquisition and control system software. The firm’s client base includes companies in the software, industrial, medical, and defense sectors.