collectdViewer: High Frequency Server Monitoring in the Browser using WebSocket


John Bergmans
Bergmans Mechatronics LLC

Aug 27, 2011


Update: The collectdViewer system is now available at collectdviewer.com

Summary

collectdViewer is a system for browser-based, high-frequency visualization of server performance data acquired by collectd.  The system combines the use of WebSockets and the new AMQP plugin for collectd 5.0.0 to achieve a data display update rate in the browser of two times per second.  This article and the accompanying video describe the operation of the system and demonstrate how the system can be used to monitor server performance under different loading conditions at very small time scales.

This is a follow-up to the July 2011 article "High Frequency Server Monitoring in the Browser using WebSocket".

Bookmark and Share

Contents

Introduction

collectdViewer System Overview

High Frequency Data Display in the Browser

Conclusions and Future Work

Video

About Bergmans Mechatronics

Comments

Introduction

Collectd is a statistics collection daemon used for server performance monitoring. Typically, this data is stored in RRD files and periodically converted by a back-end application such as drraw into image files which can then be viewed in a browser.

To allow users to visualize collectd data at high frequency and in real-time, data from collectd can now be pushed to and presented in a browser as soon as it is acquired. This approach is feasible by combining the use of

  1. the Advanced Messaging Queuing Protocol (AMQP) plugin in collectd 5.0.0, which enables data acquired by collectd to be transmitted continuously at high rates to an AMQP message broker; and,
  2. the WebSocket interface to push data continuously at high rates to a browser.

Bergmans Mechatronics LLC has developed a browser-based, high-frequency collectd data visualization system named collectdViewer which employs both WebSocket and the AMQP plugin as a demonstration of the capabilities of such a system.

collectdViewer System Overview

With reference to the system schematic shown in Figure 1, collectdViewer operates as follows. A collectd daemon runs on a target server instance. For demonstration purposes, this is an Amazon EC2 instance, but could be any collectd-compatible platform. Three "read" plugins are currently active in this demonstration system: the cpu, interface and memory plugins. The AMQP "read/write" plugin is also active, but in this case, it is only being used as a "write" plugin.

Data from the cpu, interface and memory plugins is acquired two times per second. This data is transmitted by the AMQP plugin to an Apache Qpid AMQP message broker located on a separate server, named the Messaging Server. The AMQP message broker forwards the data to a Kaazing WebSocket Gateway, also located on the Messaging Server. The Gateway pushes this information, via WebSockets to JavaScript code in the browser which in turn parses the data and displays it in three graphs, each of which is in a separate canvas element. The JavaScript and HTML code in the browser are currently served by an HTTP server on BML's main test server, mechatroner.com. In the future, the system could be simplified by locating the HTTP server on the Messaging Server.


collectdViewer Block Diagram
Figure 1. collectdViewer System Schematic (Click to enlarge)

High Frequency Data Display in the Browser

A screenshot of the browser display (Figure 2) shows the three time-history plots of data from the active collectd "read" plugins.

CPU Plot This is a plot of cpu usage in jiffies per second for each of the eight cpu plugin type instances. This is an accumulative plot meaning that the height of any data point is determined by the value of that point plus the value of the parameters plotted below it. The overall height of this plot always remains approximately constant at about 100 jiffies per second.

Memory Plot This plot presents memory usage of the EC2 instance. This plot is also an accumulative plot. The top of this plot always remains constant at 1.7 GB regardless of how the instance memory is used.

Interface Plot This final plot shows the data transferred through the eth0 plugin instance of the interface plugin. Several user-selectable views of this data are available: kilobytes per second, packets per second, and errors per second.

A pull-down menu in the lower-left corner of the browser window allows the user to adjust the time scale of the plot from 10 seconds to 5 minutes.

collectdViewer Screenshot
Figure 2. collectdViewer screenshot showing system response to several load tests. Time scale of each plot is 60 seconds. (Click to enlarge)

The plots in Figure 2 also illustrate how collectdViewer enables observation of the response of the EC2 instance to different types of simple load tests, at very small time scales. These load tests are:

Conclusions and Future Work

collectdViewer combines the use of WebSockets and the new collectd AMQP plugin to enable high frequency server performance monitoring in the browser. The capability of the system to display collectd data at high update rates in the browser could make it a useful tool for server monitoring applications.

Future work with the system could include:

Please leave any comments you might have about the system below or feel free to contact Bergmans Mechatronics if you'd like to discuss any of the above items for future work.

Video

This video gives an introduction to collectdViewer and shows the system monitoring the behavior of an Amazon EC2 instance under different loading conditions. (Duration: 5 min 56 sec.)

About Bergmans Mechatronics

Bergmans Mechatronics LLC, founded by John Bergmans in 2003, performs WebSocket application research and development and training.  In addition, BML develops custom data acquisition and control system software.  The firm’s client base includes companies in the software, industrial, medical, and defense sectors.

e-mail: jbergmans@bergmans.com
phone: 714-474-8956
twitter: @jbergmans
web: www.bergmans.com

Comments

blog comments powered by Disqus

© 2004-2012 Bergmans Mechatronics LLC