Visualising a real-time DataSift feed with Node and D3.js

March 29, 2012

Recently I have been playing with Mike Bostock’s D3.js and continue to be impressed with different examples of how people are visualising data. D3 provides an incredibly rich library, although with a relatively steep learning curve, the results can be fantastic.

I decided to mix the real-time data source of DataSift data with a volume/time graph example from the D3 library. I added a simple UI and real-time rendering using Node.js, Socket.io and  Express to allow the user to enter their stream credentials. The finished example is as follows:


Express Framework

Express is a light-weight Sinatra-inspired web development framework. Express offers features for routing, views (with templates) and comes with a built in app generation tool for rapid setup. Here Express simply renders a landing page (using the Foundation framework for layout etc) that offers a stream subscription form. Once the DataSift API credentials have been entered, the form is submitted using jQuery and the stream subscription logic is executed.

DataSift updates with Web Sockets

The DataSift side of the code is very simple and has been extracted away in to a separate file (ds_stream.js). The DataSift Node client lib provides a great example of interacting with streams, and these methods are the basis of the integration. Here we listen for the consumer.on event that is emitted when a DataSift interaction is received, and then use socket.io to push that to any connected clients:

	consumer.on("interaction", function(obj) {

		if(obj.data !== undefined) {
			//console.log(obj.data);
			io.sockets.emit('data', {
				source : obj.data
			});
		}
	});

Rendering with D3.js

The D3 code is client side and sits in /public/javascripts/graph-ic.js and is a modified example written by Mike Bostock (not by myself!). I have added additional comments to the code  so hopefully it should be relatively easy to understand and modify.

The logic of the graph update is built around a single “count” variable that gets incremented when we receive a data update (when a DataSift interaction arrives) via socket.io. The count is incremented, and we also keep a total count for display purposes:

socket.on('data', function(streamData) {
  $('#countTotal').html(countTotal++);

  if(streamData.source.interaction.id != undefined){
    ++count;
  }
});

The rest of the code is basically split in to  two sections – the graph setup and layout, and then the “tick” function that handles the transitions:

this.tick = (function() {

  // update the domains
  now = new Date();

  //  now - 0.75 seconds - (245) * 750
  x.domain([now - (n - 2) * duration, now - duration]);
  y.domain([0, d3.max(data)]);

  // push the accumulated count onto the back, and reset the count
  data.push(Math.min(100, count));

  count = 0;

  // redraw the line
  svg.select(".line")
  .attr("d", line)
  .attr("transform", null);

  // slide the x-axis left
  axis.transition()
  .duration(duration)
  .ease("linear")
  .call(x.axis);

  // slide the line left
  path.transition()
  .duration(duration)
  .ease("linear")
  .attr("transform", "translate(" + x(now - (n - 1) * duration) + ")")
  .each("end", tick);

  // Y Axis
  yaxsis.transition()
  .attr("class", "y axis")
  .ease("linear")
  .call(d3.svg.axis().scale(y).ticks(10).orient("left"));

  // pop the old data point off the front
  data.shift();

})

Code and Setup

Please note there is no error checking, exception handling or anything else! This is for demo purposes only.

  1. Download and extract the source from GitHub here.
  2. Install using NPM socket, express and DataSift
  3. start app.js with node
  4. Access http://www.localhost:8080
  5. Enter your DataSift stream credentials