Logging sensor data into Google annotated time line graphs with Python and Arduino

Posted on Dec 31, 2011 in Arduino, C++, JavaScript, PHP, Programming, Python | 1 comment

Annotated time line graphs from the Google Visualization API are a very nice way to display sensor data. With the gviz_api library for python the sensor data can be stored into JSON files that can be read by the Google Visualization API in JavaScript. Here I will explain the steps from Arduino to JavaScript.

Arduino Code

The first step is getting the sensor data from Arduino to Python. I have described setting up the serial communication between Arduino and Python here. The Arduino sends the sensor data as comma separated values, which can be read by a CSV reader in Python. As separator, I use the semicolon. The order is always the same: beer temperature, beer setting, beer annotation, fridge temperature, fridge setting, fridge annotation.

The Arduino stores the temperatures in tenths of degrees, so they have to be divided by 10 before sending them to the serial port for logging. When just the temperatures are printed, without an annotation, this function is used:

void serialPrintTemperatures(void){
      Serial.print(beerTemperatureActual/10);
      Serial.print(";");
      Serial.print(beerTemperatureSetting/10);
      Serial.print(";");
      Serial.print(";");
      Serial.print(fridgeTemperatureActual/10);
      Serial.print(";");
      Serial.print(fridgeTemperatureSetting/10);
      Serial.println(";");
}

When the Arduino wants to add an annotation, I use two very similar functions, they just print an annotation where the function above does not print anything.

void serialBeerMessage(int messageType){
      Serial.print(beerTemperatureActual/10);
      Serial.print(";");
      Serial.print(beerTemperatureSetting/10);
      Serial.print(";");
 
      switch(messageType){
        case BEER_SETTING_FROM_SERIAL:
          Serial.print("\"Beer temperature setting changed to ");
          Serial.print(int(beerTemperatureSetting+.5)/10); Serial.print("."); Serial.print(int(beerTemperatureSetting+.5)%10);
          Serial.print(" via web interface.\"");
          break;
        case BEER_SETTING_FROM_FRIDGE:
          Serial.print("\"Beer temperature setting changed to ");
          Serial.print(int(beerTemperatureSetting+.5)/10); Serial.print("."); Serial.print(int(beerTemperatureSetting+.5)%10);
          Serial.print(" via fridge menu.\"");
          break;
        case BEER_SETTING_FROM_PROFILE:
          Serial.print("\"Beer temperature setting changed to ");
          Serial.print(int(beerTemperatureSetting+.5)/10); Serial.print("."); Serial.print(int(beerTemperatureSetting+.5)%10);
          Serial.print(" according to temperature profile.\"");
          break;
        default:
          Serial.println("\"Error: Unknown Beer Message Type!\"");
      }
 
      Serial.print(";");
      Serial.print(fridgeTemperatureActual/10);
      Serial.print(";");
      Serial.print(fridgeTemperatureSetting/10);
      Serial.println(";");
}

The different types of annotations are defined in an enumeration in enums.h:

enum beerMessages{
  BEER_SETTING_FROM_FRIDGE,
  BEER_SETTING_FROM_SERIAL,
  BEER_SETTING_FROM_PROFILE
};
 
enum fridgeMessages{
  FRIDGE_SETTING_FROM_FRIDGE,
  FRIDGE_SETTING_FROM_SERIAL,
  FRIDGE_DOOR_OPEN,
  FRIDGE_DOOR_CLOSED,
  POSPEAK,
  NEGPEAK,
  POSDRIFT,
  NEGDRIFT,
  ARDUINO_START
};

Python code

A python library to create JSON files for the Google Visualization API is available from Google here. With this library imported in Python, a new empty data table can be made with the following code:

# Define Google data table description and create empty data table
description = {	"Time": ("datetime","Time"),
		"BeerTemp": 	("number",	"Beer temperature"),
		"BeerSet":	("number",  "Beer setting"),
		"BeerAnn":	("string",	"Beer Annotate"),
		"FridgeTemp":	("number",	"Fridge temperature"),
		"FridgeSet":	("number",	"Fridge setting"),
		"FridgeAnn":	("string",	"Fridge Annotate")}
dataTable = gviz_api.DataTable(description)

And here is the python code to add sensor data received from Arduino:

while(1): #read all lines on serial interface
	line = ser.readline()
	if(line): #line available?
	 	#process line
		if line.count(";")==5:
			#valid data received
			lineAsFile = StringIO.StringIO(line) #open line as a file to use it with csv.reader
			reader = csv.reader(lineAsFile, delimiter=';',quoting=csv.QUOTE_NONNUMERIC)
			for	row	in reader: #Relace empty annotations with None
				if(row[2]==''):
					row[2]=None
				if(row[5]==''):
					row[5]=None
				#append new row to data table, print it to stdout and write complete datatable to json file
				newRow= [{'Time': datetime.today(),'BeerTemp': row[0], 'BeerSet': row[1], 'BeerAnn': row[2], 'FridgeTemp': row[3], 'FridgeSet': row[4], 'FridgeAnn': row[5]}]
				print newRow
				dataTable.AppendData(newRow)
				jsonfile = open(localJsonFileName,'w')
				jsonfile.write(unicode(dataTable.ToJSon(columns_order=["Time", "BeerTemp",	"BeerSet", "BeerAnn", "FridgeTemp", "FridgeSet", "FridgeAnn"])))
				jsonfile.close()
				copyfile(localJsonFileName,wwwJsonFileName) #copy to www dir. Do not write directly to www dir to prevent blocking www file.
 
				#write csv file too
				csvFile = open(csvFileName,"a")
				lineToWrite = time.strftime("%b %d %Y %H:%M:%S;" ) + line
				csvFile.write(lineToWrite)
				csvFile.close()
			prevDataTime = time.time() #store time of last new data for interval check
		else:
			print >> sys.stderr, "Error: Received	invalid	line: " + line
	elif((time.time() - prevDataTime) >= serialRequestInterval): #if no new data has been received for serialRequestInteval seconds, request it
		ser.write("r")		#	request	new	data from	arduino
		time.sleep(1)			# give the arduino time to respond
		continue
	elif(time.time() - prevDataTime > serialRequestInterval+2*serialCheckInterval):
		#something is wrong: arduino is not responding to data requests
		print >> sys.stderr, "Error: Arduino is not responding to new data requests"
	else:
		break

This piece of code probably needs some explanation. To check whether the received line is valid sensor data, I check for 5 semicolons. The CSV reader in python can only work with files, but python has a function to open a string as a file.

I print empty annotations in the Arduino code, but this has to replaced by ‘None’ for the data table.

I add the current time to the data point, create a row in the correct format and add this row to the data table. Next I write the entire data table to a JSON file, in unicode (which is needed for the web server). Because writing to the file can take a few seconds when the data table gets large, I write to a local file first. I only copy that file to the www directory when done, which is a lot faster. Unfortunately the gviz_api library has no functionality to append to a JSON file, which would have prevented having the whole data table in memory and having to write the entire table to a file for each data point.

The data is also logged in a CSV file, for easy importing of the data in programs like Excel

If no data has been received for serialRequestInterval, python requests it from the Arduino, waits one second and jumps to the start of the while loop.

Limited memory issues

The router has only 16 MB of RAM. When logging every minute, each day generates 1440 data points. At some point, the data table is too large to be stored in RAM and the router crashes. To prevent this from happening, I start a new data table and new JSON files each day.

lastDay = day
day = time.strftime("%Y-%m-%d")
if lastDay != day:
	 #empty data table and write to new files
	dataTable.LoadData([])
	print >> sys.stderr, "Notification: New day, dropping data table and creating new JSON file."
	jsonFileName= currentBeerName + '/' + currentBeerName + '-' + day
	localJsonFileName = '/mnt/<strong>uberfridge</strong>/data/' + jsonFileName + '.json'
	wwwJsonFileName='/opt/share/www/lighttpd/data/' + jsonFileName + '.json'

All JSON files are stored in a directory with the beer’s name. When I start the script, I have to check whether a JSON file for that day already exists. If it does, I append the next unused postfix number to the filename.

day = time.strftime("%Y-%m-%d")
lastDay = day
# define a JSON file to store the data table
jsonFileName= currentBeerName + '/' + currentBeerName + '-' + day
#if a file for today already existed, add suffix
if os.path.isfile('/mnt/<strong>uberfridge</strong>/data/' + jsonFileName + '.json'):
	i=1
	while (os.path.isfile('/mnt/<strong>uberfridge</strong>/data/' + jsonFileName + '-' + str(i) + '.json')):
		i=i+1
	jsonFileName = jsonFileName + '-' + str(i)
localJsonFileName = '/mnt/<strong>uberfridge</strong>/data/' + jsonFileName + '.json'
 
# Define a location on the webserver to copy the file to after it is written
wwwJsonFileName='/opt/share/www/lighttpd/data/' + jsonFileName + '.json'
 
# Define a CSV file to store the data as CSV (might be useful one day)
csvFileName = '/mnt/<strong>uberfridge</strong>/data/' + currentBeerName + '/' + currentBeerName + '.csv'

These separate JSON files are retrieved by JavaScript later and recombined to one array to create the chart.

PHP and JavaScript Code

Because the data for each beer is stored in multiple JSON files, these files need to be are retrieved one by one in JavaScript. To do this, JavaScript has to have a list of these files. The following PHP file returns a JavaScript array with a list of the file names:

&lt;?php
	$beerName = $_POST["beername"];
	$fileNames = array();
  	$currentBeerDir = 'data/' . $beerName;
  	$handle = opendir($currentBeerDir);
  	$first = true;
  	$i=0;
  	while (false !== ($file = readdir($handle))){  // iterate over all json files in directory
		  $extension = strtolower(substr(strrchr($file, '.'), 1));
		  if($extension == 'json' ){
		  	$jsonFile =  $currentBeerDir . '/' . $file;
				$filenames[$i] = $jsonFile;
				$i=$i+1;
			}
		}
		closedir($handle);
		if(empty($filenames)){
			echo "";
		}
		else{
			echo json_encode($filenames);
		}
?&gt;

It iterates over all the files in the directory with the name of the beer, combines these in an array and echoes this array.

The JavaScript function performs a POST request on the PHP file above to get the file names. It then receives each of these JSON files. It keeps the full array for the first file. For the next files, it concatenates the new rows with the rows of the existing array. After all files have been added, it draws the chart in the DIV.

/* Give name of the beer to display and div to draw the graph in */
function drawBeerChart(beerName, div){
	var beerChart;
	var beerData;
 
	$.post("get_beer_files.php", {"beername": beerName}, function(answer){
		var combinedJson;
		var first = true;
		var files = eval(answer);
 
		for(i=0;i&lt;files.length;i++){
 
			filelocation = files[i];
			var jsonData = $.ajax({
					url: filelocation,
					dataType:"json",
		    		async: false
	      			}).responseText;
      		var evalledJsonData = eval("("+jsonData+")");
			if(first){
				combinedJson = evalledJsonData;
				first = false;
			}
			else{
				combinedJson.rows  = combinedJson.rows.concat(evalledJsonData.rows);
			}
		}
		var beerData = new google.visualization.DataTable(combinedJson);
		var beerChart = new google.visualization.AnnotatedTimeLine(document.getElementById(div));
    	beerChart.draw(beerData, {
               'displayAnnotations': true,
               'scaleType': 'maximized',
               'displayZoomButtons': false,
               'allValuesSuffix': "\u00B0 C",
               'numberFormats': "##.0",
              'displayAnnotationsFilter' : true});
	});
}

The annotated timeline chart that is created looks like this:

So there you have it, the full code to get from sensor data to charts in a browser. Getting it together was a big puzzle, especially due to the little details, like having to convert Unicode, eval’ing the POST data, combining multiple JSON files, replacing empty annotations with ‘None’, out of memory crashes, writing to a local file first and many more.
Maybe a lot of these things are straight forward for an experienced web developer, but this was my first experience with Python, PHP and JavaScript. So if you spot strange things in my code, please let me know.

One Comment

  1. Great Work!!

Leave a Reply

Your email address will not be published. Required fields are marked *