Improving the Server Density Service Map

In my first attempt at Physical Website Monitoring I used some 8 bit shift registers and tri-colour LEDs. This while fun, was hard work. Each green, red and blue along with the earth legs of the LED had to be soldered and wired up. That’s OK for a small project but I wanted to expand to 10+ monitoring locations.

So plan B was to use Neopixels, as usual Adafruit has a great guide for them. I had some no brand ones lying around from a while back. You can buy the “branded” ones from any of the major online retailers or you can go cheap from eBay.

Neopixel close up

Close up of Neopixel

Wiring them up is nice and simple. Just solder them together in serial following the arrows on their backs. They only have 3 pins: 5v, earth and data.

I roughly measured the distance between locations I am monitoring from  to measure the length of cable, and then solder the Neopixels together. I ran out of black wire pretty quick, which is why there is more red, sorry for any confusion. Below shows them attached to the back of the map.

Rear view of the Neopixels

The code for controlling the lights is listed below:

I borrowed most of it from the example in Adafruit’s library for the Neopixel. I can’t stress enough how excellent Adafruit is a resource of how to do stuff!

You can test everything is working from the serial port of the Arduino. Send “sydney down\n” and Sydney will change to red, “slow” will change it to orange instead. “up” resets it to green.

The script below is some basic Python to fetch the last response time from Server Density and decided what if at all the colour of a location should be changed to. It’s all a bit rough around the edges but you get the idea.

Below is it in action from the serial port:

Physical Website Monitor based on Server Density’s Monitoring

I wanted to make something to make monitoring more tangible. So I made a board to display the current status of this website chrishannam.co.uk as monitored from a number of remote “actors” provided by Server Density. Below is a snapshot of the monitoring setup from Server Density’s service page.

website monitoring setup

Remote monitoring actors.

The build was pretty basic and luckily I had the parts lying around from previous projects. Rather than explain the setup I’ll give you the link I used as it covers everything better than I could explain. Adafruit Shift Register is an excellent guide on wiring and programming 8 bit shift registers. The only difference is I used tri colour LEDs. The LEDs I used were almost identical to these Tri Colour LEDs from eBay. They do red, green and blue light. I just removed the blue leg as I didn’t need it.

The LEDs are mounted in a 6mm thick panel of MDF. The map was just a simple one printed off Wikipedia.

I used an Arduino Uno but any basic Arduino is up to the job. The code to control the board is listed below. It’s based on the Adafruit example from their excellent guide.

View from behind.

Wired up LEDs

Wiring for Arduino

Here it is in action.

Next I needed to talk to Server Density’s API which luckily is pretty simple. I get the last time from each actor, and test to see if it’s below 0.4 seconds. 0.4 ensures at least 1 server usually Sydney will be down, so it makes for a better display. The Arduino code flips the colour of the LED to make updates a binary change with a message over serial.

One interesting thing about the code is the number of sleeps.

WHEN OPENING A SERIAL CONNECTION TO AN ARDUINO IT RESTARTS!

Bear that in mind. You need to allow time for the Arduino to setup the connection over the serial connection is initiated. This also applies to sending data backwards and forwards as well. Adding the sleeps ensures everything runs smoothly and nothing gets lost.

This was a basic prototype, I am hoping to expand it to an A3 sized map with more locations.

How to Monitor Hard Drive SMART Status on Linux

Following on monitoring hard drive temperatures I thought I would add a check for the current SMART assessment on the status of the drive. The pySMART lib makes this trivial. The code below will output the status.

As a failure could seriously impact the life expectancy of the drive, I have added a call to the Yo service to “Yo me” if a failure is detected.

How to Monitor Hard Drive Temperatures

Monitoring hard drives is pretty similar to the work I touched on for onboard sensors. First we need the right tool for the job. In this case it’s smartctl.

smartctl is available in the smartmontools package for Ubuntu smartmontools. To install:

Once installed you will need to use sudo smartctl to test it.

By default the command outputs a lot of very useful information. Checkout a disk for example using the following:

Lots of info is outputted, have a search for “Temperature_Celsius” to find the temperature in Celsius.

To make life simpler for collecting this information I use pySMART. This Python library makes processing the output from the command so much simpler. The script below extracts all temperatures for the disks stored in the dict DISKS. The pattern is device name (just the disk not the full path) and whatever you want to know the disk as.

I run the script from root’s cron as the smartctl command requires root privileges. Do the following to add it to root’s crontab. The example below will run the script every minute.

How to Monitor Linux Server Sensors

Following on from my last post on monitoring fan speed I found PySensors.This library providers a simple method for extracting data from the sensors command. Below shows the basic usage on my server:

Extending the script from the last article, it’s now simple to record all the sensor data shown here on Grafana. I have updated the script and listed it below.

Temperature Graph from Main Board

Grafana Graphs

How to Monitor Fan Speeds

My biggest concern relocating my server to the garage was dust clogging up the fans. The server has two fans, one CPU and one mounted at the rear of the case.

The best guide for reading sensor data I have found for Ubuntu is Sensors How To. This guides you through setting up the command sensors to access the hardware sensors located on the motherboard.

I am using an ASUS main board and by default sensors wont pick up the fans attached to the board. A bit of Googling found the solution installing the correct kernel module.

$ sudo modprobe nct6775

Make sure you add nct6775 to /etc/modules to ensure it’s loaded at boot time.

sensors command now gives the following output:

$ sensors
acpitz-virtual-0
Adapter: Virtual device
temp1:        +27.8°C  (crit = +105.0°C)
temp2:        +29.8°C  (crit = +105.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +16.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:         +15.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:         +16.0°C  (high = +80.0°C, crit = +100.0°C)

nct6791-isa-0290
Adapter: ISA adapter
in0:                    +0.88 V  (min =  +0.00 V, max =  +1.74 V)
in1:                    +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in2:                    +3.31 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in3:                    +3.30 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in4:                    +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in5:                    +2.02 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in6:                    +0.64 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in7:                    +3.44 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in8:                    +3.25 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in9:                    +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in10:                   +0.21 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in11:                   +0.16 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in12:                   +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in13:                   +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in14:                   +0.20 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
fan1:                     0 RPM  (min =    0 RPM)
fan2:                  1008 RPM  (min =    0 RPM)
fan3:                     0 RPM  (min =    0 RPM)
fan4:                     0 RPM  (min =    0 RPM)
fan5:                     0 RPM  (min =    0 RPM)
SYSTIN:                  +9.0°C  (high =  +0.0°C, hyst =  +0.0°C)  ALARM  sensor = thermistor
CPUTIN:                 +11.5°C  (high = +80.0°C, hyst = +75.0°C)  sensor = thermistor
AUXTIN0:                +47.0°C    sensor = thermistor
AUXTIN1:               +111.0°C    sensor = thermistor
AUXTIN2:               +109.0°C    sensor = thermistor
AUXTIN3:               +110.0°C    sensor = thermistor
PECI Agent 0:           +15.5°C
PCH_CHIP_CPU_MAX_TEMP:   +0.0°C
PCH_CHIP_TEMP:           +0.0°C
PCH_CPU_TEMP:            +0.0°C
intrusion0:            ALARM
intrusion1:            ALARM
beep_enable:           disabled

Only the case fan is reporting, it’s fan2 on the list. I’m not sure why only one fan is reported, still better than none…

Using the script at the bottom I add the data into an InfluxDB database and use Grafana to view it. Below is a sample graph:

Fan speed

Fan Speed

This is some basic graphing that allows me to track any changes that might indicate a problem with the fan.

Below is a script to output the fan data from the above output:

 

Monitoring My House

My standard setup is an Arduino connected to PC either a Raspberry Pi or a nearby PC that can serve as a host.

I decided to relocate my server outside into the garage and I was concerned about how it would cope in the cold and sometimes dusty new environment. My first step was to step up monitoring internally for the server. Disk drive and main board temperatures along with fan speeds would need to be logged. This is covered in detail in my next post.

Once the internal stuff was done I looked at external factors, humidity and temperature being the most important. This is familiar ground for me and setting up a report Arduino was simple as I had a few left from my greenhouses.

With the basics taken care of, I added a sensor to the door to alert if it remained open too long. I also added a check based on the time. The door should be closed between 23:00 and 07:00 most of the time. If it opens between these times I get an alert to my phone to investigate.

I’ll break down each stage in future blog posts.

Analog Dials as Real Time Monitors

I have a deep love of old style analog dials. Recently I found a place selling old voltage meters. There were unusable by todays standards but they looked great. With a bit customising I set about converting them to use hobby servos and make them fully controlled by an Arduino.

The analog dials:

 

Monitors

 

Below is fairly quick transformation. Gutted the insides of the unit and used a glue gun to mount the servo and blu tac to attach the needle.

 

Monitor in bits Work of Art

Guts Monitor with Servo

 

 

After the servos were mounted, I added a tri colour LED.

Lights

It took a bit effort to get the casing back on the dials especially with the LED bent over the top the dial. Both the dials are mounted on an old shelf.

Screen Shot 2014-12-08 at 22.24.37

Above is the basic wiring, two servo controls and 6 LED controls. I kept the breadboard to make sure I was able to change things later.

Below is the simple code I used to control them:

/* 
I started with the sample from this guy

Sweep
 by BARRAGAN <http://barraganstudio.com> 
 This example code is in the public domain.

 modified 8 Nov 2013
 by Scott Fitzgerald
 http://arduino.cc/en/Tutorial/Sweep
*/ 

#include <Servo.h> 
 
Servo output_dial;  // create servo object to control a servo 
Servo input_dial;   // twelve servo objects can be created on most boards
  
boolean stringComplete = false;
String inputString = "";
int led_green_input = 2;
int led_blue_input = 3;
int led_red_input = 4;
int led_green_output = 6;
int led_blue_output = 7;
int led_red_output = 8;

void setup() 
{ 
  Serial.begin(9600);
  input_dial.attach(9);  // left
  output_dial.attach(10); // right
  pinMode(led_green_input, OUTPUT);
  pinMode(led_blue_input, OUTPUT);
  pinMode(led_red_input, OUTPUT);
  pinMode(led_green_output, OUTPUT);
  pinMode(led_blue_output, OUTPUT);
  pinMode(led_red_output, OUTPUT);
  output_dial.write(135); // reset to left side of the dial
  input_dial.write(145); // reset to left side of the dial
  
  // set green as LED colour default
  digitalWrite(led_red_input, LOW);
  digitalWrite(led_blue_input, LOW);
  digitalWrite(led_green_input, HIGH);
  digitalWrite(led_red_output, LOW);
  digitalWrite(led_blue_output, LOW);
  digitalWrite(led_green_output, HIGH);
} 
 
void loop() 
{ 
  
  if (stringComplete) {
    inputString.replace("\n","");
    int len = inputString.length();
    String servo = inputString.substring(0,1);
    String colour = inputString.substring(2,3);
    String movement = inputString.substring(4,len);
    Serial.println(stringComplete);
    
    // limit how far we move the meter to make suer we don't
    // damage the needle on the side of the casing.
    
    // right dial
    if (servo.toInt() == 1 && movement.toInt() <= 135 && movement.toInt() >= 30){
      output_dial.write(movement.toInt()); 
    }
    
    // left dial
    if (servo.toInt() == 0 && movement.toInt() <= 145 && movement.toInt() >= 55){
      input_dial.write(movement.toInt());
    }
    
    if (servo == "1"){
      if (colour.toInt() == 0){
        digitalWrite(led_red_output, LOW);
        digitalWrite(led_blue_output, LOW);
        digitalWrite(led_green_output, HIGH);
      }
      if (colour.toInt() == 1){
        digitalWrite(led_red_output, LOW);
        digitalWrite(led_blue_output, HIGH);
        digitalWrite(led_green_output, LOW);
      }
      if (colour.toInt() == 2){
        digitalWrite(led_blue_output, LOW);
        digitalWrite(led_red_output, HIGH);
        digitalWrite(led_green_output, LOW);
      }
    }
    if (servo == "0"){
      if (colour.toInt() == 0){
        digitalWrite(led_red_input, LOW);
        digitalWrite(led_blue_input, LOW);
        digitalWrite(led_green_input, HIGH);
      }
      if (colour.toInt() == 1){
        digitalWrite(led_red_input, LOW);
        digitalWrite(led_blue_input, HIGH);
        digitalWrite(led_green_input, LOW);
      }
      if (colour.toInt() == 2){
        digitalWrite(led_blue_input, LOW);
        digitalWrite(led_red_input, HIGH);
        digitalWrite(led_green_input, LOW);
      }
    }
    // clear the string:
    inputString = "";
    stringComplete = false;
  }
 
  delay(1000); 
 
} 

void serialEvent() {
  while (Serial.available()) {
    // get the new byte:
    char inChar = (char)Serial.read(); 
    // add it to the inputString:
    inputString += inChar;
    // if the incoming character is a newline, set a flag
    // so the main loop can do something about it:
    if (inChar == '\n') {
      stringComplete = true;
    } 
  }
}

https://gist.github.com/bassdread/952e9df9d820c16841f3 contains a better downloadable gist.

The above code is enough to move the needle the full range of the dial without hitting the side and damaging the needle. It also controls the colour of the LED. Currently it does a basic solid colour without any mixing.

To make it standalone I connected the Arduino to a Raspberry Pi and ran the following simple Flask app to control the dials over serial.

from flask import Flask
import serial
from datetime import datetime, timedelta
from time import sleep

serial_port = '/dev/tty.usbserial-A501S1GX'
ser = serial.Serial(serial_port, 9600)
sleep(1.5)

app = Flask(__name__)

# input is the left side dial
@app.route("/input/<colour>/<movement>")
def input(colour, movement):
    ser.write('0,{0},{1}\n' .format(colour, movement))
    return "Done!"

@app.route("/output/<colour>/<movement>")
def output(colour, movement):
    ser.write('1,{0},{1}\n' .format(colour, movement))
    return "Done!"


if __name__ == "__main__":
    app.run()

https://gist.github.com/bassdread/1a7d49adebf288c4684b is the downloadable gist.

Below is a better quality selection of pictures of it in action.

Pi Wars

This weekend (6th December) I was lucky enough to attend http://piwars.org/ at the Institute of Astronomy, Cambridge. I wasn’t entirely sure what to expect but it was a robotics event outside London so it was worth the risk.

I took my young ish son and a friend on the trip with the promise of robots and hopefully some robot wars style fun. We arrived a little bit early and had time to have a look around and chat to some of the exhibitors and robot owners before the competition kicked off.

The first that struck me was how friendly and happy to talk everyone was. Everyone was there to learn and even if they didn’t realise it, teach. My son was very impressed with http://www.cannybots.com/ these are small 3D printed bluetooth controlled like cars that could with follow a black line on white paper or be controlled manually. My friend and the guy running the display happily traded 3D printing stories while my son and I raced two little robot cars around the table.

All the robots taking part were Pi powered and varied in size. There were a number of different courses and challenges. The assault course below was a good watch with enough varied surfaces to challenge most of the robots.

Assault Course

The drag strip and the sumo fights were also great to watch. Some of the sumo matches ended up being a bit one sided.

 

 

Drag Race Sumo

 

It was a well organised and entertaining event. The boy and I intend to return, hopefully with a robot.

The venue was also very interesting, we went for a quick stroll for lunch and found an awesome old style telescope, well at least the container for it.

Outside

Tornado Testing

Tornado testing can get complex. Below is a simple snippet for testing a POST request that accesses a supplied argument. Took me ages to figure out the arguments needed to be urlencoded. Hope this saves you some time when testing your Tornado code.

from tornado import ioloop
from tornado.testing import AsyncHTTPTestCase
from tornado.web import Application, RequestHandler
import urllib


class MainHandler(RequestHandler):
    def get(self):
        self.write("Hello, world")

    def post(self):
        name = self.get_argument('name')
        self.write("Hello, {0}".format(name))


class TestHelloWorld(AsyncHTTPTestCase):

    def get_app(self):
        return Application([(r"/", MainHandler)])

    def test_hello_world(self):
        # add any useful header stuff here
        headers = {}

        data = {'name':'Tony'}
        body = urllib.urlencode(data)

        self.http_client.fetch(
            self.get_url('/'),
            self.stop,
            method="POST",
            body=body,
            headers=headers
        )
        response = self.wait()
        self.assertEquals(response.body, 'Hello, Tony')


if __name__ == "__main__":
    application = Application([
        (r"/", MainHandler),
    ])
    application.listen(8888)
    ioloop.IOLoop.instance().start()