Monthly Archives: February 2016

Physical Website Monitor based on Server Density’s Monitoring

I wanted to make something to make monitoring more tangible. So I made a board to display the current status of this website chrishannam.co.uk as monitored from a number of remote “actors” provided by Server Density. Below is a snapshot of the monitoring setup from Server Density’s service page.

website monitoring setup

Remote monitoring actors.

The build was pretty basic and luckily I had the parts lying around from previous projects. Rather than explain the setup I’ll give you the link I used as it covers everything better than I could explain. Adafruit Shift Register is an excellent guide on wiring and programming 8 bit shift registers. The only difference is I used tri colour LEDs. The LEDs I used were almost identical to these Tri Colour LEDs from eBay. They do red, green and blue light. I just removed the blue leg as I didn’t need it.

The LEDs are mounted in a 6mm thick panel of MDF. The map was just a simple one printed off Wikipedia.

I used an Arduino Uno but any basic Arduino is up to the job. The code to control the board is listed below. It’s based on the Adafruit example from their excellent guide.

View from behind.

Wired up LEDs

Wiring for Arduino

Here it is in action.

Next I needed to talk to Server Density’s API which luckily is pretty simple. I get the last time from each actor, and test to see if it’s below 0.4 seconds. 0.4 ensures at least 1 server usually Sydney will be down, so it makes for a better display. The Arduino code flips the colour of the LED to make updates a binary change with a message over serial.

One interesting thing about the code is the number of sleeps.

WHEN OPENING A SERIAL CONNECTION TO AN ARDUINO IT RESTARTS!

Bear that in mind. You need to allow time for the Arduino to setup the connection over the serial connection is initiated. This also applies to sending data backwards and forwards as well. Adding the sleeps ensures everything runs smoothly and nothing gets lost.

This was a basic prototype, I am hoping to expand it to an A3 sized map with more locations.

How to Monitor Hard Drive SMART Status on Linux

Following on monitoring hard drive temperatures I thought I would add a check for the current SMART assessment on the status of the drive. The pySMART lib makes this trivial. The code below will output the status.

As a failure could seriously impact the life expectancy of the drive, I have added a call to the Yo service to “Yo me” if a failure is detected.

How to Monitor Hard Drive Temperatures

Monitoring hard drives is pretty similar to the work I touched on for onboard sensors. First we need the right tool for the job. In this case it’s smartctl.

smartctl is available in the smartmontools package for Ubuntu smartmontools. To install:

Once installed you will need to use sudo smartctl to test it.

By default the command outputs a lot of very useful information. Checkout a disk for example using the following:

Lots of info is outputted, have a search for “Temperature_Celsius” to find the temperature in Celsius.

To make life simpler for collecting this information I use pySMART. This Python library makes processing the output from the command so much simpler. The script below extracts all temperatures for the disks stored in the dict DISKS. The pattern is device name (just the disk not the full path) and whatever you want to know the disk as.

I run the script from root’s cron as the smartctl command requires root privileges. Do the following to add it to root’s crontab. The example below will run the script every minute.

How to Monitor Linux Server Sensors

Following on from my last post on monitoring fan speed I found PySensors.This library providers a simple method for extracting data from the sensors command. Below shows the basic usage on my server:

Extending the script from the last article, it’s now simple to record all the sensor data shown here on Grafana. I have updated the script and listed it below.

Temperature Graph from Main Board

Grafana Graphs

How to Monitor Fan Speeds

My biggest concern relocating my server to the garage was dust clogging up the fans. The server has two fans, one CPU and one mounted at the rear of the case.

The best guide for reading sensor data I have found for Ubuntu is Sensors How To. This guides you through setting up the command sensors to access the hardware sensors located on the motherboard.

I am using an ASUS main board and by default sensors wont pick up the fans attached to the board. A bit of Googling found the solution installing the correct kernel module.

$ sudo modprobe nct6775

Make sure you add nct6775 to /etc/modules to ensure it’s loaded at boot time.

sensors command now gives the following output:

$ sensors
acpitz-virtual-0
Adapter: Virtual device
temp1:        +27.8°C  (crit = +105.0°C)
temp2:        +29.8°C  (crit = +105.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +16.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:         +15.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:         +16.0°C  (high = +80.0°C, crit = +100.0°C)

nct6791-isa-0290
Adapter: ISA adapter
in0:                    +0.88 V  (min =  +0.00 V, max =  +1.74 V)
in1:                    +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in2:                    +3.31 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in3:                    +3.30 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in4:                    +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in5:                    +2.02 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in6:                    +0.64 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in7:                    +3.44 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in8:                    +3.25 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in9:                    +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in10:                   +0.21 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in11:                   +0.16 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in12:                   +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in13:                   +1.01 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
in14:                   +0.20 V  (min =  +0.00 V, max =  +0.00 V)  ALARM
fan1:                     0 RPM  (min =    0 RPM)
fan2:                  1008 RPM  (min =    0 RPM)
fan3:                     0 RPM  (min =    0 RPM)
fan4:                     0 RPM  (min =    0 RPM)
fan5:                     0 RPM  (min =    0 RPM)
SYSTIN:                  +9.0°C  (high =  +0.0°C, hyst =  +0.0°C)  ALARM  sensor = thermistor
CPUTIN:                 +11.5°C  (high = +80.0°C, hyst = +75.0°C)  sensor = thermistor
AUXTIN0:                +47.0°C    sensor = thermistor
AUXTIN1:               +111.0°C    sensor = thermistor
AUXTIN2:               +109.0°C    sensor = thermistor
AUXTIN3:               +110.0°C    sensor = thermistor
PECI Agent 0:           +15.5°C
PCH_CHIP_CPU_MAX_TEMP:   +0.0°C
PCH_CHIP_TEMP:           +0.0°C
PCH_CPU_TEMP:            +0.0°C
intrusion0:            ALARM
intrusion1:            ALARM
beep_enable:           disabled

Only the case fan is reporting, it’s fan2 on the list. I’m not sure why only one fan is reported, still better than none…

Using the script at the bottom I add the data into an InfluxDB database and use Grafana to view it. Below is a sample graph:

Fan speed

Fan Speed

This is some basic graphing that allows me to track any changes that might indicate a problem with the fan.

Below is a script to output the fan data from the above output:

 

Monitoring My House

My standard setup is an Arduino connected to PC either a Raspberry Pi or a nearby PC that can serve as a host.

I decided to relocate my server outside into the garage and I was concerned about how it would cope in the cold and sometimes dusty new environment. My first step was to step up monitoring internally for the server. Disk drive and main board temperatures along with fan speeds would need to be logged. This is covered in detail in my next post.

Once the internal stuff was done I looked at external factors, humidity and temperature being the most important. This is familiar ground for me and setting up a report Arduino was simple as I had a few left from my greenhouses.

With the basics taken care of, I added a sensor to the door to alert if it remained open too long. I also added a check based on the time. The door should be closed between 23:00 and 07:00 most of the time. If it opens between these times I get an alert to my phone to investigate.

I’ll break down each stage in future blog posts.