Raspberry Pi Python Adventures

In an orbit near you

At the last PYPTUG meeting, I demoed Google's Tensorflow deep learning toolkit. I did that using the Jupyter notebook. If you are not familiar with this, check out try.jupyter.org and you'll be  able to play with Python, R, Ruby, Scala, Bash etc.

To install jupyter on your computer, pip3 is your friend (more detail at http://jupyter.readthedocs.org/en/latest/install.html):

pip3 install jupyter

By installing jupyter, you'll also get the ipython kernel, so you'll be able to create new jupyter notebooks for python. There are over 50 programming languages supported by jupyter. But that is not all. You can also create specific environments and associate notebooks with them. It works great on pretty much any platform, including the Raspberry Pi, Windows, the Mac, Linux etc. Each kernel has a varying degree of availability, and the same can be said of python modules. Tensorflow will not run on the Pi at this time...

Tensorflow notebook

New notebook dropdown in Jupyter 4

What we are going to do here is to install Tensorflow in a virtual environment, and create a notebook configuration so we can have the choice in the new-> dropdown, as pictured above.

If you've tried to install Tensorflow, particularly on a Mac,  you have probably found it challenging, as Tensorflow requires Python 2.7.

Install

brew install python

virtualenv -p /usr/local/bin/python2.7 tensor
source tensor/bin/activate

pip install numpy scipy sklearn pandas matplotlib
pip install https://storage.googleapis.com/tensorflow/mac/tensorflow-0.5.0-py2-none-any.whl

Configure Jupyter

(tensor)LTOXFDION:~ francois$ pip install ipython ipykernel
LTOXFDION:kernels francois$ pwd
/Users/francois/Library/Jupyter/kernels
LTOXFDION:kernels francois$ ls
ir python2
LTOXFDION:kernels francois$ cp -r python2/ tensor
LTOXFDION:kernels francois$ cd tensor/
LTOXFDION:tensor francois$ vi kernel.json

In the editor you'll have to modify the path to python to point to your directory. If you dont have a python2 directory to copy, just create a tensor directory and create the kernel.json file. In the end, your kernel.json file should look something like:


You should be good to go now. Start jupyter:

  jupyter notebook

You'll be able to create a new notebook for Tensorflow. From there, all you have to do is import it:

    import tensorflow as tf

We'll continue on this next time.

Francois Dion
@f_dion

“Saving history, one tech gadget at a time”

Outside the walls

The mission page

So it would be easier for people to find it, I've made that blog post a separate page on PYPTUG.ORG:

http://www.pyptug.org/p/python-piedmont-triad-user-group.html

In short

Listed on that page is the goal. I like to go back and read it from time to time to make sure i'm still aligned with that goal:

Our goal is to promote and advance computing, electronics and science in general in North Carolina using the Python programming language, and through workshops and project nights where people can learn, get help and mentor others. A secondary goal is to make you, the artist, the engineer, the investor, the manager, the programmer, the scientist, the teacher, better at your job.

The above link also talks about why Python was chosen. It made a ton of sense some years ago, and it makes as much sense now. Good.

Francois Dion
@f_dion

- You're flying! How? - Python! ( https://xkcd.com/353/ )

Continued

Part one of this series of articles on Team Near Space Circus flight NSC-01 can be found  >here<. It covered the physical layout of the 7 Raspberry Pis, 8 cameras and power system. It left off as we were about to go into the details of the network topology itself.

A satellite or a bowl of spaghetti?

Plan A: 2 Pi in a Pod...

The initial plan was for 2 Raspberry Pi using a serial communication protocol. The Model A+ does not have an ethernet port. Of course a Raspberry Pi 2 has one, and is more powerful, but it also consumes a lot more power than the A+. Our battery pack would have had to be much larger if we went down that path, and that meant significantly heavier. We had a goal of 2Kg for the payload itself, and we were getting close, so serial it was...

Serial communication

DB25 (serial or parallel)

DB9 (serial)

Many early personal computers typically had two interfaces, either built in or available through an expansion card: a serial (either a DB9 or DB25 connector) and a parallel interface (either a DB25 or 36 pins Centronics connector, aka IEE1824). The parallel interface was typically used for printers (and great for DIY DACs with resistor ladders, but that's a story for another time), while the serial port was mostly used for modems or scientific equipment.

RS-232 was introduced in 1962, so it's been around for quite a while now. Many serial connections were actually TTL (5V), such as on the Digital Decwriter III, but it was quite easy to convert them to a +/- voltage range (and even more so since the mid 80s with ICs such as the MAX232). The initial purpose was to connect a DTE (Data Terminal Equipment), a fancy way of saying a computer, and a DCE (Data Communication or Circuit-terminating Equipment).

On a DB9 connector, the pins would be (DB9 male connector pictured):

RxD is Receive Data and TxD is Transmit Data. What if you need to connect two DTE together? You need a Null-Modem adapter or cable. The simplest Null-Modem circuit requires 3 wires (2 if you can guarantee the ground on each end to be the same):

GND(7) to GND(7), TxD(5) to RxD(4), RxD(4) to TxD(5)

Raspberry Pi Header

The Raspberry Pi doesn't have a DB9 or DB25 connector. But if you look at the 40 pin header, notice on the top row, the 4th and 5th pins:

Raspberry Pi Model A+ with pinout as overlay

TXD and RXD. It is a serial port, but it is *not* RS-232 compatible as far as voltage levels are concerned (a 1 is marked by a negative voltage in the range -3V to -15V, and a 0 by a positive voltage in the range +3V to +15V), as it is a TTL interface (a 1 is marked by a positive voltage, either 3V3 or 5V and a 0 is marked by 0V). But that is fine, we are only connecting to another Raspberry Pi.

The primary issue with this serial port is that the Raspbian operating system has a console (TTY) associated it. Meaning that you could connect this to a computer, with the right cable, and see the boot process and get a console prompt, and be able to type commands, as if you had a keyboard and monitor connected directly to the Raspberry Pi.

In our case, however, we want to use the port for something else, our own communication network, so we have to disable this feature. It was once a manual process, and I had written a script to do it, but...

Configuration

sudo raspi-config

The raspi-config now has an option under the Advanced Options selection to disable the serial port console, so we can use it directly:

Advanced -> Serial

Once we have reconfigured the serial port, we can quit raspi-config and reboot. For each compute node, beside disabling the serial port console, we also enabled on the advanced page the SPI and I2C options, and on the main page we configured the time zone, enabled the camera and expanded the filesystem. We also overclocked nodes 1-3 to 800 MHz while nodes 4-6 ran at 700 MHz (so we can measure the impact of overclocking).

Plan B: A Pi cluster

By using a pair of Raspberry Pi, the communication process is pretty straightforward. With two nodes, we can use SLIP or Point to Point Protocol (PPP). Initially, this was the thought for NSC-01 and we would not have to write the code for the network layer.

6 Raspberry Pi cameras

1 Infrared Raspberry Pi camera (pointing down)

But without a functioning camera switcher, the only option was to use one node per camera (each Raspberry Pi has 1 CSI camera connector, even thought the Broadcom chip supports 2 - most cell phones have a front and rear camera). With 7 CSI cameras in total, that meant 7 Raspberry Pi.

Time to dig in our bag of tricks...

Token Ring

4 Node token ring network architecture

As I was going through the options, I started thinking about good old IBM Token Ring (and FDDI). Conceptually, at least. I wasn't going to make something compatible with 802.5, but instead reuse the concept with the serial UART, transmit to receive from one node to the next.

Conceptually, a node is always listening to the serial port. Is this data for me? (if it's a broadcast or if it's addressed to me) If not or if a broadcast, I send it right away to the next node, and process it if it is addressed to me. So let's get into the details.

The diagram

7 Raspberry Pi networked together

In the previous article, one of the pictures clearly showed 7 Raspberry Pi connected together inside the payload. I've included here a fritzing diagram to help in detailing what is going on here.

In the diagram above, I didn't have a Raspberry Pi model A+ for fritzing, so I used the closest thing for our purpose, a Raspberry Pi model B+. Both of them have a 40 pins GPIO header. First row at the top, from left to right has 5V, 5V, GND, TxD and RxD. We'll stop there for right now.

Here I've simply connected node 1 TxD to node 2 RxD (blue wire), node 2 TxD to node 3 RxD (pink), node 3 TxD to node 4 RxD (brown), node 4 TxD to node 5 RxD (cyan), node 5 TxD to node 6 RxD (orange), node 6 TxD to node 7 RxD (yellow) and finally node 7 TxD to node 1 RxD (green), in a nice circular (or ring) fashion.

The GND are not daisy chained from one Pi to the next, because they are all powered by the same source, through their microusb plugs.

Initially, I tried to power the Pi directly from the 5V pin on the GPIO header, but the signal was decoding only partly. By further connecting the 3.3V (that was key, the UART operates at 3V3 so the HIGH signal was not detected reliably when powered with only 5V) and GND together, it made it more reliable, but also more complicated. I reverted back to powering all the 7 nodes using microusb (as detailed in the previous article).

Python

What's great about Raspbian is that not only do we have Python installed, but Python 2.x and 3.x are there, along with many things that are normally optional, such as pygame, pyserial, RPi.GPIO etc. Batteries are quite definitely included with Python on the Raspberry Pi.

Still, for the networking software, we did install two modules from pypi using pip3 (we are using Python 3), docopt and sh.

For the bulk of the development, I used a Raspberry Pi Model B+ connected to my laptop and to a second Raspberry Pi using the serial port and the network switch. That way my laptop could see both Raspberry Pi and I could push new versions of the software using scp.

I also had a "field" version, in case I needed to do last minute changes:

Raspberry Pi sidekick in sidecar configuration (DHCP server on laptop)

Docopt

docopt: helps you to define interface for your command-line app, and automatically generate a parser for it.

For example, in the case of the network controller, token.py, it accepted one argument, the node number. In the future, we'll probably derive the node number from a personality module on the GPIO header, but it was a simple solution that could be done in software, not adding any weight to the payload. The downside was having to keep track of all the microsd cards:

micro SD card transport - For nodes 1 to 7 plus 1 backup

And, so back to docopt, here is how it was used for token.py. At the top of the file, after the shebang line (so the script can be an executable), I included the docstring which defines the interface (lines 2 to 5), which is a required field <address>. This is a simple case, but as cases get more complicated, it really pays off to use docopt.

Then on line 14 I import the docopt module (I group imports alphabetically first by what's included, then a blank line, then again alphabetically I add the imports that have to be installed - just a convention):

1 :  #!/usr/bin/env python3  
2 :  """  
3 :    
4 :  Usage: token.py <address>  
5 :  """  
6 :  from datetime import datetime  
7 :  import logging  
8 :  from logging.handlers import SysLogHandler  
9 :  import RPi.GPIO as gpio  
10:  import serial  
11:  import subprocess  
12:  from time import sleep, time  
13:    
14:  from docopt import docopt  

I then conclude the script by passing the arguments that docopt figured out are there (and validated) in the docstring (__doc__), to the main function:

168:  if __name__ == "__main__":  
169:    arguments = docopt(__doc__)  
170:    main(arguments)  
171:    

And how does one use it in the code itself? In the body of the main function I do the following:

97 :    
98 :  def main(args):  
99 :      my_addr = int(args['<address>'])  
100:

In case an air traveler wants to check our website...

SH

sh: is a full-fledged subprocess replacement for Python 2.6 - 3.4 that allows you to call any program as if it were a function:

`python from sh import ifconfig print ifconfig("eth0") `

Continuing with the code in token.py, you simply import what you need from the shell and sh transparently builds a function for you to call. In this case I needed access to date and rmmod:

1:  #!/usr/bin/env python3  
2:  """  
3:    
4:  Usage: token.py <address>  
5:  """  
6:  from datetime import datetime  
7:  import logging  
8:  from logging.handlers import SysLogHandler  
9:  import RPi.GPIO as gpio  
10:  import serial  
11:  import subprocess  
12:  from time import sleep, time  
13:    
14:  from docopt import docopt  
15:  from sh import date, rmmod  

Then further down in the code I can call date() to call the actual date command from the OS so it is not only in the right format, but in the right timezone (there are other ways to do this, but sh was going to be used heavily for gathering stats, so it made sense to introduce it here):

154:      # synchronize time  
155:      s = str(date())+' \n' # from the OS  
156:      bin = bytes(s, "ascii")  

Pyserial

So, how about accessing a serial port in Python? And are there things to keep in mind when using Python 3?

First thing first, we import serial (line 10). This module is already installed on the Raspberry Pi.

Then, within the body of the main function, we establish a serial connection (line 101). We specify the device that corresponds to the serial UART. This is /dev/ttyAMA0. We also specify the bitrate, expressed in bits per seconds (bps). 115200 is the fastest we can use with the default clock settings for the UART, and is what was used for the flight. As a reference, acoustic coupler modems worked at 300 bps...

We felt that at higher rates it might not give us as reliable a signal, but we will experience with faster rates on future flights (we only had time to do 1 flight within the Global Space Balloon Challenge window). Back in 2012, Gert Van Loo stated:

"The UART and the SPI can be driven by DMA so there is no SW limitation.The real maximum for all those interfaces is only set by the signal integrity.This is certainly the case for I2C as it has to use pull-up resistors to get the signals high again.I know the clocking is rather confused. I built the damn chip and I still have no idea how to control all the clocks.The clocking structure is very complex with a limited number of resources (PLL's) which are used for aplethora of things to drive. (A quick count shows 36! clocks being generated) There will be many cases where it is just not possible to give you ALL the frequencies you want."

From Wikipedia:

"A dead man's switch (for other names, see alternative names) is a switch that is automatically operated if the human operator becomes incapacitated, such as through death, loss of consciousness or being bodily removed from control. Originally applied to switches on a vehicle or machine, it has since come to be used to describe other intangible uses like in computer software."

NSC-01

For Team Near Space Circus' participation in the Global Space Balloon Challenge with NSC-01, we wanted to make sure that once powered up, and enclosed, we would still have control of the computer cluster and the timeline of events, as to what would happen when. As we devised a network with 7 nodes (see part 1 and also part 2 of the computer network in near space) with a master controller and many slave nodes, we needed a way for the system to wait on a signal from us.

A plain old switch would do the trick of course. But what if the balloon took off and we forgot to flip the switch? Or somebody tripped and let the balloon go? 

Martin DeWitt demoes why we need a deadman's switch

Then the master controller would have stayed in standby mode, sending keep alive frames to the other nodes and we would have had no images, no video, no gps data etc. Not good.

Tethered switch

If you've ever spent any time on a threadmill, no doubt you've encountered a tethered switch. Usually it's a magnetic plug, with a string, terminated by a clip. You clip it to yourself, so if you ever fall down/off the threadmill, you end up yanking the plug off the machine, and it stops right away.

Simple deadman's switch, just needing a string

Since the Raspberry Pi has many GPIO pins, all we needed to do was create a simple circuit between two of them that, once interrupted, would signal that it was time for the master controller to go to work.

We will need two GPIO pins, how about GPIO17 and 27?

Why two GPIOS? We could have connected a GPIO to 3V3 and GND through resistors (to protect/limit current), to ensure LOW and HIGH modes. But, trying to keep things as simple as possible from a hardware perspective (software doesn't weigh anything, hardware does), using two GPIOs made a lot of sense. Set one GPIO as input, one as output, raise the output HIGH, the input now sees a value of 1. Open the circuit (by pulling the wire out), and the input no longer sees HIGH, it now reads 0.

Our sponsors (bottom half of payload) with IR cam radar reflector on the left, deadman's switch on right w/carabiner

A string attached on one end to the wire sticking out of the side of the payload and on the other end to a carabiner clip is really all that was needed.

Deadman's switch code

We've already covered the serial port, sh and docopt in part 2 of the computer network in near space. If you are wondering about these things (and lines 98 to 101 in the code below) I urge you to go and read that article first. I've left here the core of what is needed for implementing the software side of the deadman's switch. We first need the GPIO module. On line 9, we import it as gpio (because uppercase is usually reserved for constants). Then on line 102 we set up the gpio module to operate in BOARD mode. That means physical board pin numbers. In the field, you can always count pins, but you might not have a GPIO map handy... But if you prefer Broadcom nomenclature there is a mode for that too.



So GPIO17 is the 6th pin on the bottom and GPIO27 is the 7th. Since odd pins are at the bottom and even at the top, this means pin 11 and 13. Line 103 sets up pin 11 to be an output, and line 104 sets up pin 13 to be an input. Finally, line 105 "arms" the deadman's switch by setting pin 11 to HIGH. Pin 13 should now be reading a 1 instead of a 0.

9:  import RPi.GPIO as gpio 
[...]
98:  def main(args):  
99:    my_addr = int(args['<address>'])  
100:    
101:    ser = serial.Serial('/dev/ttyAMA0', 115200, timeout=1)  
102:    gpio.setmode(gpio.BOARD)  
103:    gpio.setup(11, gpio.OUT)  
104:    gpio.setup(13, gpio.IN)  
105:    gpio.output(11, gpio.HIGH)  

We are now going into the actual functionality of the switch. We establish a simple while loop on line 161, reading pin 13. As long as we see 1, the circuit is closed and on line 162 we send a KEEPALIVE frame to the other nodes (see previous article). We also sleep for 1 second on line 163 (no need to poll faster than that). The moment we get a 0, the while loop exits, and before starting the master controller on line 165, we log our launch time on line 164 (the logging aspects will be in a future article).

[...]
159:    
160:    # deadman's switch  
161:    while gpio.input(13) == 1:  
162:      ser.write(KEEPALIVE)  
163:      sleep(1)  
164:    logger.debug("We are launching! (on {})".format(datetime.now()))  
165:    master_controller(ser, counter) # forever controlling the Pi network until I die    

And that is it!

This can be implemented in all kinds of different projects. Have fun with it!

Francois Dion, Rich Graham, Brent Wright, Chris Shepard and Jeff Clouse Carabiner clipped to my belt while Jeff adds redundant safety strings

Francois Dion
@f_dion

Pictures brought back by NSC01

In the coming days and weeks we'll publish charts and graphs, and we'll be working on images and videos. For right now though, I don't really feel like typing much, so I'll leave you with a few pictures... (Captured using a webcam and pygame and an infrared Raspberry Pi camera, all from node #7 of the compute cluster)

Infrared image, lakes are easy to see on those (and our radar reflector)

About to be launching

Official launch image, April 21st 7:57:24

Francois Dion
@f_dion

Space = rockets

That's what we tend to equate, cause we love them rockets... And, to reach space, it does require a rocket of some sort, at this point. To "explore" space however, at a minimum, only requires a cloudless night, away from the city, and there, with the naked eyes, you can watch the stars, not just a few, but hundreds (some apparently can see upwards of 3000!). Or perhaps you choose a full moon. One can get pretty close even with a 35mm or digital camera with a modest zoom and a tripod.

The Moon, Sept. 9th 2014, DSLR at 270MM

Albedo 0.39

The Moon is nice and all, and is great for black and white photography. And there is nothing wrong with that. In fact, there was an interesting talk back in 2012 by Fred Alger, at PyCarolinas: "Sysadmining Python to the Moon". But the blue planet, the earth is a lot more impressive (picture from that talk):



Except that we are back to rockets again? Or are we? Both Fred's talk and mine that year talked about a near space option.

Near Space

Weather balloons will get you into near space. Space officially starts at 100KM, but one can reach 35-40KM with a latex balloon filled with Helium. Add a parachute, a radar reflector and a payload, and you have your own little private launch and recovery system.

And nowadays, the payload options are numerous. The main thing is that we are no longer limited to sending microcontrollers, but can send actual computers running an operating system, and executing Python code.

An obvious candidate is the Raspberry Pi. Dave Akerman (whom I had mentionned in my talk) has been sending them into space for well over two years now. So when he sent a tweet earlier this month about the Global Space Balloon Challenge, I knew this would be the perfect project for our local Python user group, using a payload with Raspbian Linux and Python on a Raspberry Pi model A+. So many projects are possible within this project.

Hence, Team Near Space (Flying) Circus was born. Let's hope the snake and the penguin will play nice up there...

François
@f_dion

Analytics and Visualization

That's the talk I'll be doing tonight (monday the 27th) at Wake Forest University, in Winston Salem, NC. We'll talk trends, SAS, SPSS, Matlab, R, Python, Pandas, Excel, SQL, Manta, Vincent and all kinds of other stuff, all in 30 minutes. I love a challenge.

A twitter live feed

Hoping to tweet through the talk. Curious to see how that will go. Let's try #pythondata (pydata was shorter, but there's that conference...)


Those interested in attending in person:

http://www.meetup.com/PYthon-Piedmont-Triad-User-Group-PYPTUG/events/159261502/

François
@f_dion

Project nights

Do you attend project nights organized by your local maker group, hackerspace, python user group, raspberry pi user group or other similar tech meet?

No? Why not, is it because there are none? Suggest it, then. Or perhaps it is because you do not have anything to present, or do not need help with any projects. Having said that...

Innovation

Is not inventing something brand new from scratch. It's about standing on the shoulder of giants (a true, if somewhat overused metaphor). Taking many different things and bringing them together into a coherent entity, either a finished good, a software, a consumable or a building block for something else, in a new, innovative way.

It is not easy to achieve that. Are you familiar with all the bleeding and leading edge stuff happening in the tech space, in you area of expertise? Outside your area of expertise?

By attending project nights, and exchanging with people with different backgrounds and fields of expertise, the probability is much higher that you will come up with a solution, or even a new idea. But, more than that, it's at a personal level that you may benefit...

Personal scale innovation

While we all (well, a majority) would like to create the next big thing that will revolutionize the well being of mankind, truth is, what is more likely is to innovate at a personal level, household level or local community level. And your innovation or discussion may trigger another one, that is, if you are involved in some way in your community.

As an example, PYPTUG had a recent project night. One project was exploring the python picamera module. What started as that ended up creating two new projects, one based on Pi NoIR, the Raspberry Pi camera with no IR, as a way to detect heat loss (we'll see how well that works), and the other, as a helper to solder SMD devices.

Each month, there are many such moments of personal scale innovation. Perhaps not iPhone or Pebble (or Raspberry Pi) worldwide game changing innovation, but personal and local scale innovation.

François
@f_dion