PyData Carolinas 2016

At PyData Carolinas 2016 I presented the talk Stemgraphic: A Stem-and-Leaf Plot for the Age of Big Data.

Intro

The stem-and-leaf plot is one of the most powerful tools not found in a data scientist or statistician’s toolbox. If we go back in time thirty some years we find the exact opposite. What happened to the stem-and-leaf plot? Finding the answer led me to design and implement an improved graphical version of the stem-and-leaf plot, as a python package. As a companion to the talk, a printed research paper was provided to the audience (a PDF is now available through artchiv.es)

The talk

Thanks to the organizers of PyData Carolinas, videos of all the talks and tutorials have been posted on youtube. In just 30 minutes, this is a great way to learn more about stemgraphic and the history of the stem-and-leaf plot for EDA work. This updated version does include the animated intro sequence, but unfortunately the sound was recorded from the microphone, and not the mixer. You can see the intro sequence in higher audio and video quality on the main page of the website below.

Stemgraphic.org

I've created a web site for stemgraphic, as I'll be posting some tutorials and demo some of the more advanced features, particularly as to how stemgraphic can be used in a data science pipeline, as a data wrangling tool, as an intermediary to big data on HDFS, as a visual validation for building models and as a superior distribution plot, particularly when faced with non uniform distributions or distributions showing a high degree of skewness (long tails).

Github Repo

https://github.com/fdion/stemgraphic

Francois Dion
@f_dion

Two topics will be covered:

Chipmusic, limitations and creativity

In an orbit near you

At the last PYPTUG meeting, I demoed Google's Tensorflow deep learning toolkit. I did that using the Jupyter notebook. If you are not familiar with this, check out try.jupyter.org and you'll be  able to play with Python, R, Ruby, Scala, Bash etc.

To install jupyter on your computer, pip3 is your friend (more detail at http://jupyter.readthedocs.org/en/latest/install.html):

pip3 install jupyter

By installing jupyter, you'll also get the ipython kernel, so you'll be able to create new jupyter notebooks for python. There are over 50 programming languages supported by jupyter. But that is not all. You can also create specific environments and associate notebooks with them. It works great on pretty much any platform, including the Raspberry Pi, Windows, the Mac, Linux etc. Each kernel has a varying degree of availability, and the same can be said of python modules. Tensorflow will not run on the Pi at this time...

Tensorflow notebook

New notebook dropdown in Jupyter 4

What we are going to do here is to install Tensorflow in a virtual environment, and create a notebook configuration so we can have the choice in the new-> dropdown, as pictured above.

If you've tried to install Tensorflow, particularly on a Mac,  you have probably found it challenging, as Tensorflow requires Python 2.7.

Install

brew install python

virtualenv -p /usr/local/bin/python2.7 tensor
source tensor/bin/activate

pip install numpy scipy sklearn pandas matplotlib
pip install https://storage.googleapis.com/tensorflow/mac/tensorflow-0.5.0-py2-none-any.whl

Configure Jupyter

(tensor)LTOXFDION:~ francois$ pip install ipython ipykernel
LTOXFDION:kernels francois$ pwd
/Users/francois/Library/Jupyter/kernels
LTOXFDION:kernels francois$ ls
ir python2
LTOXFDION:kernels francois$ cp -r python2/ tensor
LTOXFDION:kernels francois$ cd tensor/
LTOXFDION:tensor francois$ vi kernel.json

In the editor you'll have to modify the path to python to point to your directory. If you dont have a python2 directory to copy, just create a tensor directory and create the kernel.json file. In the end, your kernel.json file should look something like:


You should be good to go now. Start jupyter:

  jupyter notebook

You'll be able to create a new notebook for Tensorflow. From there, all you have to do is import it:

    import tensorflow as tf

We'll continue on this next time.

Francois Dion
@f_dion