Raspberry Pi Python Adventures

Yes, it works

Go and check out Solaris Desktop for the details. The only challenge is in downloading the file. Then you will be able to enjoy PyCharm on IllumOS, stormOS, OpenIndiana, Solaris, Nexenta, SmartOS etc.

François
@f_dion

2012

This year, just looking at what is directly relevant to this blog, it hasn't been a bad year at all on the Python and Raspberry Pi front, considering I do this in my free (where? when?) time, outside of running an IT department, a company, a household, and helping in my community...

• mentored a few individuals
• a lightning talk at PyCarolinas
• a full length talk at PyCarolinas
• Got a whole team to embrace Python
• started PYPTUG, The PYthon Piedmont Triad User Group
• started PYPTUG PyHack Workshops @ Fablocker
• organized PYPTUG meetings
• over 250 posts on Raspberry Pi forum
• started this blog (over 56,000 visitors in last 4 months)
• over 120 blog articles in 4 languages
• published a few repositories on Bitbucket
• translated and proofread part of The Mag Pi magazine (in French)

That is a lot. And I plan to continue. Plus I contributed to a very very cool project, Brython (a Welsh word for a Breton, someone originating from the region of Brittany in France). Brython is Python scripting in your web browser.

Personally, I think this is a very important project for the Python community, and I put my money where my mouth is by investing my own time on this:

• english translation
• spanish translation
• spanish group (brython-es on google group)
• writing a few demos
• writing a few articles
• testing functionality
• sending bug reports
• fixing stuff
• some public relation and promo

Of course, this pale in comparison to the amount of time that Brython's creator Pierre Quentel has spent on it. I encourage my readers to get involved in that project and make it a success in 2013.

2013

In 2013, I would like to see dtrace (one of the most important technology of the last 15 years) bundled with Python on all platform that support it (the BSDs, Mac OS/X, Solaris and IllumOS based like OpenIndiana). The patch has existed for a long time, so it is not a technical issue. We will see if politics can be set aside and the patch be mainstreamed. I do owe some people from OpenIndiana some work on the regular Python 2.7.x, but quite frankly I am less than enthused to work on this without making dtrace (Jesus Cea's patches) part of it.

How about IllumOS on the Rasberry Pi? The time is right, with the 512MB Pi now being the norm. There is some background noise already on this, so I'm hoping we can get this thing going.

Finally, I'd like to see the disaster of 2012 (Ten's Complement Zevo zfs file system sandbagged) get reversed, and ZFS be brought back to Mac OS/X. That is probably most unlikely, since Apple are not interested in trailblazing anymore. HFS+ in 2013, really? really?

François
@f_dion

FISH

I do a good bit of hardware integration with the web, with manufacturing equipment, with embedded systems and with big data set, or that can sustain multiple failures. Not necessarily all at once, but typically, people expect FISH from me :)

FISH is Fully Integrated Software and Hardware (btw, as a side note, the internal project at Sun to create appliances based on ZFS was known as FISHWorks). The Raspberry Pi is a cool piece of hardware, but I typically need stuff that is only (or mostly) found on Solaris and derived OSes, such as ZFS. I've been using ZFS for many years now, since the first public release on Solaris Nevada. ZFS scales and give you data integrity. And it can run on the largest systems known to man.

It scales

For example, I'm listening right now to ZFS Day's live video stream and hearing a talk about ZFS on the Sequoia supercomputer, which is the fastest supercomputer out there. They are using it as a native port, not using FUSE.

What is ZFS? 

Wikipedia: "ZFS is a combined file system and logical volume manager designed by Sun Microsystems. The features of ZFS include data integrity verification against data corruption modes, support for high storage capacities, integration of the concepts of filesystem and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z and native NFSv4 ACLs. ZFS is implemented as open-source software, licensed under the Common Development and Distribution License (CDDL)."

From Supercomputers to $35 computers

So, ZFS scales at the highest level obviously. Well, it also scales down: I've been using a bit ZFS on the Raspberry Pi using FUSE, until I can get a Solaris derived OS ( such as illumos, smartos, openindiana, opensolaris etc) on the Raspberry Pi. That way, at least I have ZFS. Still missing zones, smf and dtrace, but it is a start.

Now just a reminder, the Pi only has 256MB total ram, and a BCM arm processor. So first thing first, we need to give as much ram to the OS as possible, and reduce the video buffer size:



I'm using a 240MB split on that Raspberry Pi since it is running only in text mode at the console, and I remote to it using ssh -X.


If you use the composite out you might want to use the 224MB split and definitely 192 or 128 using HDMI, but then at that point, you are chocking ZFS. That's 128 for OS and ZFS and whatever apps you are running...

Fully loaded

Altough Raspbian comes with a good amount of stuff preloaded, it was not intended to be used with FUSE out of the box, and ZFS was probably never on the radar screen of anybody. So let's start with adding the FUSE stuff and the libraries and tools we will need to build ZFS. This is the shortlist:

fdion@raspberrypi ~/zfs $ sudo apt-get install fuse-utils libfuse-dev libfuse2
fdion@raspberrypi ~/zfs $ sudo apt-get install libaio-dev libattr1-dev attr
fdion@raspberrypi ~/zfs $ sudo apt-get install git scons

If you build it...

So we have the prerequisites. Let's get the code, compile it and install the tools:

fdion@raspberrypi ~ $ mkdir zfs
fdion@raspberrypi ~ $ cd zfs
fdion@raspberrypi ~/zfs $ git clone https://bitbucket.org/cli/zfs-fuse-arm.git
fdion@raspberrypi ~/zfs $ cd zfs-fuse-arm/
fdion@raspberrypi ~/zfs/zfs-fuse-arm $ cd src
fdion@raspberrypi ~/zfs/zfs-fuse-arm/src $ scons
[a lot of stuff will scroll by]
fdion@raspberrypi ~/zfs/zfs-fuse-arm/src $ sudo scons install
[again, more stuff will scroll by]

Wow, it compiled (scons). And installed (sudo scons install). It's a good thing we are using the zfs-fuse-arm version, because the mainline wont go very far on the compile.

A demonstration, if you please? 

Well of course! Let's start the zfs-fuse daemon and create two virtual disks. I'm creating two 100M disks here using dd/ (this is on a slow SD card, rated 10MB/s). You could also use an actual /dev (like a pair of USB keys):

fdion@raspberrypi ~/zfs/zfs-fuse-arm/src/zfs-fuse $ sudo sh run.sh &

fdion@raspberrypi ~/zfs/zfs-fuse-arm/src/zfs-fuse $ cd
fdion@raspberrypi ~ $ cd zfs
fdion@raspberrypi ~/zfs $ mkdir test
fdion@raspberrypi ~/zfs $ cd test
fdion@raspberrypi ~/zfs/test $ dd if=/dev/zero of=fakedisk1 bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 10.2747 s, 10.2 MB/s
fdion@raspberrypi ~/zfs/test $ dd if=/dev/zero of=fakedisk2 bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 10.7517 s, 9.8 MB/s

Up to now we haven't done anything with ZFS per say. And basically to mirror two drives in ZFS and create a new storage out of that, all we have to do:

fdion@raspberrypi ~/zfs/test $ sudo zpool create mymirror mirror /home/fdion/zfs/test/fakedisk1 /home/fdion/zfs/test/fakedisk2

Now let's create a filesystem on that new zpool device, and mount it to a local folder in my home directory, change permissions so I can write to it and finally copy some files from /etc to my new filesystem:

fdion@raspberrypi ~/zfs/test $ cd
fdion@raspberrypi ~ $ mkdir myfilesystem
fdion@raspberrypi ~ $ sudo zfs create mymirror/myfilesystem -o mountpoint=/home/fdion/myfilesystem
fdion@raspberrypi ~ $ sudo chown fdion:pi myfilesystem/
fdion@raspberrypi ~/myfilesystem $ cp /etc/*.conf .
cp: cannot open `/etc/fuse.conf' for reading: Permission denied
fdion@raspberrypi ~/myfilesystem $ ls
adduser.conf          gssapi_mech.conf  libaudit.conf   pnm2ppa.conf
asound.conf           hdparm.conf       logrotate.conf  resolv.conf
ca-certificates.conf  host.conf         mke2fs.conf     rsyslog.conf
colord.conf           idmapd.conf       mtools.conf     sensors3.conf
debconf.conf          insserv.conf      nsswitch.conf   sysctl.conf
deluser.conf          ld.so.conf        ntp.conf        ts.conf
gai.conf              libao.conf        pam.conf        ucf.conf
fdion@raspberrypi ~/myfilesystem $ sudo zfs list
NAME                    USED  AVAIL  REFER  MOUNTPOINT
mymirror                191K  63.3M    22K  /mymirror
mymirror/myfilesystem  89.5K  63.3M  89.5K  /home/fdion/myfilesystem
fdion@raspberrypi ~/myfilesystem $ sudo zpool list
NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
mymirror  95.5M   196K  95.3M     0%  1.00x  ONLINE  -
fdion@raspberrypi ~/myfilesystem $ 

How cool is that? I now have a mirrored backup of my .conf files. Well, not quite. We are using fake disks, so if the SD card dies I loose all.

So next time we'll demo with actual USB drives.