Showing posts with label python. Show all posts
Showing posts with label python. Show all posts

Saturday, August 1, 2020

Python ASGI CLI

Call ASGI Python application module from command line (without an application server), just like CURL. Interested?

Wednesday, May 27, 2015

Asynchronous Python in Web Applications

Asynchronous or non-blocking IO allows worker to process other requests before current request has finished. That is possible in case a processing of current web request is related to an operation that is IO bound.

Prerequisites

What is important to understand about applicability of async pattern for IO bound operations, they must:
  1. Utilize the same event loop, in this case worker's event loop can switch to next in the queue (otherwise it is blocked).
  2. Have long latency (otherwise you are wasting event loop queue for short requests).
If operation is not IO bound it is considered blocking.

Saturday, April 4, 2015

wheezy template: static website generator

Sometimes you need a quick way to generate a static web site and put it up on a server. Here is how you can generate a static website with wheezy.template. Let's suppose the following structure of the site:
 content/
 `- index.html
  - main.html
  shared/
  `- master.html
   snippet/
   `- script.html
    - widget.html

Wednesday, October 29, 2014

wheezy web: RESTful API Design

In this article we are going to explore a simple RESTful API created with wheezy.web framework. The demo implements a CRUD for tasks. Includes entity validation, content caching with dependencies and functional test cases. The source code is structured with well defined actors (you can read more about it here).

Design

The following convention is used with respect to operation, HTTP method (verb) and URL:
List:   GET    /api/v1/tasks
Add:    POST   /api/v1/tasks
Get:    GET    /api/v1/task/{task_id}
Update: PUT    /api/v1/task/{task_id}
Remove: DELETE /api/v1/task/{task_id}
The task entity consists of the following attributes: task_id, title and status. The source code is available here. Please download before proceeding any further.

Monday, August 4, 2014

wheezy web: deploy nginx + uwsgi + memcached

wheezy.web is a lightweight, high performance, high concurrency WSGI web framework with the key features to build modern, efficient web. Here we will deploy quick start with nginx, uwsgi and memcached to clean debian stable installation.

Thursday, November 14, 2013

How to build Python deb package from source

You need a newer python version that is not available with the Debian package management system. You can compile it from source and install but that usually takes a lot of time. It would be better to build a deb instead and install it on as many machines as you need. You can use checkinstall for this.
apt-get install checkinstall
The instruction how to compile python from source is here. Once you compiled python here is a simple command that does the rest for you:
checkinstall -y --pkgname=python2.7
Once it finishes you will get a deb file in the current directory. You can install it this way:
dpkg -i python2.7_2.7.6-1_i386.deb
Here is how to remove it:
apt-get remove --purge python2.7
Note, packages built with checkinstall are not updated by OS package management system, thus updates need to be provided manually.

Sunday, October 13, 2013

Python HTTP Client API

The modern web services expose public API to the world and JSON is de-facto standard in this communication. Here is a simple use case that integrates with buildbot public API.

Lets setup a virtual environment and install wheezy.core package:
virtualenv env
env/bin/easy_install wheezy.core
Launch python from virtual environment (env/bin/python) and try this:
>>> from wheezy.core.httpclient import HTTPClient
>>> c = HTTPClient('http://buildbot.buildbot.net/json/')
>>> c.get('project')
200
>>> project = c.json
>>>> str(project.title)
Buildbot
Here is another example that demonstarates etag handling (the second time we request events the server responds with HTTP status code 304, not modified):
>>> base = 'https://api.github.com/repos/python/cpython/'
>>> c = HTTPClient(base)
>>> c.get('events')
200
>>> c.headers['content-encoding']
['gzip']
>>> c.get('events')
304
The HTTPClient supports HTTP(S) GET/HEAD/POST verbs, follows redirects, handles cookies and etags between requests, gzip content decoding.

Wednesday, October 2, 2013

How to manage Git or Mercurial repositories

Managing version control repositories can be a challenge in multi-user environment especially when simplification of user collaboration is your goal. There are usually two primary concerns while considering enterprise deployment for version control repositories: access control and safety of your data. Both are not directly addressed by version control itself, thus a sort of security facade is necessary.

Tuesday, September 10, 2013

wheezy web: Actors

The application design of wheezy.web application consists of several actors, each playing its unifying role. Actors are grouped into packages, e.g. models, repository, service, web, content, tests, etc. These packages (or layers) shape a subsystem.

The diagram below visualizes relations between actors, boundaries between packages, subsystems and application, giving a better picture of whole:
The sequence diagram explains request processing and interaction between actors:
You can download a pdf copy here.

Wednesday, February 6, 2013

Thoughts on SQL vs ORM

The question of persistence implementation arise often. I found repository pattern very valuable due to separation of concerns, mediate between domain model and data source (mock, file, database, web service, etc). The database data source is somewhat specific since you can proceed with SQL functions or ORM. Here are some thoughts why you might prefer SQL functions over ORM in your next project:

Wednesday, January 16, 2013

wheezy web: Quick Start i18n Project

wheezy.web is a lightweight, high performance, high concurrency WSGI web framework with the key features to build modern, efficient web. Here we will use an i18n (multilingual internationalization) project quick start to build a new project.
  1. Download quickstart-i18n.zip and extract.
  2. Rename extracted directory `quickstart-i18n` to meet your project name, e.g. mysite and open terminal in that directory.
The i18n project quick start comes with Makefile with several handy targets. Setup development environment:
make env
Internationalization is built using gettext, thus you need extract gettext messages and compile `*.po` files that you can find in i18n directory.
make po
Let run test, look at coverage and finally benchmark tests:
$ make test nose-cover benchmark
....................................
..........
Name                 Stmts   Miss  Cover   Missing
--------------------------------------------------
public                   3      0   100%   
public.web               0      0   100%   
public.web.profile       5      0   100%   
public.web.urls         10      0   100%   
public.web.views        26      0   100%   
--------------------------------------------------
TOTAL                   44      0   100%   
-----------------------------------------------------
Ran 10 tests in 0.458s

OK
public: 5 x 1000
baseline throughput change target
  100.0%    5303rps  +0.0% test_root
   94.4%    5005rps  -5.6% test_home
   98.2%    5206rps  -1.8% test_error_400
   99.2%    5258rps  -0.8% test_error_403
   97.9%    5191rps  -2.1% test_error_404
static: 3 x 1000
baseline throughput change target
  100.0%    5172rps  +0.0% test_static_files
   47.4%    2453rps -52.6% test_static_file_not_found
   52.5%    2715rps -47.5% test_static_file_forbidden
-----------------------------------------------------
Ran 2 tests in 2.286s

OK
The tests are passed let run it in web browser:
$ make run
Visit http://localhost:8080/
Alternatively you can run it using uwsgi
env/bin/easy_install uwsgi
make uwsgi
Enjoy!

Tuesday, January 15, 2013

wheezy web: Quick Start Empty Project

wheezy.web is a lightweight, high performance, high concurrency WSGI web framework with the key features to build modern, efficient web. Here we will use an empty project quick start to build a new project.
  1. Download quickstart-empty.zip and extract.
  2. Rename extracted directory `quickstart-empty` to meet your project name, e.g. mysite and open terminal in that directory.
The empty project quick start comes with Makefile with several handy targets. Setup development environment:
make env
Let run test, look at coverage and finally benchmark tests:
$ make test nose-cover benchmark
....................................
..........
Name                 Stmts   Miss  Cover   Missing
--------------------------------------------------
public                   3      0   100%   
public.web               0      0   100%   
public.web.profile       5      0   100%   
public.web.urls         10      0   100%   
public.web.views        20      0   100%   
--------------------------------------------------
TOTAL                   38      0   100%   
--------------------------------------------------
Ran 10 tests in 0.423s

OK
public: 5 x 1000
baseline throughput change target
  100.0%    5041rps  +0.0% test_root
   81.3%    4097rps -18.7% test_home
   99.4%    5008rps  -0.6% test_error_400
  100.3%    5055rps  +0.3% test_error_403
   98.7%    4975rps  -1.3% test_error_404
static: 3 x 1000
baseline throughput change target
  100.0%    4846rps  +0.0% test_static_files
   49.8%    2415rps -50.2% test_static_file_not_found
   55.0%    2663rps -45.0% test_static_file_forbidden
-----------------------------------------------------
Ran 2 tests in 2.375s

OK
The tests are passed let run it in web browser:
$ make run
Visit http://localhost:8080/
Alternatively you can run it using uwsgi
env/bin/easy_install uwsgi
make uwsgi
Enjoy!

Monday, December 3, 2012

How to Send Mail in Python

There are several use cases how you can send an email message using Python:
  1. Plain Mail: an email message with plain text or html content.
  2. Mail with Attachment: an email message with attached document.
  3. Alternative Mail Views: you provide a convenient way to email recipients to view your message in plain text or html with optional rich content including images, etc.
Here is an example of alternative email with HTML and image:
Let see how this can be accomplished.

Wednesday, November 28, 2012

How to ship eggs with pyo files only in Python

There is sometimes a need to ship python egg distribution with pyo files only. There is confusion using bdist_egg command since it doesn't have any options to do that; instead, you can instruct install_lib command. Here is what you need in setup.cfg file:
[install_lib]
compile = 0
optimize = 2

[bdist_egg]
exclude-source-files = 1
Issue the following command to build egg.
python setup.py -q bdist_egg
Note, the egg file has only python byte code optimized module files (look at dist directory). You can read about compiled modules here.

Tuesday, November 20, 2012

Python Web Frameworks Excessive Complexity

Cyclomatic (or conditional) complexity is a metric used to indicate the complexity of a source code. In this post we will take a look at web frameworks source code and estimate excessive complexity, something that is beyond recommended level of 10 (threshold that points to the fact the source code is too complex and refactoring is suggested). Here is a list of web frameworks examined:
  1. bottle
  2. cherrypy
  3. circuits
  4. django
  5. falcon
  6. flask
  7. pyramid
  8. pysi
  9. tornado
  10. turbogears
  11. web.py
  12. web2py
  13. webapp2
  14. wheezy.web
The source code is hosted on bitbucket, let clone it into some directory and setup virtual environment (this will download source code per framework listed above).
hg clone https://bitbucket.org/akorn/helloworld
cd helloworld/04-pep8 && make env up
The make file has a target for mccabe metric, so issue make mccabe.

Thursday, November 15, 2012

Lazy Attribute in Python

A lazy attribute is an attribute that is calculated on demand and only once. Here we will see how you can use lazy attribute in your Python class. Setup environment before you proceed:
$ virtualenv env
$ env/bin/pip install wheezy.core
Let assume we need an attribute that is display name of some person Place the following code snippet into some file and run it:
from wheezy.core.descriptors import attribute

class Person(object):
    def __init__(self, first_name, last_name):
        self.first_name = first_name
        self.last_name = last_name
        self.calls_count = 0
        
    @attribute
    def display_name(self):
        self.calls_count += 1
        return '%s %s' % (self.first_name, self.last_name)

if __name__ == '__main__':
    p = Person('John', 'Smith')
    print(p.display_name)
    print(p.display_name)
    assert 1 == p.calls_count
Notice display_name function is decorated with @attribute. The first call promotes function to attribute with the same name. The source is here.

Thursday, November 8, 2012

Duck Typing Assert in Python

People who come from strongly typed languages that offer interfaces often are confused by lack of one in Python. Python, being dynamic typing programming language, follows duck typing principal. Here we will see how programmer can assert duck typing between two Python classes. Setup environment before proceed:
$ virtualenv env
$ env/bin/pip install wheezy.core
Let play a bit with duck test `looks like`. Place the following code snippet into file `test.py`:
from wheezy.core.introspection import looks

class IFoo(object):
    def foo(self):
        pass

class Foo(object):
    def bar(self):
        pass

assert looks(Foo).like(IFoo)

Friday, November 2, 2012

How to generate account number?

Requirements for `account number` generator:
  1. Issue pseudo random consistent number (must be unique for dozen millions of records)
  2. Easy check validity (without a need to make a database call)
We will use Feistel cipher to generate pseudo random number (positive only) from a sequential numbers (e.g. returned by nextval() for a posgresql sequence). This algorithm is taken as basis for the `make_feistel_number` function available in wheezy.core package. Setup environment before proceed:
$ virtualenv env
$ env/bin/easy_install wheezy.core
$ env/bin/python
Let play a bit how numbers are generated (notice, the function is reversal and consistent, all numbers are unique, no collision):
>>> from wheezy.core.feistel import make_feistel_number
>>> from wheezy.core.feistel import sample_f
>>> feistel_number = make_feistel_number(sample_f)
>>> feistel_number(1)
573852158
>>> feistel_number(2)
1788827948
>>> feistel_number(1788827948)
2
We will use Luhn algorithm to generate a single digit checksum. This algorithm is taken as basis for the module available in wheezy.core package.
>>> from wheezy.core.luhn import luhn_checksum
>>> luhn_checksum(123456789)
7
There are two more useful functions to sign a number and also check it validity:
>>> from wheezy.core.luhn import luhn_sign
>>> from wheezy.core.luhn import is_luhn_valid
>>> luhn_sign(123456789)
1234567897
>>> is_luhn_valid(1234567897)
True
>>> is_luhn_valid(34518893)
False
Now let just make account number looking nice (pad with zeros, prefix with something meaningful, etc):
>>> account_number = lambda n: 'Z%011d' % luhn_sign( \
...     feistel_number(n))
>>> account_number(1)
'Z05738521581'
>>> account_number(2)
'Z17888279480'
>>> account_number(3)
'Z07395350007'
Per discussion in python mail list, there was discovered alternative, human readable representation of the same number:
>>> from base64 import b32encode
>>> human_format = lambda n: 'Z%s-%s' % (b32encode( \
...     chr((n >> 24) & 255) + \
...     chr((n >> 16) & 255))[:4], \
...     b32encode(chr((n >> 8) & 255) + \
...     chr(n & 255))[:4])
>>> human_format(5738521581)
'ZKYFA-4PWQ'
>>> human_format(17888279480)
'ZFI4Q-PO4A'
>>> human_format(7395350007)
'ZXDGA-CX3Q'
Side by side:
Z05738521581 = ZKYFA-4PWQ
Z17888279480 = ZFI4Q-PO4A
Z07395350007 = ZXDGA-CX3Q
Yes, it optimized for speed, you must provide your own `sample_f` implementation for security reasons. Source code available here.

Friday, October 26, 2012

wheezy web: introduction

The key of success for any medium to high complexity system is in separation of domain concerns. Given that choice in architectural design for web framework, the development activities are split by distinct, non-dependent parts. The wheezy.web is a lightweight WSGI web framework and serves a glue purpose between various other packages developed under wheezy.* umbrella in loosely coupled way, it combines things essential for web application developer (presentation slides are here):