Underscode

Now ``__`` escapes the space character instead of the underscore, the main reason of the change being that spaces are expected to happen much more frequently than underscores in strings which *need encoding*, because underscores in the original string are themselves a signal that it has already been converted to some indentifier-compatible form and it doesn't need further encoding.

Besides that, as underscores are frequently used to represent spaces (e.g. ``get_foo`` for "get foo"), it makes more sense to keep them in that role (i.e. ``get__foo_`` instead of ``get_x20foo_``).

This change justifies the bumping of version from 0.1.1.dev to 0.2.0.dev. check-in: f267aa66d2 user: ivan@selidor.net tags: trunk

Thus, the version has been bumped to 0.2 (0.2.0.dev to be exact). check-in: e7a9982d98 user: ivan@selidor.net tags: trunk

Packages are to be stored in the PyPi, so the URL is not needed and confusing. check-in: 673468ba65 user: ivan@selidor.net tags: trunk

Since the LGPL 3 is written in reference to the terms of the GPL 3, I'm including both of them with the source (but the license file and notices clearly state that the license which holds is the LGPL). check-in: 89ff967636 user: ivan@selidor.net tags: trunk

Automatic support of external codecs in Python just by dropping a module or package in the $PYTHONPATH worked by pure chance. With Python 2.5, it has been fixed and the trick does no longer work. See SF item #1569084. check-in: e9ce3779ba user: ivan@selidor.net tags: trunk

This is more flexible than strictly checking the type of the input argument, and allows things like:

>>> unicode('this_x20is_x20_u201ca_x20test_u201d_', 'underscode') u'this is \u201ca test\u201d'

Which created an UnderscodeDecoder with a buffer object. Function and codec docstrings have been updated and small doctests have been added.

I somehow felt that the type checking in r33 was broken in a way. Now it has spoiled my lucky strike to make the first release on revision 42! :P check-in: 75622b6a15 user: ivan@selidor.net tags: trunk

A link to the Underscode home page has been added to the package docstring, and a note about the DOT file containing the FSM description of underscoded strings has been added to the core module docstring. check-in: d79a17ac21 user: ivan@selidor.net tags: trunk

This is the "Nirvana commit". ;) New decorators have been added that allow to derive calls to methods with an underscoded argument to other methods with the same argument decoded. Also, decorators to allow redirecting access to attributes with an underscoded name to other methods while keeping the instance ``__dict__`` and ``__members__`` up to date have been added.

See the module docstring for complete examples. Have fun! check-in: d49f9f46ea user: ivan@selidor.net tags: trunk

Also, tiny aesthetical fix to ``codec.py``. check-in: 596ac09a72 user: ivan@selidor.net tags: trunk

The old format, ``_METHOD_NAME_used``, could create class-private variables (like ``__private_method_used``) with private methods. The new format, ``_used_METHOD_NAME``, avoids that. check-in: 26a81fb211 user: ivan@selidor.net tags: trunk

The new codec support is under the new ``codec.py`` module, which implements basic ``Codec``, ``StreamWriter`` and ``StreamReader`` classes. The last ones are limited to one read and one write per instance. New doctests have been added.

The new codec is automatically detected by Python just by exporting the ``getregentry()`` function, which returns the tuple of *(encoder, decoder, stream_reader, stream_writer)* required by the Python codec API. check-in: b47f3f56b0 user: ivan@selidor.net tags: trunk

Encoding and decoding code has been moved into the ``core`` module, and exceptions have been placed into the ``exceptions`` module. The main test suite has been adapted to also run doctests from the new modules. check-in: fa19f74a44 user: ivan@selidor.net tags: trunk

Encoding support has been implemented via the ``UnderscodeEncoder`` class and the ``underscode_encode()`` function in `underscode/__init__.py`. A new test suite ``EncodingTestSuite`` has been added to `underscode/tests.py`. check-in: c5ed1dddc4 user: ivan@selidor.net tags: trunk

`underscode/__init__.py` As forecasted in the message of revno 10, the ``UnderscodeDecoder`` readily supported extending the encoded string. The new ``append()`` method has been properly documented with ``doctest`` code in the class docstring. check-in: 7923623a08 user: ivan@selidor.net tags: trunk

`underscode/tests.py` The new method ``DecodingTestCase.test_valid_character()`` tests for all possible valid encodings of a Unicode character, while the method ``test_invalid_character()`` tests for all invalid and incomplete versions of a valid encoded Unicode character.

The methods ``test_decode_valid()`` and ``test_decode_invalid()`` have been renamed to ``test_valid_string()`` and ``test_invalid_string()`` to match the previous ones.

Since the new methods take care of encoded characters, all the plethora of repeated cases in the tests on whole strings are no longer necessary, so the only encoded character used in them now is ``_``.

The tests in the main body of `underscode/__init__.py` are now implemented using ``unittest`` in the new file `underscode/tests.py`. The new class ``DecodingTestCase`` checks all cases of possible valid and invalid encoding strings according to the number of transitions made by the finite state machine (FSM), taking a whole encoded character as one transition. This reduces the FSM to only five transitions, simplifying the tests.

Of course, this implies that new test units must be written in order to check for incomplete encoded characters. Also, the interface of the class ``UnderscodeDecoder`` is not checked yet. check-in: f81ab4d2aa user: ivan@selidor.net tags: trunk

The Graphviz file `docs/underscode.dot` defines the finite state machine recongnising and decoding underscoded strings. check-in: e6d02ca097 user: ivan@selidor.net tags: trunk

Sections are separated from the previous one using a blank line, a form feed character, a reStructuredText-like title in a comment, and another blank line, like this::

... code of the previous section ...

^L # Title of the next section # =========================

... code of the next section ...

Thus, when printing the file, each section starts with a title and ends with a blank line. check-in: c1ede26207 user: ivan@selidor.net tags: trunk

`underscode/__init__.py` The Unicode characters in the module docstring have been replaced by ASCII characters or Unicode escape sequences, so the file is no longer tagged as being UTF-8 encoded. check-in: e1b423fb1d user: ivan@selidor.net tags: trunk

`underscode/__init__.py` Timing a simple loop of string decoding yields that using a list comprehension and using a generator expression in ``underscode_decode()`` is equally fast, so the expression is chosen since it may lead to a better memory usage. This is not really important since the strings will surely be short. By the way, string length does not alter the results significantly. check-in: a27ffbd209 user: ivan@selidor.net tags: trunk

Join-style separates the delimiters of a multi-line structure just as if the “join line” (J) order of ``vi`` had been used, i.e. squashing leading whitespace of the joined line into a single space. Some examples::

some_list = [ 'foo', 'bar', 'baz', ]

nested_list = [ [ 'foo', 'bar', ], [ 'foo', 'bar', ], ]

nested_list = [ [ 'foo', 'bar', ], 'baz' ]

result = function( first_arg, (second_arg_0, second_arg_1), third_arg, fourth_arg )

From the second example one can see that structures fitting in a single line are *not* affected by join-style.

IMHO, this style:

1. Looks quite pythonic (no structure terminators on the same level as the beginning of the structure). 2. Makes multi-line structures easy to spot (their beginning character is followed by a space character). 3. It does not make a mess of subsequent terminators (they are separated by spaces).

`underscode/__init__.py` Use join-style for multi-line structures. check-in: 9382cdfdf5 user: ivan@selidor.net tags: trunk

`underscode/__init__.py` Multi-line structures (dictionaries, lists, function calls…) with the termination character have been altered so that the terminator is placed at the end of the last line, LISP-style. Indenting it or aligning it to the beginning of the block makes code look confusing or unpythonic.

For instance::

something = [ [0, 1, 2, 3], [4, 5, 6], ] another_thing

or::

something = [ [0, 1, 2, 3], [4, 5, 6], ] another_thing

is now written as::

something = [ [0, 1, 2, 3], [4, 5, 6]] another_thing

`underscode/__init__.py` The comment containing the copyright and comment of the module has been splitted into the variables ``__copyright__`` (a list of copyright holders, with e-mail addresses and a list of years) and ``__license__`` (a small piece of text). check-in: 2ccbbdf2aa user: ivan@selidor.net tags: trunk

`underscode/__init__.py` Replaced placeholder link in module docstring with the real URL documenting Python identifiers (http://docs.python.org/ref/identifiers.html). check-in: ab42868d41 user: ivan@selidor.net tags: trunk

`underscode/__init__.py` The new docstring for ``UnderscodeDecoder`` contains some usage samples showing how to iterate and what happens on errors.

The new tests helped to find out that non-encoded characters were being returned as normal characters instead of Unicode ones. Fixed.

`underscode/__init__.py` The docstring of ``UnderscodeDecoder.next()`` points out that it is not changing the state of the decoder when an error occurs. A paragraph on ``StopIteration`` is also added. check-in: dd29feccfc user: ivan@selidor.net tags: trunk

`underscode/__init__.py` The representation of an ``UnderscodeDecoder`` looks like ``<UnderscodeDecoder of '__foobar_' at offset 4>``. This can not be directly interpreted into Python, but it still allows one to try to get an object with the same state by using the same encoded string and repeatedly calling ``next()`` until one gets the same ``offset`` (or a greater value or an exception). check-in: 9b20e8fefc user: ivan@selidor.net tags: trunk

`underscode/__init__.py` The docstrings of ``InvalidChar`` and ``EndOfInput`` include some code for doctesting the value of instance variables. check-in: 3723186678 user: ivan@selidor.net tags: trunk