reorder vars

ddd2f011 removed this.

.gitignore

@@ -1,5 +1,8 @@
 *.pyc
 *.pyo
-build/*
+.DS_Store
-dist/*
+build
+dist
+tmp
 MANIFEST
+sendapatch.se

MANIFEST.in

@@ -1,4 +1,4 @@
-include README.rst LICENSE MANIFEST MANIFEST.in
+include README.rst LICENSE MANIFEST MANIFEST.in pooling.rst
-include _pylibmcmodule.c _pylibmcmodule.h
+include _pylibmcmodule.c _pylibmcmodule.h pylibmc-version.h
 include pylibmc.py
 include setup.py tests.py

README.rst

@@ -1,99 +1,126 @@
-`pylibmc` is a Python wrapper around the accompanying C Python extension
+`pylibmc` is a quick and small Python client for memcached__ written in C.
-`_pylibmc`, which is a wrapper around `libmemcached` from TangentOrg.
 
-You have to install `libmemcached` separately, and have your compiler and
+__ http://memcached.org/
-linker find the include files and libraries.
 
-With `libmemcached` installed and this package set up, the following basic
+It builds on the famous `libmemcached`__ C client from TangentOrg__, notable for
-usage example should work::
+its speed and flexibility.
 
-    >>> import pylibmc
+__ http://tangent.org/552/libmemcached.html
-    >>> mc = pylibmc.Client(["127.0.0.1:11211"])
+__ http://tangent.org/
-    >>> mc.set("foo", "Hello world!")
-    True
-    >>> mc.get("foo")
-    'Hello world!'
 
-The API is pretty much `python-memcached`. Some parts of `libmemcached` aren't
+`libmemcached` must be installed separately, and be available to the compiler
-exposed yet. I think.
+and linker.
 
-Behaviors
+For installation instructions, usage notes and reference documentation, see
-=========
+pylibmc__'s home at http://sendapatch.se/projects/pylibmc/.
-
-`libmemcached` has ways of telling it how to behave. You'll have to refer to
-its documentation on what the different behaviors do.
-
-To change behaviors, quite simply::
-
-    >>> mc.behaviors["hash"] = "fnv1a_32"
-
-For a list of the defined behavior key names, see what the keys of a client is.
-For example::
-
-    >>> mc.behaviors.keys()  # doctest: +NORMALIZE_WHITESPACE
-    ['hash', 'connect timeout', 'cache lookups', 'buffer requests',
-     'verify key', 'support cas', 'poll timeout', 'no block', 'tcp nodelay',
-     'distribution', 'sort hosts']
-
-The ``hash`` and ``distribution`` keys are mapped by the Python module to constant
-integer values used by `libmemcached`. See ``pylibmc.hashers`` and
-``pylibmc.distributions``.
 
+__ http://sendapatch.se/projects/pylibmc/
 
 Comparison to other libraries
 =============================
 
 Why use `pylibmc`? Because it's fast.
 
-`See this (a bit old) speed comparison <http://lericson.blogg.se/code/2008/november/pylibmc-051.html>`_.
+`See this (a bit old) speed comparison`__, or `amix.dk's comparison`__.
 
+__ http://lericson.blogg.se/code/2008/november/pylibmc-051.html
+__ http://amix.dk/blog/viewEntry/19471
+
+Installation
+============
+
+Building needs libmemcached and optionally zlib, the path to which can be
+specified using command-line options to ``setup.py``
+
+``--with-libmemcached=DIR``
+    Build against libmemcached in DIR
+``--with-zlib=DIR``
+    Build against zlib in DIR
+``--without-zlib``
+    Disable zlib (disables compression)
+
+So for example, if one were to use MacPorts to install libmemcached, your
+libmemcached would end up in ``/opt/local``, hence
+``--with-libmemcached=/opt/local``.
 
 IRC
 ===
 
 ``#sendapatch`` on ``chat.freenode.net``.
 
-Compiling on Snow Leopard
-=========================
-
-Since, for some reason, compiling Python extensions under Mac OS X 10.6 won't
-use the proper ``-arch`` flag to ``gcc``, here's how you'd do it::
-
-    $ CFLAGS="-arch x86_64" LDFLAGS="-arch x86_64" python setup.py build
-
 Change Log
 ==========
 
+New in version 1.1
+------------------
+
+- Removed deprecated space-based behavior names.
+- Acquire and release the GIL properly, thanks ketralnis__
+- Add support for ``libmemcached 0.40``
+- Included a more useful command-line interface
+- Fixed handling of NUL-byte keys in ``get_multi`` in binary protocol
+- Fixed some valgrind-reported memory warnings
+- Fixed bogus usage of time argument for delete.
+- 1.1.1: Fixed tests under Python 2.5
+
+__ http://www.ketralnis.com/
+
+New in version 1.0
+------------------
+
+- Lots of documentation fixes and other nice things like that.
+- Nailed what appears to be the last outstanding memory leak.
+- Explicitly require libmemcached 0.32 or newer.
+
+New in version 0.9
+------------------
+
+- Added a ``get_stats`` method, which behaves exactly like
+  `python-memcached`'s equivalent.
+- Gives the empty string for empty memcached values like `python-memcached`
+  does.
+- Added exceptions for most `libmemcached` return codes.
+- Fixed an issue with ``Client.behaviors.update``.
+
+New in version 0.8
+------------------
+
+- Pooling helpers are now available. See ``pooling.rst`` in the distribution.
+- The binary protocol is now properly exposed, simply pass ``binary=True`` to
+  the constructor and there you go.
+- Call signatures now match `libmemcached` 0.32, but should work with older
+  versions. Remember to run the tests!
+
 New in version 0.7
 ------------------
 
- - Restructured some of the code, which should yield better performance (if not
+- Restructured some of the code, which should yield better performance (if not
-   for that, it reads better.)
+  for that, it reads better.)
- - Fixed some memory leaks.
+- Fixed some memory leaks.
- - Integrated changes from `amix.dk`, which should make pylibmc work under
+- Integrated changes from `amix.dk`, which should make pylibmc work under
-   Snow Leopard.
+  Snow Leopard.
- - Add support for the boolean datatype.
+- Add support for the boolean datatype.
- - Improved test-runner -- now tests ``build/lib.*/_pylibmc.so`` if available,
+- Improved test-runner -- now tests ``build/lib.*/_pylibmc.so`` if available,
-   and reports some version information.
+  and reports some version information.
- - Support for x86_64 should now work completely.
+- Support for x86_64 should now work completely.
- - Builds with Python 2.4, tests run fine, but not officially supported.
+- Builds with Python 2.4, tests run fine, but not officially supported.
- - Fixed critical bugs in behavior manipulation.
+- Fixed critical bugs in behavior manipulation.
 
 New in version 0.6
 ------------------
 
- - Added compatibility with `libmemcached` 0.26, WRT error return codes.
+- Added compatibility with `libmemcached` 0.26, WRT error return codes.
- - Added `flush_all` and `disconnect_all` methods.
+- Added `flush_all` and `disconnect_all` methods.
- - Now using the latest pickling protocol.
+- Now using the latest pickling protocol.
 
 New in version 0.5
 ------------------
 
- - Fixed lots of memory leaks, and added support for `libmemcached` 0.23.
+- Fixed lots of memory leaks, and added support for `libmemcached` 0.23.
- - Also made the code tighter in terms of compiler pedantics.
+- Also made the code tighter in terms of compiler pedantics.
 
 New in version 0.4
 ------------------
 
- - Renamed the C module to `_pylibmc`, and added lots of `libmemcached` constants
+- Renamed the C module to `_pylibmc`, and added lots of `libmemcached` constants
-   to it, as well as implemented behaviors.
+  to it, as well as implemented behaviors.

TODO

@@ -1,3 +1,3 @@
-- double-check all INCREF and DECREFs
+ - make a get_stats function
-- make a get_stats function
+ - add servers after object creation (also related: set_servers)
-- add servers after object creation (also related: set_servers)
+ - UDP and the binary protocol

_pylibmcmodule.h

@@ -41,6 +41,8 @@
 #include <Python.h>
 #include <libmemcached/memcached.h>
 
+#include "pylibmc-version.h"
+
 /* Py_ssize_t appeared in Python 2.5. */
 #ifndef PY_SSIZE_T_MAX
 typedef ssize_t Py_ssize_t;
@@ -51,21 +53,117 @@ typedef ssize_t Py_ssize_t;
 #define PYLIBMC_SERVER_UDP   (1 << 1)
 #define PYLIBMC_SERVER_UNIX  (1 << 2)
 
-/* Key flags from python-memcache. */
+/* {{{ Key flags from python-memcached
+ * Some flags (like the compression one, ZLIB) are combined with others.
+ */
 #define PYLIBMC_FLAG_NONE    0
 #define PYLIBMC_FLAG_PICKLE  (1 << 0)
 #define PYLIBMC_FLAG_INTEGER (1 << 1)
 #define PYLIBMC_FLAG_LONG    (1 << 2)
-#define PYLIBMC_FLAG_BOOL    (1 << 3)
+/* Note: this is an addition! python-memcached doesn't handle bools. */
-
+#define PYLIBMC_FLAG_BOOL    (1 << 4)
-#define PYLIBMC_INC  (1 << 0)
+#define PYLIBMC_FLAG_TYPES   (PYLIBMC_FLAG_PICKLE | PYLIBMC_FLAG_INTEGER | \
-#define PYLIBMC_DEC  (1 << 1)
+                              PYLIBMC_FLAG_LONG | PYLIBMC_FLAG_BOOL)
+/* Modifier flags */
+#define PYLIBMC_FLAG_ZLIB    (1 << 3)
+/* }}} */
 
 typedef memcached_return (*_PylibMC_SetCommand)(memcached_st *, const char *,
         size_t, const char *, size_t, time_t, uint32_t);
+typedef memcached_return (*_PylibMC_IncrCommand)(memcached_st *,
+        const char *, size_t, unsigned int, uint64_t*);
+
+typedef struct {
+  char key[MEMCACHED_MAX_KEY];
+  size_t key_len;
+  char *value;
+  size_t value_len;
+  uint32_t flags;
+} pylibmc_mget_result;
+
+typedef struct {
+  char *key;
+  Py_ssize_t key_len;
+  char* value;
+  Py_ssize_t value_len;
+  time_t time;
+  uint32_t flags;
+
+  /* the objects that must be freed after the mset is executed */
+  PyObject* key_obj;
+  PyObject* prefixed_key_obj;
+  PyObject* value_obj;
+
+  /* the success of executing the mset afterwards */
+  int success;
+
+} pylibmc_mset;
+
+typedef struct {
+  char* key;
+  Py_ssize_t key_len;
+  _PylibMC_IncrCommand incr_func;
+  unsigned int delta;
+  uint64_t result;
+} pylibmc_incr;
+
+typedef struct {
+    PyObject *self;
+    PyObject *retval;
+    memcached_server_st *servers;  /* DEPRECATED */
+    memcached_stat_st *stats;
+    int index;
+} _PylibMC_StatsContext;
 
 /* {{{ Exceptions */
 static PyObject *PylibMCExc_MemcachedError;
+
+/* Mapping of memcached_return value -> Python exception object. */
+typedef struct {
+    memcached_return rc;
+    char *name;
+    PyObject *exc;
+} PylibMC_McErr;
+
+static PylibMC_McErr PylibMCExc_mc_errs[] = {
+    { MEMCACHED_FAILURE, "Failure", NULL },
+    { MEMCACHED_HOST_LOOKUP_FAILURE, "HostLookupError", NULL },
+    { MEMCACHED_CONNECTION_FAILURE, "ConnectionError", NULL },
+    { MEMCACHED_CONNECTION_BIND_FAILURE, "ConnectionBindError", NULL },
+    { MEMCACHED_WRITE_FAILURE, "WriteError", NULL },
+    { MEMCACHED_READ_FAILURE, "ReadError", NULL },
+    { MEMCACHED_UNKNOWN_READ_FAILURE, "UnknownReadFailure", NULL },
+    { MEMCACHED_PROTOCOL_ERROR, "ProtocolError", NULL },
+    { MEMCACHED_CLIENT_ERROR, "ClientError", NULL },
+    { MEMCACHED_SERVER_ERROR, "ServerError", NULL },
+    { MEMCACHED_CONNECTION_SOCKET_CREATE_FAILURE, "SocketCreateError", NULL },
+    { MEMCACHED_DATA_EXISTS, "DataExists", NULL },
+    { MEMCACHED_DATA_DOES_NOT_EXIST, "DataDoesNotExist", NULL },
+    //{ MEMCACHED_NOTSTORED, "NotStored", NULL },
+    //{ MEMCACHED_STORED, "Stored", NULL },
+    { MEMCACHED_NOTFOUND, "NotFound", NULL },
+    { MEMCACHED_MEMORY_ALLOCATION_FAILURE, "AllocationError", NULL },
+    //{ MEMCACHED_PARTIAL_READ, "PartialRead", NULL },
+    { MEMCACHED_SOME_ERRORS, "SomeErrors", NULL },
+    { MEMCACHED_NO_SERVERS, "NoServers", NULL },
+    //{ MEMCACHED_END, "", NULL },
+    //{ MEMCACHED_DELETED, "", NULL },
+    //{ MEMCACHED_VALUE, "", NULL },
+    //{ MEMCACHED_STAT, "", NULL },
+    //{ MEMCACHED_ITEM, "", NULL },
+    //{ MEMCACHED_ERRNO, "", NULL },
+    { MEMCACHED_FAIL_UNIX_SOCKET, "UnixSocketError", NULL },
+    { MEMCACHED_NOT_SUPPORTED, "NotSupportedError", NULL },
+    { MEMCACHED_FETCH_NOTFINISHED, "FetchNotFinished", NULL },
+    //{ MEMCACHED_TIMEOUT, "TimeoutError", NULL },
+    //{ MEMCACHED_BUFFERED, "Buffer, NULL },
+    { MEMCACHED_BAD_KEY_PROVIDED, "BadKeyProvided", NULL },
+    { MEMCACHED_INVALID_HOST_PROTOCOL, "InvalidHostProtocolError", NULL },
+    //{ MEMCACHED_SERVER_MARKED_DEAD,
+    { MEMCACHED_UNKNOWN_STAT_KEY, "UnknownStatKey", NULL },
+    //{ MEMCACHED_E2BIG, "TooBigError", NULL },
+    { 0, NULL, NULL }
+};
 /* }}} */
 
 /* {{{ Behavior statics */
@@ -75,27 +173,33 @@ typedef struct {
 } PylibMC_Behavior;
 
 static PylibMC_Behavior PylibMC_behaviors[] = {
-    { MEMCACHED_BEHAVIOR_NO_BLOCK, "no block" },
+    { MEMCACHED_BEHAVIOR_NO_BLOCK, "no_block" },
     { MEMCACHED_BEHAVIOR_TCP_NODELAY, "tcp_nodelay" },
     { MEMCACHED_BEHAVIOR_HASH, "hash" },
-    { MEMCACHED_BEHAVIOR_KETAMA, "ketama" },
-    { MEMCACHED_BEHAVIOR_DISTRIBUTION, "distribution" },
-    { MEMCACHED_BEHAVIOR_SOCKET_SEND_SIZE, "socket send size" },
-    { MEMCACHED_BEHAVIOR_SOCKET_RECV_SIZE, "socket recv size" },
-    { MEMCACHED_BEHAVIOR_CACHE_LOOKUPS, "cache_lookups" },
-    { MEMCACHED_BEHAVIOR_SUPPORT_CAS, "support_cas" },
-    { MEMCACHED_BEHAVIOR_POLL_TIMEOUT, "poll_timeout" },
-    { MEMCACHED_BEHAVIOR_BUFFER_REQUESTS, "buffer_requests" },
-    { MEMCACHED_BEHAVIOR_SORT_HOSTS, "sort_hosts" },
-    { MEMCACHED_BEHAVIOR_VERIFY_KEY, "verify_key" },
-    { MEMCACHED_BEHAVIOR_CONNECT_TIMEOUT, "connect_timeout" },
-    { MEMCACHED_BEHAVIOR_RETRY_TIMEOUT, "retry_timeout" },
-    { MEMCACHED_BEHAVIOR_KETAMA_WEIGHTED, "ketama_weighted" },
     { MEMCACHED_BEHAVIOR_KETAMA_HASH, "ketama_hash" },
-    { MEMCACHED_BEHAVIOR_BINARY_PROTOCOL, "binary_protocol" },
+    { MEMCACHED_BEHAVIOR_KETAMA, "ketama" },
-    { MEMCACHED_BEHAVIOR_SND_TIMEOUT, "snd_timeout" },
+    { MEMCACHED_BEHAVIOR_KETAMA_WEIGHTED, "ketama_weighted" },
-    { MEMCACHED_BEHAVIOR_RCV_TIMEOUT, "rcv_timeout" },
+    { MEMCACHED_BEHAVIOR_DISTRIBUTION, "distribution" },
-    { MEMCACHED_BEHAVIOR_SERVER_FAILURE_LIMIT, "server_failure_limit" },
+    { MEMCACHED_BEHAVIOR_CACHE_LOOKUPS, "cache_lookups" },
+    { MEMCACHED_BEHAVIOR_SUPPORT_CAS, "cas" },
+    { MEMCACHED_BEHAVIOR_BUFFER_REQUESTS, "buffer_requests" },
+    { MEMCACHED_BEHAVIOR_VERIFY_KEY, "verify_keys" },
+    { MEMCACHED_BEHAVIOR_CONNECT_TIMEOUT, "connect_timeout" },
+    { MEMCACHED_BEHAVIOR_SND_TIMEOUT, "send_timeout" },
+    { MEMCACHED_BEHAVIOR_RCV_TIMEOUT, "receive_timeout" },
+    { MEMCACHED_BEHAVIOR_SERVER_FAILURE_LIMIT, "failure_limit" },
+    { MEMCACHED_BEHAVIOR_IO_MSG_WATERMARK, "_io_msg_watermark" },
+    { MEMCACHED_BEHAVIOR_IO_BYTES_WATERMARK, "_io_bytes_watermark" },
+    { MEMCACHED_BEHAVIOR_IO_KEY_PREFETCH, "_io_key_prefetch" },
+    { MEMCACHED_BEHAVIOR_HASH_WITH_PREFIX_KEY, "_hash_with_prefix_key" },
+    { MEMCACHED_BEHAVIOR_NOREPLY, "_noreply" },
+    { MEMCACHED_BEHAVIOR_AUTO_EJECT_HOSTS, "_auto_eject_hosts" },
+    { MEMCACHED_BEHAVIOR_NUMBER_OF_REPLICAS, "_number_of_replicas" },
+    { MEMCACHED_BEHAVIOR_SORT_HOSTS, "_sort_hosts" },
+    { MEMCACHED_BEHAVIOR_RETRY_TIMEOUT, "_retry_timeout" },
+    { MEMCACHED_BEHAVIOR_POLL_TIMEOUT, "_poll_timeout" },
+    { MEMCACHED_BEHAVIOR_SOCKET_SEND_SIZE, "_socket_send_size" },
+    { MEMCACHED_BEHAVIOR_SOCKET_RECV_SIZE, "_socket_recv_size" },
     { 0, NULL }
 };
 
@@ -107,8 +211,10 @@ static PylibMC_Behavior PylibMC_hashers[] = {
     { MEMCACHED_HASH_FNV1A_64, "fnv1a_64" },
     { MEMCACHED_HASH_FNV1_32, "fnv1_32" },
     { MEMCACHED_HASH_FNV1A_32, "fnv1a_32" },
-    { MEMCACHED_HASH_HSIEH, "hsieh" },
     { MEMCACHED_HASH_MURMUR, "murmur" },
+#ifdef MEMCACHED_HASH_HSIEH
+    { MEMCACHED_HASH_HSIEH, "hsieh" },
+#endif
     { 0, NULL }
 };
 
@@ -132,33 +238,59 @@ static PylibMC_Client *PylibMC_ClientType_new(PyTypeObject *, PyObject *,
 static void PylibMC_ClientType_dealloc(PylibMC_Client *);
 static int PylibMC_Client_init(PylibMC_Client *, PyObject *, PyObject *);
 static PyObject *PylibMC_Client_get(PylibMC_Client *, PyObject *arg);
+static PyObject *PylibMC_Client_gets(PylibMC_Client *, PyObject *arg);
 static PyObject *PylibMC_Client_set(PylibMC_Client *, PyObject *, PyObject *);
 static PyObject *PylibMC_Client_replace(PylibMC_Client *, PyObject *, PyObject *);
 static PyObject *PylibMC_Client_add(PylibMC_Client *, PyObject *, PyObject *);
 static PyObject *PylibMC_Client_prepend(PylibMC_Client *, PyObject *, PyObject *);
 static PyObject *PylibMC_Client_append(PylibMC_Client *, PyObject *, PyObject *);
+static PyObject *PylibMC_Client_cas(PylibMC_Client *, PyObject *, PyObject *);
 static PyObject *PylibMC_Client_delete(PylibMC_Client *, PyObject *);
 static PyObject *PylibMC_Client_incr(PylibMC_Client *, PyObject *);
 static PyObject *PylibMC_Client_decr(PylibMC_Client *, PyObject *);
+static PyObject *PylibMC_Client_incr_multi(PylibMC_Client*, PyObject*, PyObject*);
 static PyObject *PylibMC_Client_get_multi(PylibMC_Client *, PyObject *, PyObject *);
 static PyObject *PylibMC_Client_set_multi(PylibMC_Client *, PyObject *, PyObject *);
+static PyObject *PylibMC_Client_add_multi(PylibMC_Client *, PyObject *, PyObject *);
 static PyObject *PylibMC_Client_delete_multi(PylibMC_Client *, PyObject *, PyObject *);
 static PyObject *PylibMC_Client_get_behaviors(PylibMC_Client *);
 static PyObject *PylibMC_Client_set_behaviors(PylibMC_Client *, PyObject *);
+static PyObject *PylibMC_Client_get_stats(PylibMC_Client *, PyObject *);
 static PyObject *PylibMC_Client_flush_all(PylibMC_Client *, PyObject *, PyObject *);
 static PyObject *PylibMC_Client_disconnect_all(PylibMC_Client *);
+static PyObject *PylibMC_Client_clone(PylibMC_Client *);
 static PyObject *PylibMC_ErrFromMemcached(PylibMC_Client *, const char *,
         memcached_return);
 static PyObject *_PylibMC_Unpickle(const char *, size_t);
 static PyObject *_PylibMC_Pickle(PyObject *);
 static int _PylibMC_CheckKey(PyObject *);
 static int _PylibMC_CheckKeyStringAndSize(char *, Py_ssize_t);
+static int _PylibMC_SerializeValue(PyObject* key_obj,
+                                   PyObject* key_prefix,
+                                   PyObject* value_obj,
+                                   time_t time,
+                                   pylibmc_mset* serialized);
+static void _PylibMC_FreeMset(pylibmc_mset*);
+static PyObject *_PylibMC_RunSetCommandSingle(PylibMC_Client *self,
+        _PylibMC_SetCommand f, char *fname, PyObject *args, PyObject *kwds);
+static PyObject *_PylibMC_RunSetCommandMulti(PylibMC_Client* self,
+        _PylibMC_SetCommand f, char *fname, PyObject *args, PyObject *kwds);
+static bool _PylibMC_RunSetCommand(PylibMC_Client* self,
+                                   _PylibMC_SetCommand f, char *fname,
+                                   pylibmc_mset* msets, size_t nkeys,
+                                   size_t min_compress, size_t custom_flag);
+static int _PylibMC_Deflate(char* value, size_t value_len,
+                            char** result, size_t *result_len);
+static bool _PylibMC_IncrDecr(PylibMC_Client*, pylibmc_incr*, size_t);
+
 /* }}} */
 
 /* {{{ Type's method table */
 static PyMethodDef PylibMC_ClientType_methods[] = {
     {"get", (PyCFunction)PylibMC_Client_get, METH_O,
         "Retrieve a key from a memcached."},
+    {"gets", (PyCFunction)PylibMC_Client_gets, METH_O,
+        "Retrieve a key and cas_id from a memcached."},
     {"set", (PyCFunction)PylibMC_Client_set, METH_VARARGS|METH_KEYWORDS,
         "Set a key unconditionally."},
     {"replace", (PyCFunction)PylibMC_Client_replace, METH_VARARGS|METH_KEYWORDS,
@@ -169,26 +301,37 @@ static PyMethodDef PylibMC_ClientType_methods[] = {
         "Prepend data to  a key."},
     {"append", (PyCFunction)PylibMC_Client_append, METH_VARARGS|METH_KEYWORDS,
         "Append data to a key."},
+    {"cas", (PyCFunction)PylibMC_Client_cas, METH_VARARGS|METH_KEYWORDS,
+        "Attempt to compare-and-store a key by CAS ID."},
     {"delete", (PyCFunction)PylibMC_Client_delete, METH_VARARGS,
         "Delete a key."},
     {"incr", (PyCFunction)PylibMC_Client_incr, METH_VARARGS,
         "Increment a key by a delta."},
     {"decr", (PyCFunction)PylibMC_Client_decr, METH_VARARGS,
         "Decrement a key by a delta."},
+    {"incr_multi", (PyCFunction)PylibMC_Client_incr_multi, METH_VARARGS|METH_KEYWORDS,
+        "Increment more than one key by a delta."},
     {"get_multi", (PyCFunction)PylibMC_Client_get_multi,
         METH_VARARGS|METH_KEYWORDS, "Get multiple keys at once."},
     {"set_multi", (PyCFunction)PylibMC_Client_set_multi,
         METH_VARARGS|METH_KEYWORDS, "Set multiple keys at once."},
+    {"add_multi", (PyCFunction)PylibMC_Client_add_multi,
+        METH_VARARGS|METH_KEYWORDS, "Add multiple keys at once."},
     {"delete_multi", (PyCFunction)PylibMC_Client_delete_multi,
         METH_VARARGS|METH_KEYWORDS, "Delete multiple keys at once."},
     {"get_behaviors", (PyCFunction)PylibMC_Client_get_behaviors, METH_NOARGS,
         "Get behaviors dict."},
     {"set_behaviors", (PyCFunction)PylibMC_Client_set_behaviors, METH_O,
         "Set behaviors dict."},
+    {"get_stats", (PyCFunction)PylibMC_Client_get_stats,
+        METH_VARARGS, "Retrieve statistics from all memcached servers."},
     {"flush_all", (PyCFunction)PylibMC_Client_flush_all,
         METH_VARARGS|METH_KEYWORDS, "Flush all data on all servers."},
     {"disconnect_all", (PyCFunction)PylibMC_Client_disconnect_all, METH_NOARGS,
         "Disconnect from all servers and reset own state."},
+    {"clone", (PyCFunction)PylibMC_Client_clone, METH_NOARGS,
+        "Clone this client entirely such that it is safe to access from "
+        "another thread. This creates a new connection."},
     {NULL, NULL, 0, NULL}
 };
 /* }}} */
@@ -227,23 +370,23 @@ static PyTypeObject PylibMC_ClientType = {
     0,
     0,
     PylibMC_ClientType_methods,
-    0,                                   
+    0,
-    0,                                   
+    0,
-    0,                                   
+    0,
-    0,                                   
+    0,
-    0,                                   
+    0,
-    0,                                   
+    0,
-    0,                                   
+    0,
     (initproc)PylibMC_Client_init,
-    0,                                   
+    0,
-    (newfunc)PylibMC_ClientType_new,              
+    (newfunc)PylibMC_ClientType_new, //PyType_GenericNew,
-    0,          
+    0,
-    0,          
+    0,
-    0,          
+    0,
-    0,          
+    0,
-    0,          
+    0,
-    0,          
+    0,
-    0,          
+    0,
     0
 };
 

cmemcache_tests.py

@@ -1,67 +0,0 @@
-import pylibmc
-import unittest
-import pickle
-
-class TestCmemcached(unittest.TestCase):
-
-	def setUp(self):
-		self.mc = pylibmc.Client(["127.0.0.1:11211"])
-		#self.mc = pylibmc.Client(["127.0.0.1:11211","127.0.0.1:11212", "127.0.0.1:11213", "127.0.0.1:11214", "127.0.0.1:11215"])
-
-	def testSetAndGet(self):
-		self.mc.set("num12345", 12345)
-		self.assertEqual(self.mc.get("num12345"), 12345)
-		self.mc.set("str12345", "12345")
-		self.assertEqual(self.mc.get("str12345"), "12345")
-
-	def testDelete(self):
-		self.mc.set("str12345", "12345")
-		#delete return True on success, otherwise False
-		ret = self.mc.delete("str12345")
-		self.assertEqual(self.mc.get("str12345"), None)
-		self.assertEqual(ret, True)
-
-		ret = self.mc.delete("hello world")
-		self.assertEqual(ret, False)
-
-	def testGetMulti(self):
-		self.mc.set("a", "valueA")
-		self.mc.set("b", "valueB")
-		self.mc.set("c", "valueC")
-		result = self.mc.get_multi(["a", "b", "c", "", "hello world"])
-		self.assertEqual(result, {'a':'valueA', 'b':'valueB', 'c':'valueC'})
-	
-	def testBigGetMulti(self):
-		count = 10
-		keys = ['key%d' % i for i in xrange(count)]
-		pairs = zip(keys, ['value%d' % i for i in xrange(count)])
-		for key, value in pairs:
-			self.mc.set(key, value)
-		result = self.mc.get_multi(keys)
-		assert result == dict(pairs)
-
-	def testFunnyDelete(self):
-		result= self.mc.delete("")
-		self.assertEqual(result, False)
-
-	def testAppend(self):
-		self.mc.delete("a")
-		self.mc.set("a", "I ")
-		ret = self.mc.append("a", "Do")
-		print ret
-		result = self.mc.get("a")
-		print result
-		self.assertEqual(result, "I Do")
-		
-	def testPrepend(self):
-		self.mc.delete("a")
-		self.mc.set("a", "Do")
-		ret = self.mc.prepend("a", "I ")
-		print ret
-		result = self.mc.get("a")
-		print result
-		self.assertEqual(result, "I Do")
-
-if __name__ == '__main__':
-	unittest.main()
-

coders.rst

@@ -0,0 +1,8 @@
+The List of Honored Men
+=======================
+
+* Ludvig Ericson <ludvig@lericson.se>
+* ketralnis <ketralnis@reddit.com>
+* Ruda Moura <ruda.moura@corp.terra.com.br>
+* Noah Silas <noah@mahalo.com>
+* Johan Bergström <johan@bergstroem.nu>

pooling.rst

@@ -0,0 +1,82 @@
+========================
+ Pooling with `pylibmc` 
+========================
+
+:Author: Ludvig Ericson <ludvig circled-a lericson dot se>
+:See also: `Pooling with pylibmc`__ (this document)
+:See also: `Pooling with pylibmc pt. 2`__ (follow-up)
+
+__ http://lericson.blogg.se/code/2009/september/draft-sept-20-2009.html
+__ http://lericson.blogg.se/code/2009/september/pooling-with-pylibmc-pt-2.html
+
+.. This is really a blog post, I do write them in ReST occasionally. Provided
+   here for the sake of convenience.
+
+I was discussing how to implement pooling for `pylibmc` when I realized what
+`libmemcachedutil`'s pooling is - or rather, what it isn't.
+
+It's not a magical solution for concurrently doing anything at all, it's not
+anything like that -- it just helps you with thread-safety.
+
+In Python, however, we've got the global interpreter lock, the GIL. This lock
+must always be held by the thread that is dealing with anything Python. The
+Python interpreter itself isn't thread-safe, or rather, it is with the GIL.
+
+This means that whenever Python code is running, you'll be sure to have
+exclusive access to all of Python's memory (unless something is misbehaving.)
+In turn, this means that the usecase for using `libmemcachedutil` in a Python
+library is rather slim.
+
+Let's have a look at some code for doing the equivalent in pure Python. This is
+a Werkzeug-based WSGI application which would be run in multiple threads,
+concurrently:
+
+.. code-block:: python
+
+    # Configuration
+    n_threads = 12
+    mc_addrs = "10.0.1.1", "10.0.1.2", "10.0.1.3"
+    mc_pool_size = n_threads
+
+    # Application
+    import pylibmc
+    from contextlib import contextmanager
+    from pprint import pformat
+    from werkzeug import Request, Response
+    from werkzeug.exceptions import NotFound
+
+    class ClientPool(list):
+        @contextmanager
+        def reserve(self):
+            mc = self.pop()
+            try:
+                yield mc
+            finally:
+                self.append(mc)
+
+    mc = pylibmc.Client(mc_addrs)
+    mc_pool = ClientPool(mc.clone() for i in xrange(mc_pool_size))
+
+    @Request.application
+    def my_app(request):
+        with mc_pool.reserve() as mc:
+            key = request.path[1:].encode("ascii")
+            val = mc.get(key)
+            if not val:
+                return NotFound(key)
+            return Response(pformat(val))
+
+It's fully-functional example of how one could implement pooling with
+`pylibmc`, and very much so in the same way that people do with
+`libmemcachedutil`. To start it, you could use Spawning like so:
+``spawn -s 1 -t 12 my_wsgi_app.my_app``
+
+I don't know if I think the above methodology is the best one though, another
+possibility is to have a dict with thread names as keys and client objects for
+values, then, each thread would look up its own client object in the dict on
+each request, and if none exists, it clones a master just like the pooling
+thing above.
+
+It'd be neat if there was a generic Python API for doing any variant of
+pooling, per-thread or the list-based version, and then you'd be able to switch
+between them seamlessly. Hm.

pylibmc-version.h

@@ -0,0 +1 @@
+#define PYLIBMC_VERSION "1.1.1"

pylibmc.py

@@ -1,117 +0,0 @@
-"""`python-memcached`-compatible wrapper around `_pylibmc`.
-
-The interface is pretty much exactly the same as python-memcached, with some
-minor differences. If you should happen to spot any, file a bug!
-
->>> import pylibmc
->>> mc = pylibmc.Client(["127.0.0.1"])
->>> b = mc.behaviors
->>> list(sorted(b.keys()))  # doctest: +NORMALIZE_WHITESPACE
-['binary_protocol', 'buffer_requests', 'cache_lookups', 'connect_timeout',
- 'distribution', 'hash', 'ketama', 'ketama_hash', 'ketama_weighted',
- 'no block', 'poll_timeout', 'rcv_timeout', 'retry_timeout',
- 'server_failure_limit', 'snd_timeout', 'socket recv size', 'socket send size',
- 'sort_hosts', 'support_cas', 'tcp_nodelay', 'verify_key']
->>> b["hash"]
-'default'
->>> b["hash"] = 'fnv1a_32'
->>> mc.behaviors["hash"]
-'fnv1a_32'
->>> super(pylibmc.Client, mc).get_behaviors()["hash"]
-6
-"""
-
-import _pylibmc
-
-__all__ = ["hashers", "distributions", "Client"]
-__version__ = "0.7.4"
-
-hashers, hashers_rvs = {}, {}
-distributions, distributions_rvs = {}, {}
-# Not the prettiest way of doing things, but works well.
-for name in dir(_pylibmc):
-    if name.startswith("hash_"):
-        key, value = name[5:], getattr(_pylibmc, name)
-        hashers[key] = value
-        hashers_rvs[value] = key
-    elif name.startswith("distribution_"):
-        key, value = name[13:].replace("_", " "), getattr(_pylibmc, name)
-        distributions[key] = value
-        distributions_rvs[value] = key
-
-class BehaviorDict(dict):
-    def __init__(self, client, *args, **kwds):
-        super(BehaviorDict, self).__init__(*args, **kwds)
-        self.client = client
-
-    def __setitem__(self, name, value):
-        super(BehaviorDict, self).__setitem__(name, value)
-        self.client.set_behaviors({name: value})
-
-    def update(self, *args, **kwds):
-        super(BehaviorDict, self).update(*args, **kwds)
-        self.client.set_behaviors(self.copy())
-
-class Client(_pylibmc.client):
-    def __init__(self, servers, *args, **kwds):
-        """Initialize a memcached client instance.
-
-        This connects to the servers in *servers*, which will default to being
-        TCP servers. If it looks like a filesystem path, a UNIX socket. If
-        prefixed with `udp:`, a UDP connection.
-        """
-        addr_tups = []
-        for server in servers:
-            addr = server
-            port = 11211
-            if server.startswith("udp:"):
-                stype = _pylibmc.server_type_udp
-                addr = addr[4:]
-                if ":" in server:
-                    (addr, port) = addr.split(":", 1)
-                    port = int(port)
-            elif ":" in server:
-                stype = _pylibmc.server_type_tcp
-                (addr, port) = server.split(":", 1)
-                port = int(port)
-            elif "/" in server:
-                stype = _pylibmc.server_type_unix
-                port = 0
-            else:
-                stype = _pylibmc.server_type_tcp
-            addr_tups.append((stype, addr, port))
-        super(Client, self).__init__(addr_tups)
-
-    def get_behaviors(self):
-        """Gets the behaviors from the underlying C client instance.
-
-        Reverses the integer constants for `hash` and `distribution` into more
-        understandable string values. See *set_behaviors* for info.
-        """
-        behaviors = super(Client, self).get_behaviors()
-        behaviors["hash"] = hashers_rvs[behaviors["hash"]]
-        behaviors["distribution"] = distributions_rvs[behaviors["distribution"]]
-        return BehaviorDict(self, behaviors)
-
-    def set_behaviors(self, behaviors):
-        """Sets the behaviors on the underlying C client instance.
-
-        Takes care of morphing the `hash` key, if specified, into the
-        corresponding integer constant (which the C client expects.) If,
-        however, an unknown value is specified, it's passed on to the C client
-        (where it most surely will error out.)
-
-        This also happens for `distribution`.
-        """
-        behaviors = behaviors.copy()
-        if behaviors.get("hash", None) in hashers:
-            behaviors["hash"] = hashers[behaviors["hash"]]
-        if behaviors.get("distribution") in distributions:
-            behaviors["distribution"] = distributions[behaviors["distribution"]]
-        return super(Client, self).set_behaviors(behaviors)
-
-    behaviors = property(get_behaviors, set_behaviors)
-
-if __name__ == "__main__":
-    import doctest
-    doctest.testmod()

pylibmc/__init__.py

@@ -0,0 +1,73 @@
+"""Snappy libmemcached wrapper
+
+pylibmc is a Python wrapper around TangentOrg's libmemcached library.
+
+The interface is intentionally made as close to python-memcached as possible,
+so that applications can drop-in replace it.
+
+Example usage
+=============
+
+Create a connection and configure it::
+
+    >>> import pylibmc
+    >>> mc = pylibmc.Client(["127.0.0.1"], binary=True)
+    >>> mc.behaviors = {"tcp_nodelay": True, "ketama": True}
+
+Basic operation::
+
+    >>> mc.set("some_key", "Some value")
+    True
+    >>> value = mc.get("some_key")
+    >>> value
+    'Some value'
+    >>> mc.set("another_key", 3)
+    True
+    >>> mc.delete("another_key")
+    True
+    >>> mc.set("key", "1")  # str or int is fine
+    True
+
+Atomic increments and decrements::
+
+    >>> mc.incr("key")
+    2L
+    >>> mc.decr("key")
+    1L
+
+Batch operation::
+
+    >>> mc.get_multi(["key", "another_key"])
+    {'key': '1'}
+    >>> mc.set_multi({"cats": ["on acid", "furry"], "dogs": True})
+    []
+    >>> mc.get_multi(["cats", "dogs"])
+    {'cats': ['on acid', 'furry'], 'dogs': True}
+    >>> mc.delete_multi(["cats", "dogs", "nonextant"])
+    False
+    >>> mc.add_multi({"cats": ["on acid", "furry"], "dogs": True})
+    []
+    >>> mc.get_multi(["cats", "dogs"])
+    {'cats': ['on acid', 'furry'], 'dogs': True}
+    >>> mc.add_multi({"cats": "not set", "dogs": "definitely not set", "bacon": "yummy"})
+    ['cats', 'dogs']
+    >>> mc.get_multi(["cats", "dogs", "bacon"])
+    {'cats': ['on acid', 'furry'], 'bacon': 'yummy', 'dogs': True}
+    >>> mc.delete_multi(["cats", "dogs", "bacon"])
+    True
+
+Further Reading
+===============
+
+See http://sendapatch.se/projects/pylibmc/
+"""
+
+import _pylibmc
+from .consts import hashers, distributions
+from .client import Client
+from .pools import ClientPool, ThreadMappedPool
+
+support_compression = _pylibmc.support_compression
+__version__ = _pylibmc.__version__
+__all__ = ["hashers", "distributions", "Client",
+           "ClientPool", "ThreadMappedPool"]

pylibmc/__main__.py

@@ -0,0 +1,48 @@
+"""Interactive shell"""
+
+import sys
+import code
+import random
+import pylibmc
+
+tips = [
+    "Want to use 127.0.0.1? Just hit Enter immediately.",
+    "This was supposed to be a list of tips but I...",
+    "I don't really know what to write here.",
+    "Really, hit Enter immediately and you'll connect to 127.0.0.1.",
+    "Did you know there's a --binary flag? Try it!",
+    "Want to use binary mode? Pass --binary as a sole argument."
+]
+
+def print_header(outf=sys.stdout):
+    outf.write("pylibmc interactive shell\n\n")
+    outf.write("Input list of servers, terminating by a blank line.\n")
+    outf.write(random.choice(tips) + "\n")
+
+def collect_servers():
+    in_addr = raw_input("Address [127.0.0.1]: ")
+    while in_addr:
+        in_addr = raw_input("Address [<stop>]: ")
+        if in_addr:
+            yield in_addr
+    else:
+        yield "127.0.0.1"
+
+banner = ("\nmc client available as `mc`\n"
+          "memface available as mf")
+def interact(servers, banner=banner, binary=False):
+    mc = pylibmc.Client(servers, binary=binary)
+    mf = pylibmc.Memface(servers, binary=binary)
+    local = {"pylibmc": pylibmc,
+             "mf": mf, "mc": mc}
+    code.interact(banner=banner, local=local)
+
+def main():
+    binary = False
+    if sys.argv[1:] == ["--binary"]:
+        binary = True
+    print_header()
+    interact(list(collect_servers()), binary=binary)
+
+if __name__ == "__main__":
+    main()

pylibmc/client.py

@@ -0,0 +1,90 @@
+"""Python-level wrapper client"""
+
+import _pylibmc
+from .consts import (hashers, distributions,
+                     hashers_rvs, distributions_rvs,
+                     BehaviorDict)
+
+class Client(_pylibmc.client):
+    def __init__(self, servers, binary=False):
+        """Initialize a memcached client instance.
+
+        This connects to the servers in *servers*, which will default to being
+        TCP servers. If it looks like a filesystem path, a UNIX socket. If
+        prefixed with `udp:`, a UDP connection.
+
+        If *binary* is True, the binary memcached protocol is used.
+        """
+        self.binary = binary
+        self.addresses = list(servers)
+        addr_tups = []
+        for server in servers:
+            addr = server
+            port = 11211
+            if server.startswith("udp:"):
+                stype = _pylibmc.server_type_udp
+                addr = addr[4:]
+                if ":" in server:
+                    (addr, port) = addr.split(":", 1)
+                    port = int(port)
+            elif ":" in server:
+                stype = _pylibmc.server_type_tcp
+                (addr, port) = server.split(":", 1)
+                port = int(port)
+            elif "/" in server:
+                stype = _pylibmc.server_type_unix
+                port = 0
+            else:
+                stype = _pylibmc.server_type_tcp
+            addr_tups.append((stype, addr, port))
+        super(Client, self).__init__(servers=addr_tups, binary=binary)
+
+    def __repr__(self):
+        return "%s(%r, binary=%r)" % (self.__class__.__name__,
+                                      self.addresses, self.binary)
+
+    def __str__(self):
+        addrs = ", ".join(map(str, self.addresses))
+        return "<%s for %s, binary=%r>" % (self.__class__.__name__,
+                                           addrs, self.binary)
+
+    def get_behaviors(self):
+        """Gets the behaviors from the underlying C client instance.
+
+        Reverses the integer constants for `hash` and `distribution` into more
+        understandable string values. See *set_behaviors* for info.
+        """
+        bvrs = super(Client, self).get_behaviors()
+        bvrs["hash"] = hashers_rvs[bvrs["hash"]]
+        bvrs["distribution"] = distributions_rvs[bvrs["distribution"]]
+        return BehaviorDict(self, bvrs)
+
+    def set_behaviors(self, behaviors):
+        """Sets the behaviors on the underlying C client instance.
+
+        Takes care of morphing the `hash` key, if specified, into the
+        corresponding integer constant (which the C client expects.) If,
+        however, an unknown value is specified, it's passed on to the C client
+        (where it most surely will error out.)
+
+        This also happens for `distribution`.
+        """
+        behaviors = behaviors.copy()
+        if behaviors.get("hash") is not None:
+            behaviors["hash"] = hashers[behaviors["hash"]]
+        if behaviors.get("ketama_hash") is not None:
+            behaviors["ketama_hash"] = hashers[behaviors["ketama_hash"]]
+        if behaviors.get("distribution") is not None:
+            behaviors["distribution"] = distributions[behaviors["distribution"]]
+        return super(Client, self).set_behaviors(behaviors)
+
+    behaviors = property(get_behaviors, set_behaviors)
+    @property
+    def behaviours(self):
+        raise AttributeError("nobody uses british spellings")
+
+    def clone(self):
+        obj = super(Client, self).clone()
+        obj.addresses = list(self.addresses)
+        obj.binary = self.binary
+        return obj

pylibmc/consts.py

@@ -0,0 +1,40 @@
+"""Constants and functionality related to them"""
+
+import _pylibmc
+
+#: Mapping of 
+errors = tuple(e for (n, e) in _pylibmc.exceptions)
+# *Cough* Uhm, not the prettiest of things but this unpacks all exception
+# objects and sets them on the package module object currently constructed.
+import sys
+modpkg = sys.modules[__package__]
+modself = sys.modules[__package__]
+for name, exc in _pylibmc.exceptions:
+    setattr(modself, name, exc)
+    setattr(modpkg, name, exc)
+
+hashers, hashers_rvs = {}, {}
+distributions, distributions_rvs = {}, {}
+# Not the prettiest way of doing things, but works well.
+for name in dir(_pylibmc):
+    if name.startswith("hash_"):
+        key, value = name[5:], getattr(_pylibmc, name)
+        hashers[key] = value
+        hashers_rvs[value] = key
+    elif name.startswith("distribution_"):
+        key, value = name[13:].replace("_", " "), getattr(_pylibmc, name)
+        distributions[key] = value
+        distributions_rvs[value] = key
+
+class BehaviorDict(dict):
+    def __init__(self, client, *args, **kwds):
+        super(BehaviorDict, self).__init__(*args, **kwds)
+        self.client = client
+
+    def __setitem__(self, name, value):
+        super(BehaviorDict, self).__setitem__(name, value)
+        self.client.set_behaviors({name: value})
+
+    def update(self, d):
+        super(BehaviorDict, self).update(d)
+        self.client.set_behaviors(d.copy())

pylibmc/pools.py

@@ -0,0 +1,97 @@
+"""Pooling"""
+
+from __future__ import with_statement
+
+from contextlib import contextmanager
+from Queue import Queue
+
+class ClientPool(Queue):
+    """Client pooling helper.
+
+    This is mostly useful in threaded environments, because a client isn't
+    thread-safe at all. Instead, what you want to do is have each thread use
+    its own client, but you don't want to reconnect these all the time.
+
+    The solution is a pool, and this class is a helper for that.
+
+    >>> mc = Client(["127.0.0.1"])
+    >>> pool = ClientPool()
+    >>> pool.fill(mc, 4)
+    >>> with pool.reserve() as mc:
+    ...     mc.set("hi", "ho")
+    ...     mc.delete("hi")
+    ... 
+    True
+    True
+    """
+
+    def __init__(self, mc=None, n_slots=None):
+        Queue.__init__(self, n_slots)
+        if mc is not None:
+            self.fill(mc, n_slots)
+
+    @contextmanager
+    def reserve(self, timeout=None):
+        """Context manager for reserving a client from the pool.
+
+        If *timeout* is given, it specifiecs how long to wait for a client to
+        become available.
+        """
+        mc = self.get(True, timeout=timeout)
+        try:
+            yield mc
+        finally:
+            self.put(mc)
+
+    def fill(self, mc, n_slots):
+        """Fill *n_slots* of the pool with clones of *mc*."""
+        for i in xrange(n_slots):
+            self.put(mc.clone())
+
+class ThreadMappedPool(dict):
+    """Much like the *ClientPool*, helps you with pooling.
+
+    In a threaded environment, you'd most likely want to have a client per
+    thread. And there'd be no harm in one thread keeping the same client at all
+    times. So, why not map threads to clients? That's what this class does.
+
+    If a client is reserved, this class checks for a key based on the current
+    thread, and if none exists, clones the master client and inserts that key.
+
+    >>> mc = Client(["127.0.0.1"])
+    >>> pool = ThreadMappedPool(mc)
+    >>> with pool.reserve() as mc:
+    ...     mc.set("hi", "ho")
+    ...     mc.delete("hi")
+    ... 
+    True
+    True
+    """
+
+    def __new__(cls, master):
+        return super(ThreadMappedPool, cls).__new__(cls)
+
+    def __init__(self, master):
+        self.master = master
+
+    @contextmanager
+    def reserve(self):
+        """Reserve a client.
+
+        Creates a new client based on the master client if none exists for the
+        current thread.
+        """
+        key = thread.get_ident()
+        mc = self.pop(key, None)
+        if mc is None:
+            mc = self.master.clone()
+        try:
+            yield mc
+        finally:
+            self[key] = mc
+
+# This makes sure ThreadMappedPool doesn't exist with non-thread Pythons.
+try:
+    import thread
+except ImportError:
+    ThreadMappedPool = None

setup.py

@@ -1,23 +1,77 @@
 import os
+import sys
 from distutils.core import setup, Extension
 
+# --with-zlib: use zlib for compressing and decompressing
+# --without-zlib: ^ negated
+# --with-zlib=<dir>: path to zlib if needed
+# --with-libmemcached=<dir>: path to libmemcached package if needed
+
+cmd = None
+use_zlib = True
+pkgdirs = []  # incdirs and libdirs get these
+libs = ["memcached"]
+defs = []
 incdirs = []
 libdirs = []
 
-if "LIBMEMCACHED_DIR" in os.environ:
+def append_env(L, e):
-    libdir = os.path.normpath(os.environ["LIBMEMCACHED_DIR"])
+    v = os.environ.get(e)
-    incdirs.append(os.path.join(libdir, "include"))
+    if v and os.path.exists(v):
-    libdirs.append(os.path.join(libdir, "lib"))
+        L.append(v)
 
-readme_text = open("README.rst", "U").read()
+append_env(pkgdirs, "LIBMEMCACHED")
+append_env(pkgdirs, "ZLIB")
+
+# Hack up sys.argv, yay
+unprocessed = []
+for arg in sys.argv[1:]:
+    if arg == "--with-zlib":
+        use_zlib = True
+        continue
+    elif arg == "--without-zlib":
+        use_zlib = False
+        continue
+    elif arg == "--gen-setup":
+        cmd = arg[2:]
+    elif "=" in arg:
+        if arg.startswith("--with-libmemcached=") or \
+           arg.startswith("--with-zlib="):
+            pkgdirs.append(arg.split("=", 1)[1])
+            continue
+    unprocessed.append(arg)
+sys.argv[1:] = unprocessed
+
+for pkgdir in pkgdirs:
+    incdirs.append(os.path.join(pkgdir, "include"))
+    libdirs.append(os.path.join(pkgdir, "lib"))
+
+if use_zlib:
+    libs.append("z")
+    defs.append(("USE_ZLIB", None))
  
 pylibmc_ext = Extension("_pylibmc", ["_pylibmcmodule.c"],
-                        libraries=["memcached"],
+                        libraries=libs, include_dirs=incdirs,
-                        include_dirs=incdirs, library_dirs=libdirs)
+                        library_dirs=libdirs, define_macros=defs)
 
-setup(name="pylibmc", version="0.7.4",
+# Hidden secret: if environment variable GEN_SETUP is set, generate Setup file.
-      url="http://lericson.blogg.se/code/category/pylibmc.html",
+if cmd == "gen-setup":
+    line = " ".join((
+        pylibmc_ext.name,
+        " ".join("-l" + lib for lib in pylibmc_ext.libraries),
+        " ".join("-I" + incdir for incdir in pylibmc_ext.include_dirs),
+        " ".join("-L" + libdir for libdir in pylibmc_ext.library_dirs),
+        " ".join("-D" + name + ("=" + str(value), "")[value is None] for (name, value) in pylibmc_ext.define_macros)))
+    open("Setup", "w").write(line + "\n")
+    sys.exit(0)
+
+readme_text = open("README.rst", "U").read()
+version = open("pylibmc-version.h", "U").read().strip().split("\"")[1]
+
+setup(name="pylibmc", version=version,
+      url="http://sendapatch.se/projects/pylibmc/",
       author="Ludvig Ericson", author_email="ludvig@lericson.se",
       license="3-clause BSD <http://www.opensource.org/licenses/bsd-license.php>",
-      description="libmemcached wrapper", long_description=readme_text,
+      description="Quick and small memcached client for Python",
-      ext_modules=[pylibmc_ext], py_modules=["pylibmc"])
+      long_description=readme_text,
+      ext_modules=[pylibmc_ext], packages=["pylibmc"])

tests.py

@@ -1,7 +1,11 @@
-"""Tests. They want YOU!!
+r"""Tests. They want YOU!!
+
 
 Basic functionality.
+>>> _pylibmc.__version__ == pylibmc.__version__
+True
 >>> c = _pylibmc.client([test_server])
+>>> c.get("hello")
 >>> c.set("test_key", 123)
 True
 >>> c.get("test_key")
@@ -12,6 +16,12 @@ True
 >>> c.get("test_key")
 >>>
 
+We should handle empty values just nicely.
+>>> c.set("foo", "")
+True
+>>> c.get("foo")
+''
+
 Now this section is because most other implementations ignore zero-keys.
 >>> c.get("")
 >>> c.set("", "hi")
@@ -26,7 +36,7 @@ Multi functionality.
 >>> c.get_multi("abc").keys() == ["a", "c", "b"]
 True
 >>> c.delete_multi("abc")
-[]
+True
 >>> c.get_multi("abc").keys() == []
 True
 >>> c.set_multi(dict(zip("abc", "def")), key_prefix="test_")
@@ -36,7 +46,7 @@ True
 >>> c.get("test_a")
 'd'
 >>> c.delete_multi("abc", key_prefix="test_")
-[]
+True
 >>> bool(c.get_multi("abc", key_prefix="test_"))
 False
 
@@ -44,9 +54,13 @@ Zero-key-test-time!
 >>> c.get_multi([""])
 {}
 >>> c.delete_multi([""])
-['']
+False
 >>> c.set_multi({"": "hi"})
 ['']
+>>> c.delete_multi({"a": "b"})
+Traceback (most recent call last):
+  ...
+TypeError: keys must be a sequence, not a mapping
 
 Timed stuff. The reason we, at UNIX times, set it two seconds in the future and
 then sleep for >3 is that memcached might round the time up and down and left
@@ -56,21 +70,36 @@ and yeah, so you know...
 True
 >>> c.get("hi")
 'steven'
->>> sleep(1.1)
+>>> sleep(2.1)
 >>> c.get("hi")
 >>> c.set("hi", "loretta", int(time()) + 2)
 True
+>>> c.set_multi({"hi2": "charlotta"}, 1)
+[]
 >>> c.get("hi")
 'loretta'
+>>> c.get("hi2")
+'charlotta'
 >>> sleep(3.1)
->>> c.get("hi")
+>>> c.get("hi"), c.get("hi2")
->>>
+(None, None)
+
+See issue #1 ``http://github.com/lericson/pylibmc/issues/#issue/1`` -- delete
+should not accept a time argument.
+>>> c.delete("foo", 123)
+Traceback (most recent call last):
+  ...
+TypeError: delete() takes exactly 1 argument (2 given)
+>>> c.delete_multi(["foo"], time=123)
+Traceback (most recent call last):
+  ...
+TypeError: 'time' is an invalid keyword argument for this function
 
 Now for keys with funny types.
 >>> c.set(1, "hi")
 Traceback (most recent call last):
   ...
-TypeError: argument 1 must be string or read-only buffer, not int
+TypeError: argument 1 must be string, not int
 >>> c.get(1)
 Traceback (most recent call last):
   ...
@@ -84,6 +113,23 @@ Traceback (most recent call last):
   ...
 TypeError: key must be an instance of str
 
+This didn't use to work, but now it does.
+>>> c.get_multi([])
+{}
+
+Getting stats is fun!
+>>> for (svr, stats) in c.get_stats():
+...     print svr
+...     ks = stats.keys()
+...     while ks:
+...         cks, ks = ks[:6], ks[6:]
+...         print ", ".join(cks)
+127.0.0.1:11211 (0)
+pid, total_items, uptime, version, limit_maxbytes, rusage_user
+bytes_read, rusage_system, cmd_get, curr_connections, threads, total_connections
+cmd_set, curr_items, get_misses, evictions, bytes, connection_structures
+bytes_written, time, pointer_size, get_hits
+
 Also test some flush all.
 >>> c.set("hi", "guys")
 True
@@ -114,36 +160,226 @@ True
 >>> c.get("tengil").bar == bla.bar
 True
 
+Cloning (ethically, I don't know about it.)
+>>> c is not c.clone()
+True
+>>> c2 = c.clone()
+>>> c.set("test", "hello")
+True
+>>> c2.get("test")
+'hello'
+>>> c2.delete("test")
+True
+>>> del c2
+
+Per-error exceptions
+>>> c.incr("test")
+Traceback (most recent call last):
+  ...
+NotFound: error 16 from memcached_increment: NOT FOUND
+>>> c.incr(chr(0))
+Traceback (most recent call last):
+  ...
+ProtocolError: error 8 from memcached_increment: PROTOCOL ERROR
 
 Behaviors.
 >>> c.set_behaviors({"tcp_nodelay": True, "hash": 6})
 >>> list(sorted((k, v) for (k, v) in c.get_behaviors().items()
 ...             if k in ("tcp_nodelay", "hash")))
 [('hash', 6), ('tcp_nodelay', 1)]
+
+Binary protocol!
+>>> c = _pylibmc.client([test_server], binary=True)
+>>> c.set("hello", "world")
+True
+>>> c.get("hello")
+'world'
+>>> c.delete("hello")
+True
+
+incr_multi
+>>> c.add_multi({'a': 1, 'b': 0, 'c': 4})
+[]
+>>> c.incr_multi(('a', 'b', 'c'), delta=1)
+>>> list(sorted(c.get_multi(('a', 'b', 'c')).items()))
+[('a', 2), ('b', 1), ('c', 5)]
+>>> c.delete_multi(('a', 'b', 'c'))
+True
+>>> c.add_multi({'a': 1, 'b': 0, 'c': 4}, key_prefix='x')
+[]
+>>> c.incr_multi(('a', 'b', 'c'), key_prefix='x', delta=5)
+>>> list(sorted(c.get_multi(('a', 'b', 'c'), key_prefix='x').items()))
+[('a', 6), ('b', 5), ('c', 9)]
+>>> c.delete_multi(('a', 'b', 'c'), key_prefix='x')
+True
+>>> c.add('a', 1)
+True
+>>> c.incr_multi(('a', 'b', 'c'), key_prefix='x', delta=1)
+Traceback (most recent call last):
+ ...
+NotFound: error 16 from memcached_increment: NOT FOUND
+>>> c.delete('xa')
+False
+
+Empty server lists are bad for your health.
+>>> c = _pylibmc.client([])
+Traceback (most recent call last):
+  ...
+MemcachedError: empty server list
+
+Python-wrapped behaviors dict
+>>> pc = pylibmc.Client(["%s:%d" % test_server[1:]])
+>>> (pc.behaviors["hash"], pc.behaviors["distribution"])
+('default', 'modula')
+>>> pc.behaviors.update({"hash": "fnv1a_32", "distribution": "consistent"})
+>>> (pc.behaviors["hash"], pc.behaviors["distribution"])
+('fnv1a_32', 'consistent')
+
+>>> pc = pylibmc.Client(["%s:%d" % test_server[1:]])
+>>> b = pc.behaviors
+>>> ks = list(sorted(k for k in b.keys() if not k.startswith("_")))
+>>> ks  # doctest: +NORMALIZE_WHITESPACE
+['buffer_requests', 'cache_lookups', 'cas', 'connect_timeout', 'distribution',
+ 'failure_limit', 'hash', 'ketama', 'ketama_hash', 'ketama_weighted',
+ 'no_block', 'receive_timeout', 'send_timeout', 'tcp_nodelay', 'verify_keys']
+>>> b["hash"]
+'default'
+>>> b["hash"] = 'fnv1a_32'
+>>> pc.behaviors["hash"]
+'fnv1a_32'
+>>> super(pylibmc.Client, pc).get_behaviors()["hash"]
+6
+
+Ormod's Zero-byte Adventure Story
+>>> bc = _pylibmc.client([test_server], binary=True)
+>>> bc.set("\0\0", "ORMOD")
+True
+>>> bc.get_multi(["\0\0"])
+{'\x00\x00': 'ORMOD'}
+
+Test server/client max length
+>>> mc.get('x'*250)
+>>> mc.get('x'*251)
+Traceback (most recent call last):
+  ...
+ValueError: key too long, max is 250
+
+Test CAS
+>>> mc.behaviors['cas'] = True
+>>> mc.delete('foo') and False
+False
+>>> mc.gets('foo')
+(None, None)
+>>> mc.set('foo', 'bar')
+True
+>>> foostr, cas = mc.gets('foo')
+>>> foostr
+'bar'
+>>> mc.cas('foo', 'quux', cas+1)
+False
+>>> mc.cas('foo', 'baz', cas)
+True
+>>> mc.get('foo')
+'baz'
+>>> mc.behaviors['cas'] = False
+>>> mc.gets('foo')
+Traceback (most recent call last):
+...
+ValueError: gets without cas behavior
+>>> mc.cas('foo', 'bar', 1)
+Traceback (most recent call last):
+...
+ValueError: cas without cas behavior
 """
 
 # Used to test pickling.
 class Foo(object): pass
 
-# Fix up sys.path so as to include the build/lib.*/ directory.
+# Fix up sys.path so as to include the correct build/lib.*/ directory.
 import sys
-import os
+from distutils.dist import Distribution
-from glob import glob
+from distutils.command.build import build
 
-dist_dir = os.path.dirname(__file__)
+build_cmd = build(Distribution({"ext_modules": True}))
-for build_dir in glob(os.path.join(dist_dir, "build", "lib.*")):
+build_cmd.finalize_options()
-    sys.path.insert(0, build_dir)
+lib_dirn = build_cmd.build_lib
-    break
+sys.path.insert(0, lib_dirn)
-else:
-    print >>sys.stderr, "Using system-wide installation of pylibmc!"
-    print >>sys.stderr, "==========================================\n"
 
 import pylibmc, _pylibmc
 import socket
 
 __doc__ = pylibmc.__doc__ + "\n\n" + __doc__
 
-test_server = (_pylibmc.server_type_tcp, "localhost", 11211)
+# {{{ Ported cmemcache tests
+import unittest
+
+class TestCmemcached(unittest.TestCase):
+    def setUp(self):
+        self.mc = pylibmc.Client(["%s:%d" % (test_server[1:])])
+
+    def testSetAndGet(self):
+        self.mc.set("num12345", 12345)
+        self.assertEqual(self.mc.get("num12345"), 12345)
+        self.mc.set("str12345", "12345")
+        self.assertEqual(self.mc.get("str12345"), "12345")
+
+    def testDelete(self):
+        self.mc.set("str12345", "12345")
+        #delete return True on success, otherwise False
+        ret = self.mc.delete("str12345")
+        self.assertEqual(self.mc.get("str12345"), None)
+        self.assertEqual(ret, True)
+
+        # This test only works with old memcacheds. This has become a "client
+        # error" in memcached.
+        try:
+            ret = self.mc.delete("hello world")
+        except pylibmc.ClientError:
+            pass
+        else:
+            self.assertEqual(ret, False)
+
+    def testGetMulti(self):
+        self.mc.set("a", "valueA")
+        self.mc.set("b", "valueB")
+        self.mc.set("c", "valueC")
+        result = self.mc.get_multi(["a", "b", "c", "", "hello world"])
+        self.assertEqual(result, {'a':'valueA', 'b':'valueB', 'c':'valueC'})
+
+    def testBigGetMulti(self):
+        count = 10
+        keys = ['key%d' % i for i in xrange(count)]
+        pairs = zip(keys, ['value%d' % i for i in xrange(count)])
+        for key, value in pairs:
+            self.mc.set(key, value)
+        result = self.mc.get_multi(keys)
+        assert result == dict(pairs)
+
+    def testFunnyDelete(self):
+        result= self.mc.delete("")
+        self.assertEqual(result, False)
+
+    def testAppend(self):
+        self.mc.delete("a")
+        self.mc.set("a", "I ")
+        ret = self.mc.append("a", "Do")
+        result = self.mc.get("a")
+        self.assertEqual(result, "I Do")
+
+    def testPrepend(self):
+        self.mc.delete("a")
+        self.mc.set("a", "Do")
+        ret = self.mc.prepend("a", "I ")
+        result = self.mc.get("a")
+        self.assertEqual(result, "I Do")
+# }}}
+
+from os import environ
+
+test_server = (
+    _pylibmc.server_type_tcp,
+    str(environ.get("MEMCACHED_HOST", "127.0.0.1")),
+    int(environ.get("MEMCACHED_PORT", "11211")))
 
 def get_version(addr):
     (type_, host, port) = addr
@@ -167,13 +403,64 @@ def is_alive(addr):
     except (ValueError, socket.error):
         return False
 
+def make_full_suite():
+    from doctest import DocTestFinder, DocTestSuite
+    suite = unittest.TestSuite()
+    loader = unittest.TestLoader()
+    finder = DocTestFinder()
+    for modname in (__name__, "pylibmc", "_pylibmc"):
+        ss = (DocTestSuite(sys.modules[modname], test_finder=finder),
+              loader.loadTestsFromName(modname))
+        for subsuite in ss:
+            map(suite.addTest, subsuite._tests)
+    return suite
+
+class _TextTestResult(unittest._TextTestResult):
+    def startTest(self, test):
+        # Treat doctests a little specially because we want each example to
+        # count as a test.
+        if hasattr(test, "_dt_test"):
+            self.testsRun += len(test._dt_test.examples) - 1
+        elif hasattr(test, "countTestCases"):
+            self.testsRun += test.countTestCases() - 1
+        return unittest._TextTestResult.startTest(self, test)
+
+class TextTestRunner(unittest.TextTestRunner):
+    def _makeResult(self):
+        return _TextTestResult(self.stream, self.descriptions, self.verbosity)
+
+class TestProgram(unittest.TestProgram):
+    defaultTest = "make_full_suite"
+    testRunner = TextTestRunner
+
+    def __init__(self, module="__main__", argv=None,
+                 testLoader=unittest.defaultTestLoader):
+        super(TestProgram, self).__init__(
+            module=module, argv=argv, testLoader=testLoader,
+            defaultTest=self.defaultTest, testRunner=self.testRunner)
+
+    def runTests(self):
+        pass
+
+    def makeRunner(self):
+        return self.testRunner(verbosity=self.verbosity)
+
+    def run(self):
+        runner = self.makeRunner()
+        return runner.run(self.test)
+
 if __name__ == "__main__":
-    print "Starting tests with _pylibmc at", _pylibmc.__file__
+    if hasattr(_pylibmc, "__file__"):
+        print "Starting tests with _pylibmc at", _pylibmc.__file__
+    else:
+        print "Starting tests with static _pylibmc:", _pylibmc
     print "Reported libmemcached version:", _pylibmc.libmemcached_version
+    print "Reported pylibmc version:", _pylibmc.__version__
+    print "Support compression:", _pylibmc.support_compression
+
     if not is_alive(test_server):
         raise SystemExit("Test server (%r) not alive." % (test_server,))
-    import doctest
+
-    n_fail, n_run = doctest.testmod()
+    res = TestProgram().run()
-    print "Ran", n_run, "tests with", n_fail, "failures."
+    if not res.wasSuccessful():
-    if n_fail:
         sys.exit(1)