The port to CouchDB has been done!
After some serious work I have finally managed to port the core code to use CouchDB as its storage back-end rather than SQLite + SQLAlchemy. Although the process has not been strictly complicated (I’ve dealt with worst things in SharePoint for example), I have been certainly stuck with some issues.
The design of the overall solution is similar to the previous one, where a singleton DatabaseManager performs all the different database operations (I still have to make if multi-thread safe). The idea is that each of the objects that are “storable” in the system will have to inherit from the CouchDbStorable class. The CouchDbStorable ensures that all the objects can be stored in CouchDB by taken care of their id and revision number:
from datetime import datetime from macaco_core.system.utils.decorators import logged import logging import uuid class CouchDbStorable(object): """ This object represents a serializable object that can be stored in a couch db taking into account the revision and the id of the object. """ _logger = logging.getLogger('Address') ########################################################################### ################################# private methods ######################### ########################################################################### def __get_id(self): """ Returns the id of the object. If the object has no ID it will generate and id. @rtype: int @return: The id of the object. """ return self._id def __set_id(self, value): """ Sets the id of the object. @type value: int @param value: The new id of the object. """ self._id = value def __get_revision(self): """ Returns the revision of the object. If the object has not yet been stored in the db the method will return -1. @rtype: int @return: The revision of the object. """ return self._rev def __set_revision(self, value): """ Sets the revision of the object. """ self._rev = value ########################################################################### ################################# end private methods ##################### ########################################################################### ########################################################################### ################################ properties ############################### ########################################################################### id = property(__get_id, __set_id) revision = property(__get_revision, __set_revision) ########################################################################### ############################### end properties ############################ ########################################################################### ########################################################################### ############################### public methods ############################ ########################################################################### @logged(_logger) def __init__(self, **kwargs): """ Inits the different values that are used to identify an object in the db. """ self._id = uuid.uuid4().hex self._rev = -1 self._updated = str(datetime.utcnow()) @logged(_logger) def get_object_data(self): """ This method returns a dictionary with the data of the object, this can be used to serialize the object in different ways. @rtype: dict @return: A dictionary that contains the data of the object. """ # we set the data of the dict that will be used to ensure that the db # knows the id and the rev data = {} data['_id'] = self.id if self.revision != -1: data['_rev'] = self.revision data['update_date'] = str(self._updated) return data @logged(_logger) def set_object_data(self, data): """ This method set the data of the object form the passed dictionary. @type data: dict @param data: A dictionary with the data of the object to be used. """ if data.has_key('_id'): self._id = data['_id'] if data.has_key('_rev'): self._rev = data['_rev'] if data.has_key('update_date'): self._modified = data['update_date'] def update(self): """ Sets the date of when the object was updated to now utc time. """ self._modified = str(datetime.utcnow()) ########################################################################### ############################# end public methods ########################## ########################################################################### def __hash__(self): return hash(self.id)
Once an object inherits from the above class the DatabaseManager can perform the very basic operations that will allow the developer to store, update and delete objects. On top of that the DatabaseManager. The code is the following one:
import uuid import logging from couchdb.client import Server from macaco_core.system.utils.decorators import logged from cache import Cache class DataManager(object): ########################################################################### ########################## class variables ################################ ########################################################################### _logger = logging.getLogger('DataManager') ########################################################################### ############################### end class ################################# ########################################################################### ########################################################################### ############################# private methods ############################# ########################################################################### @logged(_logger) def __get_db(self): return self.__db ########################################################################### ############################ end private methods ########################## ########################################################################### ########################################################################### ################################ properties ############################### ########################################################################### db = property(__get_db) """ Returns the database that is managed by the object. """ ########################################################################### ############################# end properties ############################## ########################################################################### @logged(_logger) def __init__(self, server_location, db_name): """ Creates a new manager that can be used to managed the data in the given database of the passed server. @type server_location: string @param server_location: The server of the database we are working with. @type db_name: string @param db_name: The name of the database to manage. """ # we create a server and the db object that is used to perform the # different operations. self.__server = Server(server_location) if db_name in self.__server: self.__db = self.__server[db_name] else: self.__db = self.__server.create(db_name) ########################################################################### ############################# public methods ############################## ########################################################################### @logged(_logger) def add(self, object): """ This method adds a new object to the database. After the methods has been added the id of the object is set. @type object: object @param object: An object that implements the get_object_data in order to ensure that the object can be serialized. After the object is added the id of it is set. """ if hasattr(object, "id") and hasattr(object, "get_object_data"): self.__db[object.id] = object.get_object_data() # we get the current rev of the object we stored doc = self.__db[object.id] object.revision = doc.rev else: raise ValueError("The object does not have the id or the"\ + " object attributes") @logged(_logger) def remove(self, object): """ This method removes the passed object from the db. @type object: object @param object: An object that already has been added and has its id set. """ # we first get the doc with the id doc = self.__db.get(object.id) if not doc is None: self.__db.delete(doc) @logged(_logger) def update(self, object): """ This method updates the passed object in the db. @type object: object @param object: The object that has already been stored and that we want to update. """ if hasattr(object, "id") and hasattr(object, "get_object_data"): object.update() self.__db[object.id] = object.get_object_data() # we change the revision to ensure that when we commit the # change it correctly works later. doc = self.__db[object.id] object.revision = doc.rev else: raise ValueError("The object does not have the id or the"\ + " object attributes") @logged(_logger) def query(self, view, generator, reduce): """ This method allows to perform a dynamic query to the database which on the passed type. @type view: string @param view: The name of the view that is going to be queried. @type generator: generator @param generator: A generator that should return objects of the expected type returned by the view. If the generator is none the query will pass the rows to the reduce funtion. @type reduce: callable @param reduce: A callable object that should return true or false when an object is passed to it to decide if the object will be returned by the query. """ result = [] view_results = self.view(view, generator) for current in view_results: if reduce(current): print current result.append(current) return result @logged(_logger) def view(self, name, generator): """ This method returns the results of the view with the given name and will. @type name: string @param name: The name of the view that we are querying. @type generator: generator @param generator: A generator that will return instances of the objects we expect to be returned by the view. """ rows = self.__db.view(name) if generator is None: return rows else: cache = Cache.instance() result = [] for current in rows: object = cache.get_object(current.value["_id"]) if object is None: object = generator.next() object.set_object_data(current.value) cache.add_object(object) result.append(object) return result ########################################################################### ############################ end public methods ########################### ###########################################################################
The Database Manager allows to query views as well as to perform reduce operations to queries, but I’ll be written about it other then… for now I have bored you enough 
(but I’ll be writing in the Wiki a bit more about it)
4 Comments
→
This is great news! Do you have macaco-contacts in a PPA somewhere where I can install it and test it? We’ve been working on a library that will automatically fire up a per-user instance of couchdb (nicknamed desktopcouch), the code is here: https://launchpad.net/desktopcouch.
Also, Rodrigo has made a backend for Evolution that can read and write contacts into couchdb, you can see the code here: https://launchpad.net/evolution-couchdb.
It will be great if we can do some compatibility testing so the same contacts records can be used from both evolution and macaco. It’s exciting to see all the pieces coming together!
Statik, I have not yet done any packaging (I’m quite bad at it) but I’m planing to do it this coming weekend.
I have also heard about you project and I was going to integrate it with mine since currently you have to start manually couchDb if you want to use macaco. Let me do some UI clean up and I’ll let you know where to find it.
Anyways, If you want to get the source you can find the project here: https://launchpad.net/macaco-contacts but I need to divide the core code (macaco-core) which contains the contacts definitions and the UI in two different projects in launchpad (it makes more sense).
Let me know if you guys want to do those tests and talk about the definitions of the contacts, it will be great to work on a unify solution!!!