Feb 5 10

FOSDEM + Desktopcouch

by mandel

On an unpredicted sucession of events I have moved from helping in the cloakroom of FOSDEM to give a talk on Desktopcouch. In the talk I will focus on desktopcouch and will get into detail of how to use the lib, recipes and errors that will get you on track with the lib. I hope I see more than my friends there!

Jan 27 10

FOSDEM

by mandel

Now you know:

I'm going to FOSDEM, the Free and Open Source Software Developers' European Meeting

:D

But this year will be a bit different, last year I was more involve in the organization and was one of the volunteers helping over there. This year I will be going to a number of gnome talks, the GnomeDev room and I’l be helping in the Ubuntu table… Seems is going to be a good one!

Jan 25 10

Desktopcouch on Windows

by mandel

One of my “projects” for the xmas vacations has been to port Desktopcouch to Windows. I have twothree main reasons for this:

  • I want to be able to sync my data between my Windows machine (mainly my machine at the office) and my Linux machines (all the others)
  • I want to be able to make my different applications as multiplatform as possible.
  • Should be fun.

In this post I’ll try explain the different changes that I had to be make in desktopcouch in order to be able to port it to Windows.

Changing how couchdb is started

As most of you know, desktopcouch allows to start a couchdb per user in the machine allowing applications to use it as its data storage. The starting of couchdb in quite straight on Linux. We just have to find the couchdb command line, and pass as parameter the .ini files to be used. The current code that does that is the following:

Getting the couchdb command

COUCH_EXE = os.environ.get('COUCHDB')
if not COUCH_EXE:
    for x in os.environ['PATH'].split(':'):
        if os.path.exists(os.path.join(x, 'couchdb')):
            COUCH_EXE = os.path.join(x, 'couchdb')
if not COUCH_EXE:
    raise ImportError("Could not find couchdb")

Starting the database

def run_couchdb(ctx=local_files.DEFAULT_CONTEXT):
    """Actually start the CouchDB process.  Return its PID."""
    pid = read_pidfile(ctx)
    if pid is not None and not process_is_couchdb(pid):
        print "Removing stale, deceptive pid file."
        os.remove(ctx.file_pid)
    local_exec = ctx.couch_exec_command + ['-b']
    try:
        # subprocess is buggy.  Chad patched, but that takes time to propagate.
        proc = subprocess.Popen(local_exec)
        while True:
            try:
                retcode = proc.wait()
                break
            except OSError, e:
                if e.errno == errno.EINTR:
                    continue
                raise
        if retcode < 0:
            print >> sys.stderr, "Child was terminated by signal", -retcode
        elif retcode > 0:
            print >> sys.stderr, "Child returned", retcode
    except OSError, e:
        print >> sys.stderr, "Execution failed: %s: %s" % (e, local_exec)
        exit(1)
 
    # give the process a chance to start
    for timeout in (0.4, 0.1, 0.1, 0.2, 0.5, 1, 3, 5):
        pid = read_pidfile(ctx=ctx)
        if pid is not None and process_is_couchdb(pid):
            break
        time.sleep(timeout)
 
    # Loop for a number of times until the port has been found, this
    # has to be done because there's a slice of time between PID being written
    # and the listening port being active.
    for timeout in (0.1, 0.1, 0.2, 0.5, 1, 3, 5, 8):
        try:
            port = desktopcouch.find_port(pid=pid, ctx=ctx) # only returns valid port
            break
        except RuntimeError, e:
            pass
        time.sleep(timeout)
 
    ctx.ensure_files_not_readable()
    return pid, port

Of course this is not that easy on Windows. If anyone has played around with couchdb on windows, specially the with the binary package, you will find that in order to start the database you have to have a batch file doing something like this (I borrowed the batch from the couchdb binary package):

@echo off
rem Licensed under the Apache License, Version 2.0 (the "License"); you may not
rem use this file except in compliance with the License. You may obtain a copy
rem of the License at
rem
rem   http://www.apache.org/licenses/LICENSE-2.0
rem
rem Unless required by applicable law or agreed to in writing, software
rem distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
rem WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
rem License for the specific language governing permissions and limitations
rem under the License.

setlocal
rem First change to the erlang bin directory
cd %~dp0

rem Allow a different erlang executable (eg, werl) to be used.
if "%ERL%x" == "x" set ERL=erl.exe

echo CouchDB 0.10.0 - prepare to relax...
%ERL% -smp auto -sasl errlog_type error ^
      -eval "application:load(crypto)" ^
      -eval "application:load(couch)" ^
      -eval "crypto:start()" ^
      -eval "couch_server:start([""../etc/couchdb/default.ini"", ""../etc/couchdb/local.ini""]), receive done -> done end."

As you can image this is not the type of command that is found on Linux, and therefore we have to make a better solution.

One of the dangers that a developers finds himself when porting an application like desktopcouch to Windows is the tendency to copy the design decitions used in Linux. Linux and Windows have complete different programming models and we should try to make the port as native as possible.

If I were to design desktopcouch from scratch to work on Windows I would do the following:

  1. Create a Windows Service that will take care of the starting and stopping of the different couchdb processeses. This service will take care of the management of the different processes as well as allowing other applications querying the system.
  2. Provide an API to query the current couchdb process for the user and retrieve the required data.

Windows Service Implementation

There are two different ways to implement a service that can be used to manage the couchdb instances:

  • A simple Windows Service (NT Service)
  • A Windows service that uses WCF service.

When implementing the service we have to think about what kind of applications we want to be able to communicate with it. On one hand we have the traditional Windows Service that can be easily implemented and any application written in the CLR can easily communicate with it. On the other hand, we can use a more “fancy” implementation with WCF. WCF allows to implement a service that can easily use different protocols for communication with not too much effort. I’d say that in our situation we prefer to use a WCF since we would like as many different languages and systems to be able to communicate with the service and not only C# (what about python!).

Some people might complain because we are using SOA, but SOA does not only involve Web Services is a broader point of view. In this example we could argue that being able to start a couchdb instance and returns its port can be identified as a service. By creating a new service I’m trying to achieve two different things:

  • Isolate as much diff code as possible from the Linux solution
  • Provide a interop interface that can be used to access the couchdb instance from any language.

In summary, in order to simplify the creation of couchdb instance in a Windows environment we will create a Windows Service that uses WCF to allow the different client applications to contact the service and request the port and auth of the db and start it if necessary.

The following code shown the service contract that will be used to allow client applications to query information regarding the cuchdb instance for the user. In this step we are not looking at the user, that step is left for the auth part of the system we are just determining the contract.

using System.Runtime.Serialization;
using System.ServiceModel;
 
namespace DesktopcouchServie
{
	/// <summary>
	/// Basic service contract that provides a number of methods that client applications can execute to query the information 
	/// of the couchdb instance of the current user.
	/// </summary>
	[ServiceContract]
	public interface IDesktopcouchService
	{
		/// <summary>
		/// This method allow a client application to query the port in which the couchdb instance is executing. 
		/// </summary>
		/// <returns>The post in which the couchdb instance can be found.</returns>
		[OperationContract]
		string GetCouchdbPort();
 
		/// <summary>
		/// This method allows a client to starta couchdb instance for the current user. It is recommended to just allow the
                /// desktopcouch python
		/// library to take care of the start of the instance.
		/// </summary>
		/// <param name="startUpData">The data to be used to start the couchdb instance.</param>
		/// <returns>A boolean value that communicates the client that the couchdb instace was started.</returns>
		[OperationContract]
		bool StartCouchdbInstance(CouchdbInstanceStartUpData startUpData);
 
		/// <summary>
		/// This method allows a client to stop the current instance of couchdb. It is recommended to just 
                /// allow the desktopcouch python library 
		/// to take care os stoping the couchdb instance.
		/// </summary>
		/// <returns>A boolean value that communicates the client that the couchdb instance was stopped.</returns>
		[OperationContract]
		bool StopCouchdbInstance();
 
	}
 
	// Use a data contract as illustrated in the sample below to add composite types to service operations
	[DataContract]
	public class CouchdbInstanceStartUpData
	{
		#region Properties
 
		/// <summary>
		/// Gets and sets the full path of the ini file to be use to start the couchdb instance.
		/// </summary>
		[DataMember]
		public string IniFile { get; set; }
 
		/// <summary>
		/// Gets and sets the full path of the file that will be used by the stdout of the couchdb instance.
		/// </summary>
		[DataMember]
		public string OutputFile { get; set; }
 
		/// <summary>
		/// Gets and sets the full path of the file that will be used by the stderr of the couchdb instance.
		/// </summary>
		[DataMember]
		public string ErrorFile { get; set; }
 
		/// <summary>
		/// Gets and sets the full path of the file that will be used to write the logs of the couchdb instance.
		/// </summary>
		[DataMember]
		public string LogFile { get; set; }
 
		#endregion
 
	}
}

The implementation of the service is very straight forward.The most interesting part of the code is the one related with the port used by the process:

		#region Get port from PID
 
		#region Structures
 
		/// <summary>
		/// Enumerator that is used to list the different class for the TCP table.
		/// </summary>
		public enum TCP_TABLE_CLASS
		{
			TCP_TABLE_BASIC_LISTENER,
			TCP_TABLE_BASIC_CONNECTIONS,
			TCP_TABLE_BASIC_ALL,
			TCP_TABLE_OWNER_PID_LISTENER,
			TCP_TABLE_OWNER_PID_CONNECTIONS,
			TCP_TABLE_OWNER_PID_ALL,
			TCP_TABLE_OWNER_MODULE_LISTENER,
			TCP_TABLE_OWNER_MODULE_CONNECTIONS,
			TCP_TABLE_OWNER_MODULE_ALL,
		}
 
		/// <summary>
		/// Struct used to show the information of the woner of a TCP connection.
		/// </summary>
		[StructLayout(LayoutKind.Sequential)]
		public struct MIB_TCPROW_OWNER_PID
		{
			public uint State;
			public uint LocalAddr;
			public byte LocalPort1;
			public byte LocalPort2;
			public byte LocalPort3;
			public byte LocalPort4;
			public uint RemoteAddr;
			public byte RemotePort1;
			public byte RemotePort2;
			public byte RemotePort3;
			public byte RemotePort4;
			public int OwningPid;
		}
 
 
		[StructLayout(LayoutKind.Sequential)]
		public struct MIB_TCPTABLE_OWNER_PID
		{
			public uint dwNumEntries;
			MIB_TCPROW_OWNER_PID table;
		}
 
		#endregion
 
		// We use the funtion from the iphlapi dll to get the talbe f tcp connections.
		[DllImport("iphlpapi.dll", SetLastError = true)]
		static extern uint GetExtendedTcpTable(IntPtr pTcpTable, ref int dwOutBufLen, bool sort, int ipVersion, TCP_TABLE_CLASS tblClass, int reserved);
 
 
		// Returns an array with the rwos with the dta of the different connecitons.
		private static MIB_TCPROW_OWNER_PID[] GetAllTcpConnections()
		{
			MIB_TCPROW_OWNER_PID[] tTable;
			// IP_v4
			var AF_INET = 2;    
			var buffSize = 0;
 
			// how much memory do we need?
			var ret = GetExtendedTcpTable(IntPtr.Zero, ref buffSize, true, AF_INET, TCP_TABLE_CLASS.TCP_TABLE_OWNER_PID_ALL, 0);
			var buffTable = Marshal.AllocHGlobal(buffSize);
 
			try
			{
				ret = GetExtendedTcpTable(buffTable, ref buffSize, true, AF_INET, TCP_TABLE_CLASS.TCP_TABLE_OWNER_PID_ALL, 0);
				if (ret != 0)
				{
					return null;
				}
 
				// get the number of entries in the table
				var tab = (MIB_TCPTABLE_OWNER_PID) Marshal.PtrToStructure(buffTable, typeof(MIB_TCPTABLE_OWNER_PID));
				var rowPtr = (IntPtr) ((long) buffTable + Marshal.SizeOf(tab.dwNumEntries));
 
				// buffer we will be returning
				tTable = new MIB_TCPROW_OWNER_PID[tab.dwNumEntries];
 
				for (var index = 0; index < tab.dwNumEntries; index++)
				{
					var tcpRow = (MIB_TCPROW_OWNER_PID) Marshal.PtrToStructure(rowPtr, typeof(MIB_TCPROW_OWNER_PID));
					tTable[index] = tcpRow;
					rowPtr = (IntPtr) ((long) rowPtr + Marshal.SizeOf(tcpRow));
				}
 
			}
			finally
			{
				// Free the Memory
				Marshal.FreeHGlobal(buffTable);
			}
 
			return tTable;
		}
 
		#endregion

In order to be able to start a new Couchdb instance we use the Process class in C#. But this supposes a problem in the solution. If you start a process using the Process class the owner of the process will be the process that it started it, and that is not nice at all. Ideally we would like to have the process to be owned by the user account that started it. For now this is where I;m stuck :( but will probably have some more done regarding the start of the process and the move away from gnome-keyring.

Jan 25 10

Django and XML-RPC

by admin

Recently I have been working with Django and I wanted a simple way to execute a number of server side methods. As you can imaging from the title, I wanted to use XML-RPC from javascript to be able to execute the methods. If you google “django xml-rpc” the first page that will appear is this.

After reading the post I decided to give it a go. My idea was the following:

  • Set XML-RPC on Django following the post.
  • Us mimic js library to call the XML-RPC method.
  • Be happy :D

Unfortunately I got stock in between the first and second step for about 3 hours :( due to a small but in the example! If you look at the view that takes care of the XML-RPC you have:

def rpc_handler(request):
	"""
	the actual handler:
	if you setup your urls.py properly, all calls to the xml-rpc service
	should be routed through here.
	If post data is defined, it assumes it's XML-RPC and tries to process as such
	Empty post assumes you're viewing from a browser and tells you about the service.
	"""
 
	response = HttpResponse()
	if len(request.POST):
		response.write(dispatcher._marshaled_dispatch(request.raw_post_data))
	else:
		response.write("<b>This is an XML-RPC Service.</b><br>")
		response.write("You need to invoke it using an XML-RPC Client!<br>")
		response.write("The following methods are available:<ul>")
		methods = dispatcher.system_listMethods()
 
		for method in methods:
			# right now, my version of SimpleXMLRPCDispatcher always
			# returns "signatures not supported"... :(
			# but, in an ideal world it will tell users what args are expected
			sig = dispatcher.system_methodSignature(method)
 
			# this just reads your docblock, so fill it in!
			help =  dispatcher.system_methodHelp(method)
 
			response.write("<li><b>%s</b>: [%s] %s" % (method, sig, help))
 
		response.write("</ul>")
 
	response['Content-length'] = str(len(response.content))
	return response

Most of the code above is right except when we are processing the XML-RPC call. The main bug I had to deal with was related with the fact that the HttpResponse was returning text/html as a response instead of XML!! This is a problem since a number of libs in javascript do care about the mimetype (lame :( ). The way to solve it is easy, use the “mimetype” named parameter.

response = HttpResponse(mimetype="application/xml")
response.write(dispatcher._marshaled_dispatch(request.raw_post_data))
Jan 20 10

Skiing 2010

by mandel

So, this is yet another kinda personal blog post so that people can know me a little bit better. After about 10 years without skiing (Last time I went was when I was 16), I finally got the chance to go with a group of friends. To be honest I was not sure if I would remember anything at all, but it seems I do!

Ski 2010

Pictures by my friend Christopher (Have you meet the Standaert?)