Desktopcouch on Windows

by mandel on January 25th, 2010

One of my “projects” for the xmas vacations has been to port Desktopcouch to Windows. I have twothree main reasons for this:

  • I want to be able to sync my data between my Windows machine (mainly my machine at the office) and my Linux machines (all the others)
  • I want to be able to make my different applications as multiplatform as possible.
  • Should be fun.

In this post I’ll try explain the different changes that I had to be make in desktopcouch in order to be able to port it to Windows.

Changing how couchdb is started

As most of you know, desktopcouch allows to start a couchdb per user in the machine allowing applications to use it as its data storage. The starting of couchdb in quite straight on Linux. We just have to find the couchdb command line, and pass as parameter the .ini files to be used. The current code that does that is the following:

Getting the couchdb command

COUCH_EXE = os.environ.get('COUCHDB')
if not COUCH_EXE:
    for x in os.environ['PATH'].split(':'):
        if os.path.exists(os.path.join(x, 'couchdb')):
            COUCH_EXE = os.path.join(x, 'couchdb')
if not COUCH_EXE:
    raise ImportError("Could not find couchdb")

Starting the database

def run_couchdb(ctx=local_files.DEFAULT_CONTEXT):
    """Actually start the CouchDB process.  Return its PID."""
    pid = read_pidfile(ctx)
    if pid is not None and not process_is_couchdb(pid):
        print "Removing stale, deceptive pid file."
        os.remove(ctx.file_pid)
    local_exec = ctx.couch_exec_command + ['-b']
    try:
        # subprocess is buggy.  Chad patched, but that takes time to propagate.
        proc = subprocess.Popen(local_exec)
        while True:
            try:
                retcode = proc.wait()
                break
            except OSError, e:
                if e.errno == errno.EINTR:
                    continue
                raise
        if retcode < 0:
            print >> sys.stderr, "Child was terminated by signal", -retcode
        elif retcode > 0:
            print >> sys.stderr, "Child returned", retcode
    except OSError, e:
        print >> sys.stderr, "Execution failed: %s: %s" % (e, local_exec)
        exit(1)
 
    # give the process a chance to start
    for timeout in (0.4, 0.1, 0.1, 0.2, 0.5, 1, 3, 5):
        pid = read_pidfile(ctx=ctx)
        if pid is not None and process_is_couchdb(pid):
            break
        time.sleep(timeout)
 
    # Loop for a number of times until the port has been found, this
    # has to be done because there's a slice of time between PID being written
    # and the listening port being active.
    for timeout in (0.1, 0.1, 0.2, 0.5, 1, 3, 5, 8):
        try:
            port = desktopcouch.find_port(pid=pid, ctx=ctx) # only returns valid port
            break
        except RuntimeError, e:
            pass
        time.sleep(timeout)
 
    ctx.ensure_files_not_readable()
    return pid, port

Of course this is not that easy on Windows. If anyone has played around with couchdb on windows, specially the with the binary package, you will find that in order to start the database you have to have a batch file doing something like this (I borrowed the batch from the couchdb binary package):

@echo off
rem Licensed under the Apache License, Version 2.0 (the "License"); you may not
rem use this file except in compliance with the License. You may obtain a copy
rem of the License at
rem
rem   http://www.apache.org/licenses/LICENSE-2.0
rem
rem Unless required by applicable law or agreed to in writing, software
rem distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
rem WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
rem License for the specific language governing permissions and limitations
rem under the License.

setlocal
rem First change to the erlang bin directory
cd %~dp0

rem Allow a different erlang executable (eg, werl) to be used.
if "%ERL%x" == "x" set ERL=erl.exe

echo CouchDB 0.10.0 - prepare to relax...
%ERL% -smp auto -sasl errlog_type error ^
      -eval "application:load(crypto)" ^
      -eval "application:load(couch)" ^
      -eval "crypto:start()" ^
      -eval "couch_server:start([""../etc/couchdb/default.ini"", ""../etc/couchdb/local.ini""]), receive done -> done end."

As you can image this is not the type of command that is found on Linux, and therefore we have to make a better solution.

One of the dangers that a developers finds himself when porting an application like desktopcouch to Windows is the tendency to copy the design decitions used in Linux. Linux and Windows have complete different programming models and we should try to make the port as native as possible.

If I were to design desktopcouch from scratch to work on Windows I would do the following:

  1. Create a Windows Service that will take care of the starting and stopping of the different couchdb processeses. This service will take care of the management of the different processes as well as allowing other applications querying the system.
  2. Provide an API to query the current couchdb process for the user and retrieve the required data.

Windows Service Implementation

There are two different ways to implement a service that can be used to manage the couchdb instances:

  • A simple Windows Service (NT Service)
  • A Windows service that uses WCF service.

When implementing the service we have to think about what kind of applications we want to be able to communicate with it. On one hand we have the traditional Windows Service that can be easily implemented and any application written in the CLR can easily communicate with it. On the other hand, we can use a more “fancy” implementation with WCF. WCF allows to implement a service that can easily use different protocols for communication with not too much effort. I’d say that in our situation we prefer to use a WCF since we would like as many different languages and systems to be able to communicate with the service and not only C# (what about python!).

Some people might complain because we are using SOA, but SOA does not only involve Web Services is a broader point of view. In this example we could argue that being able to start a couchdb instance and returns its port can be identified as a service. By creating a new service I’m trying to achieve two different things:

  • Isolate as much diff code as possible from the Linux solution
  • Provide a interop interface that can be used to access the couchdb instance from any language.

In summary, in order to simplify the creation of couchdb instance in a Windows environment we will create a Windows Service that uses WCF to allow the different client applications to contact the service and request the port and auth of the db and start it if necessary.

The following code shown the service contract that will be used to allow client applications to query information regarding the cuchdb instance for the user. In this step we are not looking at the user, that step is left for the auth part of the system we are just determining the contract.

using System.Runtime.Serialization;
using System.ServiceModel;
 
namespace DesktopcouchServie
{
	/// <summary>
	/// Basic service contract that provides a number of methods that client applications can execute to query the information 
	/// of the couchdb instance of the current user.
	/// </summary>
	[ServiceContract]
	public interface IDesktopcouchService
	{
		/// <summary>
		/// This method allow a client application to query the port in which the couchdb instance is executing. 
		/// </summary>
		/// <returns>The post in which the couchdb instance can be found.</returns>
		[OperationContract]
		string GetCouchdbPort();
 
		/// <summary>
		/// This method allows a client to starta couchdb instance for the current user. It is recommended to just allow the
                /// desktopcouch python
		/// library to take care of the start of the instance.
		/// </summary>
		/// <param name="startUpData">The data to be used to start the couchdb instance.</param>
		/// <returns>A boolean value that communicates the client that the couchdb instace was started.</returns>
		[OperationContract]
		bool StartCouchdbInstance(CouchdbInstanceStartUpData startUpData);
 
		/// <summary>
		/// This method allows a client to stop the current instance of couchdb. It is recommended to just 
                /// allow the desktopcouch python library 
		/// to take care os stoping the couchdb instance.
		/// </summary>
		/// <returns>A boolean value that communicates the client that the couchdb instance was stopped.</returns>
		[OperationContract]
		bool StopCouchdbInstance();
 
	}
 
	// Use a data contract as illustrated in the sample below to add composite types to service operations
	[DataContract]
	public class CouchdbInstanceStartUpData
	{
		#region Properties
 
		/// <summary>
		/// Gets and sets the full path of the ini file to be use to start the couchdb instance.
		/// </summary>
		[DataMember]
		public string IniFile { get; set; }
 
		/// <summary>
		/// Gets and sets the full path of the file that will be used by the stdout of the couchdb instance.
		/// </summary>
		[DataMember]
		public string OutputFile { get; set; }
 
		/// <summary>
		/// Gets and sets the full path of the file that will be used by the stderr of the couchdb instance.
		/// </summary>
		[DataMember]
		public string ErrorFile { get; set; }
 
		/// <summary>
		/// Gets and sets the full path of the file that will be used to write the logs of the couchdb instance.
		/// </summary>
		[DataMember]
		public string LogFile { get; set; }
 
		#endregion
 
	}
}

The implementation of the service is very straight forward.The most interesting part of the code is the one related with the port used by the process:

		#region Get port from PID
 
		#region Structures
 
		/// <summary>
		/// Enumerator that is used to list the different class for the TCP table.
		/// </summary>
		public enum TCP_TABLE_CLASS
		{
			TCP_TABLE_BASIC_LISTENER,
			TCP_TABLE_BASIC_CONNECTIONS,
			TCP_TABLE_BASIC_ALL,
			TCP_TABLE_OWNER_PID_LISTENER,
			TCP_TABLE_OWNER_PID_CONNECTIONS,
			TCP_TABLE_OWNER_PID_ALL,
			TCP_TABLE_OWNER_MODULE_LISTENER,
			TCP_TABLE_OWNER_MODULE_CONNECTIONS,
			TCP_TABLE_OWNER_MODULE_ALL,
		}
 
		/// <summary>
		/// Struct used to show the information of the woner of a TCP connection.
		/// </summary>
		[StructLayout(LayoutKind.Sequential)]
		public struct MIB_TCPROW_OWNER_PID
		{
			public uint State;
			public uint LocalAddr;
			public byte LocalPort1;
			public byte LocalPort2;
			public byte LocalPort3;
			public byte LocalPort4;
			public uint RemoteAddr;
			public byte RemotePort1;
			public byte RemotePort2;
			public byte RemotePort3;
			public byte RemotePort4;
			public int OwningPid;
		}
 
 
		[StructLayout(LayoutKind.Sequential)]
		public struct MIB_TCPTABLE_OWNER_PID
		{
			public uint dwNumEntries;
			MIB_TCPROW_OWNER_PID table;
		}
 
		#endregion
 
		// We use the funtion from the iphlapi dll to get the talbe f tcp connections.
		[DllImport("iphlpapi.dll", SetLastError = true)]
		static extern uint GetExtendedTcpTable(IntPtr pTcpTable, ref int dwOutBufLen, bool sort, int ipVersion, TCP_TABLE_CLASS tblClass, int reserved);
 
 
		// Returns an array with the rwos with the dta of the different connecitons.
		private static MIB_TCPROW_OWNER_PID[] GetAllTcpConnections()
		{
			MIB_TCPROW_OWNER_PID[] tTable;
			// IP_v4
			var AF_INET = 2;    
			var buffSize = 0;
 
			// how much memory do we need?
			var ret = GetExtendedTcpTable(IntPtr.Zero, ref buffSize, true, AF_INET, TCP_TABLE_CLASS.TCP_TABLE_OWNER_PID_ALL, 0);
			var buffTable = Marshal.AllocHGlobal(buffSize);
 
			try
			{
				ret = GetExtendedTcpTable(buffTable, ref buffSize, true, AF_INET, TCP_TABLE_CLASS.TCP_TABLE_OWNER_PID_ALL, 0);
				if (ret != 0)
				{
					return null;
				}
 
				// get the number of entries in the table
				var tab = (MIB_TCPTABLE_OWNER_PID) Marshal.PtrToStructure(buffTable, typeof(MIB_TCPTABLE_OWNER_PID));
				var rowPtr = (IntPtr) ((long) buffTable + Marshal.SizeOf(tab.dwNumEntries));
 
				// buffer we will be returning
				tTable = new MIB_TCPROW_OWNER_PID[tab.dwNumEntries];
 
				for (var index = 0; index < tab.dwNumEntries; index++)
				{
					var tcpRow = (MIB_TCPROW_OWNER_PID) Marshal.PtrToStructure(rowPtr, typeof(MIB_TCPROW_OWNER_PID));
					tTable[index] = tcpRow;
					rowPtr = (IntPtr) ((long) rowPtr + Marshal.SizeOf(tcpRow));
				}
 
			}
			finally
			{
				// Free the Memory
				Marshal.FreeHGlobal(buffTable);
			}
 
			return tTable;
		}
 
		#endregion

In order to be able to start a new Couchdb instance we use the Process class in C#. But this supposes a problem in the solution. If you start a process using the Process class the owner of the process will be the process that it started it, and that is not nice at all. Ideally we would like to have the process to be owned by the user account that started it. For now this is where I;m stuck :( but will probably have some more done regarding the start of the process and the move away from gnome-keyring.

From Python

Leave a Reply

You must be logged in to post a comment.