Mar 21 12

Interesting issues with the PyQt twisted reactor

by mandel

On Windows Ubuntu One uses the twisted reactor to run the Qt UI. The main reason for this is that the IPC protocol that is used was written in twisted. This has been giving us a number of head aches like the one we experienced today.

The following code simply shows a dialog that will as the user for his proxy credentials and will store them in the key-ring of which ever platform is being used:

def main():
    """Main method used to show the creds dialog."""
 
    if sys.platform == 'win32':
        import qt4reactor
        qt4reactor.install()
        logger.debug('Qt reactor installed.')
 
    app = QApplication(sys.argv)
    args = parse_args()
    win = ProxyCredsDialog(domain=args.domain,
                           retry=args.retry)
    return_code = win.exec_()
    if sys.platform == 'win32':
        from twisted.internet import reactor
        reactor.run()
    sys.exit(return_code)

From the dialog the most interesting part is the one in which the credentials are stored:

@defer.inlineCallbacks
def _on_save_clicked(self, *args):
    """Save the new credentials."""
    username = unicode(self.ui.username_entry.text()).encode('utf8')
    password = unicode(self.ui.password_entry.text()).encode('utf8')
    creds = dict(username=username, password=password)
    try:
        logger.debug('Save credentials as for domain %s.', self.domain)
        yield self.keyring.set_credentials(self.domain, creds)
    except Exception, e:
        logger.exception('Could not set credentials:')
        self.done(EXCEPTION_RAISED)
    logger.debug('Stored creds')
    self.done(USER_SUCCESS)

And to give even more details, the following is what is used to spawn a thread to store the credentials on windows:

def set_credentials(self, app_name, cred):
    """Set the credentials of the Ubuntu SSO item."""
    # the windows keyring can only store a pair username-password
    # so we store the data using ubuntu_sso as the user name. Then
    # the cred will be stored as the string representation of the dict.
    return deferToThread(self.keyring.set_password, app_name, USERNAME,
                             dumps(cred))

A priori there is nothing wrong with the code, or is it? Doing an IRL test you will see that the credentials are never stored, what’s even more none of the deferreds are called. But why? In theory the qt reactor should be taking care of everything which includes the deferreds, the deferToThread and the execution of the ui.. well, it is not. When we look a little closer we can see that we use the exec_ method from the QDialog and this is the root of the bug. Lets put an example, the following is possible in Qt:

import sys
 
from PyQt4.QtGui import QApplication, QDialog
 
app = QApplication(sys.argv)
dialog = QDialog()
dialog.exec_()

As you can see we are launching the dialog but we did not execute the application, but why is that? The main reason is found in the implementation of exec in the QDialog class (this time in C++, ouch!):

int QDialog::exec()
{
     Q_D(QDialog);
     if (d->eventLoop) {
         qWarning("QDialog::exec: Recursive call detected");
         return -1;
     }
 
     bool deleteOnClose = testAttribute(Qt::WA_DeleteOnClose);
     setAttribute(Qt::WA_DeleteOnClose, false);
 
     bool wasShowModal = testAttribute(Qt::WA_ShowModal);
     setAttribute(Qt::WA_ShowModal, true);
     setResult(0);
 
     show();
 
     QEventLoop eventLoop;
     d->eventLoop = &eventLoop;
     (void) eventLoop.exec();
     d->eventLoop = 0;
 
     setAttribute(Qt::WA_ShowModal, wasShowModal);
 
     int res = result();
     if (deleteOnClose)
         delete this;
     return res;
 }

As you can see the implementation uses a QEventLoop, if you read the documentation you will noticed that using exec of the event loops and the default flag all the events will me processed by this event loop, which is not the main event loop, but a child one (small brain fuck here). This means that, due to the implementation of the qtreactor, when using a dialog exec_ (or better say a QEventLoop.exec method) the reactor main loop is not processing the events which means that all your deferreds, deferToThread etc.. will not be processed :(

Nevertheless there is a way to work around this bug in the qtreactor implementation (because is not the qt fault and it is well documented) which is doing the following workaround:

win = ProxyCredsDialog(domain=args.domain, retry=args.retry)
win.show()
win.finished.connect(exit_app)

The above code ensures that the events will be handeled by the reactor main loop and not by a child QEventLoop.

In summary, qtreactor is buggy and will give you problems.. but if you really have to use it (like we do) do remember this detail, DO NOT use exec.

Mar 2 12

Liceo frances – the little ones

by mandel

Here you have or little guys playing rugby:

Is in Spanish, sorry, but is worth watching :)

Feb 28 12

Generating a self-signed SSL certificate

by mandel

I have been writing some integration tests lately between Ubuntu One and proxies which use SSL certificates. The idea behind this tests was to be able to test that we deal correctly with those certificates that are not correct (notify the user, remember exceptions, etc..) For that I wrote this small function that I used to generate the certificates.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
import os
 
from socket import gethostname
 
from OpenSSL import crypto
 
 
def generate_self_signed_cert(cert_dir, is_valid=True):
    """Generate a SSL certificate.
 
    If the cert_path and the key_path are present they will be overwritten.
    """
    if not os.path.exists(cert_dir):
        os.makedirs(cert_dir)
    cert_path = os.path.join(cert_dir, 'squid.crt')
    key_path = os.path.join(cert_dir, 'squid.key')
 
    if os.path.exists(cert_path):
        os.unlink(cert_path)
    if os.path.exists(key_path):
        os.unlink(key_path)
 
    # create a key pair
    key = crypto.PKey()
    key.generate_key(crypto.TYPE_RSA, 1024)
 
    # create a self-signed cert
    cert = crypto.X509()
    cert.get_subject().C = 'UK'
    cert.get_subject().ST = 'London'
    cert.get_subject().L = 'London'
    cert.get_subject().O = 'Canonical'
    cert.get_subject().OU = 'Ubuntu One'
    cert.get_subject().CN = gethostname() if is_valid else gethostname()[::-1]
    cert.set_serial_number(1000)
    cert.gmtime_adj_notBefore(0)
    cert.gmtime_adj_notAfter(10 * 365 * 24 * 60 * 60) 
    cert.set_issuer(cert.get_subject())
    cert.set_pubkey(key)
    cert.sign(key, 'sha1')
 
    with open(cert_path, 'wt') as fd: 
        fd.write(crypto.dump_certificate(crypto.FILETYPE_PEM, cert))
 
    with open(key_path, 'wt') as fd: 
        fd.write(crypto.dump_privatekey(crypto.FILETYPE_PEM, key))
 
    return cert_path, key_path

I leave to the reader to modify the function to match their needs.

Feb 17 12

QArrow a widget similar to GtkArrow in PyQt

by mandel

Recently in Ubuntu One we have been working or using PyQt for the UI of our application so that we could keep a smaller code base. While doing the move I noticed that we needed to have a widget similar to GtkArrow and to my surprise there is not one.

The following is a small implementation fo such a widget that might help other people that are in need of it. Even maybe someone that cares enough will write it in C++ and will propose it to Qt, sorry but I don’t have the time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
 
from PyQt4.QtGui import QPainter, QStyle, QStyleOption, QWidget
 
 
class QArrow(QWidget):
    """Qt implementation similar to GtkArrow."""
 
   UP = 0
   DOWN = 1
   LEFT = 2
   RIGHT = 3
 
   def __init__(self, direction, parent=None):
       """Create a new instance."""
       QWidget.__init__(self, parent)
       if not direction in (QArrow.UP, QArrow.DOWN,
                 QArrow.LEFT, QArrow.RIGHT):
           raise ValueError('Wrong arrow direction.')
       self._direction = direction
 
    def paintEvent(self, event):
        """Paint a nice primitive arrow."""
        opt = QStyleOption()
        opt.initFrom(self)
        p = QPainter(self)
        if self._direction == QArrow.UP:
            primitive = QStyle.PE_IndicatorArrowUp
        elif self._direction == QArrow.DOWN:
            primitive = QStyle.PE_IndicatorArrowDown
        elif self._direction == QArrow.LEFT:
            primitive = QStyle.PE_IndicatorArrowLeft
        else:
            primitive = QStyle.PE_IndicatorArrowRight
        self.style().drawPrimitive(primitive, opt, p, self)

Took longer to think the correct way to do it than the actual coding, at the end is very simple.

Feb 16 12

JQuery accordion with check box.

by mandel

I have been playing around with jquery a little lately and I found that it is not very well documented how to add an input field like a checkbox inside the header of an accordion so that you can click on the header and the accordeon works as expected and the checkbox can be clicked. I’m not an expert in this things but I found the way to work around the issue:

The HTML:

1
2
3
4
5
6
7
8
9
10
11
12
<div id="accordion">
    <div>
        <h3 id='example'>
            <a href='#'>
                <label>Title<input class='header-checkbox' type='checkbox' /></label>
            </a>
        </h3>
        <div>
            <p></p>
        </div>
    </div>
</div>

The Javascript:

$('#example').find('input').click(
    function(e){
        e.stopPropagation();
    }
);

I’m sure this is very easy for any advance js guy, but since I googled and did not find it I though I would be a nice guy and post it here for the next guy.

Feb 13 12

Add a custom context menu in a WebView

by mandel

Yesterday I was working on a small UI that uses a WebKit.WebView to show certain information to the user. My idea yesterday was to try and show a native context menu on the view that does not have the default menu items that the WebView has, the only way I’ve found to this is a as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
from gi.repository import GObject, Gdk, Gtk, WebKit
from utils import get_data_file
 
 
class CustomView(Gtk.EventBox):
    """Main window used to show the ui and generate signals."""
 
    def __init__(self):
        """Create a new instance."""
        super(CustomView, self).__init__()
        self.set_events(Gdk.EventMask.BUTTON_PRESS_MASK)
        self._create_context_menu()
        self.scroll = Gtk.ScrolledWindow()
        self.scroll.show()
        self.view = WebKit.WebView()
        self.view.show()
        self.scroll.add(self.view)
        # load the html used for the main window, from that point
        # on we will let the html and js do the work and raise
        # signals via the title
        self.view.open(get_data_file(MAIN_WINDOW_HMTL))
        # lets no use the webkit context menu
        settings = self.view.get_settings()
        settings.set_property('enable-default-context-menu',
                              False)
        self.view.set_settings(settings)
        self.add(self.scroll)
        # connect the on button pressed event of the event box
        # so that we can add a context menu
        self.connect('button_press_event', self._on_button_press_event)
 
    def _create_context_menu(self):
        """Create the context menu."""
        self.menu = Gtk.Menu()
        delete_menu = Gtk.MenuItem("Delete Task")
        self.menu.append(delete_menu)
 
    def _on_button_press_event(self, widget, event):
        """Deal with the button press event."""
        if event.button == 3:
            self.menu.popup(None, None, None, None, event.button, event.time)
            self.menu.show_all()

Does any one out there know a better way to do this. I know that there is a “populate-popup” signal I can listen to but I cannot get my head around on how to do exactly what I want, which is remove all defaul menu items and add my own. And, does it make sense to have to do that for every popup signal?

Feb 7 12

Python at codemotion.es

by mandel

As the codemotion.es page states, codemotion is:

Codemotion es el evento que reunirá en España a técnicos, desarrolladores y estudiantes de todas las comunidades y lenguajes. Por primera vez se celebrará en España después de 5 años de éxito en Italia.

Which badly translats to:

Codemotion is the event that will gather technicians, developers and students of different communities and languages. For the first time the event will take place in Spain after 5 years in Italy

Python is within the languages that will take part of the events thanks to the python madrid group. The group will go over the different talk proposals in the next meeting and will try to get you the best talks possible. So if you wanna take part you can do the following

Lets try and make python a better know language in Spain!.

Feb 7 12

Get/Set proxy settings in Gnome with GObject instrospection

by mandel

With GObject introspection is very simple to set the settings of your system trough python. Fist, lets use the command line to find out our current settings:

gsettings list-recursively org.gnome.system.proxy

The following script allows you to retrieve the http proxy settings that you are currently using:

from gi.repository import Gio
 
def get_settings():
    """Get proxy settings."""
    http_settings = Gio.Settings.new('org.gnome.system.proxy.http')
    host = http_settings.get_string('host')
    port = http_settings.get_int('port')
    if http_settings.get_boolean('use-authentication'):
        username = http_settings.get_string('authentication_user')
        password = http_settings.get_string('authentication_password')
    else:
        username = password = None
    return host, port, username, password

Setting them is as easy as getting them:

from gi.repository import Gio
 
def set_settings(host, port, username=None, password=None):
     """Set proxy settings."""
     http_settings = Gio.Settings.new('org.gnome.system.proxy.http')
     http_settings.set_string('host', host)
     http_settings.set_int('port', port)
     if username is not None:
         http_settings.set_boolean('use-authentication', True)
         http_settings.set_string('authentication_user', username)
         http_settings.set_string('authentication_password', password)

This is not utterly complicated but I’m notice that there are not many examples out there, so there you go. There is no code there that can be considered hard but I’d like to point out that if you use the get_value method from the Settings object you will have to call the appropriate get_* method from the returned GVariant, that is:

host = http_settings.get_string('host')

is equal to the following:

host = http_settings.get_value('host').get_string()
Jan 17 12

ReadDirectoryChangesW, if you want it asynch use the correct flag

by mandel

I have had a nice riddle given to me by Tim-Erwin within the comments of Bitten in the ass by ReadDirectoryChangesW and multithreading which I really appreciate. His problem was with the following code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import win32file
import win32con
import win32event
import pywintypes
 
FILE_LIST_DIRECTORY = 0x0001
folder = r"C:some folder to watch"
 
handle = win32file.CreateFile (folder, FILE_LIST_DIRECTORY,
              win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE,
              None, win32con.OPEN_EXISTING,
              win32con.FILE_FLAG_BACKUP_SEMANTICS, None)
buffer = win32file.AllocateReadBuffer(1024)
overlapped = pywintypes.OVERLAPPED()
overlapped.hEvent = win32event.CreateEvent(None, 0, 0, None)
flags = win32con.FILE_NOTIFY_CHANGE_FILE_NAME |
                  win32con.FILE_NOTIFY_CHANGE_DIR_NAME |
                  win32con.FILE_NOTIFY_CHANGE_ATTRIBUTES |
                  win32con.FILE_NOTIFY_CHANGE_SIZE |
                  win32con.FILE_NOTIFY_CHANGE_LAST_WRITE |
                  win32con.FILE_NOTIFY_CHANGE_SECURITY
 
win32file.ReadDirectoryChangesW(handle, buffer, False, flags, overlapped)
print "never reached until a change is done in the folder"
win32event.WaitForSingleObject(overlapped.hEvent, 5000)

As Tim-Erwin said in his comments, the above code was not being blocked in the WaitForSingleObject call but in the ReadDirecotryChangesW one. This is clearly my fault because in the blog post I did not give a very important detail that is needed to get the code working async. In order for the above to work correctly it is imperative that you pass to the CreateFile function the FILE_FLAG_OVERLAPPED flag. That means, that in order to fix the code and be blocked in WaitForSingleObject we have to modify the CreateFile call in the following way:

9
10
11
12
handle = win32file.CreateFile (folder, FILE_LIST_DIRECTORY,
              win32con.FILE_SHARE_READ | win32con.FILE_SHARE_WRITE,
              None, win32con.OPEN_EXISTING,
              win32con.FILE_FLAG_BACKUP_SEMANTICS | win32file.FILE_FLAG_OVERLAPPED , None)

That fixes Tim-Erwin problem and should help any other person with the same issue. Sorry for not being explicit on such an important detail Tim-Erwin, please accept my apologies :) .

Jan 10 12

Adding an alternative storage pool

by mandel

This is here for me to remember the next time I need to do this task:

  1. Copy the default pool definition:

    virsh pool-dumpxml default > pool.xml
  2. edit pool.xml changing the following vars:

    <pool type='dir'>
      <name>{$name}</name>
      <uuid>{$id}</uuid>
      <capacity>43544694784</capacity>
      <allocation>30412328960</allocation>
      <available>13132365824</available>
      <source>
      </source>
      <target>
        <path>{$path}</path>
        <permissions>
          <mode>0700</mode>
          <owner>-1</owner>
          <group>-1</group>
        </permissions>
      </target>
    </pool>
  3. virsh pool-create pool.xml
  4. virsh pool-refresh name

Doing the above you can add a new pool, for example one that is not in you ssd.