Automated HCI testing: a research project aimed for use with KDE

Stupid wordpress. second time I’m writing this. OK, I would like to present my idea for the research project I will work on for a year from now.

Problem: Users don’t test usability in FOSS software enough. It’s too laborous and unrewarding.

Solution: Take advantage of the fact that many users enjoy trying out new features, but don’t go so far as to actually download, compile and test unreleased software. Build a solution that let’s them test unreleased software easily. Use technologies such as klik or glick to include all needed libraries in a single runnable binary. Allow for instrumenting the software with automated tests (for example, with a qt app, one could use a class that would attach to all qt buttons and menus and generate statistics on which buttons were used most, and how a certain task was performed, maybe even record all events and do some sort of a playback that the developer can observe). Distribute this instrumented software to a large group of users and generate a summary of the results that the developer can use for reference when (re)designing the software.

Implementation details: Since this is a one year research project, I aim at making a prototype of this solution, only implementing the needed parts as I go along. I do intend to write it so that the project can be continued afterwards as well. What I need from the community is help with testing this solution on a certain piece of software. What kind of software this would be is of course dependent on how I implement the prototype, but since I know KDE well I think choosing a KDE application would be a good choice. (Maybe even choose Ark, as some might remember that I also code on)

I have not yet finalized on this plan, it might change or even be replaced altogether. But either way I would like to know what the community thinks about the idea? Are there any obvious logical problems? Technical problems? Let me know in the comments.

9 Responses to “Automated HCI testing: a research project aimed for use with KDE”

  1. maninalift Says:

    Re: which app to choose, I would suggest not Ark. It is a great app but perhaps not so likely to attract enthusiastic testers, archiving isn’t as sexy as say graphics programs or media players.

    hmm… the ability for the user(tester) to attach comments, too… perhaps standardised things like, “I’m lost I cant find [text entry box]”. So add a menu dedicated to testing stuff like this

  2. jospoortvliet Says:

    Be sure to have a look at likeback: http://basket.kde.org/likeback.php
    At least some of the code and ideas might be useful for what you want to do. It’s pretty sophisticated…

  3. Harald Says:

    jospoortvliet:
    Thanks for the hint, LikeBack does indeed look interesting. I will consider using it.

  4. Richard Moore Says:

    You might find this tool I wrote a few years back worth mining for ideas:
    http://www.kdedevelopers.org/node/376

  5. Harald Says:

    @Richard:
    Thanks, that is actually one of the things I was already thinking about doing for the testing. I will take inspiration from that when the time comes.

  6. Ian Monroe Says:

    It would be really handy to have this information for Amarok. We’re always guessing how many people use a feature based on feedback from #amarok and similar. Or we might take a feature away and see if anyone complains.😀 Having real numbers would be nice, so good luck.

  7. seele Says:

    Metrics are useful to see if users are doing something (such as using a particular functionality), but not why. If you don’t what the user is thinking and why they did what they did, you can’t solve the problem. However, the data could help identify where a potential problem may be, and then futher investigation could gather the necessary data to diagnose it.

    In terms of design, I think gathering this type of data can be useful, but we have to be careful to not draw the wrong kind of conclusions from it.

  8. Ian Monroe Says:

    @seele We found this data very useful for the Amarok website. Websites are made the best case scenario for this sort of thing though, the “why” is always some variation of “find something”.

  9. randomguy3 Says:

    One thing to bear in mind is that privacy-nuts will not want to send in their usage data. And, depending on the application, you may find that there are very few people who want to divulge usage from it unless it’s clear that it won’t include personal information. Think of web browsers, where users will be doing online banking, for example.

Comments are closed.


%d bloggers like this: