Thoughts on Process Invocation with Qt and PoCo

Posted June 9, 2011 by Tyson Key
Categories: Everything

Tags: , , , , , , , , , , , , ,

Since late 2010, I’ve enjoyed developing C++-based applications with the Qt framework. Overall, I find it to be intuitive, fairly well-designed, and well-documented.

However, one area of the framework that I’ve found unsatisfactory (or at least struggled with, despite my best efforts) is the QProcess class. Ideally, I’d like to be able to instantiate it once, on a per-method (or per-class) basis with the appropriate executable path and CLI arguments, and then dump its output directly into a QString for use elsewhere.

I vaguely recall successfully resolving ~90% of the problem in Stroma (although that project’s architecture is currently rather monolithic) – but I never managed to solve the rest of it (actually dumping the output of the invoked process into a QString/multi-line text box widget).

After taking a break from that project for a while, I found myself encountering a similar problem in another project; and after reading various forum/mailing list posts and pieces of documentation, sought to find alternative means of resolution.

The most promising initial candidate was Boost::Process, although it appears that development is still ongoing – therefore, it isn’t an “official” Boost component yet.

With that I mind, I decided to install and investigate Poco::Process, after seeing a mention of it on StackOverflow, last night.

Installation under Fedora was trivial (involved running “sudo yum -y install poco-devel poco-doc” from a shell), and the appropriate header files were conveniently located in /usr/include/Poco for later perusal.

That said, I was sceptical about the quality and usability of the PoCo library itself, since the documentation and sample code felt relatively terse and incomplete; not to mention that guesswork (aided by this post) was necessary, when it came to actually integrating it into the build system.

However, once those hurdles are dealt with, basic integration with the Qt project build system simply involves appending the following to the project file:

#Headers for PoCo
HEADERS += /usr/include/Poco/Pipe.h /usr/include/Poco/Process.h \
/usr/include/Poco/PipeStream.h \
/usr/include/Poco/StreamCopier.h

#Libraries for PoCo
LIBS += -lPocoFoundation

Once the project file is configured, adding the following to your class header file (modulo existing references, of course) should work :

#include <QApplication>
#include <QString>
#include <QDebug>

#include <string>
#include <cstring>
#include <cstdio>
#include <iostream>
#include <Poco/Process.h>
#include <Poco/Pipe.h>
#include <Poco/PipeStream.h>
#include <Poco/StreamCopier.h>

using namespace Poco;
using Poco::Process;
using Poco::Pipe;
using Poco::PipeInputStream;
using Poco::PipeOutputStream;
using Poco::ProcessHandle;

Some of the aforementioned header files are unnecessary – although their presence doesn’t seem to cause any obvious problems at compilation or application execution time.

My method in question (a rather rudimentary/brute-force mechanism for returning the MIME type of a file as a QString object, based upon sample code from a presentation slide), looks like: 
QString FileTypeHandler::GetMimeType(QString aFileName) {

    qDebug() << "Inside FileTypeHandler::GetMimeType";
    qDebug() << "Have text: " << aFileName;

    std::string mimeCommand("/usr/bin/file");
    std::vector<std::string> mimeArgs;
    mimeArgs.push_back("-i");
    mimeArgs.push_back(aFileName.toStdString());
    Poco::Pipe mimeOutput;
    ProcessHandle mimeHandle = Process::launch(mimeCommand, mimeArgs, 0, &mimeOutput, 0);
    Poco::PipeInputStream mimeInput(mimeOutput);

    std::string mimeData;

    mimeInput >> mimeData;

    qDebug() << QString::fromStdString(mimeData);

    return QString::fromStdString(mimeData);

}

Theoretically/in essence, it accepts a file name as its argument, invokes the file -i command to (hopefully) look up its MIME type, stores it in a standard ANSI C++ string, and finally returns it as a QString for consumption by UI widgets.

In reality, it pretty much does just that – with the caveat that it doesn’t quite return the correct output. (It returns “/dev/null:” instead of “/dev/null: application/x-character-device; charset=binary“, for example).

Still, I hope that these notes are useful for others who are facing the same problem…

Maybe others can chime in with a better alternative, or other suggestions?

Contributing an AT Commands Dissector to Wireshark

Posted May 11, 2011 by Tyson Key
Categories: Everything

Tags: , , , , , , , , ,

Whilst working on enhancing the Wireshark ISI dissector to add support for the USB encapsulation thereof, and a number of new resource dissectors, I felt that it would be useful to cleave off a generic dissector framework based upon the main packet-isi.c source file.

Whilst I never actually committed and released that generic code, I decided to hack up an AT commands dissector plug-in based upon it (plus some quick-and-dirty hacks to the main Makefile), and left it alongside the ISI dissection code in my BitBucket repository for a while…

That code was mostly written “in anger”, as a means of identifying and filtering out the uninteresting AT commands traffic from other, more interesting stuff; and it generally served its purpose well – although it was extremely rudimentary (due to unfamiliarity with certain APIs and uncertainty over the best way to handle strings of an unknown length).

For example:

  • The initial version didn’t support displaying commands text in Wireshark’s “Info” column.
  • Because I only had access to trace files containing AT commands packets where the text was wrapped in CRLF bytes, the initial heuristics used to detect these packets were rather weak.

Still, despite those limitations, I signalled my intentions on the 26th of April to contribute the aforementioned dissector on the Wireshark developers’ mailing list; and two days later, I decided to file Bug #5868.

Shortly afterwards, Chris Maynard chimed in with some suggestions on ways of improving the code – which I implemented over the course of a few hours; and I ended up submitting copies of my modified versions of various source files that the new dissector relied upon.

On the 5th of May, I managed to stumble upon a trace file containing an AT commands packet sans the CRLF encapsulation – and before I could even finish addressing the suggestions that were made earlier, Chris stormed ahead and delivered an enhanced version of the dissector, a few hours later.

As a prerequisite, an updated version of the the patch from Bug #4814 (which implements support for dissection of USB-encapsulated MPEG-2 Transport Stream packets, and generic infrastructure for USB Bulk URB heuristics support) was finally committed.

Not too long afterwards, Chris checked in his modified version of the dissector (in SVN revision 37045, and slightly updated in 37046) – which meant that my first ever upstream submission to Wireshark was complete. 🙂

(As an aside, some additional fixes have been performed; and the old Ethernet CDC dissector was finally committed last night).

Here We Go Again

Posted January 13, 2011 by Tyson Key
Categories: Everything

Tags: , , , , , , , , , , , , , , ,

As I compose this post, I realise that I’m extremely fortunate to have made it this far through life – especially when considering others living in developing countries, for instance.

After all – although I haven’t got the support of a wealthy, stable family, I’ve still got:

  • Food and potable drinking water
  • Heating, electricity and other necessities
  • A dry roof over my head
  • Broadband Internet connectivity
  • Good friends, and a handful of family members who mean well – even if I don’t always agree with them

In just over 6 months from now, I’ll have reached the 2 decades old milestone – which is somewhat worrying to contemplate; although I’m cautiously excited about future possibilities.

With that in mind, I’d like to reflect on the happenings of 2010, and the beginning of 2011.

In many aspects, 2010 was just another unspectacular, run-of-the-mill year – a monotonic continuation of 2009, to be blunt; although it brought change and progress in many ways.

However…

From a positive perspective, it was a good year for academia, software development, travelling, and personal relationships, amongst other things.

  • I was able to return to London, shortly after my 19th birthday in order to spend some time volunteering at the Symbian Foundation – details of what I did are available as part of my LinkedIn profile.
  • I received a number of references from several people, which were fairly useful (thanks!)
  • I finally managed to obtain a part-time, intensive placement on a 4-5 year long Computer Science course at the University of Bradford – and completed my first semester, shortly before Christmas 2010.
  • I learned the fundamentals of Java, and managed to write a number of C++-based applications using Qt – some of which I published the source code for on BitBucket.
  • Towards the end of 2010, I released a modified version of Sebastian Reichel’s ISI dissector for Wireshark with support for USB-encapsulated packets. I have since refactored the USB handling code and integrated it into the main dissector, in addition to writing new dissectors for the SIM, GSM Stack Server and Supplementary Services resources; and worked with Sebastian on incorporating these changes into his version successfully.
  • I also managed to reconnect with several people whom I haven’t heard from in a while.

But…

From a negative perspective, it was a bad year for older personal projects, family and financial-related issues, injuries, and the Symbian Foundation.

  • The server hosting DNS records for one of my domains (house404.co.uk) and Web services for several projects, which Sjors Gielen generously provided access to for several years finally succumbed to hardware failure – so I’ve lost some old data, some of which was of dubious utility, and some of which was fairly useful.
  • In November, I was unfortunate enough to have been involved in a hit-and-run traffic accident, whilst returning home from the supermarket in Boroughbridge. Thankfully, I sustained only minor injuries (from which I later fully recovered); although the suspect was never identified, after filing a police report.
  • In December, as a result of the harsh realities of the current economic climate, and decisions from handset manufacturers to slowly withdraw from the Symbian Foundation, the decision was made to effectively cease operations – which left community members such as myself to pick up the pieces.

I remain pessimistically hopeful that things improve in 2011.

Thanks to everyone who’s helped in various ways; provided advice and interesting discussion points; and otherwise persisted with me so far.

Hopefully, I’ve been useful to others in some way, too – and I’m glad, if that’s the case.

An ISI/PhoNet-over-USB dissector for Wireshark

Posted December 25, 2010 by Tyson Key
Categories: Everything

Tags: , , , , , , , , , , , , ,

Whilst working on Project Iris, I have found Sebastian Reichel‘s Wireshark dissector plug-in invaluable for identifying the content of ISI packets generated by my handset.

However, it relies upon the ability to use the Linux PhoNet stack – which isn’t always possible under certain circumstances.

For example, the stack may not be available at all under the running Linux kernel version; or the USB device generating ISI traffic may be connected to a virtual machine running a Windows-based application – which is obviously invisible to the host’s network stack.

With that in mind, I’ve decided to release a modified version of the aforementioned plug-in on BitBucket (in source code form only, at present), and I’ve uploaded a sample trace file to test it against, here.

Rough instructions for building it against an SVN release version of Wireshark under Fedora are provided in the repository; as are a copy of my colouring rules for working with USB and ISI traffic.

At present, the dissector has the following features:

  • Basic support for dissection of ISI/PhoNet packets encapsulated in USB framing (AKA “CDC PhoNet”) – for USB CDC_DATA class packets
  • Basic support for dissecting ISI GPS and SIM Authentication packets (inherited from the original version of the dissector)
  • Basic support for identifying specific types of CDC_DATA packets (works for ISI, PPP and AT/Hayes commands)

However, there are also a number of limitations and bugs – especially when compared to the original version:

  • ISI packets encapsulated in Linux Cooked framing are currently unsupported
  • Due to lack of heuristics, this dissector will override the PPP dissector (and the ISO/IEC 13818-1 dissector) when working with USB trace files
  • The length indicator may not always be accurate – although a lot of effort was spent on attempting to make it work

When working with this dissector, I recommend either using the isi.usbtype == 0x1b display filter, or individually filtering out various other types of USB packets, in order to avoid confusion.

For curious folks, a screenshot of the dissector in action is provided:

I hope that others find this useful for something.

That aside, I’d like to thank the following:

  • Chris Maynard for his USB patches (especially the CDC Ethernet one), which were useful for figuring out how to integrate with the USB dissector
  • Sebastian for providing the initial version of the dissector
  • William Roberts for providing the Nokia N73 that’s serving me well as my primary handset (and its USB cable, of course), and for persisting with me whilst I grappled with various stupid mistakes during learning C and C++

I wish readers a happy Christmas, and all of the best for 2011! 🙂

Lighting the Mines

Posted November 29, 2010 by Tyson Key
Categories: Everything

Tags: , , , , , , , , , ,

With the recent news of the Symbian Foundation board of directors deciding to gradually reduce the scope of the organisation to that of a mere licensing body, and talk of the community Websites closing imminently, there has understandably been somewhat of an uproar within this small-yet-vocal community.

A case in point being a post and poll from Julien Fourgeaud on his blog, which spawned this frantic discussion on the DevCo mailing list.

Despite reassurances of plans to make content available to the public (including mailing list posts if desired, from what I’ve been told), there’s still a great deal of uncertainty over what the future holds for not only the codebase and supporting resources, but for the incredible community that has coalesced around it over the past 2 years or so.

Because I consider the Symbian Platform codebase to be a rich, valuable resource for learning from and working with; and believe that it deserves to be publicly accessible in some way for both current developers, and for future generations, I have taken an interim step to preserve a large subset of it.

Over the past few days, I’ve started a Google Code project that initially aimed to preserve the “incubation project” repositories primarily, as they were the most vulnerable due to being transient in nature; although it now also hosts copies of the Kernel & Hardware Services, and Classic UI FCL repositories.

There’s obviously no point in listing every single repository here, although the majority of the “incubation project” ones should be covered; and I’ve also provided copies of various licensing documents for future reference.

With that in mind, I’ve kept things open-ended – because I don’t know how others will want to use these resources.

If others feel like working within the repositories and using the project’s wiki and issue tracker resources, I’m happy to open them up.

Of course, if others feel like downloading or cloning the repositories to fork them, I won’t complain – after all, it means that the codebase continues to spread in a healthy manner!

If there’s interest, I’m also willing to work with the Symbian Foundation team on a transition/temporary migration plan, whilst the situation with Nokia is resolved.

All aside, I hope that others enjoy using these resources as much as I did making them available; and I wish everyone involved during these difficult times the best of luck! 🙂

Project Iris: Affordable, Instant Connectivity for Syborg/QEMU

Posted November 1, 2010 by Tyson Key
Categories: Everything

Tags: , , , , , , , , , , , ,

Apologies for not updating here as often as I wanted – although in order to keep things concise, I won’t detail the reasons for my hiatus in this post.

That aside, whilst I can remember the details, I’d like to share a proposal for a novel (in my humble opinion – but I’m prepared to be corrected) method of potentially using unmodified, off-the-shelf Nokia handsets as a modem under Symbian OS running on QEMU.

Please note that I have so far been unable to implement this, or test certain individual components (e.g. the Linux PhoNet stack); although I believe from the research that I’ve done that individual components should work in isolation.

Additionally, this isn’t intended to be a competitor to the excellent Wild Ducks project, or the ad-hoc efforts surrounding getting regular modems utilising Hayes/AT commands to work, either. (It’s for folks who for whatever reason either can’t afford to acquire a fully fledged Wild Ducks set-up, don’t want to commit themselves for the long-term, or just want a quick-‘n’-dirty way to test stuff that requires network connectivity).

With that in mind, I’ll introduce the architecture diagram, and hopefully try to provide further details – because a picture is apparently worth a thousand words:

Click to view full size

The system itself consists of the following components, in no specific order:

  • A version of QEMU with customisations specific to the Symbian Platform, as detailed in my ancient post on the Symbian Blog – and a few others, since then!
  • Two brand new components, which will be described in further detail later (the TI SSI bus “pseudo-modem” and the raw PhoNet-to-SSI bridge)
  • The Linux PhoNet protocol stack, which was contributed to the mainline Linux kernel by Nokia on behalf of members of what was once known as the “Maemo Computers” department (if memory serves correct)
  • Your favourite Nokia device, providing that it supports USB connectivity and the “PC Suite” profile – since that’s how we can access certain baseband services via PhoNet! (A well-kept secret, so it seems)…
  • The Symbian Platform (which consists of the Symbian OS, UI framework, middleware and other components) and the baseport – Syborg, in the case of Project Iris
  • Nokia’s baseband “TSY” (telephony support plug-in), which should work in conjunction with a well-designed TI SSI bus “pseudo-modem” and the raw PhoNet-to-SSI bridge to simulate the presence of a real Nokia baseband by proxy 🙂

The most interesting components are the TI SSI bus “pseudo-modem” and the raw PhoNet-to-SSI bridge, which are pivotal to making this thing work.

The raw PhoNet-to-SSI bridge can potentially either be integrated into QEMU, or left standalone –  although designing the IPC mechanism for the latter use-case is left as an exercise for the reader.

Communication with the device could occur via either a /dev/phonet0 device node (if such a thing existed, but according to this IRC log, it seems that it doesn’t under certain circumstances), or directly bound low-level datagram/pipe sockets to communicate with the user’s handset via raw PhoNet/ISI packets encapsulated in USB frames.

Obviously, the raw PhoNet-to-SSI bridge will encapsulate and decapsulate PhoNet packets that are transmitted/received by the handset into Texas Instruments-proprietary SSI frames for consumption by the “pseudo-modem”.

The “pseudo-modem” works in conjunction with the Nokia TSY (as mentioned earlier) and the raw PhoNet-to-SSI bridge; and will be a brand new, integral component of QEMU. It has minimal state of its own; and other than creating the illusion of a genuine Nokia/TI modem’s presence, it serves solely to transport packets between the bridge and the TSY.

Finally, the interaction between the TSY, network and telephony stacks and other parts of Symbian OS are extensively documented elsewhere.

For those curious about the title, the “instant” bit refers to the fact that as of recent versions of the Linux kernel and NetLink stuff, things should Just Work™ when a PhoNet device is connected (according to this page and this presentation from 2009), and that limited hardware knowledge is necessary to use one – just plug it in and switch it on.

The “affordable” bit refers to the fact that Nokia devices are relatively low-cost, easy to obtain, and plentiful (unlike specialist hardware such as the BeagleBoard and standalone GSM modems – as great as they are, for example).

A Quick-ish Update

Posted August 23, 2010 by Tyson Key
Categories: Everything

Tags: , , , , , , , , ,

Since folks have asked nicely, I thought that I’d compose a quick post, whilst being in the process of making last-minute preparations for a trip to London.

The grand plan is that I’ll be stopping in York with a relative tonight; and then taking a coach down to London for roughly two days, in order to spend some time at the Symbian Foundation as part of an ongoing volunteering arrangement.

I probably shouldn’t discuss the agenda at this stage, although the arrangement thus far is that I’ll be meeting with either William or Sebastian on the 24th to continue working on enhancing the developer documentation, in addition to doing something (hopefully exciting!) with the build team.

On the 25th, I’ll have an opportunity to catch-up with others, and meet folks whom I haven’t already seen (including Victor); plus I might have time to do some sightseeing on the London Eye.

Finally, I also have some (mostly negative) news regarding my ongoing university application saga to share, upon my return.

Reverse-engineering an IrDA/USB Multiplexing Protocol : Part 1

Posted August 1, 2010 by Tyson Key
Categories: Everything

Tags: , , , , , , , , , , , , ,

Last month, I acquired a USB IrDA transceiver based upon a chipset from a seemingly defunct vendor (either the S110 or S110-SG1 from Tanic Electronics Ltd.), since it was fairly cheap (just under £12), and I had a spare phone with a washed-out display and an IrDA port sitting in my drawer to test it against.

Unfortunately, it seems that not only does this chipset/device not comply with the USB IrDA Bridge Device spec, but there is also no public data sheet, which makes writing a driver for non-Windows operating systems difficult. Theoretically, the Windows driver could be used under a modified version of NDISWrapper, or a similar NDIS API/ABI implementation – although that’s suboptimal in many ways (which should be obvious to some readers).

Those caveats aside, the device works fairly well when coupled with Windows XP’s primitive IrDA stack and associated utilities…

However, being the curious sort; and since the state of IrDA and USB debugging/tracing under Windows is dire, I decided to use Wireshark for analysis, coupled with the USB tracing functionality in recent versions of LibPCap and the Linux kernel – but in order to keep this post concise, I’ll provide more details in a follow-up post.

It’s the End of an Era

Posted June 25, 2010 by Tyson Key
Categories: Everything

Tags: , , , ,

Apologies for not updating things here as often as I’d have liked, since I’ve been very busy with other, more pressing issues as of late.

I also wanted to try and get this out sooner, rather than later – but I’ve been dealing with a cold, and trying to catch up on sleep; and ended up procrastinating as far as posting is concerned.

That aside, as of 23/06/2010, I’m no longer an FE student of Harrogate College – after spending roughly 3 years there.

But what does that mean for me, from a positive perspective?

  • I finally get a limited amount of time to relax, recover from stress, and contemplate future plans
  • As I now have more time to spare, I am now open to offers from people/organisations who may be interested in working with me
  • I can finally dedicate more time and attention to projects that I am involved with
  • I may even have slightly more time on my hands to blog – who knows?

On the flip-side, this now creates some brand-new issues for me:

  • I now have to reconsider my initial plans of attending my initial choice of university to study either computer science, or computer security; due to not quite managing to obtain 200 UCAS Points – which means that I’ll either have to plead my case, or go elsewhere
  • Due to the aforementioned problem, and the fact that I’m currently living in the middle of nowhere, and pretty much isolated from the rest of society; trying to find meaningful employment is next to impossible (I don’t care so much about pay, as much as not totally losing dignity, and going insane trying to cope with the menial crap involved with being a supermarket monkey, or a burger inversion specialist)
  • Since I can’t travel very far – at least not without being stranded somewhere else for ages, until another bus arrives, I’ll have to find some other way of occupying the next couple of months…

I hope that gives the curious an idea of what’s been going on, so far. Since this post is getting pretty long, I’ll discuss the college and university situation in greater detail in some follow-up posts.

Playing with WiTango Server 5.5 and Apache 2.2.15 under Windows XP

Posted March 28, 2010 by Tyson Key
Categories: Everything

Tags: , , , , , , , , ,

I don’t particularly have a need to use this stuff; and the WiTango software itself is prehistoric (the last release for Windows was made in 2006, and the last release for UNIX clones/relatives was made in 2005), but I felt like a challenge.

After obtaining and unpacking the approximately 9MB Witango_App_Server_for_Win_5.5.020.zip archive; I preceded to run the installer, and followed most of the instructions provided – stopping only to obtain a “Lite” licence key using a spare GAfYD GMail account, as well as a copy of the Apache installer.

During the installation process, I ended up having to answer “Other” to a prompt regarding HTTP daemons; since Apache wasn’t installed, and I’ve been burnt previously by the outdated and otherwise dodgy WiTango Apache modules (the Linux ones don’t even work with Apache 2.2 as supplied by Fedora, and most of the supporting tools in the Linux package are broken – but that’s by-the-by).

Eventually, I was greeted with a dialogue with rather useless (incomplete) information, and peeked inside the directory at C:\Program Files\WitangoServer\5.5, to discover a Witango.exe executable that did nothing whatsoever when invoked, a rough ReadMe file, and various other DLLs, configuration files and directories.

I then installed Apache – ensuring that it wasn’t started as a service, and operated on TCP port 8080 by default, before returning to WiTango’s ReadMe file to look for the magic httpd.conf lines that would supposedly make things work.

Said ReadMe file recommends using the following – which wouldn’t work, unless you copied a ton of DLLs around:

# Witango Apache Plug-in configuration

# This loads the Witango 5.5 client for Apache 2.2 to
# enable communication with the Witango Application Server
LoadModule WitangoModule modules/mod_witango55_apache22.so
WitangoModule mod_witango55_apache22.so
AddType application/witango-application-file taf tml thtml tcf

As an initial compromise, I resorted to appending the following to the bottom of the httpd.conf file, and restarting Apache:

#WiTango Bits

LoadModule WitangoModule "C:\Program Files\WitangoServer\5.5\PlugIns\mod_witango55_apache22.so"

WitangoModule mod_witango55_apache22.so

AddType application/witango-application-file .taf .tml .thtml .tcf

After creating an empty file at C:\Program Files\Apache Software Foundation\Apache2.2\htdocs\test.thtml, and navigating to http://localhost:8080/test.thtml in Chrome, I was greeted with this:

The lights are on, but no-one's home.

I resorted to altering the WitangoModule path to refer to the absolute path of the module’s DLL file (““C:\Program Files\WitangoServer\5.5\PlugIns\mod_witango55_apache22.so”” – a single pair of double quotes are used within httpd.conf), and restarting Apache yet again, only to receive the same “Client Error” message again.

According to the installation guide – which doesn’t even ship with the WiTango Server product, a Windows service (“Witango Server 5.5“) is supposed to be running (although it was never started in my case).

Unfortunately, the ReadMe never stated that, but I’m sure that it was obvious to the developers…

That aside, after starting the aforementioned service, I was greeted with this:

I guess that I can call it a success, given that I’m unfamiliar with the Tango/WiTango language, and haven’t got around to trying the associated developer tools yet…