Saturday, 5 December 2009

Using skypephone modem in Linux

I've been having a frustrating few days trying to get the 3G modem in my mobile phone to work with my Linux boxes. For anyone with similar issues, I'm documenting here what I did.

The phone I have is made by the Chinese company Amoi, marketed in the UK by 3 as a skypephone, currently version 2 of this. Basically, this was the cheapest 3G phone/modem I could find. 3 include 150MB of internet access in their topup charge, either used via the internet programs on the phone itself (there are various apps including a browser which uses Google Wireless Transcoder to try and render full-size webpages for the mobile form factor), or by using the phone as a modem. The modem can be used via either bluetooth or a usb cable provided with the phone. My Asus netbook does not have a bluetooth connection ootb, so I wanted to use usb.

Like most of these setups, the usb interface is heavily oriented to Windows users, to such an extent that it presents a cdrom drive which automatically installs Windows drivers when you initially plug the cable in. When you do so, the cdrom drive disappears and you can access the modem. As usual with Linux, there's no need for any drivers as these are already provided either in the kernel or as separate modules. So the main question is how to ignore the cdrom and access the modem. The best documentation I've found of this process, which varies depending on the device, is on the website for usb_modeswitch, a program which attempts to do this.

  1. The first thing to do is to switch on the phone and plug in the usb cable (a pitfall I fell into was that when you plug in the cable the phone immediately starts charging, which can give the impression that the phone is on even if it isn't - so make sure the phone's operating system is running).


  2. Then run lsusb on Linux. This should show the Amoi device, with an id of either 1614:1000 or 1614:0407. If the latter, you're in luck: that is the modem - proceed to the next step. If not, try unplugging the usb cable and plugging it back in again. With luck, it should now show 1614:0407. 'dmesg | tail' will show "Product: USB MMC Storage" for 1614:1000, or "Product: S2" for 1614:0407. I've found that you get different results depending on distro: according to the last post on this thread, both Ubuntu and Fedora find 0407 straightaway, my Debian testing/squeeze partition may need unplugging/replugging one or more times, and my Slackware partition doesn't seem able to find the modem at all, no matter how many times I replug the cable. Presumably, this is due to the way the kernel is configured, so if anyone can tell me what I have to configure on my Slackware kernel to enable this I'd be grateful. Fortunately, once the modem is set up it stays that way until you reinitialise the phone by switching it off or unplugging the cable, so once I have it set up in Debian I can then reboot in Slackware and it will then show as 1614:0407.


  3. Once you have 1614:0407, you can then load the driver. There are 2 possibilities: the old standard usbserial module and the newer option module, which it seems is now recommended for these kind of phone modems (tho IMHO it would be better if they renamed it).

    usbserial can be loaded with "modprobe usbserial vendor=0x1614 product=0x0407". option does not have the vendor/product (er) options, but thanks to a tip from usb_modeswitch, you can set them directly on sys bus:
    modprobe option
    echo "1614 0407" >/sys/bus/usb-serial/drivers/option1/new_id

    In both cases, udev then creates /dev/ttyUSB0 which pppd can use.


  4. For completeness, here is my pppd control file, which I put in /etc/ppp/peers/3:

    /dev/ttyUSB0
    115200
    crtscts
    idle 7200
    noipdefault
    defaultroute
    persist
    modem
    noauth
    nobsdcomp
    user three
    password three
    connect-delay 20000
    connect "/usr/sbin/chat -V -f /etc/ppp/chat3"

    I don't believe the modem speed has any effect for these devices, and I don't think user and password are used either.

    Here is /etc/ppp/chat3, the chatscript I use:
    ABORT        BUSY
    ABORT "NO CARRIER"
    ABORT ERROR
    REPORT CONNECT
    TIMEOUT 10
    "" ATZ
    OK 'ATE0V1Q0S0=0&C1&D2+FCLASS=0'
    OK 'AT+CGDCONT=1,"IP","three.co.uk"'
    SAY "Calling..."
    TIMEOUT 60
    OK 'ATDT*99#'
    CONNECT \c

    The important thing here is "three.co.uk", the APN, which is the important part of user validation; it will not work without this.


  5. The alert will notice that I do not have 'usepeerdns'. This is because, rather bizarrely, 3's nameservers do not appear to work. When I used 'usepeerdns' I got a connection and could do, for example, 'ping 74.125.47.103' (one of Google's servers, if you don't know) but no dns resolution was working. So I set up /etc/resolv.conf with OpenDNS's servers instead:
    nameserver 208.67.222.222
    nameserver 208.67.220.220


  6. You then start ppp with 'pppd call 3' and wait for the connection to be set up. You can monitor this from the logfile, /var/log/messages in the case of Slackware: when 'tail /var/log/messages' shows the local and remote IP addresses, the connection should be ready. On my setup, this takes about 30 seconds.


  7. When you have finished, you can kill pppd with "kill `cat /var/run/ppp0.pid`". And, as pppd also logs how much data transfer there was, you can do something like "cat /var/log/messages | grep 'bytes, received'" to find out how much of the 150MB you used during that connection.

Thursday, 12 November 2009

KDE4 multimedia in Slackware

Multimedia has always been a bit of a pain in Linux distros. It's riddled with patents and other restrictions, which free-software purists dislike. So even simple things like playing mp3 files often produce a lot of hassle. KDE3 had its own sound server, called arts, which in recent years has been unmaintained and generally unsatisfactory. KDE4 replaced this with a new multimedia framework called Phonon. This can work with different backends, of which the usual ones are gstreamer and xine.

In both Debian and Slackware, gstreamer is the default. When using xine in kde3, I used the Windows codecs from mplayer, but gstreamer uses its own plugins, of which there are several packages: gst-plugins-base includes quality open-source codecs like ogg and theora, gst-plugins-good adds others, including containers like flv and avi. The mpg codecs are in gst-plugins-ugly, i.e. those which are good-quality but have license/patent issues. Debian includes all these plugins somewhere in its repos, but Slack only has base and good, so you cannot use mp3 files with gstreamer in the standard Slack distro. Fortunately, ugly is available on slacky.eu which also includes the various libraries needed which are also not in standard Slack. Install all those and you can then listen to mp3 files in KDE programs like Juk.

Mind you, Slack does include mpg321 which enables you to play mp3 files on the command line, so you could set up a file association in kde to use mpg321 to play mp3 files.

Another of those areas where Linux distros create a lot of hassle for something that should be simple.

Back to Slack

In a previous post, I said that I was using Debian on my EEE. I have though become increasingly frustrated at the length of time it takes Debian to move programs from sid to testing. When I first installed Debian, I used the default option, which is stable (i.e. Lenny at the moment) with Gnome desktop. That was, however, not really what I wanted, so I went off and RTFM, and found that you can change this by using expert mode. So, I kept Lenny on his own partition, and installed KDE on another partition. I first tried with sid, but couldn't get X to load - gave me strange error msgs which I didn't have time to try and fathom - so I installed testing instead. In theory, programs are quickly moved from sid to testing unless there are major issues, but the new versions of KDE seem to have got stuck in sid. In the meantime, KDE have released 4.3.3, but Debian have still not moved 4.3.2 to testing, let alone 4.3.3. As Slackware made 4.3.3 available in current just a couple of days after the KDE folk released it, I decided to go back to using Slackware. In fact, I kept testing, and installed Slack on top of the Lenny partition, as I wasn't really using this for anything.

In principle, the big advantage of Debian is the vast package repos, which Slack doesn't have. However, there is now an excellent collection of additional Slackware packages at http://repository.slacky.eu/ I've so far found only 2 packages I use that are in Debian and not available for Slack: mapserver, and Google Gadgets for Linux, which can be used as KDE plasmoids. I could compile for Slack, but, if necessary, can also boot my testing partition and use that. There are a couple of curious little bugs in Slack I've not got to the bottom of yet: those icons that are in Hicolor but not in Oxygen are not displayed; and Ark seems unable to open tar.gz files (I think this is because a compile option was missing).

Thursday, 5 November 2009

Zoho remote api

In a previous blog on strategies for replacing the desktop, I wrote: "In theory, there is another alternative - data local, software on the web - but I never heard of anyone implementing that!"

Turns out that zoho.com's Remote API does precisely that. This enables you to keep all your data on your own server but use Zoho's editing program to change them. Piotr Malinski has example code to do that with Curl in Django or PHP

Monday, 2 November 2009

KDE and HTML5

I've just been looking at Mark Pilgrim's interesting pages on HTML5, especially his browser tests for various new features. Here's the results for the various browsers that I use on my KDE setup.

Edit: 14.11: I have now updated this to show Slackware Firefox as well as Debian's Iceweasel which appears to be far less capable. I've also included Arora (not surprisingly as they both use QtWebKit, this is the same as KDE plasmoid).











Konqueror (KHTML)Plasmoid/Arora (QtWebKit)Firefox (Gecko)Iceweasel (Gecko)Chrome (WebKit)
Canvasyyyyy
Canvas textnyyny
Videonny (Ogg Theora)ny (Ogg Theora and H.264)
Storagenyyny
Web workersnnyny
Offline appsnyyyy
Geolocationnnynn
Input typessearch, rangesearch, tel, url, email, number, range, color
Placeholder textnynny
Form autofocusnynny

Clearly, Konqueror is worst. After only testing Iceweasel I thought Gecko wasn't much better, but Firefox is far superior at these tests, so it looks like this is browser dependent, not browser engine. Firefox even manages geolocation, though I tested this via my ADSL link which is pretty useless as it gives the ISP's location not mine. The WebKit browsers also score well at these tests.

Centralising bookmarks

It's always the way that no sooner do you post something than you discover additional information. Since writing my previous post, I've discovered two attempts to provide exactly the kind of centralised bookmarks and password managers I've been looking for.

The first is Xmarks which provides exactly this kind of service, albeit only for Firefox (though Chrome is promised). This can sync both bookmarks and passwords. Although it's run by a private company and by default stores the data on their server, you're not tied to it - you can use your own WebDAV or ftp server to store the data, and you can export the stored bookmarks at any time. An additional nice feature is the ability to create 'sync profiles' , which enable you to limit certain bookmark folders to a particular computer.

Meanwhile, Google recently introduced their sync facility, though frustratingly this does not seem to be available on the Linux version as yet. This stores the bookmarks in with the Google account, though it seems they plan to open-source all this, so presumably could be done elsewhere too. It will also use the standard XMPP for syncing.

Now all that's needed is to persuade KDE to use this instead of the built-in bookmarks.

Friday, 30 October 2009

Farewell, desktop? Different strategies

Although I used to use my PC for my business, which required some specialised software, mostly my needs aren't that much different from everyone else: web browsing, email, documents, spreadsheets, news feeds, digital music - though I do also do a lot of web development, which requires its own development and server software.

Up until recently, like most people, I've been doing all this on my desktop PC. An obvious disadvantage to a desktop is that it's tied to one place; laptops and their various derivatives like netbooks remove this to some extent, as you can carry your data around with you, but inevitably there are situations where you cannot take your laptop/netbook and have to use another device. If you have data on some other device, such as a mobile phone, you have to do regular synchronisation with your desktop/laptop. Another disadvantage is that your data will be lost if you have a disk crash or if your computer is stolen. The traditional way of combating the latter problem is to make regular backups. USB sticks nowadays provide quite large storage capacities for little cost, and are the obvious solution for audio files and the like which don't change and don't have to be synchronised. An alternative is online storage - I use Humyo - as an offsite backup. This has the advantage that files can be shared, but I find it less convincing for large items like audio files which take up a lot of bandwidth to transmit to and fro. It also only addresses the question of data, not the programs needed to create/maintain/view/listen to that data.

For that you need software, and as with online backup of data a growing alternative to desktop software is web software. This means there are 3 basic strategies:
  1. data and software both on the local machine, be that desktop or laptop, mobile phone or whatever other gadgets manufacturers come up with
  2. data and software both on the web
  3. data on the web; software local
[In theory, there is another alternative - data local, software on the web - but I never heard of anyone implementing that!]

(1) is the traditional way. Google is the best-known exponent of (2), with applications such as Google Mail and Google Docs. It has the great advantage that Google looks after all the server infrastructure and programs; it's also free. One obvious disadvantage is the need to still use the data when there is no network link. Google offers Gears to get round this problem, and the storage abilities of HTML5 will probably replace this eventually.

There is also the security/privacy aspect: do I really want Google having responsibility for all my private correspondence? Although Google is unlikely to go bust in the foreseeable future, and is unlikely to abandon its Mail and Docs services either, there is also the problem that I have no control over what features or facilities they allow or introduce in the future. For someone used to the open-source way of doing things, where you can change any program you want whenever you want, this might be problematic.

There are plenty of alternatives to Google, the most extreme being to use one of the 'webtop' or 'web desktop' systems. These provide a pseudo-desktop on the web, giving you file space and a series of desktop-style applications. There is a list of these on Wikipedia and, like Google, they give you a central repository for your data, which you can access from anywhere. They cover a much wider range of software than Google, though, aiming to completely replace the desktop. However, they suffer the same disadvantages of tying you to someone else's vision, and many of them being start-ups without much obvious income, it's questionable how viable they are. Several have already disappeared.

One of the more interesting ones is the rather bizarrely-named G.ho.st, which runs under Flash; it's interesting as much for being a joint Israeli-Palestinian venture as for its software, about the only software company I might support for political reasons. It also includes the ability to access data on other servers, such as Google's, so you can store other material on G.ho.st, but still store your mail or docs on Google and access them through G.ho.st.

An interesting attempt to address the issue of non-viable business models is eyeOS, an open-source system in PHP/AJAX which you can download and run on your own PHP-capable server. This gives you a central repository but also means you can install and run, or change, any modules you like, resolving the "tied-to-someone-else's-vision" problem. However, you are then back with the problem of maintaining the installation and paying for the server infrastructure. eyeOS points out that you can also install the software on your own PC, but that seems to me to be missing the point!

Another system worth mentioning is OpenGoo, though this is geared to collaborative working, so is oriented to small companies and organisations rather than individuals.

Wikipedia's page lists speed as one of the disadvantages of this technology. This would certainly be the case with the webtops, which are slow to use. However, Google seem to have found a solution to this, as I have not found using Google Mail or Docs instead of desktops to be noticeably slower, at least on a reasonably capable broadband connection.

Google's total vision, though, is rather different from the webtop. It's proposing replacing the desktop operating system with its own cut-down version, which according to the sketchy details so far, will be based on a browser and probably not much else. So all the data would be on the web, and all the software would be browser-based, and heavily oriented to AJAX and similar web-service technologies. Like many other people, I look forward to hearing more details of this.

Meanwhile, back at KDE, their vision, as is emerging from their Project Silk, is closer to option (3). The browser, and HTTP in general, is not an ideal way of processing data. It's oriented to pages which are read, not to reading or changing discrete chunks of data. It also has no state, i.e. the server treats each page request separately, so there is no concept of a transaction comprising more than one request. To some extent, AJAX gets round this, and there are various projects to improve the basics of web applications, but it's still essentially mis-using the web. So, even if data is stored on the web, it can still be better to adapt existing desktop programs to read and process it. As long as there's a standard way of accessing data over HTTP - an API - an application, wherever it might be, can treat it just like a local file, so it doesn't really matter to the application whether a chunk of data is coming from a local or remote source. This is a basic insight that KDE has always had; Konqueror has never cared whether files are local or accessed via ftp or even secure ftp, or whether an html file is local or accessed via HTTP - it displays them in the same way, and in the case of files, can even edit them in the same way too. A Google spreadsheet, for example, can be read via an API, even to access a specific cell, so this can be done by any program that knows how to use the API; it does not have to be a spreadsheet or browser program.

The 3 strategies I've listed here are often presented as alternatives, but they don't have to be; in practice, there's no reason why you can't mix or use several at the same time. In fact, if you're using online services, it's better not to limit yourself to one but store your data in several places then, if one of them goes out of business, you just go to the other. So you can have an online backup storage on a specialised site like humyo, have the same documents on a webtop, and also run eyeOS on your own website. Overkill, perhaps, but perfectly possible.

At the moment, I've not come to any firm conclusions from all this, apart from that having competing visions is 'A Good Thing'. I don't find the webtop approach very convincing, particularly for those like me who already have a real desktop. I can see it might have its uses for those who don't have a desktop/laptop of their own, but use other people's, in libraries, netcafes or whatever. Then you can have your own personal desktop without any support hassles or expense. But even here, I find the Google approach more convincing: having individual apps as separate services, only loosely linked. After all, it's the individual services I'm interested in; having them all together in some fake 'desktop' is only important for those who are stuck in that mindset. eyeOS seem to be acknowledging this with their new version, which seems to be moving away from the desktop paradigm.

I'll put my current status in a separate post.

Farewell, desktop? Personal background

There's a lot of work going on at the moment rethinking the desktop. Google's best known for this, and its new OS will take this further - though it's hard to know exactly how, as it's keeping tight-lipped about what it will actually consist of. In the meantime, other ideas are being developed, so I thought I'd document a couple I've looked at.

First, some background
I've had my own PC for the best part of 20 years now - which makes me feel very ancient - and I've never been a great fan of Microsoft's software. When I started, I was using a DOS program for my business which would not run under Windows (version 3 as it was then) because of the way it used memory. So I looked around for an alternative. I found it in OS/2, which was not only able to run the DOS program; it could also run other things at the same time. This was an eye-opener: a proper multi-tasking operating system in a PC! Although Windows became more capable over time, it remained technically far behind OS/2. Sadly, Microsoft realised early on the importance of getting the operating system pre-installed, so it cornered the market and OS/2 never got the market share it deserved. In addition, Microsoft always had the marketing savvy to ensure that Windows was 'good enough' for most people's needs as they (the users) saw it at the time.

By the late nineties, IBM had ceased to support OS/2 - they had other things to worry about, such as escaping bankruptcy - but fortunately by this time a new alternative to Windows had emerged, namely open-source software based round Linux. So, around 2000 I started to investigate what was available. There were 2 basic types of desktop: simple window managers, and more integrated desktop environments. The former were small and fast, but required rather fiddly configuration in text files; they were though ideal for old PCs without much oomph, so I used them on some old hardware as backups etc. For the main machine, I wanted a full desktop, and there were 2 main competitors for this: GNOME and KDE. These were similar in what they could do, and even at that time were able to do most of what could be done on Windows. KDE seemed to be more tightly integrated, whereas GNOME gave the impression of being more a bundle of not particularly related programs; so I went for KDE.

I used a variety of distros, starting with Red Hat, and using Mandrake (now Mandriva) for a while. However, whilst this was easy to install, I also like to know more what's going on behind the scenes, or 'under the hood' as Americans call it. When I discovered some program was sending out requests over the internet, and I had no idea what or why, I decided it was time for a distro where I had more control over what exactly was installed. I tried Linux from Scratch, and learnt a lot from this experience - though the main thing it told me was that compiling everything needed in a modern desktop system takes an awfully long time, and you're much better off leaving the compilations to those nice people in the distros who do it all for you. Eventually I settled on Slackware, which has the great advantage of simplicity and gave me more or less the complete control I wanted.

The disadvantage of Slackware, though, is that it only has a limited number of packages, and whilst there are independent orgs which package up other binaries, if you need other programs, you have to compile them yourself.

In the meantime, I also started using the internet in the mid-1990s - one of OS/2's advantages was its good networking support. Seems hard to believe now, but I started off with a 2400bps modem. OS/2 included some examples and, although at first I didn't really understand what the WWW consisted of, one of those examples was a link to the Botanical Gardens in Canberra, where you could see what was currently in flower. I am not particularly interested in either gardens or Australia, but the light went on regarding the potential of the web, and I became an early adopter of many web technologies, introducing them to some of the many voluntary groups I belong to, particularly the open-source ones that can be used for free.

There have been many claims over the years about how internet technologies would replace the desktop. I didn't understand Sun's slogan 'the network is the machine' when they first introduced it, but it later became a good metaphor for the interconnected world of computing. Sun also introduced Java, and applets were at one time supposed to revolutionise desktop computing; however, when I saw the speed at which they operated over my 2400bps modem, my main thought was: 'they have got to be kidding'. Oracle too unveiled a thin client which was supposed to put Microsoft out of business, but it disappeared without trace. More recently, there has been a lot of hype about 'Web 2.0', and this time it seems to me the many claims about revolutionising the desktop are actually becoming realisable.

Like many others, I recently decided to invest in a netbook; however, unlike many others, I was not interested in running Windows on it. I chose an EEE (primarily because of price), and this came with a version of KDE pre-installed. However, whilst this worked well enough, it was rather limited in what it could do, and the distro didn't have much in the way of additional packages either. So, time to install another distro. The best-known for these netbooks at the moment is Ubuntu's Netbook Remix, but that required me to download a large CD image (pretty daft, as few netbooks actually have a CD reader) and I'm not a great fan of Ubuntu anyway - particularly its lack of a root signon. So, I decided to go for Debian, which gives more control and has the great advantage of a vast library of binary packages.

This gave me the opportunity to have a serious think about whether I actually need a 'desktop' any more, or whether I could go over more to the web. What programs do I want or need nowadays anyway?

But that, as they say, is another story, which I'll put in a separate post.

Thursday, 29 October 2009

Farewell, desktop? My current status

I mentioned in the previous post what applications I currently use the desktop for - web browsing, email, documents, spreadsheets, news feeds, digital music - but there's actually quite a few other smaller bits of software I use that are valuable parts of the desktop without really being 'applications': contacts, bookmarks, the collection of cookies and passwords I need to access websites or that speed up the process of filling in forms. It would be better to have a centralised version of these too.
  1. email: I decided straightaway that, as I was using Google Mail anyway, I might as well go completely over to that and not install an email program on the EEE. As all my emails come in via my domain name anyway, they are stored there and this can serve as a backup
  2. news feeds: here, there is no 'data' at all (well, not my data anyway). I was using a reader on my PC, but here too, it's an obvious move to use Google Reader and not bother installing one on the EEE. The only thing to backup is the list of feeds I subscribe to
  3. documents and spreadsheets: I have also been experimenting with Google Docs for quite some time, so my current project is to move all my documents there and not install any office software on the EEE either. I have a large number of old docs and quite a few spreadsheets lurking around, so it could take me quite a while. Backup for the moment is the PC, but longer-term I might debate setting up some program that every so often automatically syncs with, say, humyo. I'm also not sure whether I will need some way to input whilst offline, though I'll try out Gears or HTML5 storage; in any case, there's always a text editor! Some of the documents are produced on behalf of various orgs I belong to, and here using an online data store has the big advantage over the desktop of being sharable with the other members of the org concerned
  4. music: the webtops claim that you can store all your music on their sites, but to me this makes little sense: if you only want to listen to a file once, then fine, by all means online. But downloading something every time you listen to it seems like a large waste of bandwidth to me. So, music files will continue to be stored at home, and copied over to other devices if needed. Backup on USB stick(s). I still have a host of old CDs and even a large collection of vinyl I need to copy over, but it's a question of finding time to do that. Now that you can download tracks from places like Amazon for a pound or two, it may be simpler to just get rid of the old stuff and go with a new copy

There are a couple of other areas that I am thinking of maybe digitising some time soon (when I have time, mañana, mañana). Like music, they would be stored at home, with USB stick backup:
  • books: I have literally thousands of books. Some of these, such as travel guides, can probably already be replaced by online versions. Others, like novels, can probably soon be replaced by a portable reader of some sort
  • photos: at one time I was quite a keen photographer, and have large numbers of slides lying around in cupboards. I will probably digitise the best of these, and dump the rest

This leaves the question of what to do with the other desktop utilities. KDE has a good integrated system, with Konqueror providing an all-singing all-dancing browser/file manager/file viewer/bookmark manager/cookie manager/password controller/etc/etc. Indeed, for me Konqueror has always been KDE's killer app, but its browser element always lags behind other browsers, so I'm finding increasingly that I use Chrome (for speed, particularly with AJAX apps) or Firefox (for Firebug and general javascript debugging) instead. They aren't coordinated with KDE's desktop system, so I end up with 3 separate bookmark/cookie/history/password mgrs which don't match up. A centralised version somewhere that could be used from everywhere would be good, so I've been experimenting with storing bookmarks on Yahoo. This is however not very satisfactory, and I'll be interested to see whether KDE comes up with some solution to this. I might even end up writing something myself and storing it on my website.

Cookies and especially password managers are more problematic because of the obvious security issues. A centralised version of these would certainly be convenient, but the more convenient it is for me, the more convenient it is for anyone who hacks into any centralised store on the web, or steals my EEE. Encryption might solve at least some of that, but would also slow things down. No solution to this so far.

So, I'm now essentially just running KDE without any 'applications' as such, just a browser/file mgr/viewer, text editor, simple music player. This occupies some 2GB of my SSD, the web development side taking another 500MB or so, leaving some 13GB for music/books/whatever else I want to have with me when I go wandering with EEE in hand.

I'll post periodic updates on my progress with this little project.