Saturday, 11 August 2012

Maps on mobile devices - current status

Standard web maps

As I mentioned in my post on the mobile version of my maps back in January, there are two major issues with using standard web maps on mobile devices: online connection, and battery (well, all right, there's a third issue: whether you can actually read the maps in direct sunlight). These are linked: the longer you maintain a connection, whether by wifi or through a mobile network, the quicker the battery will go flat. No doubt batteries will improve, and their life between charges lengthen, but at the moment you are unlikely to be able to use them with the network connection on for longer than a couple of hours. As I said, a possible strategy is to just maintain the connection long enough to display the map for your next stage, then turn it off and only turn it on again when you move out of that area. However, if you are out in the wilds somewhere you may not be in signal range.


So it would be better if you could save the maps on your mobile device so you can download what you need when you have a connection, and then use this saved version without the online connection. This is essentially what you do with a GPS device, which doesn't have an online connection.

To some extent, you can do this 'caching' with existing web maps, such as mine. At the moment, map servers generally serve rectangular images ('tiles' in the jargon) of say 256x256 pixels, georeferenced so it's clear which piece of the earth they represent and hence at what scale (zoom level). The browser software then requests however many of these images it needs for the current browser window, and displays them in the correct position; any feature/vector data for points and routes, for example from a gpx file, can then be rendered in the correct position on top of these images.

So, if you want to cache this, you need to save (a) the software, (b) the images, and (c) the vector data. Browsers generally have their own cache space and do this saving automatically. However, neither users nor developers have much control over this. It saves everything it reads regardless of whether the user considers it important or worth caching. Users can only clear the cache completely, and cannot say, for example, I want to clear everything from this site and not from that one. When the cache space is full, it deletes old pages, again regardless of whether the user would like to keep them. The cache is browser-specific, so if you use more than one browser, each one will save the same pages in its own cache. In addition, it can only load a page when there is a connection.

For this last problem, there is now a solution: modern browsers can now save (a), the software, in something called the appcache (though this suffers from the same problem as the normal cache in that it is browser-specific, each browser storing the same 'app' in its own appcache).

(c), the vector data, is relatively easy to handle: the user can simply save it in whatever storage the device has, and modern web browsers can then load it from there. The main problem is with (b) the images.

Storage space

There are 2 issues: firstly, the images are large, some 20-40kB apiece, so the more you save, the more space you need. Google's tiling system, for example, needs more than 350 billion tiles to cover the world at 20 zoom levels (I'll leave you to work out how long it would take to download the lot). Browsers do have the ability to store a certain amount of data in another type of storage called localStorage; this is however limited in size, and unlikely to be much more than those necessary for a day or two's walk at one zoom level. So it offers little advantage over the normal browser cache, other than that developers (and to some extent users) can control what is stored and what isn't.

There are (native) apps available which will cache the tiles for you as you fetch them over the internet. Because they are run directly by the operating system and not in the browser, they are not subject to browser caching limits and can use any spare storage available in the device. However, this is not limitless (not even a fraction of Google's 350 billion tiles), so again if you are on a long trip and need maps to cover a lot of ground and/or at different zoom levels, you will have to delete the tiles regularly to make space for new ones.


They are also subject to the second issue: copyright. Almost all decent mapping costs money to create and keep up-to-date, so is subject to copyright. Although copyright generally permits small amounts of copying for personal use, the legality of large amounts of caching is dubious. Most commercial providers, such as Google, Bing, and the APIs from the Ordnance Survey and the French IGN, explicitly forbid it. Many mapping agencies supply datasets for use with GPS devices, and some of these are also available for mobile apps (for example, the ViewRanger app), but unlike the maps on say my website, you have to pay for that, maybe even more than you would pay for internet access (though probably less than you would pay for paper maps).

Vector mapping

Mapping agencies maintain their maps in vector format (i.e. the definition of what will appear in the images, not the actual images themselves). Back in the early days of web maps, servers were powerful things and client devices like PCs feeble, so the best way to serve the map data was to 'render' it on the server as images which client browsers could easily display. Nowadays, even mobile phones are pretty powerful, so are better able to render the data themselves. Also, vector data is way smaller in size, so is faster to transmit and takes up less room when cached. For this reason, web maps are beginning to be supplied as vectors, leaving the client to convert this into the map image. The first major player to do this was Google: Google Maps for Android now sends its maps not as images but as vectors, meaning the transmission bandwidth and storage needed for caching are much less. Apple is planning a similar change in its new mapping app for iOS 6, promised for later this year. It remains to be seen whether the national mapping agencies do likewise for their map data.


One major exception to the copyright issue is OpenStreetMap. Although they do maintain tiled images similar to Google et al, their map data is freely available in vector form and can be downloaded on a country or region basis to your mobile device. Although these files are much smaller than the equivalent image files would be, they are still many megabytes in size, so are best downloaded in a compressed format (using the cheapest internet connection you have available). AFAIA, there is not yet any browser software for reading this compressed data, but various native apps are now appearing which can read these vector files, and display them along with any vector data from gpx or kml files; one example I have used on Android is OsmAnd. It looks like at present there is no standard format for this compression, and different apps use different incompatible ones. I would hope this situation will improve. And different apps also supply different data; OsmAnd for example supplies separate contour data which is a significant improvement for use in hilly countryside. Although OSM is improving rapidly, outside major towns it remains of limited use; proper topographical maps from the national/regional agencies are generally far superior for country walks.


In towns, my preferred solution is to use an app that can read OSM vector data; download this data before your journey and then use the app offline. For the countryside, OSM maps are not very detailed, so my preferred solution is to use the maps from national mapping agencies with my mobile map pages with or without the caching option.

For those countries where these map pages are not available, such as Switzerland, there may be an app for your device licensed to cache their tile images for use offline.

For the future, it's a pretty safe bet that things will have changed considerably in 2 or 3 years time.

Friday, 16 March 2012

VML now deprecated on map pages

Only applies to users of Internet Explorer versions older than 9.

Most modern browsers use Scalable Vector Graphics for rendering map data. For many years Microsoft refused to implement this, and used its own renderer, VML, in IE. This meant special code to deal with VML had to be included in map pages to cater for users of IE. Microsoft finally changed this policy and included SVG in IE9. They have now deprecated VML, and it will no longer be included in IE10.

So that users of browsers that support SVG are not slowed down loading unnecessary code, I have now removed VML support from the standard pages. It will be loaded separately for users of IE < 9.

As SVG rendering is much faster than VML anyway, users of IE < 9 are strongly recommended to either upgrade to version 9 or use another browser such as Chrome or Firefox.

Thursday, 5 January 2012

Mobile map pages waiting to be tested

Although my mapping pages have been usable on mobile phones for a while, they've not been very mobile-friendly. I have now written more mobile-friendly versions:
* simple map display pages can be reached from:
* using geolocation (GPS) to display your current position on a map:

Advantage of using a phone for this sort of map display is you can do a lot of other things with a phone which you can't with a dedicated GPS device. Disadvantage is you need access to both internet (which may be a problem in remote areas) and GPS satellite signal; having these on for any length of time soon chews up battery power.

An alternative strategy to preserve battery when your position changes only slowly (for example, when on foot) is to turn on internet access and GPS at or before you start, go to your position on the map, then turn off both internet connection and GPS and only turn them on when you need to refresh your GPS position or you need to pan the maps. Once you have the page in your browser, the map display will stay in the browser cache, so you could also pan the map over your full journey at or before you start, which will cache those map tiles you need later; this strategy won't however work for the routes option, as the route lines are not cached.

Alternatively, you can get the maps while you have internet access, and use screen capture to save a snapshot of the map, which you can then use in the field leaving internet access off. However, these snapshots are not georeferenced, so you can't use GPS with them.

Feedback welcome in comments below. Please state which device and browser you are using.

Saturday, 5 December 2009

Using skypephone modem in Linux

I've been having a frustrating few days trying to get the 3G modem in my mobile phone to work with my Linux boxes. For anyone with similar issues, I'm documenting here what I did.

The phone I have is made by the Chinese company Amoi, marketed in the UK by 3 as a skypephone, currently version 2 of this. Basically, this was the cheapest 3G phone/modem I could find. 3 include 150MB of internet access in their topup charge, either used via the internet programs on the phone itself (there are various apps including a browser which uses Google Wireless Transcoder to try and render full-size webpages for the mobile form factor), or by using the phone as a modem. The modem can be used via either bluetooth or a usb cable provided with the phone. My Asus netbook does not have a bluetooth connection ootb, so I wanted to use usb.

Like most of these setups, the usb interface is heavily oriented to Windows users, to such an extent that it presents a cdrom drive which automatically installs Windows drivers when you initially plug the cable in. When you do so, the cdrom drive disappears and you can access the modem. As usual with Linux, there's no need for any drivers as these are already provided either in the kernel or as separate modules. So the main question is how to ignore the cdrom and access the modem. The best documentation I've found of this process, which varies depending on the device, is on the website for usb_modeswitch, a program which attempts to do this.

  1. The first thing to do is to switch on the phone and plug in the usb cable (a pitfall I fell into was that when you plug in the cable the phone immediately starts charging, which can give the impression that the phone is on even if it isn't - so make sure the phone's operating system is running).

  2. Then run lsusb on Linux. This should show the Amoi device, with an id of either 1614:1000 or 1614:0407. If the latter, you're in luck: that is the modem - proceed to the next step. If not, try unplugging the usb cable and plugging it back in again. With luck, it should now show 1614:0407. 'dmesg | tail' will show "Product: USB MMC Storage" for 1614:1000, or "Product: S2" for 1614:0407. I've found that you get different results depending on distro: according to the last post on this thread, both Ubuntu and Fedora find 0407 straightaway, my Debian testing/squeeze partition may need unplugging/replugging one or more times, and my Slackware partition doesn't seem able to find the modem at all, no matter how many times I replug the cable. Presumably, this is due to the way the kernel is configured, so if anyone can tell me what I have to configure on my Slackware kernel to enable this I'd be grateful. Fortunately, once the modem is set up it stays that way until you reinitialise the phone by switching it off or unplugging the cable, so once I have it set up in Debian I can then reboot in Slackware and it will then show as 1614:0407.

  3. Once you have 1614:0407, you can then load the driver. There are 2 possibilities: the old standard usbserial module and the newer option module, which it seems is now recommended for these kind of phone modems (tho IMHO it would be better if they renamed it).

    usbserial can be loaded with "modprobe usbserial vendor=0x1614 product=0x0407". option does not have the vendor/product (er) options, but thanks to a tip from usb_modeswitch, you can set them directly on sys bus:
    modprobe option
    echo "1614 0407" >/sys/bus/usb-serial/drivers/option1/new_id

    In both cases, udev then creates /dev/ttyUSB0 which pppd can use.

  4. For completeness, here is my pppd control file, which I put in /etc/ppp/peers/3:

    idle 7200
    user three
    password three
    connect-delay 20000
    connect "/usr/sbin/chat -V -f /etc/ppp/chat3"

    I don't believe the modem speed has any effect for these devices, and I don't think user and password are used either.

    Here is /etc/ppp/chat3, the chatscript I use:
    ABORT        BUSY
    TIMEOUT 10
    "" ATZ
    OK 'ATE0V1Q0S0=0&C1&D2+FCLASS=0'
    OK 'AT+CGDCONT=1,"IP",""'
    SAY "Calling..."
    TIMEOUT 60
    OK 'ATDT*99#'
    CONNECT \c

    The important thing here is "", the APN, which is the important part of user validation; it will not work without this.

  5. The alert will notice that I do not have 'usepeerdns'. This is because, rather bizarrely, 3's nameservers do not appear to work. When I used 'usepeerdns' I got a connection and could do, for example, 'ping' (one of Google's servers, if you don't know) but no dns resolution was working. So I set up /etc/resolv.conf with OpenDNS's servers instead:

  6. You then start ppp with 'pppd call 3' and wait for the connection to be set up. You can monitor this from the logfile, /var/log/messages in the case of Slackware: when 'tail /var/log/messages' shows the local and remote IP addresses, the connection should be ready. On my setup, this takes about 30 seconds.

  7. When you have finished, you can kill pppd with "kill `cat /var/run/`". And, as pppd also logs how much data transfer there was, you can do something like "cat /var/log/messages | grep 'bytes, received'" to find out how much of the 150MB you used during that connection.

Thursday, 12 November 2009

KDE4 multimedia in Slackware

Multimedia has always been a bit of a pain in Linux distros. It's riddled with patents and other restrictions, which free-software purists dislike. So even simple things like playing mp3 files often produce a lot of hassle. KDE3 had its own sound server, called arts, which in recent years has been unmaintained and generally unsatisfactory. KDE4 replaced this with a new multimedia framework called Phonon. This can work with different backends, of which the usual ones are gstreamer and xine.

In both Debian and Slackware, gstreamer is the default. When using xine in kde3, I used the Windows codecs from mplayer, but gstreamer uses its own plugins, of which there are several packages: gst-plugins-base includes quality open-source codecs like ogg and theora, gst-plugins-good adds others, including containers like flv and avi. The mpg codecs are in gst-plugins-ugly, i.e. those which are good-quality but have license/patent issues. Debian includes all these plugins somewhere in its repos, but Slack only has base and good, so you cannot use mp3 files with gstreamer in the standard Slack distro. Fortunately, ugly is available on which also includes the various libraries needed which are also not in standard Slack. Install all those and you can then listen to mp3 files in KDE programs like Juk.

Mind you, Slack does include mpg321 which enables you to play mp3 files on the command line, so you could set up a file association in kde to use mpg321 to play mp3 files.

Another of those areas where Linux distros create a lot of hassle for something that should be simple.

Back to Slack

In a previous post, I said that I was using Debian on my EEE. I have though become increasingly frustrated at the length of time it takes Debian to move programs from sid to testing. When I first installed Debian, I used the default option, which is stable (i.e. Lenny at the moment) with Gnome desktop. That was, however, not really what I wanted, so I went off and RTFM, and found that you can change this by using expert mode. So, I kept Lenny on his own partition, and installed KDE on another partition. I first tried with sid, but couldn't get X to load - gave me strange error msgs which I didn't have time to try and fathom - so I installed testing instead. In theory, programs are quickly moved from sid to testing unless there are major issues, but the new versions of KDE seem to have got stuck in sid. In the meantime, KDE have released 4.3.3, but Debian have still not moved 4.3.2 to testing, let alone 4.3.3. As Slackware made 4.3.3 available in current just a couple of days after the KDE folk released it, I decided to go back to using Slackware. In fact, I kept testing, and installed Slack on top of the Lenny partition, as I wasn't really using this for anything.

In principle, the big advantage of Debian is the vast package repos, which Slack doesn't have. However, there is now an excellent collection of additional Slackware packages at I've so far found only 2 packages I use that are in Debian and not available for Slack: mapserver, and Google Gadgets for Linux, which can be used as KDE plasmoids. I could compile for Slack, but, if necessary, can also boot my testing partition and use that. There are a couple of curious little bugs in Slack I've not got to the bottom of yet: those icons that are in Hicolor but not in Oxygen are not displayed; and Ark seems unable to open tar.gz files (I think this is because a compile option was missing).

Thursday, 5 November 2009

Zoho remote api

In a previous blog on strategies for replacing the desktop, I wrote: "In theory, there is another alternative - data local, software on the web - but I never heard of anyone implementing that!"

Turns out that's Remote API does precisely that. This enables you to keep all your data on your own server but use Zoho's editing program to change them. Piotr Malinski has example code to do that with Curl in Django or PHP