TfL Kill Two of My Apps

UPDATE: Access to the Journey Planner has now been re-enabled, more details here.

On Monday 18th Oct TfL turned off their journey planner XML API. I have been using this successfully for more than 6 months for both London Bus (now called London Travel Deluxe) and London Tube Deluxe Pro for iPad. TfL are well aware of the success & reach my applications have achieved; sometimes I’ll even get a friendly heads up as to status changes coming up to ensure things keep running smoothly (99.98% uptime).

However it seems some areas of TfL are not so amicable. The removal of the API located at http://journeyplanner.tfl.gov.uk/user/XML_TRIP_REQUEST2?sessionID=0&language=en came without any warning whatsoever. Effectively dropping all journey plan requests for London Travel Deluxe & Deluxe Pro, as of this writing they return zero results. London Travel Deluxe has been downloaded 100,000 times & has a solid user base of 35,000 – 40,000 users producing 180,000 journey plans a month. They have been well battle-tested having been through 2 recent tube strikes. Blind & visually impaired people rely on specific features I have implemented for them using VoiceOver on the iPhone. It has got people using the often ignored bus system because they could plan on the run with complete confidence and even TfL’s own staff use the app.

It’s an uncomfortable coincidence that the XML API was closed down a week after the removal of MDV’s release of the ‘My TfL’ app. MDV run the journey planner for TfL and it’s a very impressive system. The My TfL app used a cut down version of the XML API located here http://www.journeyplanner.org:80/ultralite/XML_TRIP_REQUEST2?language=en&sessionID=0 You’ll notice this hasn’t been closed down. Nor has the scraping of the HTML version of the journey planner (as done by another popular, well developed & supported iOS app London Journey Planner).

I have repeatedly sought clarification on access to what was an open API from Transport for London. Emer Coleman from the London Datastore part of Greater London Authority has also done the same to no avail. Yes there are commercial issues around using the journey planner in third party applications and I am not asking for a free ride (I have sub-royalty agreements with the likes of Traveline) and already pay £3,200 a year to licence official TfL maps. MDV should be rewarded for the decades of hard work they’ve put into a commercial product they sell & support. It makes sense for TfL to find a way for this API to be accessed by numerous developers under an open, non-exclusive, authorised & managed arrangement (i.e a developer key). The benefits are enormous for everyone involved. London Travel Deluxe would cost the best part of £100,000 to develop commercially. But the past & prevailing attitude of certain departments and key individuals has amounted to nothing but corporate stone-walling, dodging, ducking, diving & weaving. Why?

Is it control, protectionism, shortsightedness or just plain stupidity? Take your pick.

Some in TfL will allude to the complexity of opening up the API, commercials, DoS attacks, vulnerabilities or making sure the data is 100% correct. These are the very tactics used to stagnate the opening up of any official API, I have witnessed these excuses first hand in several forums.

TfL has just released a mobile version of their site at http://m.tfl.gov.uk (you are automatically redirected to on an iPhone). A good initiative, but no doubt the html mobile version of TfL will be used as yet another excuse to further bury developer & London Datastore requests for open API access. My iPhone apps allow the offline saving of journey plans (very handy considering 45% of the network has no network signal) & have 750 searchable bus routes installed for offline reference as well as high resolution maps. An html mobile version will always cater to the lowest common denominator; a plethora of features in todays native apps is not possible in html, or even html5.

Do I want TfL to pull the html mobile version? No. For some the html version will be very useful, I support it. At the very least, I want TfL not to take away choice & stifle innovation; yet that’s exactly what they’ve done. This effects me & other developers, some have even started documenting the various API’s here http://wiki.opentfl.co.uk/Frontends

Taking away a perfectly functioning API for no reason is ridiculous. Although reasons maybe given after this writing, I can tell you they are certain to be 100% total bullshit. Why am I so sure? Because they haven’t closed down all of them, all of these XML API’s still work on the exactly same server farm the journey planner was removed from…

Find stops near Victoria Underground Station…
http://journeyplanner.tfl.gov.uk/user/XML_STOPFINDER_REQUEST?language=en&type_sf=any&name_sf=Victoria%20Underground%20Station&coordOutputFormat=WGS84%5BDD.ddddd%5D&locationServerActive=1

Get summary route details by a partial or route number or code search…
http://journeyplanner.tfl.gov.uk/user/XML_SEL_STT_REQUEST?sessionID=0&language=en&execInst=normal&mode=line&selectAssignedStops=0&net=tfl&itdLPxx_motCode=3&lineName=115

Get departure times for a specific stopId & all it’s services…
http://journeyplanner.tfl.gov.uk/user/XML_DM_REQUEST?language=en&sessionID=0&ptOptionsActive=&itdLPxx_tubeMap=&itdLPxx_request=&command=&lsShowTrainsExplicit=1&name_dm=1001180&nameState_dm=notidentified&place_dm=London&type_dm=stopID&itdDate=20100503&itdTimeHour=13&itdTimeMinute=8

Go ahead try them yourself. Although it wouldn’t surprise me if these also mysteriously disappeared.

If there were any real problems with the XML API you’d close them all down. Or like most normal IT departments…simply fix them, technical & security problems can be overcome. These API’s have existed for years and taken a hammering by many developers, if TfL check the logs they’ll see just how well this API has flawlessly performed! There are no problems or vulnerabilities with these API’s & I have never done anything to compromise this system. I know how to code against an API.

So where to from here? TfL’s actions go against their own boards digital strategy, see http://www.tfl.gov.uk/assets/downloads/corporate/Item04-Commissioners-report-sept-2010.pdf point 8.2. We are at an impasse. Senior management at TfL could lead the UK in terms of developer engagement & open data. Myself & many others see that as an opportunity that benefits commuters, developers & TfL alike, but they see it as a threat.

Who made this decision to turn off the journey planner API? Why?

If you would like to ask TfL this question you can email them enquire@tfl.gov.uk

UPDATE 24/10/2010 @ 6pm: I have been informed that the change was part of a security update & that TfL are working on restoring the service ASAP and are also fully committed to open data. TfL have recently released data sets & a developer area.

I sincerly hope ‘we are working to restore’ or ‘ASAP’ is a matter of days, not weeks dragging into months. If I need to change something in my apps, that’s not a problem.

UPDATE 26/10/2010 @ 8am: Late yesterday I had a really positive meeting with senior people at TfL, they explained they fully committed to open data. They are indeed working extremely hard to get the popular TrackerNet feed back online (unrelated to the feeds I mention in this posting) and are actively working out ways to engage further with the development community to deliver more data.

I am told access to the API will be re-enabled. I have no timeframe for this, but given it’s a simple configuration change, sometime today is not unrealistic.

This entry was posted in apple, ipad, iPhone, iPod Touch. Bookmark the permalink.

14 Responses to TfL Kill Two of My Apps

  1. smallcheese says:

    This must have happened at not long after midnight on the 18th, as I was trying to use the app to get home as was getting the same messages from it as I am getting now. I just tried to plan the same journey as I did last week to see if it would work (thinking of popping over to a friend’s later) and thought I’d check out your blog and see if there was any reason for the problem. I was trying to get home at around 00:45 on the 18th and got no joy from the app, and ended up getting a £20 cab home rather than a couple of buses. TfL cost me money!

    A friend of mine had the same issue with National Rail (http://chrisroos.co.uk/blog/2009-01-31-turning-off-my-national-rail-twitter-bot).

    It seems strange to pull this with no explanation whatsoever, at least National Rail had the decency to email Chris. I will email TfL and encourage everyone to do the same.

    Tom

  2. marksteward says:

    I assumed it was a knee-jerk response from whoever’s involved in pulling MyTFL. However, if that app uses /ultralite, which is still working, it looks a bit cynical.

    As somebody wrote recently, TfL is barely emerging from being a black box which takes funding and spews out transport. What’s frustrating is that if they opened up their feeds, developers would deal with all the difficult bits for them. There would quickly be wrappers, syndication and a large pool of enthusiasts that they could call on.

    The least TfL could do is a message on the “developer” page, or an email to everyone signed up for feeds to explain the situation. If they can’t get this right when there are relatively few users, what will happen when there are hundreds of apps?

    Hope you manage to get your apps updated soon.

  3. marksteward says:

    P.S. while I don’t know of any exploits in the API, there are XSS attacks: http://wiki.opentfl.co.uk/XSS (TfL have been emailed, with no reply).

  4. Malcolm Barclay says:

    As far as I know, this is not a problem with the existing API.

  5. jaa_lon says:

    This is horrendous. I’ve recommended your apps to everyone I know with an iPhone who’s ever expressed an intention to visit London. People are so often overwhelmed by the complexity of the London Underground if they’ve never visited, a journey planner in their pocket is a fantastic aid. And it’s SO much more simple to tell at a glance which lines they’re letting people know aren’t working, before you leave work/ home.

    Glad you’ve got Wired UK supporting your cause. I’ll be adding my name to those emailing TFL directly.

  6. potwalloper says:

    If you look here

    http://www.whatdotheyknow.com/request/cancellation_of_open_access_to_d

    It seems that someone has now lodged a Freedom of Information Act request on this decision. It would be sensible for more people to do this I think…

  7. andytizer says:

    I hope you get the support you need from TfL to make your apps and other transport apps work.

  8. Terence Eden says:

    Thanks for the heads up.
    Interesting to see that they’ve now launched m.tfl – I’ve been using http://wap.tfl.gov.uk/ for years – wasn’t even aware that there was a new version out.

    Good luck getting the feeds back.

  9. DamianB says:

    Malcolm

    Working for another private business (Kizoom) with an interest in the public transport information sector, and some experience of what it feels like to have a feed removed (see exchanges on MyRailLite passim), I wanted to share my thoughts on your recent issues with TfL.

    First of all, it doesn’t seem right that you were cut off without warning (if this is what happened). In the absence of any official TfL journey planning application you have done Londoners a terrific service in providing them with a another tool to plan journeys on the move. You have also developed a strong following and public voice as an advocate of the mobile internet sector and, if I had been in TfL’s place, I would tried to work with you to manage the change in policy or to put in place suitable commercial terms for you to continue operating your app.

    And commercial terms are the important element that I feel have been missing from these discussions. I guess that you are running a business venture for commercial gain. You have charged members of the public to provide a product which depends upon a 3rd party’s (TfL) data and service, but without any back-to-back commercial agreement or license in place with TfL. Doing this you must have known that you were taking a risk, but it was your judgement that it was a good one. That risk has (for the time being at least) gone sour and it is understandable that you and your users are upset. You shouldn’t mistake though your irritation at being exposed, or your users’ irritation at losing a service that they have paid you for, as establishing you as the wronged party, and TfL as the villain.

    A distinction that it feels hasn’t been explored enough is between freedom of access to data and freedom of access to services. TfL has been made great progress in providing open access to its data. You and I can spend our time and money on building software that makes use of that data to provide valuable services. You seem to be implying that because TfL provides access to its data it should be providing free access to its journey planning services too. That’s like saying that because the government provides access to historic National Census data, it should be obliged to provide free software to visualise and interpret that data. In my view that should be left to 3rd party private sector companies who should compete to provide the best data visualiser, and earn profit by so doing. How would you feel if you were a private company that sold software to interpret National Census data. Or more specifically, would you really be making the same argument now if you had already developed or purchased your own Journey Planning software?

    My preferred environment would be one in which TfL offers free access to raw data and charges for licensed access to their Journey Planning service if you want it. I hope the situation is cleared up soon. It’s important that people get their transport information. You’ve done a great job.

  10. marksteward says:

    DamianB: While I agree with your distinction between data and services, I’d put Malcolm’s app on the “visualiser” side. The Journey Planner has to be used as raw data, because is not made available by any other means.

    TfL made the same data/services mistake with the Trackernet feed. Instead of releasing a feed of units and locations, they only provided a departure-board of trains approaching a particular station. Getting a snapshot of the entire network meant asking each station for its view, vastly increasing the number of requests.

    This is data that should be accessible to the public, and in both cases disabling the feed cut off a known userbase. Until TfL make the raw data available, I don’t think there could be any justification charging for it.

  11. Malcolm Barclay says:

    Hi DamianB

    All very good points & I agree with you. There’s a distinction between raw data, visualisation and the turning of that data into useful applications or services. I am not suggesting that services such as the Journey Planner should or need to be automatically free. And companies like MDV well deserve to be paid for the system they’ve developed, sell & support. TfL already pays for this service. What needs to be addressed is the commercials of TfL allowing developers to access these services. I have seen nothing from TfL that indicates any seriousness in making this happen, on the contrary, such discussions are avoided & actively shut down. If there’s a clause in the contract between MDV & TfL to the effect “TfL you will not allow journey planner API access to third parties”, surely it would be worthwhile pursuing a change to this arrangement? The London Datastore has been waiting on an answer to this simple question for months.

    True, we could all dive into TransXchange data & start building our own journey planners. Right now, there would be rather large gaps in this, i.e no National Rail Data. And potential IP issues. This is not ideal; but push come to shove, if enough TransXchange data is released & it includes rail, this would probably happen (think OpenStreetMap).

    I am all for licensed access to the Journey Planning services, but it needs to be open for all & managed (easily done through a developer key program)

  12. gregjackson says:

    DamianB and Malcolm:
    The big risk of commercialised government data is that it so often inhibits low cost innovation. Where developers have to go through an application licensing procedure, or start paying for a service, the only people who can realistically innovate are those who have commercial backing, or are already in the sphere. The cost is an issue, but the ensuing complexity can be a real problem (witness the impenetrable licensing conditions for postcode data or OS data).
    Risking one’s time, or company’s time, on developing speculative ideas utilising data is already a barrier. Adding licensing complexities just makes it harder to experiment. Public authorities should certainly release “base data” for free. If they choose to create their own value-added services utilising this data, then perhaps charging may be appropriate – but they should not use restrictions on base data to generate sales benefits for their value-added products.
    There is an argument that where there is additional cost to providing base data, the public sector provider may charge those who are profiting from its use, but this needs to be done in such a way that it only gets charged once someone is actually generating a significant profit and provides no barrier to service innovation. Given the usually negligible extra cost of providing open data this should be a rarity and avoided wherever possible.
    Of course, this sort of thing can provide a threat to companies like Kizoom (who’ve done so much great work) but it should also provide opportunities to them and so many others. Most importantly, it provides tremendous societal benefits at incredibly low cost.

  13. James says:

    Malcome,

    Can you not move people over to the Traveline South East Data, which you do have a licence for, and covers the whole of the London area, as well as the South East.

  14. Malcolm Barclay says:

    Hi James,

    Unfortunately not without changing parts of the app, resubmitting to the app store & going through the approval process again (1-2 weeks). Also my licence with Traveline S&E is strictly for their application. Whilst they may of course be understanding if I take it up with them. True, the changes required could be a simple as a URL, but needs full testing to make sure what I have coded works (i.e some parts of the API might not be in the Traveline S&E instance of MDV).

Leave a Reply