The Guardian on your Kindle

Update: It is now possible to buy an official subscription to The Guardian and The Observer. The rest of this post is now largely of historical interest if you just want The Guardian on your Kindle, but I’ve left the rest of the content unchanged for people who are interested in how I generated my unofficial version.


If you just want a copy of today’s copy of The Guardian or The Observer for you Kindle, you can download one from this automatically generated page.  This post describes the script that generates that generates the file and the motivation for it.

Since moving to Switzerland, I’ve found that I really miss being able to get The Guardian in the morning on my way into work.  Unfortunately, reading the website on a phone (or any other device) is no substitute if you’re relying on data over the mobile phone networks – one really wants all the articles cached for fast navigation through the paper.  The solution for this should be my shiny new Kindle, but sadly subscriptions to The Guardian aren’t available in the Kindle store.  (There are many other papers available.)  Fortunately, The Guardian has an excellent API for accessing its content, and the lovely interface produced by Phil Gyford for reading the paper in a cleaner interface suggested that I could similarly generate a bare bones version of the paper for my Kindle.  I believe that this is permitted under the terms and conditions of the Guardian Open Platform, since I’m (a) including the advertisement linked to from each article, (b) linking back to the original article and (c) acknowledging that the data is supplied by that service.  If I’ve misunderstood, and in fact this is not allowed, please let me know.

To generate a book in the Kindle’s preferred format, you have to generate a .opf file describing the contents of the book, which refers to other files describing its text, images, structure, etc.  Then you can run a binary called “kindlegen” to generate a .mobi archive from those files that will work on your Kindle.  (The samples in the kindlegen archive and the Amazon Kindle Publishing Guidelines are quite sufficient to figure out how to do this.)  My script to generate the .opf and supporting files is far from elegant, but I’m very happy with the results that it produces – it’s a really lovely reading experience.  You can use the normal page forward / back buttons to go from page to page, while the left and right buttons on the five-way skip to the next article’s headline.  This means you can skip quickly through the articles that you aren’t interested in, but each article you do want to read is presented very clearly on the amazing eInk display:

There are a few articles for which the API won’t return the text, saying that rights for redistribution are not available – I’m still including the other metadata for these article and the link to the original article, so that you know what’s missing:

At the end of each article is the advertisement image that’s included – this is to comply with the requirements of the Guardian Open Platform:

You can download the generated .mobi file for today’s Guardian (or The Observer on Sunday) from this page:

A Kindle version of today’s Guardian or Observer

You can bookmark that page in your Kindle’s web browser. Then, whenever you select the bookmark and then “Reload”, then it’ll be refreshed with a link to that day’s generated edition of the newspaper for your Kindle, which you can download straight from that page.

If you’re interested in this project, or have any comments or suggestions, you can contact me by email at:

"mark" followed by a dash, then "guardian" then "kindle" then an "at" sign then longair dot net

… or leave a comment below.  The script for generating this version of The Guardian is available at github.


Posted

in

,

by

Tags:

Comments

30 responses to “The Guardian on your Kindle”

  1. Mike Sheldon Avatar

    Great job :)

    It’d be really nice if this were offered as a web service where people can register their kindle usernames and have a daily cron job run that creates a guardian mobipocket file and e-mails it to everyone who registers via their @kindle address. Resulting in anyone interested getting the Guardian delivered directly to their kindle each day and removing the onerous requirement of each user needing to apply for their own guardian open platform key.

    1. mark Avatar

      Mike: Thanks for the suggestion. The problem I see with that is that even after compression the .mobi files are several megabytes in size, which at $0.99 per megabyte could be a significant cost. I’m reluctant to set up a web service that would end up costing people an unpredictable amount of money.

      My real hope is that someone at the Guardian notices this and sets up an official Kindle version, of course. :)

        1. mark Avatar

          Great, thanks for letting me know! I’ve subscribed to that now, and it seems to be very nicely done. I’ll update my various pages with pointers to the official version.

  2. Will Moindrot Avatar
    Will Moindrot

    Hi, I might be really wrong on this but I’ve seen elsewhere that it’s possible to get Calibre setup on you PC to pull together the guardian articles (I don’t know how well it does this) or probably even load it with the download you offer, and it will automatically send it out daily to your Kindle email address. In my case if I could get it working, I would be picking up that email over wifi therefore for free.

    1. mark Avatar

      Will: is there some way to say that you only want to receive things sent to your Kindle email address when you’re on wireless LAN? (I haven’t tried sending anything to it in case I get charged.)

    2. Dan Avatar
      Dan

      Is this the sort of thing you’re looking for? http://blog.garethj.com/2011/01/21/free-daily-news-to-a-kindle/

  3. Will Avatar
    Will

    HI Mark, yes I think there is an email address you use (http://www.amazon.com/gp/help/customer/display.html?nodeId=200375630) such as name@free.kindle.com – I think you need to set it up following instructions at that site. It’s not explained massively well. My Kindle is wifi only which as far as I know doesn’t get any charges applied (not using 3G). Presumably if you sent it to the free email it will use either 3G or wifi hopefully for free. And there may be a network ‘order of preference’ type setting so you can turn 3G off or something. But for me this would work very well in updating my Kindle with content before leaving the house of a morning.

  4. Pablo Avatar
    Pablo

    Thank you, man! Been looking a long time for this.
    Enjoy and thank you, thank you, thank you!!
    I can’t understand why is The Guardian the only major newspaper missing in the Kindle. I would gladly subscribe…
    Best.

  5. Ingmar Avatar
    Ingmar

    This is amazing!

    Would love to see a Blind [Carbon] Copy (BC) mailinglist, where we could leave our “name”@FREE.kindle.com addresses (for free delivery). We have to add the @longair address to the Kindle approved e-mail list though, so that you are authorized to send attachments to our Kindle.

  6. Nick Avatar
    Nick

    Ingmar – just what I’m looking for as well. Have you found anything yet?

    Have to leave my laptop on at the moment to do it at the moment, but would be great if someone set something up…..

  7. roisin o'hara Avatar

    i have just bought a kindle for the sole purpose of being able to download the guardian every day when i am out of the u.k. on trips. so it is a huge disappointment to find that the guardian is the one newspaper that cannot be downloaded to the kindle.

    please make the guardian available to kindle users as i would gladly subscribe.

    n.b. downloading the guardian onto a kindle is more environmentally friendly than buying the tree paper.

    1. mark Avatar

      roisin o’hara: Just to make it completely clear, I don’t work for the Guardian, so it would be better to complain to the newspaper about that directly rather than here. What this page does gives you, however, is a link you can bookmark in your Kindle’s web browser that on reloading each day will have a link to a version of that day’s Guardian with much of the content you would find in the newspaper itself.

  8. Luo Genie Avatar
    Luo Genie

    Guys, you can have any newspaper on your Kindle automatically and for free – as long as they publish article summaries in RSS. There are plenty of sites that let you build your RSS list then send every new article to your e-reader. And guess what: some of them even send you full articles, not just the RSS summaries (one that I know of is http://www.tomykindle.com). Personally I don’t read anything that’s longer than 10-20 lines on a monitor – I just sent it to my Kindle to read later. On a toilet, presumably :D

  9. Tara Avatar
    Tara

    Thanks a lot for this Mark.

    I also hope the Guardian provides a subscription service soon… I haven’t gotten my Kindle yet, but this was my primary worry in getting one… And you have just solved it!

    Cheers.

  10. Gammakid Avatar
    Gammakid

    Hi Mark,

    This is great! Thanks very much!

    Just to be clear: does this method require you to have a Guardian digital subscription (I currently have one) and is the content the same as the print edition?

    Guardian not working on Kindle has been the only reason I was holding off buying one! (as the “read on kindle” option from the on-line edition is truly awful). If this works I will order my Kindle today!

    Thanks so much for this!

    GK

    1. mark Avatar

      Gammakid: thanks for you kind remarks. To answer your questions:

      * No, you don’t need a Guardian digital subscription.

      * The content isn’t the same as the print edition because some proportion of the articles are not available via the API for free – I guess this is usually for content that is sourced from another news provider whose terms and conditions don’t allow the articles to be made available for free. This is sometimes frustrating, but I don’t mind too much. In addition, the pictures are only small thumbnails, and you don’t get the crosswords, puzzles and lovely typesetting of the digital edition.

      What do you mean by the “read on kindle option from the on-line edition”?

  11. Gammakid Avatar
    Gammakid

    Hi Mark,

    Thanks for the reply,

    Sorry if that was unclear, by “on-line edition” I meant the digital (subscriber) edition, which has the option to read off-line and export to any e-reader format. Until very recently the formatting of this was dreadful. It seems like they have adressed this now (possibly in the last few days in fact) but there still seems to be no table of contents to help navigate the articles. I’ve only tried this on the desktop version of the kindle however, and although it’s an improvement on their previous offering it still doesn’t seem as nice to use as the versions via your link or from Caliber.

    So if there is any way to gauge the level of content that is excluded in the API content it would be useful in making a decision as to which path to stick with. What type of content would they typically source from other providers?

    GK

    1. mark Avatar

      Gammakid: thanks for explaining about the digital subscriber Kindle version – I didn’t know that existed.

      You can get a sense of how much is missing on any given day by going to Phil Gyford’s “Today’s Guardian” and flicking through the articles with the right arrow key looking for those whose text is just “Redistribution rights for the article body are unavailable.”

      My generated Kindle version uses the same source data, so the same articles should be unavailable in both.

      The other thing to bear in mind, of course, is that currently my version isn’t delivered automatically, as the periodicals you would buy from Amazon are. I’m working on an option to have it delivered to @free.kindle.com email addresses but that’s not done yet. On the other hand, I think this version is a very nice reading experience.

  12. Gammakid Avatar
    Gammakid

    Thanks for the link, it seems like all the “main” articles are there, with bits missing mainly from the comments/world news sections…

    The digital edition of the Guardian is also not delivered through Amazon, it comes from http://newspaperdirect.com/ and must be downloaded each day manually, so the convenience is about the same but without the cost of monthly subscription :)

    I presume you are also aware of the version available for download through Caliber? This is also free but can be auto delivered to your Kindle through the Caliber software, i’m unsure if the articles are the same as your version however…? (i’ll have a check over the next days)

    Anyway, great work! i’m off to order a Kindle! :)

    GK

  13. Zhao Avatar

    Great Work!
    I’m trying to do some thing with the periodical format mobi file. But the output is always in book format. What version of kindlegen are you using? Could you please sent me a copy?
    Thank you very much.

  14. sanrid_01 Avatar
    sanrid_01

    Hello, your tool is really brillaint, but for two days the mobi file hasn’t been updated. Would it work?

    1. mark Avatar

      Sorry, this was due to The Guardian switching domain to theguardian.com – it should be fixed now: https://github.com/mhl/guardian-for-kindle/commit/b469bf884495c55eae0e006bdb71457373c1cdd1

  15. James Avatar
    James

    Hi Mark,

    The Guardian kindle file you produce is great, I use it a lot.

    Currently however the latest file is gdn-2014-09-15.mobi, but it should be the one for 20th September. So I assume the script did not work, nor did it for the previous four days? I have noticed this happening occasionally, not too often though, just once in a while.

    Another minor point – I have noticed that the file does not seem to pick up every single article in a section. Sometimes all the stories are picked up, but occasionally one or two or more are missing. I’m comparing the articles in your kindle file with those shown in “Today’s paper” at http://www.theguardian.com/theguardian. I don’t know if you are aware or would know why this happens. Perhaps it’s a quirk of the Guardian Open Platform.

    Cheers

    1. mark Avatar

      Hi James,

      Yes, I’m afraid the Guardian had upgraded their API to a new version and (although they gave fair warning of this) I just hadn’t had time to deal with getting a new API key. I’ve done this now, and it should be working again from tomorrow.

      As for missing articles: if an article can’t be fetched because of licensing reasons or it being inexplicably missing from the API, I include the headline anyway but with the error in place of the content. So, I think the distinction is in the source of which articles to include – rather than the URL you mention, the script uses http://www.guardian.co.uk/theguardian/all which I believe is an authoritative list of what’s in the paper edition of The Guardian, whereas the one you quoted includes materials that’s only on the web. I could well be wrong about that though.

      1. mark Avatar

        Actually, those two sources look like they have the same articles listed. If you do spot something missing, then please email me and I’ll investigate.

        1. James Avatar
          James

          Hi Mark

          Great – thanks for fixing. Large pictures are showing in articles now, also the Sudoku diagram is there.

          Re missing articles – in today’s Kindle file, 4 articles are missing from the G2 section: Bay City Rollers, The Himalayan Boy, TV Highlights and Steve Bell.

          It seems that all articles that are presumably without distribution rights were returned by the Guardian Open Platform with 404 errors in today’s file and so don’t have article links.

          On a lesser note, I notice that articles can be in a different order. For example, the first article in the Main section today is the Ed Miliband one, but in today’s Kindle file Miliband is the second article, and the first article is about Tesco. Not that this matters of course, both articles there.

          Interesting that the Lego article shown at the end of http://www.theguardian.com/theguardian/all seems to be missing from http://www.theguardian.com/theguardian/g2 which does however have the Lego picture gallery page which is not included in http://www.theguardian.com/theguardian/all.

  16. Paul Symmonds Avatar
    Paul Symmonds

    Hi – does this still work? Recently I’ve noticed that the filesize has reduced dramatically and it only includes a small number of articles

    1. mark Avatar
      mark

      Oops, the Guardian slightly changed the format of the links on the “today’s Guardian” page, so it wasn’t picking any up. I’ve updated the script now and regenerated today’s edition. Is that better now?

      1. Paul Symmonds Avatar
        Paul Symmonds

        Hi Mark – sorry for the late reply, I completely forgot about this.

        Works perfectly, thank you very mych. Great work!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.