Twingly Blog LiveFeed API v5

Still using the legacy version LiveFeed API v4?

Introduction

Blog LiveFeed API is a HTTP based API used for receiving a continuous real-time stream of blog data. To be able to retrieve data through LiveFeed, an API key issued by Twingly must be used. The API key then grants access to blog data for one or more languages.

The responsibility of retrieving the data in a correct manner so that nothing is missed lies entirely on the client. To facilitate this, LiveFeed API exposes one endpoint, GetData, which is polled using a timestamp that’s extracted from the last response.

It is possible to request historic data from the LiveFeed API up to 30 days back in time.

API Endpoint

A GET request to /GetData will return posts from (and forward in time) a given timestamp. This method should be polled continuously, each time with a new timestamp from last results nextTimestamp, effectively creating a “live feed” of blog data.

Note that the input timestamp relates to when Twingly first made the data available through the LiveFeed API.

HTTP options

Request Parameters

Allowed GET parameters:

Parameter Format Example Notes
apikey String E67EFC65-08A9-4086-BE92-074BBD7F78EA Required
timestamp ISO 8601 (UTC) 2017-04-24T10:22:38.361Z Required
maxPosts Integer 1000 Optional, defaults to 1000. Maximum value is 5000.
format String xml Optional, defaults to xml (which is the only allowed value)

Example Request

Example request for the GetData method with maxPosts set to 1.

curl -s "https://api.twingly.com/blog/livefeed/api/v5/GetData?apikey=KEY&timestamp=2017-04-24T10:22:37.354Z&maxPosts=1"

Response

The response is an XML document containing one or more posts. See below for an example and explanation of all the elements and attributes.

Note that in older versions of the LiveFeed API, the result can look different. Only the latest version is documented here.

Example response from the example request shown above.

<twinglydata ts="2017-04-25T08:33:07.053785Z" from="2017-04-24T10:22:37.354Z" numberOfPosts="1" maxNumberOfPosts="1" firstPost="2017-04-24T10:22:38.36Z" lastPost="2017-04-24T10:22:38.36Z" nextTimestamp="2017-04-24T10:22:38.361Z">
  <post>
    <id>17052901411729148785</id>
    <author>Henrik Foto</author>
    <url>
    http://henriksandgren.blogspot.com/2017/04/norrsken.html
    </url>
    <title>Norrsken</title>
    <text>
    Äntligen har jag lyckats fånga norrsken på bild (här nere på 57:e breddgraden dyker det inte upp så jätteofta). Bilden är tagen inatt.
    </text>
    <languageCode>sv</languageCode>
    <locationCode>se</locationCode>
    <coordinates/>
    <links>
      <link>
        https://1.bp.blogspot.com/-SmKpunfW8Tg/WPxQ-nd_PSI/AAAAAAAACbo/-uiqAE-ZbpsOrcgmatB1C0gNiRIvqbjtwCLcB/s1600/IMG_1044%25282%2529.jpg
      </link>
    </links>
    <tags>
      <tag>aurora</tag>
      <tag>aurora borealis</tag>
      <tag>götaland</tag>
      <tag>ljurhalla</tag>
      <tag>norrsken</tag>
      <tag>northern lights</tag>
      <tag>vårgårda</tag>
    </tags>
    <images/>
    <indexedAt>2017-04-24T10:22:26Z</indexedAt>
    <publishedAt>2017-04-23T07:03:00Z</publishedAt>
    <reindexedAt>2017-04-24T10:22:26Z</reindexedAt>
    <inlinksCount>0</inlinksCount>
    <blogId>8079022330337403774</blogId>
    <blogName>Henriks fotoblogg</blogName>
    <blogUrl>http://henriksandgren.blogspot.com</blogUrl>
    <blogRank>1</blogRank>
    <authority>2</authority>
  </post>
</twinglydata>

Elements and attributes:

<twinglydata> is the root element with the following attributes:

Within the root element, <twinglydata>, there can be zero or more <post> elements. Note that posts that are indexed/inserted at the same millisecond have an undefined sorting order, meaning that the same query might return documents in slightly different order.

<post> is the root element for a post and contains the following child elements:

Errors

We strive to respond with the correct HTTP status code and respond with valid XML. But since computers can be tricky at times, you should ensure your client don’t blow up if we give a broken response (please contact us if we do).

4xx Client errors

Client errors, most likely your client sending invalid requests but please contact us if you can’t figure it out.

5xx Server errors

You should retry later.


Best practices

Integration example

This shows the basics required to continuously fetch data. It’s a fully functional example in Ruby 2.4, but primary purpose is to quickly illustrate how an integration should work.

# Remember to set the environment variable key `TWINGLY_API_KEY`
require "net/http"
require "time"
require "rexml/document"

BASE_URL = "https://api.twingly.com/blog/livefeed/api/v5/GetData"
params = {
  timestamp: (Time.now - 3600).utc.iso8601, # Start one hour ago
  apikey: ENV["TWINGLY_API_KEY"], # API key to access Blog LiveFeed
}

loop do
  uri = URI(BASE_URL)
  uri.query = URI.encode_www_form(params)

  Net::HTTP.get_response(uri) do |result|
    # No error handling code in this example, just stop
    break unless result.is_a?(Net::HTTPSuccess)

    document = REXML::Document.new(result.body)

    # Extract post URLs, print to stdout
    document.elements.each("twinglydata/post/url") do |url_element|
      puts url_element.text
    end

    # Extract helper attributes to control the polling
    attributes = document.root.attributes

    next_timestamp      = attributes["nextTimestamp"].value
    number_of_posts     = attributes["numberOfPosts"].value.to_i
    max_number_of_posts = attributes["maxNumberOfPosts"].value.to_i

    # Update timestamp for next iteration
    params[:timestamp] = next_timestamp

    # Sleep a while if there's no more documents available
    sleep 60 if number_of_posts < max_number_of_posts
  end
end

With this approach you will get a continuous stream of all the data that the API key grants access to. Of course, error handling must be added to handle network outages, server failure et cetera.


Available data

As soon as we get new blog content it will be available through the LiveFeed API. Most often it takes just a few seconds for data to flow through our system, it may be a few minutes during maintenance though.

Some notes:


Known issues


Clients


Changes from previous version

We have focused on simplifying the API request and extending the response.

Changes to output XML:

Migrating from LiveFeed v4

These are the minimum required steps to move from v4 to v5:

Then you should get the same functionality as before, but feel free to utilize the new data now provided by Twingly Blog LiveFeed API v5.


Documentation changelog

API changelog