Static Indieweb pt1: Syndicating Content

One of the core principles of the IndieWeb is that people should own their own content. Controlling how and where they publish makes users more independent from big content silos.

However, the main reason why people publish on Twitter / Medium or other platforms is that they can reach a much bigger audience there - everyone's on them, so you have to be too. Publishing on a personal site can cut you off from those readers. That's why it might be a good idea to automatically post copies of your content on these sites whenever you publish something new.

This practice is known as "POSSE" (Publish on your Own Site, Syndicate Elsewhere). It enables authors to reach people on other platforms while still keeping control of the original content source.

For the recent relaunch of my personal website, I wanted to embrace some of these ideas. I included a section called notes featuring small, random pieces of content - much like tweets. These notes are perfect candidates for syndication to Twitter.

Syndication on Static Sites

My site is built with Eleventy, a static site generator based on node, and hosted on Netlify. Static sites are awesome for a variety of reasons, but interacting with other platforms typically requires some serverside code - which they don't have.

Luckily though, Netlify provides a service called "Functions", which lets you write custom AWS lambda functions without the hassle of dealing with AWS directly. Perfect! 🤘

A content feed

The first step is to publish a machine-readable feed of the content we want to syndicate. That's exactly what RSS-Feeds are for - but they're usually in XML format, which is not ideal in this case.

For my own site, I chose to provide notes as a simple JSON object. I already have an atom feed for content readers, and JSON makes the note processing easier later on.

My feed looks something like this:

// notes.json
[
{
"id": 1,
"date": "2018-12-02T14:20:17",
"url": "https://mxb.at/notes/2018-12-02/",
"content": "Here's my first note!",
"syndicate": true
},
{...}
]

All entries also include a custom syndicate flag that overrides the auto-publishing behaviour if necessary.

Event-Triggered Functions

Now for the tricky part: we need to write a lambda function to push new notes to Twitter. I won't go into detail on how to build lambda functions on Netlify, there are already some great tutorials about this:

💡 Hint: also check out the netlify-lambda cli, a very handy tool to test and build your functions in development.

To trigger our custom function everytime a new version of the site was successfully deployed, we just need to name it deploy-succeeded.js. Netlify will then automatically fire it after each new build, while also making sure it's not executable from the outside.

Whenever that function is invoked, it should fetch the list of published notes from the JSON feed. It then needs to check if any new notes were published, and whether they should be syndicated to Twitter.

// deploy-succeeded.js
exports.handler = async () => {
return fetch('https://mxb.at/notes.json')
.then(response => response.json())
.then(processNotes)
.catch(err => ({
statusCode: 422,
body: String(err)
}))
}

Since we will have to interact with the Twitter API, it's a good idea to use a dedicated helper class to take some of that complexity off our hands. The twitter package on npm does just that. We will have to register for a developer account on Twitter first though, to get the necessary API keys and tokens. Store those in your project's .env file.

// Configure Twitter API Client
const twitter = new Twitter({
consumer_key: process.env.TWITTER_CONSUMER_KEY,
consumer_secret: process.env.TWITTER_CONSUMER_SECRET,
access_token_key: process.env.TWITTER_ACCESS_TOKEN_KEY,
access_token_secret: process.env.TWITTER_ACCESS_TOKEN_SECRET
})

Right. Now we need to look at the notes array and figure out what to do. To keep it simple, let's assume the latest note is a new one we just pushed. Since the JSON feed lists notes in descending date order, that would be the first item in the array.

We can then search twitter for tweets containing the latest note's URL (we will include that in every syndicated tweet to link back to the original source). If we find anything, then it's already been published and we don't need to do anything. If not, we'll go ahead.

const processNotes = async notes => {
// assume the last note was not yet syndicated
const latestNote = notes[0]

// check if the override flag for this note is set
if (!latestNote.syndicate) {
return {
statusCode: 400,
body: 'Latest note has disabled syndication.'
}
}

// check twitter for any tweets containing note URL.
// if there are none, publish it.
const search = await twitter.get('search/tweets', { q: latestNote.url })
if (search.statuses && search.statuses.length === 0) {
return publishNote(latestNote)
} else {
return {
statusCode: 400,
body: 'Latest note was already syndicated.'
}
}
}

Next, we need to prepare the tweet we want to send. Since our self-published note does not have the same restrictions that twitter has, we should format its content first.

My implementation simply strips all HTML tags from the content, makes sure it is not too long for Twitter's limit, and includes the source url at the end. It's also worth noting that Eleventy will escape the output in the JSON feed, so characters like " will be encoded to " entities. We need to reverse that before posting.

// Prepare the content string for tweet format
const prepareStatusText = note => {
const maxLength = 200

// strip html tags and decode entities
let text = note.content.trim().replace(/<[^>]+>/g, '')
text = entities.decode(text)

// truncate note text if its too long for a tweet.
if (text.length > maxLength) {
text = text.substring(0, maxLength) + '...'
}

// include the note url at the end.
text = text + ' ' + note.url
return text
}

When everything is done, we just need to send our note off to Twitter:

// Push a new note to Twitter
const publishNote = async note => {
const statusText = prepareStatusText(note)
const tweet = await twitter.post('statuses/update', {
status: statusText
})
if (tweet) {
return {
statusCode: 200,
body: `Note ${note.date} successfully posted to Twitter.`
}
}
}

Hopefully that all worked, and you should end up with something like this in your timeline:

🎉 You can find the finished lambda function along with the rest of the source code for this site on Github.

Further Resources

Webmentions

  1. Marty McGuire
    Great write-up! I've been meaning to play with Netlify functions and this post is full of learning goodies!
  2. Max Böck
    Thanks marty! glad it's useful.
  3. Michael Bishop
    Thanks for sharing!
  4. René Stalder
    What a beautiful idea: Writing and publishing tweets on the own website and then send copies to Twitter. And a very nice use case of serverless and FaaS.
  5. NorthStack™
    Syndicate content from static websites using AWS lambda functions #AWSlambdafunctions #serverless mxb.at/blog/syndicati… Via HackerNews news.ycombinator.com/item?id=188413…
  6. Netlify
    Excellent work by @mxbck - Syndicating content from your #JAMstack site out to Twitter with use of Netlify Functions. An excellent walkthrough.
  7. Lobsters
    Static Indieweb pt1: Syndicating Content via @flyingfisch_ lobste.rs/s/j2pp1s #web mxb.at/blog/syndicati…
  8. CSS-Tricks
    > One of the core principles of the IndieWeb is that people should own their own content. Controlling how and where they publish makes users more independent from big content silos. mxb.at/blog/syndicati…
  9. Roberto Iriondo
    > One of the core principles of the IndieWeb is that people should own their own content. Controlling how and where they publish makes users more independent from big content silos. mxb.at/blog/syndicati…
  10. @robiriondo: > One of the core principles of the IndieWeb is that people should own their own content. Controlling how and where they publish makes users more independent from big content silos. mxb.at/blog/syndicati…
  11. Kevin McGillivray
    Setting up note syndication to Twitter following this guide from @mxbck: mxb.at/blog/syndicati… kevinmcgillivray.net/notes/2

Other things I've written