indie-org

Putting Org mode on the Indieweb.

This manual corresponds to indie-org version 0.5.2.

Table of Contents


1 Introduction

Many of us maintain personal websites using Org Mode. An Org-generated static site has advantages over full-blown Content Management Systems: authoring pages in Org, working in Emacs Lisp, unparalleled control & customizability over the process, and the simplicity of not having to leave your Emacs environment to name a few.

However, that simplicity comes with costs such as fewer features; such a site will not be able to handle commenting or support user login, for instance. Furthermore, in an age of social media, an independent site can come to feel isolated altogether, existing outside the conversations taking place on sites like Twitter & Mastodon.

Enter the Indieweb: “The IndieWeb is a community of independent & personal websites connected by simple standards, based on the principles of: owning your domain & using it as your primary identity, publishing on your own site (optionally syndicating elsewhere), and owning your data.”

The Indieweb (See The Indieweb) offers protocols, idioms & services for connecting your site to those of others, and even to other ecosystems such as Twitter & Mastodon (“silos” in the Indieweb’s parlance).

indie-org is an Emacs package for integrating an Org-generated, static site with the Indieweb. This manual describes indie-org for the interested site author. Throughout, it will reference a simple static site, indie-org.sh. It is live on the web, and its source is available at Github. The reader may wish to refer to the source while reading this manual.


2 Background


2.1 The Indieweb

Per indieweb.org: “the Indieweb is a collection of protocols for connecting to other independent sites, pushing your content to social media sites, collecting likes, comments & responses from other sites _back_ to yours, and many other things as well.”

Many of the Indieweb principles would likely appeal to the Emacs Org Mode user, and the Org Mode static site owner in particular:

  1. Own your data; your content, your metadata & your identity.

    The Org Mode static site owner has already taken a large step in this direction.

  2. Make what you need & use what you make

    Nothing more need be said.

  3. Document your stuff

    This manual is a step in that direction on the author’s part.

This section covers a few aspects of the Indieweb that are pertinent to indie-org.


2.1.1 Microformats

Microformats are a fundamental building block of the Indieweb. “Designed for humans first and machines second, microformats are a set of simple, open data formats built upon existing and widely adopted standards”. They are a set of tags which one may add to one’s HTML to indicate semantic rather than formatting information. For instance, consider mentioning a person in your prose. In plain text, one could say:

Barnaby Walters

and in HTML, to indicate a location on the internet, one could say:

<a href="http://waterpigs.co.uk">Barnaby Walters</a>

Microformats2 adds additional information to that link, the information that this names a person, by adding a particular class to the HTML element:

<a class="h-card" href="http://waterpigs.co.uk">Barnaby Walters</a>

Microformats allow you to build structure, for instance giving this reference to a person both a location on the web and a photo:

<div class="h-card">
    <p><img class="u-photo" href="/me.png" alt="" /></p>
    <p class="p-name"><a href="u-url" href="http://waterpigs.co.uk">Barnaby Walters</a></p>
</div>

You can find a complete vocabulary here, and an on-line Microformats parser here.

Of particular note is that, as static site authors, the onus is on us to mark-up our HTML with microformts. We’ll use the page template in the Org Export back-end to do so (see below).

The Microformats Wiki is here and mf2py is a commonly used Python library for parsing Microformats.


2.1.2 IndieAuth

In a world in which everyone owns their own identity, how shall we authenticate one another? OAuth 2.0 has become increasingly common, but suffers from a few shortcomings that make it unsuitable for use on the Indieweb.

In the first place, the protocol requires an app to register with the authorization server to be used (see the RFC, section 2). This makes sense when, say, building a Facebook app, or a Twitter app, but breaks-down in a world in which everyone is their own authorizing authority.

The second issue is simply that OAuth 2.0 is not an authentication protocol– it is a protocol designed to authorize applications to access resources on a principal’s behalf. Without augmenting the protocol in some way, the app will have no information that can identify the user on their app– just a grant of permissions to carry out certain operations on the resource provider.

This led Aaron Parecki and others to develop a related protocol– IndieAuth. Here, your identity is your domain name. When you wish to authenticate with a site, you no longer pick one of a few silos (“sign-in with Google”, “sign-in with Facebook”, and so forth), but instead provide your domain name. The site will fetch your page (or, at least, its head) to discover your IndieAuth provider endpoints.

From there, the flow largely proceeds in the same manner as OAuth with the additional bit that the authorization endpoint will provide not only a token, but a canonical URL identifying the user. The full authorization flow is described here.

This is an important step forward, but making everyone on the Indieweb responsible for implementating an authentication server would not be very practical. Fortunately, there are multiple standalone implementations available, beginning with IndieAuth.com. This is an IndieAuth authentication implementation that will allow one to authenticate using either one’s existing social media accounts or PGP keypair, even (or, perhaps, especially) when using a static site.

In order to integrate with IndieAuth.com, the site author needs to add a link to each account to the home page with the attribute rel="me", and ensure that those accounts list your home page in their profiles:

<ul>
  <li>
    <a href="https://twitter.com/jdoe" rel="me">Twitter</a>
  </li>
  <li>
    <a href="https://github.com/jdoe" rel="me">Github</a>
  </li>
  ...
</ul>

This manual will cover the details for a sample site See below.


2.1.3 Webmentions

Webmention is an open standard for distributed interactions among independent web sites.

webmentions-general

Suppose Alice posts to her site. The content is presumably HTML, but could be something else (plain/text, say). Bob now wants to mention that content on his site. He makes a post (the source) that contains the URL of Alice’s post (the target). Again, the source is presumably an HTML document, but may be anything that can express an URL.

Bob, or more likely his publishing software, must now discover Alice’s Webmention endpoint. He does this by fetching Alice’s document via HTTP– if it has a Link header with a rel value of “webmention” that’s Alice’s webmention endpoint (this is how sending webmentions of non-HTML documents is supported). If there is no such header, and the document is HTML, Bob must look through the document itself for <link> or <a> elements with rel values of “webmention”– the first one in the document is Alice’s Webmention endpoint.

Bob then posts x-www-form-urlencoded source & target parameters to Alice’s Webmention endpoint (the target parameter being Alice’s post and the source being Bob’s mention thereof). Alice’s Webmention endpoint will typically do some initial validation of the request, and on success return either “201 Created” or “202 Accepted” to indicate that further processing will take place asynchronously (in the former case, the Location header in the response will include an URL at which the request processing status may be checked).

Asynchronously, Alice verifies the Webmention by issuing an HTTP GET on the source and verify that the source document contains an exact match for the URL given in the target parameter of the Webmention.

At this point, Alice may choose to publish content from Bob’s mention along with any other data it obtained (e.g. “Bob liked...”).

You can see an example of this in action (in conjunction with see POSSE) here.


2.1.4 POSSE

“POSSE” stands for Publish on your Own Site, Syndicate Elsewhere” (or Everywhere). POSSE is less a formal protocol (like see Webmentions) than a feature you may want to build into your site.

The idea is that, after posting to one’s site, one then pushes copies of, or links to, that new post to assorted third-party sites (Twitter, for instance). This allows readers who prefer those sites to remain aware of your activity without leaving their prefered experience. This feature can be extended to collect responses to the POSSE’d content within the third-party sites (likes, replies and so forth) and feed that back to your site: a list of Tweets and/or Toots that have replied to you underneath your original post, for instance.

You can see an example of this in action (in conjunction with see Webmentions) here.

It stems from the Indieweb principle of owning your data rather than trusting it to a third-party site that may go away or simply decide to deny you access to your own content. It also stems from the hard fact that many of our friends & colleagues are still on such third-party sites such as Twitter or Mastodon: this is a way to connect your content thereto.

Implementations are generally bespoke; indie-org supports this for the site author via brid.gy (see see Supporting POSSE).


2.2 OrgMode

This manual assumes that the reader is generally familiar with Org mode. If they are not, the Org Manual, site and Worg are excellent resources for getting started.

This section will instead concern itself with deeper explanations of a few aspects of Org mode of particular interest to indie-org: the Org export process and the question of standardizing Org syntax and its implications for static site generation based on Org mode documents.


2.2.1 The Export Process

This subsection outlines the generic Org Export process with an eye toward customizing it for our purposes. While we defer the fully-customized implementation to See Customizing the Publication Process, it is here that we introduce indie-org.sh, a live example of an Org-generated static site built on top of indie-org.

The Org Export publication process, in broad terms, looks like this:

generic-org-pub-flow

The main entrypoint to the Org Export facility is See org-publish-all. org-publish-all refers to the variable See org-publish-project-alist, which describes our publication “project” to Org Mode. For each project (the root project can have sub-projects), Org Export will iterate over each file, copying it to its configured destination and, in general, transforming the contents in some way.

Of particular note is the info, or plist argument which is initialized at the beginning of the process of publishing each project (in org-publish-file) and is passed down through the entire call stack. It begins as the project plist (i.e. the project entry in org-publish-project-alist) and is augmented at each step. It is a property list that the different pieces of the publication pipeline use to communicate, and indie-org will use it as well.

The process of transforming the Org Mode document to another format is governed by the :publishing-function attribute for each project. It is invoked with the input filename, the publication directory, and the project plist.

indie-org.sh has two sub-projects salient to this discussion: “pages” and “posts”. The former contains a few top-level pages (home, about & so forth) whereas the latter contains all the site posts. Each employs a lambda as its publishing-function; the lambdas simply provide one more argument to the true publishing implementation, iosh/publish-page. The additional argument is the Org Export back-end to use ( see Back-end in Adding Export Back-ends ).

iosh/publish-page passes that back-end on to org-publish-org-to and thence to org-export-to-file and org-export-as, which is where the real work begins. The function has a lot of functionality packed into its 171 lines; of note to us, it will:

  1. Parse the buffer being exported into Org elements (see see org in Org Syntax).
  2. Give the export backend the opportunity to filter the resulting parse tree; we’re not doing that so far in this example, but will take advantage of this below (see Controlling Content).
  3. Transcodes the resulting parse tree to HTML
  4. Extract the template from the backend & ask it to produce an HTML document from that HTML
  5. Offers the backend the opportunity to “finalize” the resulting page; that is, to run arbitrary code after the HTML document has been produced.

We introduce an Emacs Lisp file, indie-org.sh.el to the project where we can setup this boilerplate & invoke org-publish-all:

(defun iosh/publish (prod)
  "Publish indie-org.sh to production if PROD is non-nil, locally else."
  ;; Define a derived backend to insert our template:
  (org-export-define-derived-backend
      'iosh/page-html
      'html
      :translate-alist '((template . iosh/page-html)))
  (org-export-define-derived-backend
      'iosh/post-html
      'html
      :translate-alist '((template . iosh/post-html)))
  ;; ...
  (let* ((publishing-root (concat (file-name-as-directory project-dir) "www"))
         (org-publish-project-alist
          `(("indie-org.sh" :components ("pages"))
            ("pages"
             :base-directory             ,(concat project-dir "pages")
             :publishing-directory       ,(concat project-dir "www")
             :publishing-function        ,(list
                                           (lambda (plist filename pub-dir)
                                             (iosh/publish-page plist filename pub-dir 'iosh/page-html)))
             :html-doctype               "html4-strict"
             ...
             )
            ("posts"
             :base-directory             ,(concat project-dir "posts")
             :publishing-directory       ,(concat project-dir "www/posts")
             :publishing-function        ,(list
                                           (lambda (plist filename pub-dir)
                                             (iosh/publish-page plist filename pub-dir 'iosh/post-html)))
             :html-doctype               "html4-strict"
             ...
             ))))
      (org-publish-all t)
      ;; ...
      ))

2.2.2 Can Org Mode Documents Exist Outside of Emacs?

It has been suggested to the author that he has built far too much functionality in Emacs Lisp, and that if one is interested in putting a staticly generated site on the Indieweb, it is better implemented as a purpose-built program written in a general-purpose programming language.

Perhaps. However, if one is committed to writing in Org Mode, that avenue of attack on the problem leads us to an interesting question: how can we parse Org Mode files outside of Emacs?

Today, the reference implementation of an Org Mode parser is... Emacs Org Mode. The syntax is documented here and the implementation, in org-element.el is unusually readable, but it all exists within Emacs.

There are (numerous) parsers available, in a wide variety of languages, with various levels of fidelity to Org Mode claimed. There is an effort ongoing at the time of this writing on the emacs-orgmode list to standardize a set of test cases to make compliance more rigorous. Karl Voit independently gave a talk at EmacsConf 2021 proposing a specification which he called “Orgdown”.

So it may be that in the not too distant future, we will have Org Mode parsers in general-purpose languages which can be validated against some sort of specification, or at least a well-known test suite. However, and this has been discussed at length on the emacs-orgmode list, such a parser will either need to re-implement much of Emacs or be incomplete: how should such a parser handle Babel, for instance? Table formulas that invoke arbitrary Emacs Lisp functions? The author uses a custom Org Mode property, #+AUTODATE in his documents; if set to t, the #+DATE property will be updated every time the buffer is saved– how should that be handled?

Some argue that such a specification could only cover the “static” portions of an Org Mode document, but it’s not clear to the author exactly how that would work in the case of the “dynamic” portions generating document content.

The answer is not clear to me at this time. However, having implemented the functionaly to support putting my own sites on the Indieweb, it seemed reasonable to make that functionality available to others.


2.3 The Limitations of Org Mode and Where indie-org Comes In

We’ve seen that Org Mode is a flexible and extensible way to publish HTML from within Emacs. There are numerous ways in which one could push the resulting HTML to a server and thereby serve a static website. That said, a static site can be... limiting.

It is a way of broadcasting content to a passive audience, but no more. It provides no way for readers to respond to the content. Commenting as a service is available (from Disqus, for example), but in the first place such services always feel grafted-on to the host site, and in the second, if you’re going to the trouble to host your own site, are you really comfortable offloading the resulting conversations to a third party?

Additionally, although consumers are becoming increasingly chary of entrusting their online identities (and data) to sites like Twitter, the reality is that a great deal of public coversation continues to take place there (and on Facebook, Reddit, &c). A self-hosted, static web site offers no connection to such fora.


3 Building a Static Site on Org Mode & indie-org

The indie-org.sh project was introduced in See The Export Process. This chapter will flesh-out the implementation details, including authoring Microformats. The next chapter, See Adding Indieweb Support, will cover truly adding the site to the Indieweb by adding things like Indieauth, Webmentions & POSSE support.


3.1 The Project Structure

The site itself is structured with a single top-level page and a series of posts beneath that (in the posts directory). It is hosted in an AWS S3 bucket fronted by Cloud Front.

The source for each page or post is an Org document. The top-level page sources can be found in pages and that for posts in posts. The Org documents are published to HTML via HTML export into a directory named www.

The publication process is run by invoking Emacs, telling it to load the file lisp/indie-org.sh.el, and telling it to invoke the function iosh/publish therein:

emacs --batch -l lisp/indie-org.sh.el --eval '(iosh/publish)'

When publishing to the live site, the www directory is simply copied to the S3 bucket using the aws CLI.

Since it is tedious to type these commands over & over, a Gnu Makefile is provided: building the site locally is make and publishing it is make prod.

The remainder of this chapter is a guided tour of the code intended to leave the reader in a position to put their own Org-generated static site on the Indieweb. Much of the work has been encapsulated in the indie-org Emacs package, but there is still a fair amount of per-site customization required.


3.2 Customizing the Publication Process

We customize See The Export Process in order to produce microformats in our generated HTML and to send Webmentions & POSSE requests:

customized-org-pub-flow

This section will broadly sketch the customizations; subsequent sections will detail implementing each aspect.

In order to emit See Microformats, we’re going to need to get our fingers into the process of transcoding Org Mode to HTML; this entails defining our own backend(s) and setting the :publishing-function for each project (see see Where & How to Author Microformats).

In order to support IndieAuth, we’ll need to generate a PGP keypair and/or setup accounts with one or more social media sites (see see Supporting IndieAuth).

In order to note webmentions made on each post, we’ll need to define new link types, so that we can indicate that we’d like to mention another post by authoring Org Mode such as:

as you can see [[mention:https://cool-site.org/cool-post.html][here]]...

In the export function for such link types, we’ll note the webmentions in the page’s informational property list; then, in the project :final-output filter we can extract all of that page’s mentions and add them to the site’s list (see see Supporting Webmentions.

POSSE is handled in the page template– since at this point the Org Mode document has been parsed, we can simply access them as properties and add them to the site’s list of POSSE requests.

In the foregoing, we’ve mentioned site-wide lists of webmentions made & POSSE requests. These lists reside in indie-org-state-v2. See See Publication Environments & State for details.


3.3 Where & How to Author Microformats

When one “publishes” an Org Export project, each file in the project is copied to the destination directory and perhaps transformed in the process (see Publishing in Org Mode, OrgMode) The publication operation is determined by the project’s :publishing-function property, which defaults to org-html-publish-to-html. This function, in turn, invokes org-publish-org-to with the html back-end.

The details of the Org Export process are determined by the selected back-end (see Exporting in Org Mode, Org Mode). Of particular interest is the template member of each back-end’s translate-alist attribute– this shall be a lisp form that receives the abstract syntax tree for each Org Mode document published and is responsible for returning the text to be written to the output file (HTML, in our case)– this is where we emit our microformats.

This leaves us with a quandry: how shall we support different post types? One option would be to implement it in the template function: have the template extract the post-type document property, figure-out what sort of post we’re publishing, and emit the corresponding HTML.

Alternatively, we could implement it in the publication function– that is, have a different back-end for each post type. This is less convenient because you need to know the post type before the file is parsed (that is, all you’ve got at publication time is a filename).

So, let us define two Org Export projects, one for top-level pages on our site (home, about & so forth) and one for posts. The latter shall eventually support different post types.


3.4 Now We Have a Static Web Site

At this point, we have a web site which, while static, is already marked-up with Microformats. We can publish locally by saying make and to the web by saying make prod. In the next chapter, we’ll add assorted IndieWeb features to it.


4 Adding Indieweb Support


4.1 Page Keys

Throughout indie-org, we need a way to "name" pages on the site. For the most part, indie-org requires a page-key to simply be a string that uniquely identifies each page on the site. There are a few instances, however, in which we need a bit more structure:

  1. It must be possible to map an URL naming the page back to a page-key (when retrieving Webmentions)
  2. At page publication time, we need to know the page key corresponding to the page being published is/will be (for recording Webmentions to be sent)

It would be possible to make this parameterizable by, say, providing customizable variables holding functions for performing these two operations. As a first implementation, however, indie-org simply defines "page key" in the obivous way: as the page’s path in a URL naming it on the site (i.e. if the page lives at “https://foo.net/a/b/c.html” on your site, its page-key shall be “a/b/c.html”).


4.2 Publication Environments & State

Supporting various Indieweb features such as Webmentions & POSSE requires maintaining state regarding site publication. For instance, every time one publishes the site (whether to a local test server or to the live site) one now needs to scribble down new Webmentions that need to be sent for subsequent use. Incoming webmentions also need to be recorded upon receipt for potential use in subsequent publications.

indie-org implicitly defines the idea of “publication environment” to represent the various places to which one may wish to publish. This is implicit because they are named by arbitrary symbols chosen by the site author. For instance, one may choose to examine one’s site after simply publishing HTML to the local filesystem in a browswer with a “file:” URL, and name this publication environment :local.

Perhaps one has an Apache instance running locally for testing purposes: that could be named :staging. The only assumption made by indie-org is that the default site is named :prod.

indie-org associates each production environment with an instance of the struct indie-org-state-v2.

The struct contains the fields:

  1. last-published: the last time the site was published to this environment
  2. webmentions-made: a mapping from page-key to a list of Webmentions made by that page
  3. webmentions-sent: a mapping from page-key to a list of all Webmentions sent to-date for that page
  4. webmentions-received: a mapping from page-key to a list of all Webmentions received by that page
  5. posse-requests: a mapping from page-key to a list of POSSE requests for that page (i.e. an expressed desire for the post to be POSSE’d to Twitter, or Mastodon, or wherever)
  6. posse-responses: a mapping from page-key to a list of POSSE responses for that page (i.e. a record of a completed POSSE request for the page to various siloes).

While it is up to the site author to incorporate state into their publication process, indie-org provides basic utilities for doing so.

Use indie-org-make-publication-state to create a new indie-org-state-v2 instance, indie-org-state-write to write a property list mapping publication environment to state to file, and indie-org-state-read to read it back out.

A site’s publication function might then look something like this:

(defun my/publish (prod)
  "Publish my site to production if PROD is non-nil, locally else."
  (indie-org-enable)
  (let* ((env (if prod :prod :local))
         (pub-states
          (if (file-exists-p publication-state-file)
           (indie-org-state-read publication-state-file)
         `(:prod ,(indie-org-state-make) :local ,(indie-org-state-make))))
         (publication-state (plist-get pub-states env)))
    ;; Setup the call to `org-publish-all'.
    (let* (...)
      (org-publish-all t)
      ;; Update state
      ...
      (plist-put pub-states env publication-state)
      ;; and write it all out to disk.
      (indie-org-state-write pub-states ".state"))))

4.3 Supporting IndieAuth

Adding See IndieAuth support to one’s site requires no coding. This section will instead sketch the steps necessary to support IndieAuth to the extent necessary for adding additional IndieWeb features.


4.3.1 Using PGP

You’ll need to generate a PGP keypair to identify your site, if you don’t already have a keypair you’d like to use. To generate a new keypair, say gpg --gen-key and follow the prompts. Think ahead of time about the e-mail you’d like to associate with the new keypair.

Once you’ve got your keypair, you can get its identifier by saying gpg --list-keys (the identifier is the long hexadecimal string listed for each public key on your keyring). Say gpg --output PATH-TO-YOUR-SOURCE/site-public.pgp --export KEYID to put a copy of your site’s public key somewhere in your site’s source project directory.

Add a link to your home page like:

<link rel="pgpkey" href="/site-public.pgp>

To test, go to https://indieauth.com and scroll down to the “Try It!” section.

Note that IndieAuth.com supports neither Twitter nor Mastodon for authentication purposes.

You should see a green button labelled “GPG” along with your domain & the name of your public key. Click that and you’ll be presented with a string to sign with your private key (demonstrating ownership of the private key). Copy the challenge text and say echo 'CHALLENGE-TEXT'|gpg -u KEYID --clearsign --armor. This should produce output something like:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

eyJ0eXAiOiJKV1Qi...
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCAAdFiEEr2ZwJM99NLbLOVkMca2Hl/Lwd1cFAmQqHdQACgkQca2Hl/Lw
d1e1jwv/bpIrpr7+WNfD1xfiNzkq+PzbeeMT07B8kHo3ZKXJINB420jO3P+QqM8G
S1WQF2XyhnzxKmo/ySk54HOV5iWZ62uBHIrn/Nn6YUBvVQUB6CiF0zeCvKrbreW/
3omdbdLfCryPAMd120sQi8mQ5fDr798jq8Oq7QyIA4WusIh3ZesoDYboE4VJKryK
...
-----END PGP SIGNATURE-----

Paste the entire messages into the box.


4.3.2 Using Social Media Sites

While IndieAuth does not support login via other sites, such as Twitter or Mastodon, IndieLogin does, and the services we’re going to be using use the latter, not the former.

In order to do this, the site author must mark-up their home page with links to their social media profiles with the rel="me" attribute, e.g.:

<link href="https://twitter.com/jdoe" rel="me">

On each service listed, ensure there’s a link back to the site’s home page.

Supported services:

  1. Twitter
  2. Mastodon
  3. Github
  4. PGP key

4.4 Supporting Webmentions

This section will lay-out the process of adding See Webmentions support to the static site.


4.4.1 Making & Recording Webmentions

The first step in making a Webmention is to express that intent in our Org Mode source files. indie-org implements this by See Adding Hyperlink Types in Adding Hyperlink Types, Adding Hyperlink Types.

When the site author invokes indie-org-webmentions-enable four new link types will be defined:

  1. mention
  2. reply
  3. like
  4. repost

Each link type has a few properties, in particular :follow and :export. indie-org’s link types all implement follow in the same way as standard HTML links; it’s the export aspect in which they differ. While they will all render as standard HTML, they will take the opportunity to note each of them via indie-org-webmentions-record-webmention

For example, the author can indicate a mention like so:

I am mentioning Alice's [[mention:https://alice.net/i-made-a-thing.html][post]].

The site author shall include their publication state’s indie-org-webmentions-made (see Publication Environments & State) in the project properties (under the name :indie-org/webmentions-made); at export time, the custom link types will, first, produce See Microformats appropriate to the sort of webmention being made, then invoke indie-org-webmentions-record-webmention to record them in the current publication environment’s state:

(defun iosh/publish (prod)
  "Publish indie-org.sh to production if PROD is non-nil, locally else."
  (indie-org-enable)
  ;; ...
  (let* ((env (if prod :prod :local))
         (pub-states
          (if (file-exists-p publication-state-file)
           (indie-org-state-read publication-state-file)
         `(:prod ,(indie-org-state-make) :local ,(indie-org-state-make))))
         (publication-state (plist-get pub-states env))
         (webmentions-made
          (or (indie-org-state-v2-webmentions-made publication-state)
              (indie-org-webmentions-make-made)))
         ;; ...
         )
    ;; This is all to setup the call to `org-publish-all'.
    (let* (
           ;; ...
           (org-publish-project-alist
            `(("indie-org.sh" :components ("h-feed" "pages" "posts" "rss"))
              ("posts"
               ;; ...
               :indie-org/webmentions-made ,webmentions-made))))
      (org-publish-all t)
      ;; During the course of publication, we may have updated state: in
      ;; particular, we may have made new webmentions & POSSE requests.  Update
      ;; the publication state's most recent publication time...
      (setf (indie-org-state-v2-last-published publication-state)   (current-time)
            ;; along with new requests:
            (indie-org-state-v2-webmentions-made publication-state) webmentions-made)
      (plist-put pub-states env publication-state)
      ;; and write it all out to disk.
      (indie-org-state-write pub-states publication-state-file))))

You can see an example on indie-org.sh here.


4.4.2 Sending Webmentions

At this point, assuming the site author has made & recorded one or more webmentions (see made & recorded), they presumably want to send them.

Despite the protocol’s simplicity, sending Webmentions is a non-trivial task. The sender needs to carry-out Webmention endpoint discovery on their targets (which itself involves fetching & parsing the resulting document). Then there’s the question of handling failures to send & re-trying. Finally, since Webmention processing is generally done asynchronously, the sender has to “check back” to get the final result.

Aaron Precki has built a service to handle this: Telegraph. Once you sign-up (using See IndieAuth, of course), you can send a Webmention with a single API call over HTTP.

indie-org simplifies this further with indie-org-webmentions-send a function that takes a source, target, and Telegraph API token and will send the Webmention (recall that the precise sort of Webmention is not encoded in the protocol; the receiver discovers it by parsing the sender’s post).

We can record sent Webmentions in indie-org-webmentions-sent, another attribute of indie-org-publication-state. The process of taking both & determining what Webmentions are to be sent is encapsulated in indie-org-webmentions-required.

So equipped, we can now code-up a function in our site’s Lisp that we can call post-publication that will handle sending all new Webmentions:

(defun iosh/send-webmentions (prod)
  "Send webmentions post-publication.
PROD shall be set to t for production and nil for staging."
  (indie-org-enable)
  (let* ((env (if prod :prod :staging))
         (all-pub-states
          (if (file-exists-p publication-state-file)
              (indie-org-state-read publication-state-file)))
         (publication-state
          (or (plist-get all-pub-states env)
              (indie-org-state-make)))
         (webmentions-made
          (or (indie-org-state-v2-webmentions-made publication-state)
              (indie-org-webmentions-make-made)))
         (webmentions-sent
          (or (indie-org-state-v2-webmentions-sent publication-state)
              (indie-org-webmentions-make-sent)))
         (webmentions-to-send
          (indie-org-webmentions-required webmentions-made webmentions-sent))
         (token
          (indie-org-read-token
           telegraph-io-token-file))
         (now (current-time)))
    (while webmentions-to-send
      (let* ((wm (car webmentions-to-send))
             (location
              (if prod
                  ;; The source is just the page key
                  ;; (e.g. "blog/foo.html")-- it is only here that we
                  ;; know we're sending this for prod, so prepend the
                  ;; "https://indie-org.sh" here:
                  (let ((src (car wm))
                        (dst (cdr wm)))
                    (indie-org-webmentions-send
                     (cons (concat "https://indie-org.sh" src) dst)
                     token))
                (message "Will send webmention: %s :=> %s" (car wm) (cdr wm))
                nil)))
        (indie-org-webmentions-record-sent
         wm
         now
         webmentions-sent
         location))
      (setq webmentions-to-send (cdr webmentions-to-send)))
    (setf (indie-org-state-v2-webmentions-sent publication-state) webmentions-sent)
    (plist-put all-pub-states env publication-state)
    (indie-org-state-write all-pub-states publication-state-file)))

4.4.3 Receiving Webmentions

Receiving Webmentions may seem like a particularly daunting problem for the staticly generated site owner. In order to receive them, we need a server capable of handling the HTTP posts and implement the server-side of the protocol (Webmention validation, asynchronous processing, possibly another endpoint at which senders can check their results).

It turns out, there’s a service for that: webmention.io. Once you sign-up via See IndieAuth, advertise your Webmention endpoint like so:

<link rel="webmention" href="https://webmention.io/indie-org.sh/webmention" />
<link rel="pingback" href="https://webmention.io/indie-org.sh/xmlrpc" />

Webmention senders will now POST to webmention.io who will store them on your behalf. You can retrieve them via API call to webmention.io.

indie-org handles this for you via indie-org-webmentions-check. This function takes the domain name for which you are checking Webmentions, your webmtion.io API token, and an indie-org-webmentions-received struct (another attribute of indie-org-publication-state). indie-org-webmentions-check will update the struct with any new Webmentions made.

So armed, we can add a method to our site’s Lisp, meant to be invoked periodically, for checking Webmentions made to our site:

(defun iosh/check-webmentions ()
  "Check for new webmentions; update publication state."
  (indie-org-enable)
  (let* ((env :prod) ;; no staging support ATM
         (publication-state
          (if (file-exists-p publication-state-file)
              (indie-org-state-read publication-state-file)))
         (state-for-env
          (or (plist-get publication-state env) (indie-org-state-make)))
         (webmentions-received
          (or (indie-org-state-v2-webmentions-received state-for-env)
              (indie-org-webmentions-make-received)))
         (token
          (indie-org-read-token
           webmentions-io-token-file)))
    (indie-org-webmentions-check "indie-org.sh" token webmentions-received)
    (setf (indie-org-state-v2-webmentions-received state-for-env) webmentions-received)
    (plist-put publication-state env state-for-env)
    (indie-org-state-write publication-state publication-state-file)))

The site owner will need to arrange for this method to be invoked on a regular basis, perhaps as a cron job. If the owner wishes to display mentions as part of their page template(s), they will need to re-publish on receipt of new Webmentions.


4.5 Supporting POSSE

Adding See POSSE to one’s site is a bit like See Sending Webmentions; simple in principle, but once one begins digging into the details it can quickly become daunting. Also like sending Webmentions, there’s a service that takes this off our hands: brid.gy.

brid.gy supports publishing:

  1. Twitter tweets & Mastodon toots
  2. photos to Flickr
  3. all sorts of things to Github
  4. Reddit

as well as a growing list of sites.


4.5.1 Setting-up POSSE to Mastodon

This subsection will describe the process of adding POSSE to an Org Export-generated site, using indie-org.sh as an example. It will focus on Mastodon.

The first step is to sign-up at brid.gy. There are extensive docs at https://brid.gy/about. When you land on the home page, you’ll be asked to click on a tile representing the account to which you want to enable POSSE. On clicking “Mastodon”, you’ll be asked whether you want to post to a Mastodon account or connect to the fediverse. Select the former. You’ll next be asked for the address of your Mastodon instance, where you’ll have to sign-in. Once you’ve authenticated there, you’ll be asked for the address of your site– that’s it.

As an aside, brid.gy works via See Webmentions, both for accepting POSSE requests and for backfeeding responses to your POSSE’d content, so you’ll need Webmention support, first.

indie-org integrates POSSE support into Org Mode, first by defining a new keyword, #+POSSE. When authoring one’s document, one expresses a desire for POSSE by setting it like so:

#+TITLE: My Post
#+AUTHOR: me
#+POSSE: mastodon flickr
...

During the See Publication Process, indie-org will note the POSSE requests & record them in the publication state for future reference (Cf. indie-org-posse-requests).

indie-org also handles sending the requests and recording the responses in indie-org-send-posse-request. The response can be recorded with the publication state in indie-org-record-sent-posse. Given an indie-org-posse-requests instance and and instance of indie-org-posse-responses, the outstanding requests (i.e. the POSSE requests that still need to be sent) may be found with indie-org-posse-required.

So equipped, we can give ourselves another function:

(defun iosh/send-posse-requests (prod)
  "Send POSSE requests post-publication.
PROD shall be t to select production & nil to select staging."
  (message "Sending POSSE requests (if any)...")
  (indie-org-enable)
  (let* ((env (if prod :prod :local))
         (all-pub-states
          (if (file-exists-p publication-state-file)
              (indie-org-state-read publication-state-file)))
         (publication-state
          (or (plist-get all-pub-states env)
              (indie-org-state-v2-make-publication-state)))
         (requests
          (or (indie-org-state-v2-posse-requests publication-state)
              (indie-org-posse-make-requests)))
         (responses
          (or (indie-org-state-v2-posse-responses publication-state)
              (indie-org-posse-make-responses)))
         (to-send
          (indie-org-posse-required requests responses)))
    ;; `to-send' will be a list of cons cells, each of whose car is a
    ;; page key & whose cdr is a list of POSSE symbols.
    (while to-send
      (let* ((page-key (caar to-send))
             (source
              (concat
               "https://indie-org.sh/"
               page-key))
             (symbols (cdar to-send)))
        (while symbols
          (if prod
              (let ((rsp (indie-org-send-posse-request source (car symbols))))
                (indie-org-record-sent-posse page-key rsp responses))
            (message "Will send POSSE request: %s :=> %s" source (car symbols)))
          (setq symbols (cdr symbols))))
      (setq to-send (cdr to-send)))
    (setf (indie-org-state-v2-posse-responses publication-state) responses)
    (plist-put all-pub-states env publication-state)
    (indie-org-state-write all-pub-states publication-state-file))
    (message "Sending POSSE requests (if any)...done."))

4.5.2 Handling Backfeed

brid.gy automatically polls whatever silo you’ve connected to your site, looking for likes, responses, reposts, and so forth. It will then relay that information back to your site through... See Webmentions.

As an example, the author went to the Mastodon post POSSE’d from What Is indie-org.sh? and favorited it. After the next poll, a Webmention arrived representing that like and is now displayed here.


4.5.3 Controlling Content

When POSSE’ing to space-limited fora such as Twitter & Mastodon, it can be inconvenient to have a lengthy post truncated by the target platform. brid.gy offers the ability to specify platform-specific text to be sent to the target silo.

We can take advantage of the Org Export back-end feature :filters-alist to make this more convenient. :filters-alist is a property list defining various filters to be applied to the Org document during publication. Salient to our purposes is :filter-parse-tree: this filter is applied to the Org document’s abstract syntax tree after the initial parse.

indie-org.sh sets this filter to the function iosh/post/filter-parse-tree; it will search for a top-level Org heading tagged with toot-text. If found, that heading will be removed from the parse tree, but the contents will be used when POSSE’ing to Twitter & Mastodon.

For instance:

#+TITLE: My Post
#+DESCRIPTION: I made a thing
...

* Introduction
This is a very long post....

* The First Thing
...

blah. Blahblahblah.

* Mastodon          :toot-text:
I wrote a very long post!

When this document is POSSE’d to Mastodon, only the text “I wrote a very long post!” will appear, along with a link to the original document.


5 Future & Roadmap

Both the indie-org package and the demonstration site https://indie-org.sh are early code, and the author has tried to indicate that with the 0.x version number for the package.

In order to use indie-org successfully, the site author is going to need to be familiar with the Org Export facility (see Org Export in Org Export) and comfortable with See Emacs Lisp in Emacs Lisp. That said, if one is hosting an Org Mode staticly-generated site, those requirements are likely already met.

All that said, the recent meltdown at Twitter, shutting-down API access to numerous applications including brid.gy, only reinforces the imperitive to take control of our own identity, data & platform.

Furthermore, indie-org is not just an experiment: I’m using it today with my personal site.

The author gave a talk on one small part of this at EmacsConf. The documentation is hosted on-line here.

Bug reports & feature requests are welcome at .,


Index


Function Index


Type Index


Variable Index