ActivityPub / Fediverse bridge

[shawnb61]

Sure seems feasible.  Maybe as a mod.

Any Mastodon users out there?

Some background:
https://en.m.wikipedia.org/wiki/ActivityPub
https://www.w3.org/TR/activitypub/

A first pass poc needn't be too complicated.  As a simple example, I believe you can share your music with the fediverse via a channel on Funkwhale.  Folks can subscribe to that channel, e.g., on Mastodon.  And my understanding is that Funkwhale just publishes that via a simple RSS feed.
https://www.funkwhale.audio/

Just some food for thought...

[Sesquipedalian]

More information is needed about the scope and goals of this feature request. What would it be intended to do? How should it work?

[Arantor]

There are definite holes in the ActivityPub spec that would need to be solved; there's no super solid way of migrating accounts between services in the purest sense, meaning that migration is a build-in that has to happen per instance, which is annoying.

Representing a post as an article with its replies being comments is otherwise not particularly tragic to do, and even pushing notifications out and back isn't the worst.

Treating topics in a board as if it's a federated instance that can be viewed from Mastodon as a 'server' to push/pull from is probably the least massive change you'd need to do to keep it cohesive.

The real question is whether you truly gain anything from it; Mastodon in particular operates off a Twitter like model (user posts + comments, rather than topics in the traditional sense) and, really, you lose the categorisation angle. And one of the big downsides to it is the even-more-inconsistent moderation of peer servers, it's even more scattershot than forum to forum.

[shawnb61]

At the moment, I think this is directional &  aspirational - planting a seed of thought.  Further brainstorming is needed.  In the long run, I think SMF would benefit from joining in the fediverse.  So SMF forums aren't isolated little islands - they can opt to be federated & with whom they wish to be federated. 

The good news is that ActivityPub has a finite, clearly defined approach, scope & spec.  (https://www.w3.org/TR/activitypub/)

In the short term, a baby step may be as simple as one-way subscriptions (e.g., to be able to display recent music releases from FunkWhale).  I would suggest a new class of topics/boards that participate. 

In the long term topics should be published via ActivityPub.  Comments from non-local members, if allowed, can be represented as guest posts.  Would need to be able to whitelist/blacklist servers &/or individuals.

ActivityPub looks like it has a hard OAuth requirement.  Maybe step 1 towards federation is to implement OAuth. 

Mastodon looks tempting because it is so widely adopted, and a full implementation already exists.  It looks easy to setup a Mastodon server, but it has IMO odd server requirements, e.g., postgresql, ruby.  Maybe some potential Frankenstein action there...

Ownership of sites like reddit/fb/bandcamp/etc is a problem, it's like they all grow & turn evil or something...  I think SMF sites are pretty much immune from that rot.  I think federation would bring almost immediate relevance & interest, and lure folks back from those massive, manipulative platforms.

[Sesquipedalian]

Okay, so what I understand is that this is a request:

• To implement the ability to import posts and comments via the ActivityPub API as new forum topics in a specified board. This in turn would require, for SMF's needs:
  
  • The ability to associate imported ActivityPub posts created by the forum's members with the their forum accounts.
  • The ability to create guest posts for imported ActivityPub posts created by people who are not members of the forum.
  • The ability to automatically moderate imported content based on ActivityPub user.
  • The ability to automatically moderate imported content based on source ActivityPub server.
• To implement the ability to publish topics via the ActivityPub API, with the topic's original post as a new ActivityPub post and the reply posts as ActivityPub comments.
• To implement OAuth.

Arantor makes a very good point about categorization of incoming content—namely, that dumping everything imported from ActivityPub into a board will create an undifferentiated mess of content. That is fundamentally a content management issue rather than a show-stopping technical issue, but it would nevertheless require careful consideration in order to avoid producing a bad user experience in the end.

A more pressing technical issue is the fact that SMF uses a flat topic structure rather than a branching one. Microblogging-style social media uses branched structures. Squashing a branched structure down into a flat structure is difficult to do, and nearly impossible to do well.

If I have misunderstood something, please explain further.

[Arantor]

IF the goal is about de-islanding, I think you'd immediately benefit from adding in OAuth2 support both for account creation and account authentication (i.e. to be both an service provider and an identity provider), ahead of implementing APub.

The thing about federated moderation is that, honestly, as someone who does spend time in Mastodon... it's actually a different kind of disaster where you don't just have individuals having trouble with other individuals but server to server beef too, to the point that servers can cut off their inhabitants from other servers. That and the fact that identity is tied to home instance which means if the home instance collapses, goodbye your entire existence, not just an account on an island.


The structure needn't be a problem outbound; you treat it as you treat any other single strand: it's a post with a single linear thread of replies to each other because you don't have any metadata to infer otherwise. And realistically even if you did try to capture it, that's still all you're going to end up with because users don't tend to actually use that style of branching conversation unless it's made absolutely front and centre a la Reddit, or it's made the *only* style of conversation a la Twitter/Mastodon.

Inbound is a different proposition. Realistically what you could try to do is ingest all of them (on first import) and internally store the relationships from post to post as a directed acyclic graph. At that point you can walk the tree until you read the tip of the tree, and there's your opening topic, with all replies to it implicitly in chronological order. For bonus points you must know the reply context from that, and you could display that above the post with a link back to it if you have it. (Structurally this is no more than adding a parent ID to each message).

Though this raises the bigger question: do you want to consider properly supporting the threaded conversation model? It's certainly been done historically to support it as a true hybrid, where each existing thread is really one long linear chain of replies back to the top but future topics didn't have to be.

https://xenforo.com/community/threads/will-xenforo-ever-adopt-threaded-comment-structures-like-reddit.205394/ is an interesting time capsule of thoughts on the subject. Disclaimer: I am a participant in this conversation and I am not a clear proponent of it, however there is merit to having the same discussion with different points of view joining it.

[shawnb61]

Yep.  Perfect for starters!

I'm sure we'll learn quite a bit as we go...

SMF might even adopt a bit of a role as a fediverse aggregator.  I could easily see setting up a board that shows activity from 3 other actors on 3 completely different servers.  The board follows 3 remote users.

The part that will be fun to learn is how deep & wide the "following" goes, user/board/topic, in the long term...  As an example, User A follows remote users b, c, d; user E follows remote f, g, h.  What should be visible to user A is local smf content as appropriate, plus remote b, c, d.  User E should see local smf content, plus f, g, h.  E doesn't see b, c, d, as it's not followed.  Like a subscription model almost.  Topics may need a finer permission set than "in board x".

But...  One step at a time.  Baby steps.

[Sesquipedalian]

Though this raises the bigger question: do you want to consider properly supporting the threaded conversation model?

I have no such plans, no.

I do like the suggestion you gave in another discussion a while ago to allow a post to be marked as a reply to another post, i.e. with a small "Replying to #17" metadata line in the header or something along those lines. But that would be a tweak to the existing flat structure, not a change from flat to threaded.

[Sesquipedalian]

SMF might even adopt a bit of a role as a fediverse aggregator.

Probably not.

I could easily see setting up a board that shows activity from 3 other actors on 3 completely different servers.  The board follows 3 remote users.

But what happens when those actors post content that doesn't belong with the expected content for the board?

But questions about content curation aside, I think you could accomplish this easily already using an RSS import mod, since Every Mastodon User Has an RSS Feed.

[shawnb61]

The aim here was to have SMF join the fediverse.  Just viewing simple feeds was a suggestion as a baby step down that path.  (Using RSS was even suggested as a temp hack in the first post.)

I am seeing more fediverse usage over time, as folks finally get sick of reddit, fb, etc.  Slow, but steady.  SMF was here before them, & I have a fairly strong suspicion a move in that direction will help us outlast 'em. 

As I said, food for thought.

[Sesquipedalian]

I see the attraction, sure. But before it could be done, a lot more thought would need to go into figuring out how to map concepts and structures from one to the other. 

From what I can tell, there are several vital concepts in a forum that have no obvious equivalent in ActivityPub, and vice versa. Figuring out how to translate from one to the other would therefore be challenging.

[shawnb61]

Yep. Understood. 

Just planting that seed of thought.

[Sesquipedalian]

Fair enough. But if you want that seed to grow, you can water it by providing a detailed explanation of how to map the concepts and structures from one to the other and back.

[shawnb61]

The requirements, I think, are pretty well spelled out above.  Thankfully the ActivityPub spec is mercifully short & clear.

I think there are only two gotchas on the design, which definitely need more work.  The main one being introduction of an access model that isn't 100% board based like it is today.  Not impossible at all, though almost certainly will require db changes to existing structures.  The flat vs threaded question, I think, has a proposal above that could work w/o db changes.  Threaded has always bugged me anyway.

I'll put some thought into a proposed ERD over the next few weeks.  (Or more...)  Maybe dummy up some test data for performance testing.

[shawnb61]

...and it's the kinda thing that needs to steep in everyone's brain for a while...

If you haven't already, just setup a Mastodon acct & use it instead of twitter or fb for a week or two.

It's got some pretty steep & interesting limitations, and those are baked into the spec...  At first glance, they are odd.  But the reality is the spec sets realistic expectations on communications across servers. 

An example: I created a list of folks to follow, who happened to be on other servers.  I pulled up the list, expecting to see a bunch of their recent posts...  But...  It was empty...  Until someone posted.  Slowly, over time, it filled up.  It's all a kinda "going forward" model.  There's no backfilling, as that is really expensive.  (If you really want to see history on another server, then go to that other server...)  It doesn't expect everything to be cloned everywhere; it's very targeted in its publish/subscribe model.

[NFG]

I came here to see if ActivityPub had been mentioned yet, as well as federation in general.  I think, as has been mentioned, this is going to be critical to the future success of all the forums and many websites all over the world, 'cause big social media is intent on tearing itself apart, and we don't all want to join tiny isolated communities all around the world anymore.

That said, my thinking for SMF's position in the federated future, is that it should at the very least make it easy to follow from other places.  Users, threads, categories, for sure, the whole forum as well, if someone wants that.  Basically, it's RSS for the new era.

Also, Bsky is huge, and growing very quickly, with a very different audience than you'll typically find on Mastodon.  So, IMO ATproto should be considered as well. 

A Wordpress plugin made it very easy for me to create a followable actor on the AP fediverse, so for sure there's going to be code there that can be used as a springboard.  ATproto, still a lot newer, has a wild ecosystem coming up around it.  So there's going to be plenty of code there too.

Making a follow-only system, at least with AP, shouldn't require too much work, but I confess I'm a PHP hack without the strongest awareness of how much I don't know (a lot).  I don't know if I have anything to offer WRT to these integrations but I'd love to help. 

If nothing else, consider this a vote for 'yes plz'.

[Sesquipedalian]

Making a follow-only system, at least with AP, shouldn't require too much work, but I confess I'm a PHP hack without the strongest awareness of how much I don't know (a lot).  I don't know if I have anything to offer WRT to these integrations but I'd love to help. 

Writing code isn't the issue. As you have rightly noted, there are APIs available and we have people who know how to use such things.

Rather, the hard part is this:

From what I can tell, there are several vital concepts in a forum that have no obvious equivalent in ActivityPub, and vice versa. Figuring out how to translate from one to the other would therefore be challenging.

That would be just as true for BlueSky's API as for ActivityPub.

Since you said you'd like to help, @NFG, the best thing you could do would be to work on this:

providing a detailed explanation of how to map the concepts and structures from one to the other and back.

That's a matter of abstract conceptualization, with no knowledge of PHP code required. Perhaps if you and @shawnb61 were to work together on thinking through that, you might be able come up with a solid proposal for this.

[shawnb61]

The more I thought about this, the more I felt things were similar to what we already do.  This led me to one possible relatively straightforward solution that leverages existing SMF access.

To be honest, I can't tell if this is insane or not...

To set the table, fediverse general behavior...
 - All posts are public.
 - The UI let's you choose between Local vs Remote vs All posts, when browsing your feed & when doing searches.
 - The "All" posts include those that have been of interest, i.e., followed by others, on your local server, even when not explicitly followed by you.
 - When a local user follows a remote user, both servers keep track of the follow & agree to communicate.  E.g., if (local) Susan follows (remote) George, then (remote) George's server must be told it needs to publish/share George's posts to Susan's server.  Of course, if (remote) George follows (local) Susan, then the local server must keep track of George's follow, and publish Susan's posts to George's server.
 - If those users un-follow each other, and there are no more follows between those two servers, then they no longer need to talk to each other.
 - The bottom line is that these are not live pulls over the internet...  It's just another set of posts, with a remote source...
 - It's like users opting-in to RSS feeds.

A possible SMF approach: "A board is a follow"...
 - "Recent Posts" is promoted to the main menu, as the "Feed".
 - Each local follow of a remote resource results in a new SMF board, with a new, unique, corresponding membergroup tracking access to that board.  If multiple local users follow the same remote resource, they share that same membergroup.
 - A "fedi" indicator is needed on the board to indicate whether the board/group is a "fedi" feed or normal SMF board, to easily distinguish between the two.  (Local vs Remote vs All)
 - The "Feed" needs an option to include/exclude other fediverse boards.  (Local vs Remote vs All)
 - Search needs an option to include/exclude other fediverse boards.  (Local vs Remote vs All)
 - Local SMF users choose which posts get published to the fediverse by posting on the "Feed".  These posts are stored in a special "fedi" board for local users. 
 - "Fedi" boards are hidden.  They are navigated via the feed.

The above leverages existing SMF board access.  The only tweak to the board table is the "fedi" indicator, to enable the Local|Remote|All selection in the Feed & in Search.

Currently, SMF has a limit on membergroups of ~80 groups, due to the size of the groups field on the members record.  But the number of follows can easily approach hundreds/thousands in the fediverse.  This would need to be addressed.  It may be as simple as expanding the size of the groups field.

SMF server & follow maintenance...
 - Each SMF server needs to track the other fedi servers it communicates with, which OAuth server was used, etc. 
 - Each SMF server needs to track local follows of remote users.  If unfollowed, that needs to be communicated to the remote server, so it knows to stop publishing to the local.  This associates the local fedi boards to the corresponding remote user.
 - Each SMF server needs to track remote follows of local users.  The local server must publish posts by followed users to the remote servers.
 - Remote server admin needs to handle blacklisting, e.g., of spammers or questionable remote servers.
 - OAuth.

This feels doable.  At the very least, it may help folks start to think about how to marry the two sets of access rules. 

[NFG]

Thanks, @shawnb61, for reminding me I wrote this and forgot about it.

I refined it a bunch of times as I was researching it, trying to map what I knew about AP and SMF into something that made sense, but it's basically a stream of consciousness thing, for which I apologize.  I dunno how much of it makes sense or is useful, but since I want this integration, I figured I should put in some effort.

Anyway.  Here's what I had before I mis-filed it in my own mind:

Digging into AP, with implementations other people have done and documented as a starting point.  Right off the bat it's clear that, while a lot of things don't map directly to the sorts of activities, users or data we use with SMF, a lot of it can just be skipped, or set up with default values that will satisfy external apps.

Needed things:

For a number of reasons, but also to conform to expected norms within the communities we connect to, all of this should require HTTPS.

Webfinger - used to resolve username@domain < - > specific actor URLs
It basically translates back and forth between user-friendly and absolute addresses:
[email protected] to https://example.org/forum/index.php?action=profile;u=1523

nodeinfo - gives a short list of server stats, like user/active user counts, API versions available, etc.  I'm assuming that sort of detail is immediately available based on the code itself, or a quick check of the number of records in the user table.  I don't know if '30 day active' user counts would be a computationally large hit on the fly, for large sites, or if this would be better calculated on the reg during other processes and just readily available.

Webfinger:

Webfinger defines how SMF users can be @mentioned on other servers.

Code Select Expand

    "subject": "acct:[email protected]",
    "links": [
        {
            "href": "https://my-example.com/actor"
        }
    ]

So webfinger is easy, it defines user@domain, and includes a link to the profile page.  There are other things that can be added, inbox and outbox URLs, etc.

Events:

Code Select Expand

    "object": {
        "id": "https://my-example.com/hello-world",
        "type": "Note",
        "published": "2018-06-23T17:17:11Z",
        "attributedTo": "https://my-example.com/actor",
        "inReplyTo": "https://mastodon.social/@Gargron/100254678717223630",
        "content": "<p>Hello world</p>",
        "to": "https://www.w3.org/ns/activitystreams#Public"
    }

id = the link to the post (https://example.org/forum/index.php?topic=1234.msg50534#msg50534)
inReplyTo - Does SMF even track this, or is it just Topics sorted by date?  May have to use the existing post URLs as reply-to references.  I envision a system where the forum's threads are shown as they are now with an expandable/expanded link under each post that shows external replies to it.
We'd have to convert BBcode in content (SMF 'body') to HTML equivalents. 

I really don't know how much detail I need to include here.  It's obvious that 'attributedTo' is SMF 'member', is it obvious that this field's value in AP is the link to the user profile?


The official Mastodon dev docs are pretty handy.

Actors:

Actors are things that create events, usually users but there may be bots or other sources that create events, so 'actors' is the catch-all.  So for SMF purposes, we might consider users, threads, and forums to be actors. 

Here's the relevant actor details in the 'actor' JSON-LD doc:

```
    "id": "https://my-example.com/actor",
    "type": "Person",
    "preferredUsername": "alice",
    "inbox": "https://my-example.com/inbox",
  ```

  So this is pretty simple. 
  id = the SMF user profile URL.  It's self referential, so as I understand it, doing an HTTP request for the user's profile will return the usual page, unless the header includes:

```
Content-Type of application/ld+json; profile="https://www.w3.org/ns/activitystreams".
```
  The rest of that profile is self explanatory.

Each actor has an outbox, which is where followers will pick up the events they are following.  I don't think this needs to be a tangible file, an outbox could be whipped up on demand, and include (for example) only the last 10 posts.

For SMF, I think some actors will need inboxes to receive external replies.  It's possible to just refuse replies, but receiving and displaying them would be quite a useful feature.  They could be shown differently, collapsed, different colours, different pages, etc.  I know it's pretty cool to get replies to a post on my Wordpress site from people around the world.  SMF absolutely can refuse all replies, and for boards this is probably a good idea, but an inbox must still be specified to work with Mastodon.

Posts:

Here's the data you might find in a post in an actor's outbox:

```
{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Create",
  "id": "https://example.com/alice/post/123",
  "actor": "https://example.com/alice",
  "published": "2022-12-34T12:34:56Z",
  "to": ["https://www.w3.org/ns/activitystreams#Public"],
  "cc": ["https://example.com/alice/followers"],
  "object": {
    "id": "https://example.com/alice/post/456",
    "type": "Note",
    "attributedTo": "https://example.com/alice",
    "content": "Lorem ipsum dolor sit amet.",
    "published": "2022-12-34T12:34:56Z",
    "to": ["https://www.w3.org/ns/activitystreams#Public"],
    "cc": ["https://example.com/alice/followers"]
  }
}
```

Admin Options:

Admins will need a dashboard where servers, users and other things (like tags, keywords, replies to specific threads or forums) can be managed.  A simple list would work as a starting point:  "Enter a list of domains to block, one per line" sort of thing.  This would probably evolve as admins and users encounter the real world and decide what they need.  I could, for example, see a need for refusing all external replies to particular boards or threads, and maybe some users might opt out of external replies.

Other things that might be helpful:

inbox and outbox for each user doesn't necessarily need to be functional - especially for a forum that can be followed but not follow someone else, we only need outboxes, but the inbox may still need to be specified, Mastodon requires it even when it's not used.

[NFG]

Each local follow of a remote resource results in a new SMF board

I don't see the immediate value of making every SMF site a fully fledged fedi instance.  I think it's enough to make a forum followable and replyable, without having people on that forum following other sites too.  Because that seems like a massive amount of extra work (isn't it?).

But, that's me thinking about scope and the effort, not the functionality.  I dunno, maybe it would be useful for some reason, to some people.  ¯\_(ツ)_/¯