I do note that you argue for backwards "Fediverse-compatibility" here, and I wonder how that equates with backwards #ActivityPub compatibiity.
Discussion
Loading...
Post
@hongminhee what i have found necessary (sadly) is to sometimes ignore what @\context a software produces and simply inject a corrected @\context describing what they *actually* meant instead of what they said they meant. x_x
https://gist.github.com/trwnh/d2f6fb1c87465baab8231427013a36a8
it would be an Exercise to sit down and map out the actual contexts of softwares like mastodon 4.5, mastodon 4.4, misskey 2025.12, akkoma 3.10.2, and so on...
for all else, there's shacl i guess, if you want to beat things into the correct shapes.
Re: I have deeply mixed feelings about #ActivityPub's adoption of JSON-LD, as someone who's spent way too long dealing with it while building #Fedify.
@trwnh@mastodon.social it's not an exercise, not anymore, with the Fediverse Observatory!
@hongminhee I had a similar realization early on when implementing Pinka. I almost went full JSON-LD but found that to properly expand the document I might need to make network calls. I stopped worrying about unknown terms and just hard coded a list of well-known AS and APub terms for interoperability.
@hongminhee old fart over here mumbling something about HTML, XHTML, tag soup, internet explorer, Netscape, CSS, JavaScript, and 30 years of brain damage
@hongminhee This is a conversation I've been wanting to see for a long time. As things stand you have to develop ActivityPub specially for every service you want to talk to, because of all the implementation inconsistency. I hope we can converge on a simple solution soon.
@hongminhee How hard would it be for a future version of ActivityPub to simply back out JSON-LD support? Would there be a downside to this?
@mcc I'm not sure, but that would break some ActivityPub implementations relying on JSON-LD processors. 🤔
Re: I have deeply mixed feelings about #ActivityPub's adoption of JSON-LD, as someone who's spent way too long dealing with it while building #Fedify.
@mcc@mastodon.social asking the important questions 🤣
@hongminhee You have my deepest sympathies. I've been using Linked Data (correctly, for the most part) since 2002, and I feel your pain.
@hongminhee i struggling badly with rust cuz it's rust being rust... i can imagine a duck typing lang might have easier time
@hongminhee@hollo.social from the point of view of someone who is "maintaining" a JSON-LD processing fedi software and has implemented their own JSON-LD processing library (which is, to my knowledge, the fastest in it's programming language), JSON-LD is pure overhead. there is nothing it allows for that can't be done with
- making fields which take multiple values explicit
- always using namespaces and letting HTTP compression take care of minimizing the transfer
without JSON-LD, fedi software could use zero-ish-copy deserialization for a majority of their objects (when strings aren't escaped) through tools like serde_json and Cow, or System.Text.Json.JsonDocument. JSON-LD processing effectively mandates a JSON node DOM (in the algorithms standardized, you may be able to get rid of it with Clever Programming)
additionally, due to JSON-LD 1.1 features like @type:@json, you can not even fetch contexts ahead of time of running JSON DOM transformations, meaning all JSON-LD code has to be async (in the languages which has the concept), potentially losing out on significant optimizations that can't be done in coroutines due to various reasons (e.g. C# async methods can't have ref structs, Rust async functions usually require thread safety due to tokio's prevalence, even if they're ran in a single-threaded runtime)
this is after context processing introducing network dependency to the deserialization of data, wasting time and data on non-server cases (e.g. activitypub C2S). sure you can cache individual contexts, but then the context can change underneath you, desynchronizing your cached context and, in the worst case, opening you up to security vulnerabilities
json-ld is not my favorite part of this protocol
@kopper @hongminhee As the person probably most responsible for making sure json-ld stayed in the spec (two reasons: because it was the only extensibility answer we had, and because we were trying hard to retain interoperability with the linked data people, which ultimately did not matter), I agree with you. I do ultimately regret not having a simpler solution than json-ld, especially because it greatly hurt our ability to sign messages, which has considerable effect on the ecosystem.
Mea culpa :\
I do think it's fixable. I'd be interested in joining a conversation about how to fix it.
@cwebber @kopper @hongminhee If we consider data exchange just in one application context (mastodon), then JSON-LD is overhead because data structures are fixed.
But as soon as we go out of that application context, JSON-LD will make a lot of sense. IMHO, the use will grow over time as this allows to add permissions to post on the data level using ODRL e.g. Or to have privacy considerations.
On a very short term, JSON-LD may be an overhead. But it is an investment into the future.
I don't remember it that way.
We started the WG off with AS2 being based on JSON-LD, and I don't think we ever considered removing it.
I don't think it was a decision you made on your own. I'm not sure how you would, since you edited AP and not AS2 Core or Vocabulary.
I would be strongly opposed to any effort to remove JSON-LD from AS2. We use it for a lot of extensions. Every AP server uses the Security vocabulary for public keys.
@evan @kopper @hongminhee The problem is that signing json-ld is extremely hard, because effectively you have to turn to the RDF graph normalization algorithm, which has extremely expensive compute times. The lack of signatures means that when I boost peoples' posts, it takes down their instance, since effectively *every* distributed post on the network doesn't actually get accepted as-is, users dial-back to check its contents.
Which, at that point, we might as well not distribute the contents at all when we post to inboxes! We could just publish with the object of the activity being the object's id uri
@cwebber @evan @kopper @hongminhee For the purpose of "message" wouldn't it then be easier to just sign an entire file that is transmitted? Or is this stream processing.
@cwebber @evan @kopper @hongminhee
I may be naive and am not an expert here, but in my musings on a protosocial AP extension I imagined a clean separation of "message bus" where you'd want closed-world predictable msg formats defined by some schema (perhaps JSON Schema or LinkML). These msgs would JSON-LD formatted but validated as plain JSON.
And then there would be the linked data side of the equation, where a semantic web is shaping up that is parsed with the whole set of open standards that exists here, but separate of the message bus. This is then a hypermedia, HTTP web-as-intended side. Open world and follow your nose, for those who want that, or minimum profile for the JSON-only folks.
It occurs to me these require separate/different extension mechanisms, guidelines and best-practices. The linked data part lends itself well for content and knowledge presentation, media publishing. While the msg bus gives me event driven architecture and modeling business logic / msg exchange.
@cwebber @kopper @hongminhee I talk about this in my book. Unless the receiving user is online at the time the server receives the Announce, it's ridiculous to fetch the content immediately. Receiving servers should pause a random number of minutes and then fetch the content. It avoids the thundering herd problem.
3+ more replies (not shown)
@cwebber@social.coop @hongminhee@hollo.social @evan@cosocial.ca admittedly, https://codeberg.org/fediverse/fep/src/branch/main/fep/8b32/fep-8b32.md does kind of solve this specific problem. the json canonicalization used there is much lighter than rdf canonicalization (which iceshrimp had to implement in dotNetRdf specifically for its ld signature support, so tooling availability is not really an excuse in favor of json-ld either!)
1 more replies (not shown)
@cwebber @kopper @hongminhee It would be a huge backwards-incompatible change for almost zero benefit. People would still make mistakes in their ActivityPub implementations (sorry, Minhee, but that's life on an open network). We'd need to adopt another mechanism for defining extensions, and guess what? People are going to make mistakes with that, too.
@evan @cwebber @kopper @hongminhee maybe a compromise approach could be to specify a simpler “json-ld as it is used in practice”, similar to what HTML5 was, that remains backward compatible while simplifying the spec to the point that it is actually feasible to implement
@evan@cosocial.ca @hongminhee@hollo.social @cwebber@social.coop my argument is that json-ld is way more prone to mistakes. in iceshrimp.net, for example, we ship and preload several modified contexts in order to correct some mistakes on our end, and even then we encounter a lot of software that do not, for example, include the security context in their actors
if, as per my suggestion, property names were always written in expanded form, the only mistakes you could really do would be typos, and that would fail pretty loudly compared to the current status quo where most software accept it and some software silently fail. how are those developers meant to even be aware that this is a problem?
1 more replies (not shown)
@cwebber @kopper @hongminhee The biggest downside to JSON-LD, it seems, is that it lets most developers treat AS2 as if it's plain old JSON. That was by design. People sometimes mess it up, but most JSON-LD parsers are pretty tolerant.
1+ more replies (not shown)
hm, we really need to differentiate between users responsibility and dev responsibility.
Not sure if Hong saw the draft about the AP kv thing, it supports either JSON-LD fields _or_ as:attachment / as:context …
wtf do I want to say.
user story:
We are working on 2 major and 3 projects fulltime which is
- federation of wikibase / wikidata
- federation of Public Broadcasters https://www.publicmediaalliance.org/public-broadcasters-create-public-spaces-incubator/
and these https://codeberg.org/Menschys/fedi-codebase
Let's say we want to federate a Country, then all the knowledge is sent in `attachment` with the fully qualified qikidata url in `context` [as:context - not @context ! - this is so confusing :)]
For example the according entries from the PressFreedomIndex `collection` (co-founder of freelens here :)
But anyway, the idea about having
"wd": "https://www.wikidata.org/wiki/Special:EntityData/",
"wdt": "https://www.wikidata.org/prop/direct/" in the `@context` was that any user can consume and federate wikibase
incl.
🧵 1/2
incl.
- the properties in all the languages of the world
- the knowledge of the world in all the languages
- the wikidata relations and qualified statements including the nameMap etc. and all the urls to all wikiprojects incl. their languages and knowledge
How else could I say to other softwares if they want all users qualified data, use wikidata vocabulary?
wikipedia, wikidata, EBU, Public Broadcasters, taxi data is _all_ JSON-LD …
@sl007@digitalcourage.social @hongminhee@hollo.social @julian@activitypub.space i feel like you're falling into a trap i've seen a lot around AP spaces: just because the data can be contorted to represent something does not mean software will interpret it as such.
any software who wants to support wikidata statements and relations will have to go out of their way to implement that manually with or without json-ld in the mix, and interoperability between those software will have to specify how that works. and in your specification you can indeed make it so Simply Linking to the wikidata json-ld (which i don't believe it provides out of the box, it does for xml, turtle, and n-triples, if we're talking about rdf. if not, their bespoke json format is just as authoritative) can work (but i'd say using the Qxxx and Pxx IDs and letting the software figure out how to access it would be better!)
if you have the dream of making an as:Note and having it's as:attributedTo be the wikidata entity for alan turing... sorry, nobody other than maybe your own software will even attempt interpreting that
@hongminhee@hollo.social @sl007@digitalcourage.social @julian@activitypub.space attempting to support this kind of "data contortion" (i made this up and prolly isnt the right way to describe this) would rapidly balloon the scope of every fedi software ever. i don't believe anyone would want to develop for such ecosystem
a similar example i saw was someone attempting to explain how you can partially inline an as:object you as:Like'd in order to specify you only liked that past version of it and if it changed your like shouldn't count. without describing this exact scenario i don't believe any software, json-ld capable or not, would interpret that Like as such. same thing with the long-form text FEP which attempts to support non-activitypub authors
just btw, we had many W3C Social CG meetings about the importance and how to use the as:context property - not the JSON-LD @context and we all agreed.
About 30-40 devs attended.
Between 2016 and 2024 I attended basically any meeting. I felt that using wikidata urls in as:context was nice for anyone.
1+ more replies (not shown)
@hongminhee@hollo.social take this part with a grain of salt because my benchmarks for it are with dotNetRdf which is the slowest C# implementation i know of (hence my replacement library), but JSON-LD is slower than RSA validation, which is one of the pain points around authorized fetch scalability
https://wetdry.world/@kopper/114678924693500011
@hongminhee@hollo.social if i can give one piece of advice to devs who want to process JSON-LD: dont bother compacting. you already know the schema you output (or you're just passing through what the user gives and it doesn't matter to you), serialize directly to the compacted representation, and only run expansion on incoming data
expansion is the cheapest JSON-LD operation (since all other operations depend on it and run it internally anyhow), and this will get you all the compatibility benefits of JSON-LD with little downsides (beyond more annoying deserialization code, as you have to map the expanded representation to your internal structure which will likely be modeled after the compacted one)
@hongminhee I'm reading this thread as a relative noob, but what I see again and again: almost no one "properly" implents #ActivityPub largely because #JSONLD is hard but also because the spec itself is unclear. Most people who get stuff done have to go off-spec to actually ship.
This seems a fundamental weakness of the #fediverse - and that disregarding the limitations coming from base architecture. Seems to pose a mid/long-term existential threat.
What can we do to help improve things?
@douginamug @hongminhee Hi Doug, I do have ideas to share about how to make schema on ActivtyPub more flexible and accessible to all social app makers.
@nextgraph very cool! Will have to look into your project more, I think I saw you talking as FOSDEM @hongminhee
@hongminhee@hollo.social I have the same feeling. The idea behind JSON-LD is nice, but it isn't widely available, so developing with it becomes a headache: do I want to create a JSON-LD processor, spending twice the time I wanted to, or do I just consider it as JSON for now and hope someone will make a JSON-LD processor soon? Often, the answer is the latter, because it's a big task that we're not looking for when creating fedi software.
@Varpie out of interest, did you look at Minhee's Fedify? It may be the JSON-LD processor you/people are looking for.
@hongminhee I attempted to deal with parsing json-ld and I really wish they had just gone with XML. XML is also complicated, but it has mature implementations, while JSON-LD is a new protocol that never felt to me to have compelling advantages except matching data models people were used to using.
@hongminhee @jalefkowit JSON-LD is the sole reason I gave up on building anything for the fediverse. It is ludicrously over-engineered for the purpose, all the worst parts of XML crammed into JSON.
Want to add a field to any of the objects? Better be prepared to host a schema definition from now until the end of time. Good luck figuring out how to write that definition though.
@hongminhee I agree that new developers should use a JSON-LD processor. It saves a lot of heartache.
@hongminhee do you use the activitystrea.ms module from npm? It takes a lot of the pain out.
@evan I don't remember exactly, but I think I came across it while doing research before developing Fedify. I probably didn't use it because the TypeScript type definitions were missing. In the end, I ended up making something similar in Fedify anyway.
@hongminhee that's great!
@hongminhee XML was always the better option tbh.
@hongminhee @jalefkowit huh. I’ve been pondering using it for some projects of mine, so this is good to know.
Is it a fundamental problem with JSON-LD, such that it should just be avoided, or a problem with how ActivityPub uses it?
And is there something else you’d recommend that fulfills the same goals?
@lkanies @jalefkowit To be honest, I'm not too sure myself. I just know that JSON-LD was originally planned as a foundation for the Semantic Web. I can only guess that if ontology is useful in a certain area, then JSON-LD would probably be useful there too.
Great piece of writing, and I agree with lots of what you say here. Building this stuff is super hard, so good for you doing it “the right way.”
I hate not being able to trust the data I receive.
I’m one of those who’s taking the shortcuts, which has plenty of drawbacks, so I’m glad you’re in here fighting the good fight.
@benpate reminder that you can have a trusty library that fulfills more or less the same functionality of fedify any time you want to switch. ;;)
⁂ Article
Re: I have deeply mixed feelings about #ActivityPub's adoption of JSON-LD, as someone who's spent way too long dealing with it while building #Fedify.
@hongminhee@hollo.social I'll give you my take on this... which is that my understanding of JSON-LD is that with JSON-LD you can have two disparate apps using the same property, like thread, and avoid namespace collision because one is actually https://example.org/ns/thread and the other's really https://foobar.com/ns/thread.
Great.
I posit that this is a premature optimization, and one that fails because of inadequate adoption. There are likely documented cases of implementations using the same property, and those concern the actual ActivityStreams vocabulary, and the solution to that is to communicate and work together so that you don't step on each others' toes.
I personally feel that it is a technical solution to a problem that can be completely handled by simply talking to one another... but we're coders, we're famously anti-social yes? mmmmm...
@pintoch read this thread?
@douginamug I saw it pass by, but I'm lucky enough not to have had to deal with JSON-LD too much so I don't have informed opinions to share 😊 In the group steering the reconciliation API, people have been discussing whether that API should use JSON-LD. I still don't understand what sort of interoperability would be expected from it. https://github.com/reconciliation-api/specs/issues/183
@hongminhee @julian I'm a true believer in RDF from back in the day, so I'm hardly neutral. But...
There are essentially no interesting ActivityPub extensions right now. Even Evan's chess example, no-one's actually using AP to play chess. It's just ActivityStreams + a few cute tricks now and then. Even if there were extensions, existing AP servers chop off and throw away data they don't understand. so none of these extensions could work.
I feel like most of the "WTF am I learning JSON-LD for" criticisms are coming from this status quo. That includes "if someone wants to add a gallery thing or whatever, can't they make a FEP?" The way things work now, your extension either a) works only in your software or b) has to be painfully negotiated with the whole community. We're all gonna have a big fight about it on this forum anyway. Let's not pretend JSON-LD helps us.
But if we add two things to the mix, the situation looks different. Those are 1. server software that "keeps all the bits", and 2. a whitelabel extensible app. That would make it very easy to spin up crazy new experiences for a sizeable existing userbase. Developers should not be forced to endure a FEP process, and they should not have to attract a userbase from nothing. They should be able to just build, without even worrying if they're stepping on toes. And of course, Fedify and libraries in other languages are a load-bearing part of that world, including enforcement of the JSON-LD rules.
That world does not exist at all today, but JSON-LD does, so it's pretty valid to describe this design as premature optimisation. I dunno though, we don't seem that far away.
Re: I have deeply mixed feelings about #ActivityPub's adoption of JSON-LD, as someone who's spent way too long dealing with it while building #Fedify.
@mat@friendica.exon.name that's a really interesting point of view, and has some parallels to how app development on the ATProto side is easier in many ways.
I do think that this is something C2S (aka the ActivityPub API) can enable.
I am critical of JSON-LD but I do certainly recognize I could be very wrong 😁
>There's no going back.
We absolutely must go back. Either we have a vibrant ecosystem where building stuff is a pleasant experience, or fediverse slowly dies while linked data cultists harass developers about nonresolvable URLs in @context.
JSON-LD adds nothing to ActivityPub, it only creates problems. Time to move on.
@silverpill @hongminhee can someone explain what was it supposed to bring?
It sounds like people are attached to xml with schemas whike everyone is using schemaless json (xmpp vs matrix...).
@hongminhee Not into the void. I don't know shit about JSON or programming, but I did read your post! :-)
JSON-LD 를 제대로 다루는 라이브러리를 하나 만들어 보시면 어떨까요
@chomu.dev JavaScript는 이미 JSON-LD 具顯이 있어서 Fedify는 안 그래도 그걸 가져다 쓰고 있어요 ㅋㅋㅋ
근데 왜 다른 사람들은 그걸 잘 안 쓰는 걸까요...?
@chomu.dev JSON-LD를 그냥 JSON으로 다뤄도 괜찮다고 보는 사람이 많은 것 같아요.
@hongminhee thank you for doing hard work with the plumbing! You are helping build coherence in this place and I'm grateful for it. Diverse people, unified standards 🖤
@douginamug Thank you so much for recognizing my hard work!