Understanding Re-Frame - Websockets?


#1

Hi all,
The “Understanding Re-Frame” course is absolutely terrific, I’m so glad I joined.
Right now, however, I’m trying to figure out how to use re-frame to share/sync data across clients in realtime.
I’m not familiar with it, but, presumably I need to share server-side state via websockets on an effect handler, right?
Is there any direction anyone can point me at?
BTW, what library/component that supports websockets would you use today?
Thanks so much in advance!


#2

Hi,
struggled with this last week. I used effect/event handlers that make xhrio calls. The backend is a “boring” ring/compojure app. So just regular REST calls because I don’t need anything from Websockets.

The xhrio call looks like this:

    (rf/reg-event-fx
     ::search-xyz
     (fn [{db :db} _]
       {:http-xhrio {:uri (str BASE_URI "/search")
                     :method :get
                     :params {:query-param1 "abc"}
                     :timeout 5000
                     :format (edn/edn-request-format)
                     :response-format (edn/edn-response-format) 
                     :on-success [::search-result]
                     :on-failure [::notified-error]
                     }})) 

(edn is a namespace from the clj-ajax library “(:require [ajax.edn :as edn])”)

I wrote about several problems I encountered. Maybe this is helpfult to you. https://feierabendprojekte.wordpress.com/2017/12/28/i-learned-the-inter-webs/

Cheers


#3

Thanks Azel!
As I’m leaning towards microservices for a greenfield project of mine, I have also set up a couple of separate ring/compojure services that I originally planned to call via xhrio… until I realized… oops I really need client’s realtime synchronisation for a good bunch of these calls (!)

I’m currently investigating the broader issue (I know very little about websockets) and - so far - I bumped into Sente, Chord and a couple of other integration attempts. Options can be overwhelming, especially in the early days, when you don’t know the relative merits and if certain solutions aged well.
I just hope I’m on the right path!


#4

Hey @cperrone,

Thanks for the great question! I am definitely working on a websocket/server push kind of lesson for realtime synchronization. But I’ll answer it quickly here:

Recommended servers:

Communication pattern:

The simplest to set up is that you send messages to the server, which has a thread to handle them. The client will send a “subscribe” event, such as “subscribe to XYZ chat room”, and the server will receive it, authenticate, then remember that. Whenever the chatroom changes, the server will see all subscribers and send relevant events to all clients. You could also have an “initial load” event which contains a big bunch of data for the current state of the chatroom.

The client receives those events and puts them into the database.

Re-frame side:

I would make an Effect that sends a message to the server over the websocket. Then the individual Event Handlers can trigger those events. It’s up to you whether to use optimistic or pessimistic update. For instance, you can do pessimistic update where your client sends a chat message, but it’s not shown until it roundtrips through the server.

I hope that helps!

Rock on!
Eric


#5

Awesome answer. Thank you @ericnormand !

Yeah, I know, this problem it’s somewhat at the “fringe” of re-frame (or even orthogonal), yet I suspect it’s becoming quite common with modern web apps nowadays, well beyond the canonical old “chat” scenario (I’m looking at trello-like apps where card state is propagated in realtime).

Eh eh, a good example about “how” to sync state seems like it could be a great followup to the excellent existing lesson about choosing “where” to store it :wink:

Frankly, I was already digging into HTTPKit/Sente as I saw quite a few people referring to it.
It might be my first attempt as it’s somewhat more familiar and builds upon stacks I already use.

Pedestal seems like a great contender, but I’ll have to investigate further.
I honestly had somewhat ruled out Aleph simply on the basis that the resources I found where a bit old.
Stupid, I know. If it works, it works.

Thanks indeed for the insights about how we may approach the problem in Re-frame.

One thing I truly love about re-frame is that the separation of concerns allow us to delay (or even revert) these decisions about what to use relatively easily.
Heck, I might keep working on the in-memory DB and/or plain xhrio calls for the time being :slight_smile:


#6

I’ve finished the Understanding Re-Frame course and have the beginnings of a solid mental model how everything works, but there is still a rather large gap for me as a novice Clojure programmer. I could certainly dig into sente and figure out how it works, at least to some extent, but I’m not at all sure how to plug that into the re-frame approach to handling state, which flows to and from the in-memory DB. How does persistent data flowing from and to the server intersect that circular flow?

I assume that when a form is first loaded with existing data for editing a record, for example, that data must be either added or overwritten in the in-memory DB, so one may need to check for the existence of a submap, if it’s there, overwrite, if it’s not, add. This seems somewhat cumbersome to me, but I probably just don’t understand something.

If a form is submitted with valid data, then I assume the in-memory version of that data will continue to hang around until the session is closed (unless it is pruned). So in the case of a business app that is logged into first thing in the morning and used continuously throughout the day, the in-memory DB will grow larger and larger as records are accessed? Is that how it is supposed to work?

Perhaps the reality is that the developer must prune the in-memory DB as it becomes clear the data is likely no longer needed. Is that the case? How is the in-memory DB kept in sync with the persistent database on the server, particularly when users modify data that may be cached in other users in-memory DBs? Theoretically, it seems possible to crash the browser if too much data is loaded into the in-memory DB over time. Is this a valid concern?

These all seem like naive questions to me but I haven’t been able to see through them on my own so far.


#7

Hey @nando,

Well, you bring up some great questions. The short answer to all of them is that Re-frame does not provide any facilities for these things. There’s nothing special in the framework for synching with the server or pruning old data so your db doesn’t grow out of control.

The integration point for interacting with the server is through Events. Events can send an ajax request and the ajax response can issue a new Event. Similarly, an Event can send a message over the websocket and a response can issue an Event.

The issue of Re-frame’s in-memory database being out of sync with the server is a very hard problem that all web apps face–in fact, all distributed systems face. Re-frame doesn’t promise to solve it, but it does give you the tools to deal with these things as you need to. Do you have a particular problem you’re trying to solve?

Rock on!
Eric


#8

Hi Eric,

For now, I’m only trying to develop a mental model of how everything fits together in a re-frame app, and of course, understanding the persistence part is necessary.

In a “traditional” web app where the views are generated on the server, the only data that is persisted in the client is related to a user session, whether a user is logged in, their permissions and preferences, more or less. I want to keep it as simple as possible, so to me what would make sense in the context of re-frame would be to fetch data for a particular panel anew from the server whenever it was loaded, and prune that data whenever a user navigated away from that panel. Is there a re-frame event triggered when the user navigates away from a panel that I could use to delete the panel’s data from the in-memory DB? Or is there a better strategy to continuously purge potentially stale data as the user navigates through the app? I could, for instance, place all data for a panel in a map under the keyword :data, and delete that with a “global event handler” whenever the user transitions from one panel to another. Then I’d only need to write that code once.

Consistent data across users is more important in my use case than the performance hit from pulling data afresh from the server all the time. Imagine implementing this discussion board in re-frame. If you reply to this message, how does my in-memory DB pick this change up? Is there a library that could do this, encapsulating the difficulty, without re-fetching the data for the entire discussion?

As I think through this, the answer may depend on whether users are expected to remain within one panel and data must be updated with a websocket connection, such as a chat app, or whether users are expected to fairly rapidly traverse through multiple panels within their workflow (the latter being the case I’m exploring).

Thanks for your help, Eric.


#9

Is there a re-frame event triggered when the user navigates away from a panel that I could use to delete the panel’s data from the in-memory DB?

There are no built-in events like this. But you can easily set up an event to be dispatched when you switch panels. That’s how I do it.

Now, for the most reliable and simplest way to purge old data is to consider your in-memory db to be a cache of what’s on the server. Keep a least-recently-used discipline so that you can purge stuff that hasn’t been accessed in a while.

If you store stuff for a view under a particular keyword and purge it after you switch views kind of defeats the purpose. The DB should have a unified view of the data. There is very often more shared between the panels than you might think.

Consistent data across users is more important in my use case than the performance hit from pulling data afresh from the server all the time.

Well, I’d argue that at that point, why not just use a “traditional” web app. Note that these problems of data freshness, synching with the server, and purging are present in every “single page app”. No framework has a real solution to this.

If you reply to this message, how does my in-memory DB pick this change up?

The server needs to notify the client. When the client connect to the server (through websockets or server-sent events), a “subscription” is set up for the data they are looking for. That subscription keeps track of whatever needs to update. In a chat app, you’d keep track of what chatrooms they are in. But it’s totally domain and backend dependent. No solution to this, though GraphQL may make it easier.

Is there a library that could do this, encapsulating the difficulty, without re-fetching the data for the entire discussion

No library. I think it’s very domain-specific, though GraphQL may help. The best patterns that I’ve see rely on very granular updates. So instead of sending the whole Person record when their address changes, you just send the person’s new address.

I hope this discussion helps. You’re asking really deep questions that are past the edge of our abilities to solve in the general case so far. So it sounds like you understand the issues.


#10

That’s very helpful Eric. Thanks.

If the data unique to the panel was fetched on every view, I assume an SPA would be more performant than a traditional web app simply because the packet of data needed for any panel will be significantly less than the data plus the markup plus any view generation latency the server imposes when serving a traditional web app.

I could imagine maintaining a copy of the client DB on the server. The data for each view would be stored under a keyword and hashed when it was modified on the server, and the hash sent to the client. When a user navigates to a view, the hashes are compared, and when different, the data would be refreshed from the server. That’s about as sophisticated as I can manage. Perhaps there is a caching library that might facilitate such a strategy … I’m just thinking out loud.