Users work with Devin sessions for days at a time, creating huge sessions with hundreds of thousands of events. These include messages, actions, diffs, generated artifacts, testing videos, and more. Our largest Devin sessions were taking 20+ seconds to load. When scrolling up in the chat or streaming new messages, they’d crawl with dropped frames and excessive memory consumption.
We built Devin a brand new chat renderer to make sessions feel lightning fast.
The textbook solution
Fetching and rendering 100,000+ events will never be fast. Our events contain complex JSON data; there’s no cheap way to send hundreds of MB over the wire. And rendering those events into DOM nodes will crush your browser’s frame budget. Add in database query speed and layout work on every resize — your app goes from slow to unusable very quickly.
The canonical solution to this problem is simple: don’t fetch everything, and don’t render everything.
- Fetch only the most recent events, and load older events when the user scrolls up into them.
- Render only the events that are on the user’s screen. Anything off-screen becomes a blank placeholder to keep the scroll height.
Rendering only what’s visible on screen is called virtualization. It’s a classic technique for making long lists fast.
This is as far as most chat apps go.
We needed to do better
The biggest issue with the classic approach is scroll jank.
As events load in, the heights of elements on screen change, moving the content you were in the middle of reading. Since new rows render whenever something moves on or off screen, this jank happens on every scroll. Streaming makes it worse still — the chat moves under you even when you aren’t scrolling.
The other issue is that accessing any part of the session other than the preloaded tail becomes annoyingly slow. You scroll, hit a spinner, wait, and then scroll again, and repeat over and over until you find what you want. For very long sessions, navigating up to the first message can take minutes of scrolling.
We wanted users to feel like the whole session was already on their machine. You should be able to load into a session and scroll to what you want immediately. Seeing a brief loader when you scroll up is OK, but scrolling needed to feel seamless.
To do this, we had to take a unique approach to fetching data and stabilizing our virtualizer.
Filling in the map
If we want users to be able to scroll up to any part of the chat history, we need to reserve space for them to actually scroll to. We can’t load the entire chat, but we don’t need to — we can load just enough to give users a rough “map” of their session.
We only need the type of each event; from that, we can paint a skeleton that approximates the real event’s height and shape. Concretely, this outline is just a big array of message types, from which we render a big array of skeleton components. This query stays fast, even on huge sessions, for two reasons: we only scan an index, avoiding expensive JSON materialization, and we return very little data — just an array of types.
This gives us a rough map of the entire session.
When the user scrolls up to an unloaded section of the chat, we fill in an island of the map with complete event data. We compute the event IDs of the skeletons on screen, plus a buffer in either direction, so that when the user scrolls slowly, content is already loaded by the time it comes into view. Then we fetch the full event data for those IDs, so we can swap the skeletons for real messages.
With loading done, the challenge becomes swapping in the real messages without scroll jank.
Virtualization without jank
Say you’re at the bottom of a chat and scroll up until you see some skeletons. Once those turn into real messages, they’ll swap from estimated to real height, which pushes the other messages on screen down. This jank gets even worse in a complicated app like Devin with dynamic UI components and message streaming.
To fix this, we must keep the position of the element the user was looking at fixed, even as row heights change around it. At a basic level:
- If rows resize below what you’re looking at, no problem — our scroll position is a measure from the top of the chat.
- If rows resize above what you’re looking at, measure the number of pixels gained or lost, and subtract that from the scroll position.
In practice, this becomes messy quickly. We have to ensure only one group of rows loads in at a time, and offset the height delta before paint. And the “what you’re looking at” mentioned above is hard to figure out. Is it a row on screen? If so, which one? What if there are no rows on screen and it’s all skeletons? What if there are skeletons interspersed with loaded rows?
To solve this, we came up with two rules that let us figure out “what you’re looking at,” even in these trickier situations:
- If there are visible loaded messages, choose the topmost message.
- If there are only skeletons on screen, choose the topmost skeleton.
We call this the “anchor” row. We record the position of the top of the anchor relative to the viewport. Once the browser has finished laying out the new rows, we measure the delta from before those rows rendered. We apply that delta as a scroll offset before the browser paints.
Notably, this does not address the edge case of skeletons interspersed with loaded rows. In this scenario, the user might have been looking at any of the rows on screen, and we cannot keep them all stable.
In this case, there’s no way to avoid scroll jank. So instead, we prevent these situations from arising entirely.
Whenever a load would result in an “islet” (a small enough group of skeletons that you could see both ends of it in the viewport at once), we extend the load to cover the whole islet. This keeps all load situations on the happy path, so we never have to compromise on what part of the screen will shift.
Making agents good at front-end polish
Most of these improvements were driven by finding failure cases and giving them to Devin to fix. Out of the box, Devin wasn’t great at figuring out what was going on. Computer use just isn’t good enough to understand the UI issues that humans understand intuitively.
Instead of relying on it, we had Devin build a virtualization debugger that combined visualizations of the chat state (for humans) with extensive logging of every relevant action (for agents). Whenever a new bug was reported, we could load the session up in the debugger. Many issues could be intuited from the visualization. For subtler bugs, we could give Devin the log so it could root-cause the exact issue.
Towards the end of the project, we would gather large batches of failing session logs and give them to Devin overnight. It would come up with several fixes in parallel and test them with the debugger on its computer to confirm that the issue would not reproduce. We’d come back in the morning to validate its work while we set it onto the next batch of broken logs.
We were impressed by what Devin could do when we gave it the tools to see what humans see, and validate that its fixes work, without a human in the loop.
Agents tend to stick to their existing tools, like computer use, rather than setting out to build new ones, like custom debuggers. So, for the time being, it’s up to the engineers running them to push them in an ambition-maximizing direction.
Conclusion
The new chat renderer significantly improves the user experience for large sessions. Chats load 70% faster, and layout shift is down 86%. Users can now seamlessly scroll to any position, even in the largest sessions with hundreds of thousands of events.
These changes, along with dozens of other small details, are live in the app today.
At Cognition, we’re obsessed with building the best coding agent in the world. If this mission resonates with you, come join us — we’re hiring product and infra engineers.