tl;dr We plan to explicitly call out experimental capabilities (features, products, etc) as Prototype capabilities so that our users and customers know where we are still actively iterating and gathering feedback on, without guarantees on what the end state looks like.
Background and Motivation
In the last few months at Codeium, we have reached the realm of “we don’t really know whether new capabilities will definitely drive more value to our users.” We had built autocomplete and chat, state of the art proprietary code LLMs, a full codebase aware context awareness engine, integrations into over 40 IDEs, and a number of other things that we knew would drive value going in. There was no question we would build, productionize, and maintain these. But what is next?
There are a lot of demos of new capabilities on the internet, some of which are very flashy, but we are not convinced that they truly drive value. We have worked in this space for long enough to have some humility in the face of both the novelty and complexity of this technology and the difficulty to get widespread adoption amongst developers. So, in the ideal world, we would develop these potential capabilities quickly, test their effectiveness, and make this judgment call for ourselves, armed by real data.
However, while there are some things we can A/B test without our users having to learn anything new, such as new models or changes to our context awareness engine, a lot of the things we would like to test are visible to the developer, and sometimes require explicit developer interaction. These would include recent capabilities like our GPT4 integration and Codeium Command. We have many reasons to believe that these will indeed drive value, but are they robust enough? Valuable enough? Adopted well? Cost-effective enough to serve? We don’t know until we try.
Prototype Capabilities
Enter, Prototype capabilities. We have a lot of ideas where AI in the software development life cycle is going and want to ship quickly to test these, and not just develop in the dark. At the same time, however, we want to be transparent to you about where we are still experimenting so that, if we make changes or completely roll back the capability, we don’t lose your trust. Prototype capabilities are like rapid-fire MVPs.
Why use the term “Prototype” instead of the often used “Beta”? To us, “Beta” conveys the sense that we know for a fact that the capability is going to work and just needs some tweaking and tuning, while “Prototype” more accurately communicates that we have a lot of reason to believe that something will have value, but there will be work required to reach there, if we reach there at all.
Not all capabilities will be Prototype capabilities. There are a number of capabilities and even entirely new modalities that we are either convinced will drive meaningful value, or are important to us for other reasons. Those will just be launched as normal.
What You Should Expect from Prototype Capabilities
While in the Prototype stage, to minimize our engineering overhead, the capability will likely come to only one or two surfaces, such as JetBrains and Visual Studio Code on the IDE side or GitHub SaaS on the SCM side. Access will be gated behind user lists, which will be collected via waitlists posted on Discord and social media, so that we can survey and follow up with the users trying these capabilities out. We will likely also figure out a permanent early access system so folks who like being early adopters won’t have to keep filling out waitlists, but more to come on that.
All Prototype capability announcements will be clearly marked as such, and any tags or mentions of “Prototype” will be removed once we have verified that this is something we want to continue developing, after which productionizing across IDEs and our enterprise plans will shortly follow. In these Prototype announcements, we promise that we will clearly articulate why we are building and testing the capability (the rationale behind the Prototype), why we are not sure if it would work or be adopted, and if relevant, potential directions the capability could go depending on feedback.
To be clear, there are no promises that a Prototype capability will be productionized long-term for either individuals or enterprises, or that the capability will look the same or similar to the initial release. We do not believe in shipping capabilities for the sake of shipping capabilities. If the capabilities don’t drive meaningful value, we are doing you a disservice by investing a bunch of development time productionizing the capability more broadly and maintaining it indefinitely.
Conclusion
We debated for a while how we wanted to communicate our development with you, the users that expect us to keep building more and more powerful systems. We believe Prototype capabilities will allow us to both demonstrate our speed while communicating the reality that no one has “the answers” in this space.