Neat idea. Regarding the performance, I think you could get a lot better performance by training a small classifier model, essentially an embedding model, and using the LLM as the distillation source. This would both be much smaller, addressing your desire for it to run in browser, while also being much more performant, addressing your quantization need. Using the full LLM is a bit overkill and you can extract the core of what you're looking for out of it with something a little custom.
I just wish the stupid chatbots on customer service websites would show "Get human" as an option sooner. 99% of time there is a mistake and the bots don't have authority to correct it.
Thats why I go for the nuclear option (the one that requires higest attention, like account cancellation) when presented with a list of options by a bot. More likely to get a person that way.
The problem is it's hard to come up with better examples than your toy examples of weather and maps. Goodness there are so many travel planning demos. Who actually wants the context switch of a UI popping up mid-typed-sentence? Is a date picker really more convenient than typing "next school break"? Visualizations are interesting -- but that changes the framing from soliciting input to densifying information presentation. Datagrids and charts'll be valuable.
Anyway, it's a space that's still starving for great ideas. Good luck!
Thank you for the references. I agree there is a lot of room for innovation in this space.
Main differentiator here is as-you-type UI customization, and also in this case the UI isn't generative in the sense the options are hard coded - the LLM just chooses between them.
I think we'll see a mix of all of the above in the future as we take full advantage of LLMs.
People are coming to ChatGPT with an intent to type. Its much harder to introduce text-based interface to otherwise click-based interfaces because of the friction. Thats why chatbots often fail to gain adoption, because they are a byproduct of the product team being lazy and not thinking through the UX.
I 100% agree that grafting a chatbox on an existing app that wasn't meant for it sucks. (I'm working on an AI app where I set a "no chatbox" constraint, see my profile.)
However, I think a lot of people are willing to type if it will do something useful they can't otherwise accomplish. And: typing gives you user intent, very valuable.
I think this idea of "generative UI" stems from a genuine confusion about what Spotlight, Alfred, or Chrome Omnibox are: they are a collection of utilities, not an app of themselves.
They work because a designer has put some thought into anticipating what kinds of things users search for – formulae, definitions, maps, flights, timers – and designed tiny, polished experiences for each – important in their own right, but not enough to be standalone apps. The NLP component only acts as a dispatcher, calling the right intent with the right arguments.
If these micro-apps are untested unpolished, and most likely broken, having been created here and there without a human touch, they will likely prove inadequate for the user. And if the user is now forced to debug or extend them with natural language and no understanding/control of the underlying code, allowing the power users to write community plugins will have become preferable to this. Not to mention the latency and unpredictability which defeat the standard quick search model.
This is cool. Thanks for posting.
I could see this adaptive UI / reactive intent UX become valuable for a couple use cases:
1. Tutorial modes for complicated UX that gradually introduces the user to new features as they "progress" through the website.
2. Warning users that their inputs could lead to unforeseen consequences. Think "you forgot to attach the file you talked about in the email" warning that Gmail already does.
This feels similar to the experience you get when asking ChatGPT a programming question. The UI shifts to better display code and provides buttons to copy content.
This looks like a neat idea, but I'm not too positive about it.
This makes using computers even harder to explain to people who do not spend their entire day keeping up with the latest developments. They cannot form a mental image or reuse any memory of what will happen next, because it is all context dependent.
On the other end of the spectrum, for power users, dynamically adapting user interfaces can also be quite annoying. One can't type ahead, or use shortcut keys, because one doesn't know what the context will be. Having to wait any positive amount of time for feedback is limiting.
Then again, there are probably tons of places where this is useful. I'm just a bit disappointed that we (as a society) haven't gotten the basics covered: programming still requires text files that can be sent to a matrix printer, and the latency of most applications is increasing instead of decreasing as computers become faster.
I think there is a good chance that this design is more suitable for power uses.
My hunch is that typing 5 words could be faster than going through 3 menus and screens. I don't see this as a replacement for normal UI, but as an optional shortcut - if you're happy typing.
This is a very cool idea, I think this could potentially be more powerful by (at least initially) limiting it to small sections of the UI (Like you've done in your example)
You could take it a step further and, instead of adding/removing buttons, simply updating the copy of (a) button(s) based on the question, which I imagine you could manage with a very small model.
Cool demo and a neat write-up! I've been experimenting with similar patterns to differentiate between search queries/question answering and presenting specific outputs to the user.
Interesting, but also true that intent can be much more accurately passed with words, than inferred with guessing? Visit apple.com, reorganize this list in alphabet order, find me a ...
Neat idea. Regarding the performance, I think you could get a lot better performance by training a small classifier model, essentially an embedding model, and using the LLM as the distillation source. This would both be much smaller, addressing your desire for it to run in browser, while also being much more performant, addressing your quantization need. Using the full LLM is a bit overkill and you can extract the core of what you're looking for out of it with something a little custom.
I just wish the stupid chatbots on customer service websites would show "Get human" as an option sooner. 99% of time there is a mistake and the bots don't have authority to correct it.
Thats why I go for the nuclear option (the one that requires higest attention, like account cancellation) when presented with a list of options by a bot. More likely to get a person that way.
LLMs generating just-in-time UI has a lot of interest and effort going into it. It's usually called "generative UI" or "dynamic UI generation". It was a pretty hot about a year ago. Here's a HF blog on it https://huggingface.co/blog/airabbitX/llm-chatbots-30. Also check out Microsoft's Adaptive Cards. Nielsen Group wrote about it too. https://www.nngroup.com/articles/generative-ui/
The problem is it's hard to come up with better examples than your toy examples of weather and maps. Goodness there are so many travel planning demos. Who actually wants the context switch of a UI popping up mid-typed-sentence? Is a date picker really more convenient than typing "next school break"? Visualizations are interesting -- but that changes the framing from soliciting input to densifying information presentation. Datagrids and charts'll be valuable.
Anyway, it's a space that's still starving for great ideas. Good luck!
Thank you for the references. I agree there is a lot of room for innovation in this space.
Main differentiator here is as-you-type UI customization, and also in this case the UI isn't generative in the sense the options are hard coded - the LLM just chooses between them.
I think we'll see a mix of all of the above in the future as we take full advantage of LLMs.
This is great. But the vast majority of non-gaming non-terminal-loving-developers human beings want to avoid the keyboard at all costs.
They are doing it via trackpads, mousepads, touch screens, etc. Which are all inputs that transcend language or the ability to find meaningful words.
ChatGPT has 400 million weekly users
People are coming to ChatGPT with an intent to type. Its much harder to introduce text-based interface to otherwise click-based interfaces because of the friction. Thats why chatbots often fail to gain adoption, because they are a byproduct of the product team being lazy and not thinking through the UX.
I 100% agree that grafting a chatbox on an existing app that wasn't meant for it sucks. (I'm working on an AI app where I set a "no chatbox" constraint, see my profile.)
However, I think a lot of people are willing to type if it will do something useful they can't otherwise accomplish. And: typing gives you user intent, very valuable.
I think this idea of "generative UI" stems from a genuine confusion about what Spotlight, Alfred, or Chrome Omnibox are: they are a collection of utilities, not an app of themselves.
They work because a designer has put some thought into anticipating what kinds of things users search for – formulae, definitions, maps, flights, timers – and designed tiny, polished experiences for each – important in their own right, but not enough to be standalone apps. The NLP component only acts as a dispatcher, calling the right intent with the right arguments.
If these micro-apps are untested unpolished, and most likely broken, having been created here and there without a human touch, they will likely prove inadequate for the user. And if the user is now forced to debug or extend them with natural language and no understanding/control of the underlying code, allowing the power users to write community plugins will have become preferable to this. Not to mention the latency and unpredictability which defeat the standard quick search model.
We're always coming back full circle: https://en.m.wikipedia.org/wiki/Ubiquity_(Firefox)
Was this supposed to embed maps in emails, etc?
This is cool. Thanks for posting. I could see this adaptive UI / reactive intent UX become valuable for a couple use cases: 1. Tutorial modes for complicated UX that gradually introduces the user to new features as they "progress" through the website. 2. Warning users that their inputs could lead to unforeseen consequences. Think "you forgot to attach the file you talked about in the email" warning that Gmail already does.
This feels similar to the experience you get when asking ChatGPT a programming question. The UI shifts to better display code and provides buttons to copy content.
This looks like a neat idea, but I'm not too positive about it.
This makes using computers even harder to explain to people who do not spend their entire day keeping up with the latest developments. They cannot form a mental image or reuse any memory of what will happen next, because it is all context dependent.
On the other end of the spectrum, for power users, dynamically adapting user interfaces can also be quite annoying. One can't type ahead, or use shortcut keys, because one doesn't know what the context will be. Having to wait any positive amount of time for feedback is limiting.
Then again, there are probably tons of places where this is useful. I'm just a bit disappointed that we (as a society) haven't gotten the basics covered: programming still requires text files that can be sent to a matrix printer, and the latency of most applications is increasing instead of decreasing as computers become faster.
I think there is a good chance that this design is more suitable for power uses.
My hunch is that typing 5 words could be faster than going through 3 menus and screens. I don't see this as a replacement for normal UI, but as an optional shortcut - if you're happy typing.
This is a very cool idea, I think this could potentially be more powerful by (at least initially) limiting it to small sections of the UI (Like you've done in your example)
You could take it a step further and, instead of adding/removing buttons, simply updating the copy of (a) button(s) based on the question, which I imagine you could manage with a very small model.
Cool idea. FYI the GitHub link at the bottom leads to "page not found". Maybe the repo is not public?
Thank you, you're right. Now public.
Cool demo and a neat write-up! I've been experimenting with similar patterns to differentiate between search queries/question answering and presenting specific outputs to the user.
Thanks for taking your time to write this up
Interesting, but also true that intent can be much more accurately passed with words, than inferred with guessing? Visit apple.com, reorganize this list in alphabet order, find me a ...
You get it
This seems like a very expensive way to do basic NLP
What the Windows search box wanted to be.