Since we last built Flint earlier this year (https://news.ycombinator.com/item?id=48834924), we have been wondering if interactive charts can also find they home in terms of simple high-level specifications to allow AI agents to produce them easily, reliably and with high-quality. I have always been amazed by NYT interactive stories back in school, as they can always tell a fun story without super long verbose text write-ups; thus I have been wondering what if they can just be as easily created as any basic charts to make them easier to create everywhere.
Despite the AI models now are more capable of creating interactive artifacts, we find creating interactive charts still kinda annoying (esp. if you are building them to serve users directly in their chat, in your data apps). Agents (especially with smaller models) quite often generate interactions with undesired effects, they are often quite slow, costly, and extremely annoying if to want to steer it to change something. Most times, I find that if I ask the agent to change interaction, it can randomly change style, layout, even the data.
We find the issue is that the language agents use to create interactive visualization is kinda low-level. The logic for specifying data encoding, style and interactions are all entangled (if you swap x,y axis, you are likely going to redo your interaction and layout again). Thus, despite agents have been working hard, it is essentially doing lots of stuff bottom up. We decide to design a language that can separate the visual logic and the interaction logic to make specification much cleaner, manageable, and friendly (to both human and AI). The magic here is that we are now disentangle the logic at the spec level, but make the compiler to do the heavy lifting (sorry, compiler!) to wire up logics to support easy composition and adaptation of the logics to create a expressive interactive charts. We used a trick of bidirectional transformation to help the compiler maintain the mapping between pixel/visual space to data semantics so the specs can stay at a higher-level. We made a video to explain the ideas.
- the language should be high-level and the agents can easily user them in chats/apps declaratively (with many presets).
- the language should not give up expressiveness. We also provide a programming API, so that you can use it to create charts in your app/dashboards for external interactions. The expressive API also makes it possible to compose very cool bespoke interactions (the spinning globe in the website).
- If you are building applications, ask the agent to use the library. Flint is especially suitable for building agentic apps where you want agents to bring up an interactive charts quickly in the user session following their asks.
- Btw, the flint mcp app allows you create interactive dashboard in the chat as well.
Finally, I'm thinking that despite AI agents are now so good at coding, that they seem to be able to create EVERYTHING. It might not just be the time to give up designing new languages to make it easy to describe and create stuff. In the end, language is still the way for us to communicate with the computer, I think the alignment of programming language between human and AI in could probably make building, steering easier.
Thank you for writing this yourself, it's informative and refreshing!
If an LLM wrote it, then it's the best simulation of non-LLM writing I've seen on an agent related post. They seem all to have gone through an LLM "edit" (cough) that strips the personality seen in your update.
Since we last built Flint earlier this year (https://news.ycombinator.com/item?id=48834924), we have been wondering if interactive charts can also find they home in terms of simple high-level specifications to allow AI agents to produce them easily, reliably and with high-quality. I have always been amazed by NYT interactive stories back in school, as they can always tell a fun story without super long verbose text write-ups; thus I have been wondering what if they can just be as easily created as any basic charts to make them easier to create everywhere. Despite the AI models now are more capable of creating interactive artifacts, we find creating interactive charts still kinda annoying (esp. if you are building them to serve users directly in their chat, in your data apps). Agents (especially with smaller models) quite often generate interactions with undesired effects, they are often quite slow, costly, and extremely annoying if to want to steer it to change something. Most times, I find that if I ask the agent to change interaction, it can randomly change style, layout, even the data.
We find the issue is that the language agents use to create interactive visualization is kinda low-level. The logic for specifying data encoding, style and interactions are all entangled (if you swap x,y axis, you are likely going to redo your interaction and layout again). Thus, despite agents have been working hard, it is essentially doing lots of stuff bottom up. We decide to design a language that can separate the visual logic and the interaction logic to make specification much cleaner, manageable, and friendly (to both human and AI). The magic here is that we are now disentangle the logic at the spec level, but make the compiler to do the heavy lifting (sorry, compiler!) to wire up logics to support easy composition and adaptation of the logics to create a expressive interactive charts. We used a trick of bidirectional transformation to help the compiler maintain the mapping between pixel/visual space to data semantics so the specs can stay at a higher-level. We made a video to explain the ideas.
https://youtu.be/JAJy7ToczMs
Some principles we had in the design:
- the language should be high-level and the agents can easily user them in chats/apps declaratively (with many presets).
- the language should not give up expressiveness. We also provide a programming API, so that you can use it to create charts in your app/dashboards for external interactions. The expressive API also makes it possible to compose very cool bespoke interactions (the spinning globe in the website).
To try it out:
- You can try Flint in your agent app. Simply call your agent to create it as an MCP app in your chat! (https://microsoft.github.io/flint-chart/#/mcp)
- If you are building applications, ask the agent to use the library. Flint is especially suitable for building agentic apps where you want agents to bring up an interactive charts quickly in the user session following their asks.
- Btw, the flint mcp app allows you create interactive dashboard in the chat as well.
Finally, I'm thinking that despite AI agents are now so good at coding, that they seem to be able to create EVERYTHING. It might not just be the time to give up designing new languages to make it easy to describe and create stuff. In the end, language is still the way for us to communicate with the computer, I think the alignment of programming language between human and AI in could probably make building, steering easier.
Thank you for writing this yourself, it's informative and refreshing!
If an LLM wrote it, then it's the best simulation of non-LLM writing I've seen on an agent related post. They seem all to have gone through an LLM "edit" (cough) that strips the personality seen in your update.
Also, Flint is awesome.