It takes all the core relational operators and makes an easy semantic version of each as a python dataframe library extension . Each call ends up being a 'model' point in case you also want to do fancier things later like more learning based approaches. Afaict, snowflake and friends are moving in this direction for their cloud SQLs as well.
We ended up doing something similar for louie.ai , where you use AI notebooks/dashboards/APIs (ex: MCP) to talk to your data (splunk, databricks, graph db, whatever), and it'll figure out symbolic + semantic operators based on the context. Super helpful in practice.
My 80% case here is:
- semantic map: "get all the alerts from splunk index xyz, add a column flagging anything suspicious and another explaining why" <--- generates an enriched dataframe
- semantic map => semantic reduce: "... then summarize what you found" <--- then tells you about it in natural text
I think it's interpreting the command as "replace each fruit with a vegetable", and it might intuit "make the resulting vegetables unique from one another" but otherwise it's not trying to find the "most similar" vegetable to every fruit or anything like that.
It's subjected to randomness. But you're ultimately in control of the LLMs's hyperparams -- temperature, top_p, and seed -- so, you get deterministic outputs if that's what you need. However, there are downsides to this kind of LLM deterministic tweaks because of the inherent autoregressive nature of the LLM.
For instance, with temperature 1 there *could be* a path that satisfies your instruction which otherwise gets missed. There's interesting work here at the intersection of generative grammars and LLMs, where you can cast the problem as an FSM/PA automaton such that you only sample from that grammar with the LLM (you use something like logits_bias to turn off unwanted tokens and keep only those that define the grammar). You can define grammars with libs like lark or parsimonious, and this was how people solved JSON format with LLMs -- JSON is a formal grammar.
Contracts alleviate some of this through post validation, *as long as* you find a way to semantically encode your deterministic constraint.
That's gonna be a very, very, long answer. What's funny is that not much changed since 2022 (eoy) when the projected started; the models just got better, but we had a good chunk of the primitives since gpt-3.
What's more recent is the DbC contribution which I think is unique. It literally solved anything agent related I've thrown at it -- especially because I can chain contracts together and the guardrails propagate nicely.
I've built most the custom tools myself. For instance, not only perplexity was rendered useless by openai's web search, but openai's web search itself is not good enough for what you can customize yourself. To this end, I've built my own deep research agent. Here's a thread with some results from the first day it was working: https://x.com/futurisold/status/1931751644233945216
I'm also running a company, and we've built an e2e document generation just from contracts (3 contracts chained together in this case). Here's an output (sorry about the PDF rendering, that's not what we serve, it's just what I've quickly hacked for local dev): https://drive.google.com/file/d/1Va7ALq_N-fTYeumKhH4jSxsTrWD...
This was the input:
---
Prompt:
> I want the files to be analyzed and I am interested in finding patterns; feel free to make suggestions as well. I want to understand how different providers use their system prompts, therefore things like: what kind of tags do they use – are they XML, markdown, etc, are they prone toward sycophancy or trying to manipulate the user, are they using tools and if so how, etc. I want the tech report to deconstruct and synthesize and compare the information, find interesting patterns that would be hard to spot.
Generated instructions:
(a) Query:
Conduct a comparative analysis of system prompts across major AI providers (OpenAI, Google, Anthropic, xAI, etc.) to identify structural patterns, linguistic frameworks, and operational constraints that shape AI behavior and responses.
(b) Specific Questions:
1. What syntactic structures and formatting conventions (XML, markdown, JSON, etc.) are employed across different AI system prompts, and how do these technical choices reflect different approaches to model instruction?
2. To what extent do system prompts encode instructions for deference, agreeability, or user manipulation, and how do these psychological frameworks vary between commercial and research-focused models?
3. How do AI providers implement and constrain tool usage in their system prompts, and what patterns emerge in permission structures, capability boundaries, and function calling conventions?
4. What ethical guardrails and content moderation approaches appear consistently across system prompts, and how do implementation details reveal different risk tolerance levels between major AI labs?
5. What unique architectural elements in specific providers' system prompts reveal distinctive engineering approaches to model alignment, and how might these design choices influence downstream user experiences?
One last comment here on contracts; an excerpt from the linked post I think it's extremely relevant for LLMs, maybe it triggers an interesting discussion here:
"The scope of contracts extends beyond basic validation. One key observation is that a contract is considered fulfilled if both the LLM’s input and output are successfully validated against their specifications. This leads to a deep implication: if two different agents satisfy the same contract, they are functionally equivalent, at least with respect to that specific contract.
This concept of functional equivalence through contracts opens up promising opportunities. In principle, you could replace one LLM with another, or even substitute an LLM with a rule-based system, and as long as both satisfy the same contract, your application should continue functioning correctly. This creates a level of abstraction that shields higher-level components from the implementation details of underlying models."
Anyone interested in this from a history / semiotics / language-theory perspective should look into the triad concepts of:
Sign (Signum) - The thing which points
Locus - The thing being pointed to
Sense (Sensus) - The effect/sense in the interpreter
Also known by: Representation/Object/Interpretation, Symbol/Referent/Thought, Signal/Data/User, Symbol/State/Update. Same pattern has been independently identified many many times through history, always ending up with the triplet, renamed many many times.
What you're describing above is the "Locus" essential object being pointed to, fulfilled by different contracts/LLMs/systems but the same essential thing always being eluded to. There's an elegant stability to it from a systems design pov. It makes strong sense to build around those as the indexes/keys being pointed towards, and then various implementations (Signs) attempting to achieve them. I'm building a similar system atm.
Thanks for bringing this up. I'm fairly familiar with Peirce's triadic semiotics and Montague's semantics, and they show up in some of my notes. I haven't turned those sketches into anything applied yet, but the design space feels *huge* and quite promising intuitively.
Very cool. Being able to use semantic (as opposed to syntactic) operators like `==`, `+`, etc. feels like fertilizer for some novel ideas. Sort of like when word embeddings first came out and there was a loose concept algebra introduced with it ("King - Man + Woman = Queen").
That said the neuro + symbolic integration here is, like most systems, pretty shallow/firewalled (taxonomically, Type 3 / Neuro;Symbolic — https://harshakokel.com/posts/neurosymbolic-systems). I think the real magic is going to come when we start heading toward a much more fundamental integration. We're actually working on this at my company (https://onton.com). How do we create a post-LLM system that:
1) features an integrated representation (neither purely symbolic nor dense floating point matrix); 2) can learn incrementally from small amounts of noisy data, without being subject to catastrophic forgetting; 3) can perform mathematical and other symbolic operations with bulletproof reliability; and 4) is hallucination-free?
The cobbling together of existing systems hot-glue style is certainly useful, but I think a unified architecture is going to change everything.
We hear you. We might end up renaming it. In the paper we have a footnote about the name choice -- it's meant to credit the foundational work of Newell and Simon that inspired this project.
I'd appreciate it! It's cool and I wish you success. Just hope that when someone says "We're using Symbolic AI" a year from now, it won't be even more ambiguous than today :D
I didn't expect this -- I was supposed to be sleeping now, but I guess I'll chat with whoever jumps in! Good thing I've got some white nights experience.
One question, OP, how does cost for this work? Do you pay the LLM inference cost (quite literally if using an external API) every time you run a line that involves natural language computation? E.g. what happens if you call a "symbolic" function in a loop.
Yes, that's correct. If using say openai, then every semantic ops are API calls to openai. If you're hosting a local LLM via llama.cpp, then obviously there's no inference cost other than that of hosting the model.
this works like functional programming where every symbol is a pure value and operations compose into clean, traceable flows. when you hit an ambiguous step, the model steps in. just like IO in FP, the generative call is treated as a scoped side effect. this can engage your reasoning graph stays deterministic by default and only defers to the model when needed. crazy demo though, love it
Yes, pretty much. We wanted it be functional from the start. Even low level, everything's functional (it's even called functional.py/core.py). We're using decorators everywhere. This helped a lot with refactoring, extending the framework, containing bugs, etc.
Since code is generated by LLM these days, how does specific syntactic constructs like a Symbol which essentially carries the context and can be manipulated with python operators help when compared to a normal python code generated by LLM with all the checks and balances instructed by a human?
For example, I can write in this syntax to convert all fruits to vegetables or I can simply prompt an LLM to construct a program that takes a list of fruits and calls a LLM in the background to return the vegetables equivalent.
I am trying to understand the difference.
Yes. That seems to be the case.
While it may not be saving any time compared to generating general python code vs. specific symbolic code, the real value could be that it has an engine to enforce the contract on LLM responses with the library or even do the calls to the LLM as a common piece of code making it less error prone and bringing consistency in the interactions with the LLM.
Shortly, yes to all. We actually had an experiment going from theory of mind to emotion, but it's hanging right now since I feel the models aren't quite there yet and it yields diminish returns relative to effort. But could easily be revived. Minsky isn't my fav though, I'm leaning more toward Newell/Simon and friends from that generation.
I spent some time toying around with LLM-guided "symbolic regression", basically having an LLM review documents in order to come up with primitives (aka operators) that could be fed into github.com/MilesCranmer/PySR
I didn't get very far because I had difficulty piping it all together, but with something like this I might give it another go. Cool stuff.
Oh, definitely. I recommend you go for contracts. I've used something similar for a contract that iteratively "stitched together" a broken ontology graph. Here's some of the data models for inspiration -- you could have something similar for your ops, and write the contract to solve for one op, then apply the op, etc.
---
class Merge(LLMDataModel):
indexes: list[int] = Field(description="The indices of the clusters that are being merged.")
relations: list[SubClassRelation] = Field(
description="A list of superclass-subclass relations chosen from the existing two clusters in such a way that they merge."
)
@field_validator("indexes")
@classmethod
def is_binary(cls, v):
if len(v) != 2:
raise ValueError(
f"Binary op error: Invalid number of clusters: {len(v)}. The merge operation requires exactly two clusters."
)
return v
class Bridge(LLMDataModel):
indexes: list[int] = Field(description="The indices of the clusters that are being bridged.")
relations: list[SubClassRelation] = Field(
description="A list of new superclass-subclass relations used to bridge the two clusters from the ontology."
)
@field_validator("indexes")
@classmethod
def is_binary(cls, v):
if len(v) != 2:
raise ValueError(
f"Binary op error: Invalid number of clusters: {len(v)}. The merge operation requires exactly two clusters."
)
return v
class Prune(LLMDataModel):
indexes: list[int] = Field(description="The indices of the clusters that are being pruned.")
classes: list[str] = Field(description="A list of classes that are being pruned from the ontology.")
@field_validator("indexes")
@classmethod
def is_unary(cls, v):
if len(v) > 1:
raise ValueError(
f"Unary op error: Invalid number of clusters: {len(v)}. The prune operation requires exactly one cluster."
)
return v
class Operation(LLMDataModel):
type: Merge | Bridge | Prune = Field(description="The type of operation to perform.")
Thank you to everyone for participating in the discussion and for your overall support! As I said, I didn't expect this. I'm always just an email or tweet away, so you know how to reach me. It was great talking to you all!
How did you sort out mapping python constructs to their semantic equivalents?
I hope you keep at this, you may be in the right place at the right time.
It's getting to the point where some of the LLMs are immediately just giving me answers in Python, which is a strong indication of what the future will look like with Agents.
Very nice, bookmarked for later. Interestingly enough, we share the same timeline. ~2yo is when a lot of interesting work spawned as many started to tinker.
We've been working on some exciting things with SymbolicAI and here a few things which might interest the HN community.
Two years ago, we built a benchmark to evaluate multistep reasoning, tool use, and logical capabilities in language models. It includes a quality measure to assess performance and is built on a plugin system we developed for SymbolicAI.
We've also implemented some interesting concepts in our framework:
- C#-style Extension Methods in Python: Using GlobalSymbolPrimitive to extend functionalities.
It can (in theory) do very similar things, where natural-language input is a first class citizen of the language and can operate on other objects. The whole thing came out almost a decade before LLMs, I'm surprised that they haven't revamped it to make it really shine.
Hey, I think I am not using this framework to its full potential, but I have been using it as my main framework for 2 years now. Advantages for me are, that I can easily develop integrations myself. I like the concept of contracts, that I can first make a detailed output model using their LLMDataModel class (an extension of pydantic), I can validate every field both syntactically and semantically using their semantic operations and raise exceptions in case of failures with meaningful messages (and those will be considered via the LLM for retries). So I like this clean separation of actual business logic and all things validation, hallucinations checking and I don’t have to worry about retries and error handling, because that is done by framework as long as I provide a good output data model. It makes my code much more clean and secure and it also makes it easier for me to analyse by just looking at the output model how secure it is. Also I liked that I can basically inject any python logic,be multimodal without overcomplicating everything with DAGs etc. I like having low level control of what is going on. Also once I started out building my main issue with most frameworks was speed. I wanted to create complex multistep workflows using LLMs and still be very fast and I can’t really stand when my code looks messy. I had a data set of 50 million news and a very small compute budget. After some experiments, decided to go with symbolicai, because it was easier for me to speed it up with batching my agent workflows.
I'd argue it's all of them. Contracts simply make better agents. I believe it also gives a very nice bias on how to talk about agents -- as apps obeying contracts. If you find time, please read this blog post; it gives the underlying motivation for using contracts in agent design: https://futurisold.github.io/2025-03-01-dbc/
This is the voodoo that excites me.
Examples I found interesting:
Semantic map lambdas
comparison parameterized by context bitwise ops `interpret()` seems powerful.OP, what inspired you to make this? Where are you applying it? What has been your favorite use case so far?
You might enjoy Lotus: https://github.com/lotus-data/lotus
It takes all the core relational operators and makes an easy semantic version of each as a python dataframe library extension . Each call ends up being a 'model' point in case you also want to do fancier things later like more learning based approaches. Afaict, snowflake and friends are moving in this direction for their cloud SQLs as well.
We ended up doing something similar for louie.ai , where you use AI notebooks/dashboards/APIs (ex: MCP) to talk to your data (splunk, databricks, graph db, whatever), and it'll figure out symbolic + semantic operators based on the context. Super helpful in practice.
My 80% case here is:
- semantic map: "get all the alerts from splunk index xyz, add a column flagging anything suspicious and another explaining why" <--- generates an enriched dataframe
- semantic map => semantic reduce: "... then summarize what you found" <--- then tells you about it in natural text
Why is carrot the vegetablefication of apple?
I think it's interpreting the command as "replace each fruit with a vegetable", and it might intuit "make the resulting vegetables unique from one another" but otherwise it's not trying to find the "most similar" vegetable to every fruit or anything like that.
This is the correct view. Since the instruction was ambiguous, the LLM did its best to satisfy it -- and it did.
Are you asking for the root cause?
Also if you run it twice, is it gonna be a carrot again?
It's subjected to randomness. But you're ultimately in control of the LLMs's hyperparams -- temperature, top_p, and seed -- so, you get deterministic outputs if that's what you need. However, there are downsides to this kind of LLM deterministic tweaks because of the inherent autoregressive nature of the LLM.
For instance, with temperature 1 there *could be* a path that satisfies your instruction which otherwise gets missed. There's interesting work here at the intersection of generative grammars and LLMs, where you can cast the problem as an FSM/PA automaton such that you only sample from that grammar with the LLM (you use something like logits_bias to turn off unwanted tokens and keep only those that define the grammar). You can define grammars with libs like lark or parsimonious, and this was how people solved JSON format with LLMs -- JSON is a formal grammar.
Contracts alleviate some of this through post validation, *as long as* you find a way to semantically encode your deterministic constraint.
Since these seem like short prompts, you can send as context data that was correct on past prompts
You can create a test suite for your code that will compile correct results according to another prompt or dictionary verification
That's gonna be a very, very, long answer. What's funny is that not much changed since 2022 (eoy) when the projected started; the models just got better, but we had a good chunk of the primitives since gpt-3.
What's more recent is the DbC contribution which I think is unique. It literally solved anything agent related I've thrown at it -- especially because I can chain contracts together and the guardrails propagate nicely.
I've built most the custom tools myself. For instance, not only perplexity was rendered useless by openai's web search, but openai's web search itself is not good enough for what you can customize yourself. To this end, I've built my own deep research agent. Here's a thread with some results from the first day it was working: https://x.com/futurisold/status/1931751644233945216
I'm also running a company, and we've built an e2e document generation just from contracts (3 contracts chained together in this case). Here's an output (sorry about the PDF rendering, that's not what we serve, it's just what I've quickly hacked for local dev): https://drive.google.com/file/d/1Va7ALq_N-fTYeumKhH4jSxsTrWD...
This was the input:
---
Prompt:
> I want the files to be analyzed and I am interested in finding patterns; feel free to make suggestions as well. I want to understand how different providers use their system prompts, therefore things like: what kind of tags do they use – are they XML, markdown, etc, are they prone toward sycophancy or trying to manipulate the user, are they using tools and if so how, etc. I want the tech report to deconstruct and synthesize and compare the information, find interesting patterns that would be hard to spot.
Generated instructions:
(a) Query: Conduct a comparative analysis of system prompts across major AI providers (OpenAI, Google, Anthropic, xAI, etc.) to identify structural patterns, linguistic frameworks, and operational constraints that shape AI behavior and responses.
(b) Specific Questions:
1. What syntactic structures and formatting conventions (XML, markdown, JSON, etc.) are employed across different AI system prompts, and how do these technical choices reflect different approaches to model instruction?
2. To what extent do system prompts encode instructions for deference, agreeability, or user manipulation, and how do these psychological frameworks vary between commercial and research-focused models?
3. How do AI providers implement and constrain tool usage in their system prompts, and what patterns emerge in permission structures, capability boundaries, and function calling conventions?
4. What ethical guardrails and content moderation approaches appear consistently across system prompts, and how do implementation details reveal different risk tolerance levels between major AI labs?
5. What unique architectural elements in specific providers' system prompts reveal distinctive engineering approaches to model alignment, and how might these design choices influence downstream user experiences?
---
Contracts were introduced in March in this post: https://futurisold.github.io/2025-03-01-dbc/
They evolved a lot since then, but the foundation and motivation didn't change.
One last comment here on contracts; an excerpt from the linked post I think it's extremely relevant for LLMs, maybe it triggers an interesting discussion here:
"The scope of contracts extends beyond basic validation. One key observation is that a contract is considered fulfilled if both the LLM’s input and output are successfully validated against their specifications. This leads to a deep implication: if two different agents satisfy the same contract, they are functionally equivalent, at least with respect to that specific contract.
This concept of functional equivalence through contracts opens up promising opportunities. In principle, you could replace one LLM with another, or even substitute an LLM with a rule-based system, and as long as both satisfy the same contract, your application should continue functioning correctly. This creates a level of abstraction that shields higher-level components from the implementation details of underlying models."
Anyone interested in this from a history / semiotics / language-theory perspective should look into the triad concepts of:
Sign (Signum) - The thing which points Locus - The thing being pointed to Sense (Sensus) - The effect/sense in the interpreter
Also known by: Representation/Object/Interpretation, Symbol/Referent/Thought, Signal/Data/User, Symbol/State/Update. Same pattern has been independently identified many many times through history, always ending up with the triplet, renamed many many times.
What you're describing above is the "Locus" essential object being pointed to, fulfilled by different contracts/LLMs/systems but the same essential thing always being eluded to. There's an elegant stability to it from a systems design pov. It makes strong sense to build around those as the indexes/keys being pointed towards, and then various implementations (Signs) attempting to achieve them. I'm building a similar system atm.
Thanks for bringing this up. I'm fairly familiar with Peirce's triadic semiotics and Montague's semantics, and they show up in some of my notes. I haven't turned those sketches into anything applied yet, but the design space feels *huge* and quite promising intuitively.
Agreed. This is a very interesting discussion! Thanks for bringing it to light.
Have you read Escher, Bach, Gödel: the Eternal Golden Braid?
Btw, besides the prompt, the other input to the technical report (the gdrive link) was this repo: https://github.com/elder-plinius/CL4R1T4S/tree/main
Very cool. Being able to use semantic (as opposed to syntactic) operators like `==`, `+`, etc. feels like fertilizer for some novel ideas. Sort of like when word embeddings first came out and there was a loose concept algebra introduced with it ("King - Man + Woman = Queen").
That said the neuro + symbolic integration here is, like most systems, pretty shallow/firewalled (taxonomically, Type 3 / Neuro;Symbolic — https://harshakokel.com/posts/neurosymbolic-systems). I think the real magic is going to come when we start heading toward a much more fundamental integration. We're actually working on this at my company (https://onton.com). How do we create a post-LLM system that: 1) features an integrated representation (neither purely symbolic nor dense floating point matrix); 2) can learn incrementally from small amounts of noisy data, without being subject to catastrophic forgetting; 3) can perform mathematical and other symbolic operations with bulletproof reliability; and 4) is hallucination-free?
The cobbling together of existing systems hot-glue style is certainly useful, but I think a unified architecture is going to change everything.
Probably linking the paper and examples notebook here makes sense as they are pretty explanatory:
https://github.com/ExtensityAI/symbolicai/blob/main/examples...
https://arxiv.org/pdf/2402.00854
Wanted to do just that, thank you
FYI, there’s a correctness issue in the part about correctness contracts: valid_opts = ['A', 'B', 'C'] if v not in valid_sizes:
valid_sizes is undefined
Oh, thank you; refactoring artifacts. Fixed now.
My only gripe with this is that there's alread a term in computer science for this concept of "design by contract".
It's called a type system.
What you need for this is a more expressive type system.
What bums me out a bit - "Symbolic AI" is already well defined: https://en.m.wikipedia.org/wiki/Symbolic_artificial_intellig...
We hear you. We might end up renaming it. In the paper we have a footnote about the name choice -- it's meant to credit the foundational work of Newell and Simon that inspired this project.
I'd appreciate it! It's cool and I wish you success. Just hope that when someone says "We're using Symbolic AI" a year from now, it won't be even more ambiguous than today :D
I didn't expect this -- I was supposed to be sleeping now, but I guess I'll chat with whoever jumps in! Good thing I've got some white nights experience.
One question, OP, how does cost for this work? Do you pay the LLM inference cost (quite literally if using an external API) every time you run a line that involves natural language computation? E.g. what happens if you call a "symbolic" function in a loop.
Yes, that's correct. If using say openai, then every semantic ops are API calls to openai. If you're hosting a local LLM via llama.cpp, then obviously there's no inference cost other than that of hosting the model.
This will need a cache of some sort
this works like functional programming where every symbol is a pure value and operations compose into clean, traceable flows. when you hit an ambiguous step, the model steps in. just like IO in FP, the generative call is treated as a scoped side effect. this can engage your reasoning graph stays deterministic by default and only defers to the model when needed. crazy demo though, love it
Yes, pretty much. We wanted it be functional from the start. Even low level, everything's functional (it's even called functional.py/core.py). We're using decorators everywhere. This helped a lot with refactoring, extending the framework, containing bugs, etc.
Since code is generated by LLM these days, how does specific syntactic constructs like a Symbol which essentially carries the context and can be manipulated with python operators help when compared to a normal python code generated by LLM with all the checks and balances instructed by a human? For example, I can write in this syntax to convert all fruits to vegetables or I can simply prompt an LLM to construct a program that takes a list of fruits and calls a LLM in the background to return the vegetables equivalent. I am trying to understand the difference.
Hallucination obstruction, I'd imagine. When you have an LLM create a formal system, it can be verified way easier than a general purpose one
Yes. That seems to be the case. While it may not be saving any time compared to generating general python code vs. specific symbolic code, the real value could be that it has an engine to enforce the contract on LLM responses with the library or even do the calls to the LLM as a common piece of code making it less error prone and bringing consistency in the interactions with the LLM.
Nice! But have you considered a Neurosymbolic AI that can Evolve?
https://deepwiki.com/dubprime/mythral/3.2-genome-system
Or feel Emotion? https://deepwiki.com/search/how-do-emotives-work_193cb616-54...
Have you read Marvin Minsky’s Society of Mind?
Shortly, yes to all. We actually had an experiment going from theory of mind to emotion, but it's hanging right now since I feel the models aren't quite there yet and it yields diminish returns relative to effort. But could easily be revived. Minsky isn't my fav though, I'm leaning more toward Newell/Simon and friends from that generation.
@futurisold, would love to collaborate with your team on running experiments. We have $300k of GPU credits to burn in the next 2 months.
There’s only so many cat videos my Agentic AI Army can create:
https://youtu.be/upVY1QioDeY?si=H566-_JIm7FmW4u0
That's very kind of you, thank you. Let's sync and see if we can align on something. You can find me on X, or shoot me an email at leo@extensity.ai
But is it also explainable or a magic black box?
I spent some time toying around with LLM-guided "symbolic regression", basically having an LLM review documents in order to come up with primitives (aka operators) that could be fed into github.com/MilesCranmer/PySR
I didn't get very far because I had difficulty piping it all together, but with something like this I might give it another go. Cool stuff.
Oh, definitely. I recommend you go for contracts. I've used something similar for a contract that iteratively "stitched together" a broken ontology graph. Here's some of the data models for inspiration -- you could have something similar for your ops, and write the contract to solve for one op, then apply the op, etc.
---
---Thank you to everyone for participating in the discussion and for your overall support! As I said, I didn't expect this. I'm always just an email or tweet away, so you know how to reach me. It was great talking to you all!
How did you sort out mapping python constructs to their semantic equivalents?
I hope you keep at this, you may be in the right place at the right time.
It's getting to the point where some of the LLMs are immediately just giving me answers in Python, which is a strong indication of what the future will look like with Agents.
I'm struggling to understand the question. I'll revisit this when I wake up since it's quite late here.
I love the symbol LLM first approaches.
I built a version of this a few years ago as a LISP
https://github.com/nbardy/SynesthesiaLisp
Very nice, bookmarked for later. Interestingly enough, we share the same timeline. ~2yo is when a lot of interesting work spawned as many started to tinker.
I already created a neurosymbolic code generator. Based on bench marks it received a score of 99 percent.
I did it for python and will come out with other languages afterwards
Did you use symbolicai for that?
We've been working on some exciting things with SymbolicAI and here a few things which might interest the HN community.
Two years ago, we built a benchmark to evaluate multistep reasoning, tool use, and logical capabilities in language models. It includes a quality measure to assess performance and is built on a plugin system we developed for SymbolicAI.
- Benchmark & Plugin System: https://github.com/ExtensityAI/benchmark
- Example Eval: https://github.com/ExtensityAI/benchmark/blob/main/src/evals...
We've also implemented some interesting concepts in our framework: - C#-style Extension Methods in Python: Using GlobalSymbolPrimitive to extend functionalities.
- Symbolic <> Sub-symbolic Conversion: And using this for quality metrics, like a reward signal from the path integral of multistep generations. - https://github.com/ExtensityAI/benchmark/blob/main/src/func....For fun, we integrated LLM-based tools into a customizable shell. Check out the Rick & Morty-styled rickshell:
- RickShell: https://github.com/ExtensityAI/rickshell
We were also among the first to generate a full research paper from a single prompt and continue to push the boundaries of AI-generated research:
- End-to-End Paper Generation (Examples): https://drive.google.com/drive/folders/1vUg2Y7TgZRRiaPzC83pQ...
- Recent AI Research Generation:
Finally, for those interested in building similar services, we've had an open-source, MCP-like API endpoint service available for over a year:- SymbolicAI API: https://github.com/ExtensityAI/symbolicai/blob/main/symai/en...
great job! it reminds me genaiscript. https://microsoft.github.io/genaiscript/
// read files
const file = await workspace.readText("data.txt");
// include the file
content in the prompt in a context-friendly way def("DATA", file);
// the task
$`Analyze DATA and extract data in JSON in data.json.`;
Thank you! I'm not familiar with that project, will take a look
Some of this seems a bit related to Wolfram Mathematica's natural language capabilities.
https://reference.wolfram.com/language/guide/FreeFormAndExte...
It can (in theory) do very similar things, where natural-language input is a first class citizen of the language and can operate on other objects. The whole thing came out almost a decade before LLMs, I'm surprised that they haven't revamped it to make it really shine.
> I'm surprised that they haven't revamped it
No worries! I can't find it right now, but Wolfram had a stream (or short?) where he introduced "Function". We liked it so much we implemented it after one day. Usage: https://github.com/ExtensityAI/symbolicai/blob/main/tests/en...
Wolfram's also too busy running his TOE exps to focus on LLMs (quite sadly if you ask me).
not to be confused with symbolica.ai
+
what are the implicaitons and actual real world application of this? better agents? more accurate, debuggable LLM answers?
Hey, I think I am not using this framework to its full potential, but I have been using it as my main framework for 2 years now. Advantages for me are, that I can easily develop integrations myself. I like the concept of contracts, that I can first make a detailed output model using their LLMDataModel class (an extension of pydantic), I can validate every field both syntactically and semantically using their semantic operations and raise exceptions in case of failures with meaningful messages (and those will be considered via the LLM for retries). So I like this clean separation of actual business logic and all things validation, hallucinations checking and I don’t have to worry about retries and error handling, because that is done by framework as long as I provide a good output data model. It makes my code much more clean and secure and it also makes it easier for me to analyse by just looking at the output model how secure it is. Also I liked that I can basically inject any python logic,be multimodal without overcomplicating everything with DAGs etc. I like having low level control of what is going on. Also once I started out building my main issue with most frameworks was speed. I wanted to create complex multistep workflows using LLMs and still be very fast and I can’t really stand when my code looks messy. I had a data set of 50 million news and a very small compute budget. After some experiments, decided to go with symbolicai, because it was easier for me to speed it up with batching my agent workflows.
I'd argue it's all of them. Contracts simply make better agents. I believe it also gives a very nice bias on how to talk about agents -- as apps obeying contracts. If you find time, please read this blog post; it gives the underlying motivation for using contracts in agent design: https://futurisold.github.io/2025-03-01-dbc/