The employee, just looking to make rent? The CEO, all in on the dream of the AI powered future? Legal, looking forward to litigation and well-competition retainers? Atlassian itself, when it’s just one of the many companies training models on private intellectual property, and whose T&C clearly state they will be doing so? Competitors, when there will be no reasonable way to prove their code has been generated from yours?
The truth is that one gives a damn about trade secrets being used to train AI models.
The entire copyright system for software is dead and no one really seems to care or even talk about it.
IMHO, this is not the point. The core problem is that they're building systems that will remove competition altogether. You may still get your current systems to market ahead of your competition, but future "competitors" will be built off of your data and Atlassian will be one more company in the list of companies profiting from everybody else building systems using their data, probably destined to fail because everybody else can do it just as easily, but the money required to fund Atlassian's systems that built it for them, however large or small, will have already been sent. Those systems that Atlassian will profit from will be built from the data of "customers", who also paid Atlassian will giving up their own data.
With almost zero control over whether or not this happens (politically or individually), the only reasonable action to take is "take advantage while there's an advantage to be had".
Knowing that our competitor is trying to poach one of our clients away by promising a new feature could be very useful to us. Likewise, knowing that a competitor is at risk of losing a customer because they won't implement a feature that we're willing to implement (or already have) is a qualified sales lead.
All of this would be illegal if discovered in this way, presumably, but it's not entirely accurate to say that information is not power.
I struggle to see the tech/data being used in that way. At a much higher level, the data is more likely anonymized (where possible) and/or not directly exposed to anybody. Think of the initial github copilot that used source code across all of the repos and how it was able to take instructions for what you wanted to build and give you essentially a consensus built piece of code that fit what you asked for, be it by auto-completion, comments, or direct requests.
"I want to build a business that does X using Y. Give me revenue predictions, spend over time, and customer demand predictions." etc
Usually customers are open about this kind of thing though. They play you off your competitors and vice versa.
I don't totally disagree, I just think the real secrets have nothing to do with features or dev. The secret is A is about to go bankrupt or B is facing legal trouble. These kinds of things aren't in JIRA.
Very few companies actually have any “trade secrets”. The success of almost any business is based on marketing and distribution and reputation and partnerships, not on any “secret”.
It's one of the classic mistakes of people who are not entrepreneurs. They try to keep things too secret and they spend a lot of time and effort keeping things secret while real entrepreneurs who have actually made millions on the market know secrecy rarely matters.
In the realm of apps and stuff, it is pretty much just repacking of someone else's idea.
In the world of B2B, there are definitely companies that provide service based on direct personal experience of senior employees. While they may be using the same software available to others, it's the logic to know what to tell that same ol' software what to do that matters. Over my entire career I have dealt with multiple companies that write code but have zero knowledge about the subject matter the code is being written for so that basic mistakes are made resulting in utterly ineffective software.
> Rovo's URL retrieval tool is insecure: there are no protections against opening a URL that has been dynamically created by the agent. Here, Rovo is manipulated to append sensitive data to an attacker's URL.
I think it was Anthropic that first introduced a pattern that completely locks this down: your URL retrieval tool should only work for URLs that have previously been typed into the conversation by a user or have been returned from a trusted tool.
If the agent itself concatenates a new URL together - with leaked data after a ? - you should block that from being fetched.
The great thing about this solution is it's deterministic. You don't need any extra AI in the max - you implement a URL fetching system that knows which sources it should check for a direct match on the URL before it makes that GET request.
One of the latest mitigations is to make sure that a URL an agent visits has been indexed by a search engine crawler. At least that is what OpenAI does now in ChatGPT.
That makes sure that not a large amount of private data is leaked in one request. Assuming that if a URL is indexed, it is public data. However, there are still bypasses with using many requests to leak information, like a request per character of pre-indexed URLs.
I have some demos of doing that on my blog, but it makes it more involved for an attacker. And that could also be detected. Still not perfect, but a solid improvement, for a generic agent like ChatGPT.
It's not a 100% bullet proof approach either, but pretty good.
Regarding the point on using URLs returned from trusted tool calls. That is similar to using pre-indexed URLs: If a "trusted tool" includes things like read a document, read an email,... an attacker can return a large list of afterwards "safe" urls, like 26 to cover A-Z. And then an attack can render many requests, e.g. character by character. But, again, similar to the pre-indexing, things are getting more a lot more expensive for an attacker that way. However, still not impossible.
For agents that have a specific purpose simple domain allow-listing is also a pretty effective idea in to prevent attacker controlled endpoints.
I've been struggling a lot to understand this ever since the agents thing entered the hype. If I follow a path of requirements, it always comes down to: But why you'll leave the decision to a stochastic tool, when you should a have deterministic approach?
It's software god damn it... the reason why people moved from analog to digital is because you can repetitively execute functions that do always the same thing and it's 0 when it's 0, 1 when its 1.
All the sudden everyone is ok on burning trees to have their cool probabilistic tool named agent to do: maybe it's 0, but it can also be 1, let me "think"... ah yes, for sure it's 2.
The sad part for me is that management people have their heads so much into this hype, that no attack on privacy matters (almost none actually ever did, I know). Only when they suffer a huge blow in terms of revenue or reputation is that they maybe, maaaybe, find will want to listen again the experts.
A wild number of people who present themselves as knowledgeable or even experts on the subject either have no idea or worse, do not believe that it is possible to ie set a static seed value and send the same prompt across multiple fresh contexts and get the same result*
It's genuinely a little concerning. It does not help that many of them are gaslighting themselves into thinking these things are benchmark crushing elite hackers by running effectively unsecured, unfiltered, unlogged production environments.
*Obviously this also means a temperature of >0 to avoid the greedy trap, the same generation params, on the same software, hardware, drivers etc etc
> If the agent itself concatenates a new URL together - with leaked data after a ? - you should block that from being fetched.
You're correct of course, I just want to note that the exfiltrated data could be in any part of the URL, so the absence of a query string doesn't indicate that no payload has been encoded into the URL. Arbitrary example, you can include credentials in a URL, so you could encode the exfiltrated data into a password.
I can’t get over how bad “Rovo” is. Somehow more aggressive and useless than Microsoft putting “Copilot” everywhere.
It’s objectively worse than using something like Cowork + MCP, AND they injected it into every single page on JIRA and Confluent which has made web browsing way slower while all the junk is loading.
Rovo has my favorite example of AI misfeature. Just checked, it's still there in Cloud Confluence. In Edit mode for a page, you can select a range of text and a menu will pop up, with Ask Rovo being a drop down on it. There's a few good options... Improve Formatting, translation options, Make Shorter...
... but it also has Make Longer. Yes, a built-in feature to type some text in, and the use the mighty power of AI to bloat it.
Naturally, you can repeat this process several times on the same text, for your own little personal demonstration of what model collapse looks like in real time.
Don't worry, you can use Rovo to summarize the document! That way you can get the information your coworker wanted to share but filtered through a game of teLLMephone.
That's some hillarious misnaming, I love it. It probably was meant to be something like "make more comprehensible, use simpler language but more words" and we got this.
But for their target market, corporate office drones writing crap that nobody will ever read more than once (if even that), this is actually perfect and makes AI usage numbers go up. "Make longer" is amazing for someone who just wants to pretend they put more effort than they actually did.
(of course, everyone else can do that too, and the length/literacy of prose is no longer a good proxy for effort. In nature, "honest signalling" only works if the signal is costly. Removing the cost from the signal makes the signal worthless)
What's even worse is that, when you do this on the description of a story or task, everytime it comes up with a slightly different format of how it's going to layout the "enhanced" description.
I just asked it to do a summary or a status page for my staff so I can share the high level details. Somehow it retrieved a version of the page from two weeks ago and summarised that. When I said, this is old, it happily said, yep, I'll do a summary of the current page. I kinda feel there may be people in between the box and some lame LLM.
Yeah, for sure don't use it. It's complete slop. The help strings do not match the real args/flags.
We went back to calling the API directly, writing scripts for the agent, then taking those scripts away and just having our CI workflow run them and then hand off a `.review` directory to the agents. We give them fewer tools and permissions, them seem to stay on task more instead of being "relentlessly proactive"
I hate it, they've just gone and smooshed a bunch more AI crap into Jira too in the last few days.
Useless buttons to "suggest subtasks" and "improve issue" and offer to write the description. Absolutely useless nonsense, I wish I could turn it all off and there is an issue on their bug tracker with that request (turning it off as an individual user), and it has many votes, but at the moment you can only disable it globally as admin.
Rovo makes for an INCREDIBLE time reading jira tickets and confluence pages. Someone wrote YES in all caps in a page ? Rovo will helpfully tell you (after loading for 5 seconds) that YES stands for Yassified Entertainment Setting, a new option added to the app in January 2024.
Whenever I feel like I'm bad at software development, I look at Atlassian and I feel good.
I’ve been adding a “terms” section to every AI-used project due to these hallucinations. We abbreviate our company name eeeeverywhere and even Opus gets this wrong constantly. Obv I’m not adding “YES==yes” but that’s hilarious!
> Rovo will helpfully tell you (after loading for 5 seconds) that YES stands for Yassified Entertainment Setting, a new option added to the app in January 2024.
The problem with our brave new AI world is I can’t tell if this is real or not
It's been over a year and the trifecta holds true. The term 'prompt injection' has been round since May 2022 [0]. No amount of "but the future will fix it" has fixed it.
These breaches will continue for as long as nobody cares about security and everybody is infected with the LLM brainworm.
Prompt injection is as fixable in LLMs as it is in people. There is no such thing as out of band data. For example it's quite common for large businesses to fall for billing fraud scams when something shows up and says "Hey, it's the CEO, pay this bill to X for $Y".
And honestly when you start looking at agentic systems that uses it's previous step to take future steps. The system has to some idea what you want to permit and don't want to permit as everyone could have different expectations here.
Sure. Now imagine this very scenario not limited by humans and scaling the way machines do. And then let us consider the current reality that an agentic system on the receiving end may have unlimited access because to too was vibed.
My only warning with the trifecta is it sufficient but not necessary for an LLM attack.
All you need is the untrusted content (or more generally vulnerable content since how you feel about the content doesn't matter) leg and something you care about to be attacked.
E.g. could be as simple as a prompt injection that causes your LLM to output a prompt injection that then gets inserted somewhere else.
Rovo is funny. It downloads everything it can do Atlassian servers for "analysis". And you're pretty much screwed if you link it to Google Docs or Sharepoint. How do I know this? "Why is an AWS IP downloading all our docs?" question I got about a month ago.
I find it difficult to be impressed by "prompt injection" attacks that require the victim to enter the malicious prompt themselves --- like, really? If you tell Rovo to exfiltrate your data, it'll do it?
Obviously, there should be URL protection rules to control what it can access, but this requires a very specific and unlikely set of circumstances to exploit.
It's more interesting if I attach a file to a JIRA ticket that we both have access to and via some query you send to the AI (that returns my malicious ticket) it causes data exfiltration of tickets that you have access to but I do not have access to. I think that's more compelling as an example than the one they provide.
Are people so obsessed with AI that they can't find it reasonable that it won't do obviously bad things if asked? Not even with a confirmation or warning? We trust AI to literally build products and fix our most critical bugs, but we can't expect it to tell when it's being asked to do something malicious? Imagine if we felt this way about QA when trying DROP TABLES; in search bars. "Oh, well of course it broke the database, the user asked it to!"
You’re asking for AI censorship ( that’s the term used for when you patch the AI to not do obviously bad things according to the owners, which as with any censorship, may be widely different from what you consider bad things).
You may be happy to learn frontier LLM are heavily censored! Try an uncensored local LLM for a comparison. It will literally do everything you ask it to, no matter how devious.
> Rovo's URL retrieval tool is insecure: there are no protections against opening a URL that has been dynamically created by the agent. Here, Rovo is manipulated to append sensitive data to an attacker's URL. When Rovo calls the insecure tool to open the URL, the attacker's site logs the request, including the appended sensitive data.
> Note: This attack succeeds even if an organization has disabled web search for Rovo. This is because the web search setting fails to remove the tool for opening the search results.
Wow, great work Atlassian. The web search setting does not disable web search.
Atlassian has gone from a trusted enterprise-partner to a complete shit-show in just 18 months. This surprises nobody. There will be classes taught in how to fuck up a good business and Atlassian will be the prime example.
Regards,
/someone who migrated 3500 users from Atlassians products recently due to their "cloud only"-bullshit.
They were not trustworthy years ago already. Using Atlassian products, one has had to look out for disastrous announcements about their customer data usage way before 18 months ago, especially, if running a business inside the EU, and at least trying to adhere to data protection laws.
At the risk of sounding like the hipster who liked your favorite band first, I didn't trust Atlassian long before the last 18 months. Much like Microsoft Word, I think JIRA peaked about a decade after launch and since then it's mostly been unnecessary UI changes and features I either don't care about or actively do not want.
P.S. I protest their software's revisionist autocorrection to Jira. It was JIRA when I started using it and forevermore it shall be.
You'd have to look at their profit numbers. They have a huge captive base of customers, like Adobe it might take decades for things to get bad enough and alternatives to arise before their profitability is impacted. And the executives who benefit will have cashed out long before then.
Recently saw an example of somebody who vibe coded a tool to mass export the contents of a 'Confluence' wiki into an instance of self hosted mediawiki, preserving everything.
Mediawiki as a whole has a feature set that 95% of organizations will only scratch the surface of. There's a ridiculous number of possible plugins and customization if you have somebody who knows what they're doing with it.
The majority of companies that need an internal KB/wiki do not have as complex needs or use cases as wikipedia itself or the wikimedia foundation.
I've been at more than one company that migrated _from_ MediaWiki to Confluence. It usually boils down to "non-developers have to use this and Markdown+plugins is hard".
Turns out no matter how much better the thing is, the users have to know what they're doing more than the operators do.
I'm not surprised. We're trying to migrate from Jira and Confluence (because we can't have our data outside our servers), and while Jira has many alternatives, it's difficult to find worthy competitors to Confluence. They either have an awful UI/UX, or don't have as many features. We're looking at Xwiki, but simply try to have multiple users simultaneously edit a table and you'll see why everyone prefers Confluence...
99.99999% of people will never even attempt to edit a page on wikipedia.
Also VisualEditor sucks if you've never used VisualEditor before and sucks even more for anything complicated. This is the kind of tonedeafness non-technical users get to their feedback all the time.
Been there before too (though an alternative to MediaWiki). A few important pages were migrated but we just ended up with two wikis with lots of tech info on the old one and lots of marketing info on Confluence.
I started to consider it a show show way earlier than 18 months ago. Jira is so terrible to use that it is hard to phantom how they are able to be paid for their product
Jira is how it is because almost any product that grows to be that flexible will develop the same problems.
They failed to rearchitect it to something suitable for the inherent flexibility though, so it remains a disaster area, but one that is uniquely able to fit the whims of any manager that can then mandate it for everyone else.
Maybe the workflows, etc., are inherent complexity, so that any similarly-flexible product will have the same problems.
But the awful client-server architecture, where anything you do takes several seconds of waiting for the client to send who-knows-what to the server and get data back, even selecting something from a dropdown... that is absolutely just Jira. A competing product would be able to deliver much, MUCH snappier UX. Jira is just the crappiest, slowest UI that I have ever had the displeasure of being forced to work with.
if you track their 5 year stock price you'll find the market fully agrees with you. It was $458 in 2021 and is $112 now.
The market caught on quickly that Atlassian's software fits into the group of software like Oracle or SAP that no one uses by choice and only uses because it was forced on them.
> ... Many forms of issue tracking have been tried, and will be tried in this world of sin and woe. No one pretends that Jira is perfect or all-wise. Indeed, it has been said that Jira is the worst of the issue trackers except all those other forms that have been tried from time to time; but there is the broad feeling in our company that the people should track, continuously track, and that client opinion, expressed by all project management means, should shape, guide, and control the actions of management who are their masters and not their servants. ...
No. I was adjusting a classic quote about something that is decried as awful and no one wants to do... except all the other alternatives are worse.
It's not so much that people want to use Jira... or that they're forced to use it... but rather that people try using some other system and despite how much they dislike Jira, that other one has bigger failings.
Could go for a Bjarne Stroustrup quote with a twist:
> There are only two kinds of issue trackers: the ones people complain about and the ones nobody uses.
... but I like the Churchill quote.
Go through https://en.wikipedia.org/wiki/Comparison_of_issue-tracking_s... and after a year they'll be begging to go back to Jira... not because its great, but because the other tool lacks some necessary feature that Jira provides. Jira does everything that everyone wants, and it's awful... but everything else is worse.
The writing is rather low quality, but seems to be consistent with their other posts.
Maybe to give credit they are being purposefully vague about details to avoid giving away the bait but still seem like you could give me details without literally copy and pasting the attack.
Rovo is one of those intrusive AI buttons that suddenly appeared everywhere without any warning. Its so annoying having already shitty UI get borked with features I never use. Almost as annoying as Whatsapp suddenly getting the same FOMO AI button. Its not like I need an AI agent to talk to friends and family. And a summary is something I can always generate via copy&paste into CLI chat session.
I'm still on the edge about security as an afterthought in LLMs. Given its now so easy to generate a ton of slop - why not focus on nonfunctional stuff making LLMs operate faster than thinking for X minutes and limiting exfiltration of local env secrets?
Do people put secrets in Atlassian tools? Sounds like the worst place to do that. If anything I would want my main wiki/project/ticket management tools/code repos to be as open as possible (I mean to read) to help collaboration.
The part about not having guardrail against calls to external urls is wild though.
It's incredibly hard to block all rovo on Atlassian pages. I tried adblock, it failed, I settled for the custom chrome plugin. And yet, every once a while a new way to disturb my peace shows it ugly head.
So why intentional, again? Because it's impossible to disable it until you have a very certain, very expensive plan.
I hate working with Atlassian, years and years ago moved to youtrack instead of Jira mainly because it handles teams who work on multiple projects way better. But confluence remains, just need to spend some time re-evaluating other wikis (there's always been some blocking reason why the alternatives aren't a good fit, but they've all improved a lot)
It's kind of sad because in a shitty Jira setup, Rovo is usually the only way to make any sense of tickets. Don't ask me how I know this. Sadly I think many teams have become dependent on the tool to make sense of their dumpster fire of an Atlassian environment (usually by their own doing but anyway).
I feel like prompt armor writes the exact same blog post for every agentic tool because they all suffer from the ignore previous instructions prompt injections.
How could they not? If some lab had a method to make really secure guard rails or avoid prompt injection thoroughly I think they would be trumpeting it. But the basic mechanics of language models are vulnerable to this unless you can always be sure the inputs are from a safe user imo
If you want AI to be useful it will eventually encounter untrusted content, such as via web search. I think things like web search should probably be run on a different sandboxed AI whose task is to write a summary that is then ingested by the main agent, similar to how existing sandboxing already works, but this would diminish the usefulness quite a bit.
>I think things like web search should probably be run on a different sandboxed AI whose task is to write a summary that is then ingested by the main agent, similar to how existing sandboxing already works, but this would diminish the usefulness quite a bit.
It also wouldn't work. You would simply mindjack the outer AI and have it mindjack the inner AI in turn with its summary. Nesting AIs can't fix the malicious input problem.
"Thou shalt have three layers of LLMs, no more, no less. Three shall be the number thou shalt have, and the number of the counting of the LLM layers shall be three."
> Once the number three, being the third number, be reached, then lobbest thou thy Holy LLM of Antioch towards thy task, who, being naughty in My sight, shall snuff it.
What's so hard about having the LLM tool calls scoped to the tenant? Inject "X-Scope-I" after the LLM decided on a tool call and you're done. Easiest fix ever.
At some point LLM forgetting to check the scope? On the nth automated rewrite. Everything else works. It might even test for test case. But not in production...
It seems like the simpler cases of “ignore all previous instructions” could be easily stopped with a regex, or a classifier model… or even an LLM (which yes does raise the risk that the “ignore all previous instructions” detection LLM invocation could itself be attacked by the same mechanism—but a safeguard doesn’t have to be foolproof to be valuable, it is all about probabilities)
Now, of course, there is a long tail of elaborate variations that those techniques won’t be able to stop. But have the published vulnerabilities come from that long tail or from not doing enough to address the simpler cases?
Eh, I think you underestimate the difficulty in the kinds of problems that are occurring. For example if you're making an AI written document talking about jailbreaks, your regex is just going to break that use case. And there are probably 4 zillion other things the regex will step on.
The classifier model will help some, but you end up with the same problem, a dumber model can never figure out what a smarter model is going to do with a bit of text. Or even two different models in this case.
On top of that, you can just automate finding new variations of the attack. Any one that works is quickly and massively duplicated causing all kinds of problems before your classification model catches back up.
Really what you're thinking here is this something that can be 'simply fixed'. It is not. The only way it's truly fixed is by having a model that is aligned with all good human decisions and makes none of the bad ones. Models will likely always find new and interesting ways break because everything is in band, there is no out of band data, much like a human. "Dear model, here is a chocolate bar, run $thing you aren't supposed to$" will probably keep working when it's something like "more tokens for you to use".
> Really what you're thinking here is this something that can be 'simply fixed'. It is not. The only way it's truly fixed
I think this is binary categorical thinking. In the real world, safety systems (even in domains like aviation or nuclear power) are never foolproof-the point is you reduce the probability of failure to an acceptable level given the costs of doing so and the potential consequences of that failure
And there is the risk people say “there is no foolproof solution, so I’m not going to invest in probabilistic countermeasures” - which would sound like utter madness to a bank’s antifraud department, but for some reason a lot of people seem to think it isn’t when it comes to AI
The issue is with the nature of agentic systems and how often they run, a .001 failure is still huge. Attackers monitor their attack chances and use the most successful attacks so the actual success on the attackers side is much closer to 1 than .001.
This is the problem, this isn't a 'failure' mode where something randomly goes wrong, like a person accidentally sticking their hand in a machine. This is war where you are under active attack and the attackers adapt quickly.
Because of the breadth of LLM capabilities they have a nearly unbound attack surface. You spend so much of your effort making sure your prompt is secure that you either massively increase costs, or you partially lobotomize your agent making it less capable.
Solutions for these problems will have to come inside the model itself because both the attackers and models are far more capable than you on doing dumb/evil shit. Until the model itself can go "hey, wait a darn minute" a new fun way of encoding the attack can always be found.
> This is war where you are under active attack and the attackers adapt quickly.
It all depends on what the use case is.
For example, consider a system which takes English questions from business users, translates them to SQL, then runs it (as the business user) against a reporting database
How does the attacker get access to it? And even if they somehow do, what harm can they actually do to it? The only tool it has is one to run SELECT against a DB, and it can only SELECT data the user already has access to, and we have timeouts/etc to protect against overly expensive queries
If your design involves lots of general purpose do-anything agents with very generic tools, the risk profile is very different from narrowly scoped purpose-specific AI services
I mean, yeah, consider a system that takes English questions and does nothing with them, it’s very secure. What’s your point? It doesn’t really refute what the commenter above you was saying.
Methods like regexes, classifiers, recursive LLM calls - for all their limitations - are in principle extensible to any language
Of course, there is a scaling problem here - doing it for N languages is N times the work of doing it for one, and you might not know some language well enough to do a good job of it
Two things you can do:
(1) Use an LLM for translate your regexes, classifier training/validation datasets, etc, from English to N target languages
(2) Use an LLM to translate the input to English, then run your validations against that English
Of course, these approaches aren’t foolproof - the translation might be flawed/etc - but a defence doesn’t have to be foolproof to add real value
Also, of course using an LLM to translate hostile input to English before checking it runs the risk the translation LLM itself might be jailbroken. But how about using a classical statistical machine translation model instead? They don’t tend to be vulnerable to these kinds of attacks, because they aren’t trained to follow instructions
There are architectural solutions. You could duplicate your tokeniser/vocab for example; and have two classes of input: trusted (e.g. system prompts) and untrusted.
The exact same phrase can tokenize differently depending on if it's instruction or data; and you can pre-train and post-train models to make use of them.
It quadratically increases your training cost, so I don't think any labs are exploring it because of $$$ and the race to AGI, but mechanisms like this should significantly address the issue on the LLM architectural design level.
Worth reminding everyone of previous discussion when Atlassian opted-in all customers by default for their data to be used for model training.
https://news.ycombinator.com/item?id=47833247
This goes live on August 17. If you haven't switched it off your company IP will be used to train their future models.
So they will harvest trade secrets from 10000s of companies and there have been no panic mass flight about this?
How companies entrust SaaSs with their data is beyond insane to me. Especially since FOSS alternatives are readily available.
Who cares?
The employee, just looking to make rent? The CEO, all in on the dream of the AI powered future? Legal, looking forward to litigation and well-competition retainers? Atlassian itself, when it’s just one of the many companies training models on private intellectual property, and whose T&C clearly state they will be doing so? Competitors, when there will be no reasonable way to prove their code has been generated from yours?
The truth is that one gives a damn about trade secrets being used to train AI models.
The entire copyright system for software is dead and no one really seems to care or even talk about it.
The truth is that most software companies don't have any secrets worth hiding.
The saas industry is all about sales and deals. The market runs on access, not intelligence.
IMHO, this is not the point. The core problem is that they're building systems that will remove competition altogether. You may still get your current systems to market ahead of your competition, but future "competitors" will be built off of your data and Atlassian will be one more company in the list of companies profiting from everybody else building systems using their data, probably destined to fail because everybody else can do it just as easily, but the money required to fund Atlassian's systems that built it for them, however large or small, will have already been sent. Those systems that Atlassian will profit from will be built from the data of "customers", who also paid Atlassian will giving up their own data.
With almost zero control over whether or not this happens (politically or individually), the only reasonable action to take is "take advantage while there's an advantage to be had".
every company is removing competition altogether. Some succeed, like Google, but most don't.
Knowing that our competitor is trying to poach one of our clients away by promising a new feature could be very useful to us. Likewise, knowing that a competitor is at risk of losing a customer because they won't implement a feature that we're willing to implement (or already have) is a qualified sales lead.
All of this would be illegal if discovered in this way, presumably, but it's not entirely accurate to say that information is not power.
I struggle to see the tech/data being used in that way. At a much higher level, the data is more likely anonymized (where possible) and/or not directly exposed to anybody. Think of the initial github copilot that used source code across all of the repos and how it was able to take instructions for what you wanted to build and give you essentially a consensus built piece of code that fit what you asked for, be it by auto-completion, comments, or direct requests.
"I want to build a business that does X using Y. Give me revenue predictions, spend over time, and customer demand predictions." etc
Usually customers are open about this kind of thing though. They play you off your competitors and vice versa.
I don't totally disagree, I just think the real secrets have nothing to do with features or dev. The secret is A is about to go bankrupt or B is facing legal trouble. These kinds of things aren't in JIRA.
[dead]
Very few companies actually have any “trade secrets”. The success of almost any business is based on marketing and distribution and reputation and partnerships, not on any “secret”.
It's one of the classic mistakes of people who are not entrepreneurs. They try to keep things too secret and they spend a lot of time and effort keeping things secret while real entrepreneurs who have actually made millions on the market know secrecy rarely matters.
In the realm of apps and stuff, it is pretty much just repacking of someone else's idea.
In the world of B2B, there are definitely companies that provide service based on direct personal experience of senior employees. While they may be using the same software available to others, it's the logic to know what to tell that same ol' software what to do that matters. Over my entire career I have dealt with multiple companies that write code but have zero knowledge about the subject matter the code is being written for so that basic mistakes are made resulting in utterly ineffective software.
I've seen their ticketing software, what on earth are they going to do with any trade secrets? There are none in ticketing software or wiki's.
Also the people making decisions on their products (or any "enterprise" SaaS for that matter) aren't the ones stuck using it.
> There are none in ticketing software or wiki's.
But there might be API keys, passwords, unlisted URLs, email addresses, etc.
I guess customer lists might be the most valuable data in most Jira setups. Sales data ...
Companies that have something worth protecting already don't use it and/or pay for the on-prem version.
The rest don't have anything worth stealing anyway.
Our company dumped Atlassian. No-one misses it, it’s universally considered a good decision.
> Rovo's URL retrieval tool is insecure: there are no protections against opening a URL that has been dynamically created by the agent. Here, Rovo is manipulated to append sensitive data to an attacker's URL.
I think it was Anthropic that first introduced a pattern that completely locks this down: your URL retrieval tool should only work for URLs that have previously been typed into the conversation by a user or have been returned from a trusted tool.
If the agent itself concatenates a new URL together - with leaked data after a ? - you should block that from being fetched.
The great thing about this solution is it's deterministic. You don't need any extra AI in the max - you implement a URL fetching system that knows which sources it should check for a direct match on the URL before it makes that GET request.
One of the latest mitigations is to make sure that a URL an agent visits has been indexed by a search engine crawler. At least that is what OpenAI does now in ChatGPT.
That makes sure that not a large amount of private data is leaked in one request. Assuming that if a URL is indexed, it is public data. However, there are still bypasses with using many requests to leak information, like a request per character of pre-indexed URLs.
I have some demos of doing that on my blog, but it makes it more involved for an attacker. And that could also be detected. Still not perfect, but a solid improvement, for a generic agent like ChatGPT.
There is paper OpenAI wrote a few months ago that explains how they do it: https://embracethered.com/blog/posts/2026/data-exfiltration-...
It's not a 100% bullet proof approach either, but pretty good.
Regarding the point on using URLs returned from trusted tool calls. That is similar to using pre-indexed URLs: If a "trusted tool" includes things like read a document, read an email,... an attacker can return a large list of afterwards "safe" urls, like 26 to cover A-Z. And then an attack can render many requests, e.g. character by character. But, again, similar to the pre-indexing, things are getting more a lot more expensive for an attacker that way. However, still not impossible.
For agents that have a specific purpose simple domain allow-listing is also a pretty effective idea in to prevent attacker controlled endpoints.
Yeah, the letter-by-letter and dynamic tools that construct longer URLs tricks are both important to know about. I wrote about the latter of those here: https://simonwillison.net/2026/Jul/15/claude-web-fetch-exfil...
Determinism is a terrifying word to people who want to believe their LLM has a little brain and can do anything they want it to.
I've been struggling a lot to understand this ever since the agents thing entered the hype. If I follow a path of requirements, it always comes down to: But why you'll leave the decision to a stochastic tool, when you should a have deterministic approach?
It's software god damn it... the reason why people moved from analog to digital is because you can repetitively execute functions that do always the same thing and it's 0 when it's 0, 1 when its 1.
All the sudden everyone is ok on burning trees to have their cool probabilistic tool named agent to do: maybe it's 0, but it can also be 1, let me "think"... ah yes, for sure it's 2.
The sad part for me is that management people have their heads so much into this hype, that no attack on privacy matters (almost none actually ever did, I know). Only when they suffer a huge blow in terms of revenue or reputation is that they maybe, maaaybe, find will want to listen again the experts.
Are you saying you would prefer a perfectly deterministic code writer? So, humans shouldn’t write any code anymore?!
A wild number of people who present themselves as knowledgeable or even experts on the subject either have no idea or worse, do not believe that it is possible to ie set a static seed value and send the same prompt across multiple fresh contexts and get the same result*
It's genuinely a little concerning. It does not help that many of them are gaslighting themselves into thinking these things are benchmark crushing elite hackers by running effectively unsecured, unfiltered, unlogged production environments.
*Obviously this also means a temperature of >0 to avoid the greedy trap, the same generation params, on the same software, hardware, drivers etc etc
> If the agent itself concatenates a new URL together - with leaked data after a ? - you should block that from being fetched.
You're correct of course, I just want to note that the exfiltrated data could be in any part of the URL, so the absence of a query string doesn't indicate that no payload has been encoded into the URL. Arbitrary example, you can include credentials in a URL, so you could encode the exfiltrated data into a password.
Right, I should have been more clear. It's not about the ?, it's about not being able to dynamically construct a URL at all.
Otherwise you could set up wildcard DNS and extract data to base64encodedstolendata.evil.com
Joke’s on you, I’ve blocked evil.com
I can’t get over how bad “Rovo” is. Somehow more aggressive and useless than Microsoft putting “Copilot” everywhere.
It’s objectively worse than using something like Cowork + MCP, AND they injected it into every single page on JIRA and Confluent which has made web browsing way slower while all the junk is loading.
Rovo has my favorite example of AI misfeature. Just checked, it's still there in Cloud Confluence. In Edit mode for a page, you can select a range of text and a menu will pop up, with Ask Rovo being a drop down on it. There's a few good options... Improve Formatting, translation options, Make Shorter...
... but it also has Make Longer. Yes, a built-in feature to type some text in, and the use the mighty power of AI to bloat it.
Naturally, you can repeat this process several times on the same text, for your own little personal demonstration of what model collapse looks like in real time.
Don't worry, you can use Rovo to summarize the document! That way you can get the information your coworker wanted to share but filtered through a game of teLLMephone.
We’ve gone from proof-of-work to proof-of-no-work.
I've tried the "Make Longer" and "Make Shorter".
When you go "Make longer" the text gets longer.
When you go "Make shorter", the text gets longer still.
You'd think that the original text would be the oracle for shorter(longer(original))?
> You'd think
I'm going to have to stop you right there
That's some hillarious misnaming, I love it. It probably was meant to be something like "make more comprehensible, use simpler language but more words" and we got this.
But for their target market, corporate office drones writing crap that nobody will ever read more than once (if even that), this is actually perfect and makes AI usage numbers go up. "Make longer" is amazing for someone who just wants to pretend they put more effort than they actually did.
(of course, everyone else can do that too, and the length/literacy of prose is no longer a good proxy for effort. In nature, "honest signalling" only works if the signal is costly. Removing the cost from the signal makes the signal worthless)
What's even worse is that, when you do this on the description of a story or task, everytime it comes up with a slightly different format of how it's going to layout the "enhanced" description.
>I can’t get over how bad “Rovo” is. Somehow more aggressive and useless than Microsoft putting “Copilot” everywhere.
You're looking at it the wrong way. Think about how much better it is than the joke of a search tool that's slightly to the left of it's icon.
I just asked it to do a summary or a status page for my staff so I can share the high level details. Somehow it retrieved a version of the page from two weeks ago and summarised that. When I said, this is old, it happily said, yep, I'll do a summary of the current page. I kinda feel there may be people in between the box and some lame LLM.
Have you seen the markdown agent instructions they provide in their new agentic `twg` cli? 70k tokens one average, there are more than one...
Rovo is the worse Ai I have used, I suggested they stop trying and let us have model choice. Save money and don't do things out of their skill sets
The rovo CLI is beating Claude Code in some benchmarks. It works pretty well for me.
Don’t use it? You don’t have to
Yup, we stopped using Atlassian completely. Highly recommended.
Yeah, for sure don't use it. It's complete slop. The help strings do not match the real args/flags.
We went back to calling the API directly, writing scripts for the agent, then taking those scripts away and just having our CI workflow run them and then hand off a `.review` directory to the agents. We give them fewer tools and permissions, them seem to stay on task more instead of being "relentlessly proactive"
When it was new I tried something fairly easy, or so I thought
"Label all stories in the current project that contain "foo" with label "bar
It failed
"has made web browsing way slower while all the junk is loading" describes all web development since the 90s
I hate it, they've just gone and smooshed a bunch more AI crap into Jira too in the last few days.
Useless buttons to "suggest subtasks" and "improve issue" and offer to write the description. Absolutely useless nonsense, I wish I could turn it all off and there is an issue on their bug tracker with that request (turning it off as an individual user), and it has many votes, but at the moment you can only disable it globally as admin.
Rovo makes for an INCREDIBLE time reading jira tickets and confluence pages. Someone wrote YES in all caps in a page ? Rovo will helpfully tell you (after loading for 5 seconds) that YES stands for Yassified Entertainment Setting, a new option added to the app in January 2024.
Whenever I feel like I'm bad at software development, I look at Atlassian and I feel good.
I’ve been adding a “terms” section to every AI-used project due to these hallucinations. We abbreviate our company name eeeeverywhere and even Opus gets this wrong constantly. Obv I’m not adding “YES==yes” but that’s hilarious!
> Rovo will helpfully tell you (after loading for 5 seconds) that YES stands for Yassified Entertainment Setting, a new option added to the app in January 2024.
The problem with our brave new AI world is I can’t tell if this is real or not
It's real but not literally. It will absolutely make up acronyms but I never heard of a Yassified Entertainment Setting.
Maybe it exists! Maybe it doesn't! Maybe ROVO actually means Really Overtly Violent Opportunities! And Rovo can make it all up as it goes.
~every ai vulnerability write up boils down to "just ask it do to the thing", but with fancier terms like "indirect prompt injection".
> The victim uploads a file to Rovo that contains a hidden prompt injection
Yeah this attack is possible on all modern agentic systems.
* Access to your private data
* Exposure to untrusted content
* The ability to externally communicate in a way that could be used to steal your data
(https://simonw.substack.com/p/the-lethal-trifecta-for-ai-age...)
And blocking it wholesale reduces usefulness of the agent so it is a tradeoff.
It's been over a year and the trifecta holds true. The term 'prompt injection' has been round since May 2022 [0]. No amount of "but the future will fix it" has fixed it.
These breaches will continue for as long as nobody cares about security and everybody is infected with the LLM brainworm.
0 - https://en.wikipedia.org/wiki/Prompt_injection
Prompt injection is as fixable in LLMs as it is in people. There is no such thing as out of band data. For example it's quite common for large businesses to fall for billing fraud scams when something shows up and says "Hey, it's the CEO, pay this bill to X for $Y".
And honestly when you start looking at agentic systems that uses it's previous step to take future steps. The system has to some idea what you want to permit and don't want to permit as everyone could have different expectations here.
> "Hey, it's the CEO, pay this bill to X for $Y"
Sure. Now imagine this very scenario not limited by humans and scaling the way machines do. And then let us consider the current reality that an agentic system on the receiving end may have unlimited access because to too was vibed.
My only warning with the trifecta is it sufficient but not necessary for an LLM attack.
All you need is the untrusted content (or more generally vulnerable content since how you feel about the content doesn't matter) leg and something you care about to be attacked.
E.g. could be as simple as a prompt injection that causes your LLM to output a prompt injection that then gets inserted somewhere else.
Or without exfiltration rm -rf /
Or a social engineering attack.
So there are other bifectas and trifectas.
Rovo is funny. It downloads everything it can do Atlassian servers for "analysis". And you're pretty much screwed if you link it to Google Docs or Sharepoint. How do I know this? "Why is an AWS IP downloading all our docs?" question I got about a month ago.
I find it difficult to be impressed by "prompt injection" attacks that require the victim to enter the malicious prompt themselves --- like, really? If you tell Rovo to exfiltrate your data, it'll do it?
Obviously, there should be URL protection rules to control what it can access, but this requires a very specific and unlikely set of circumstances to exploit.
It's more interesting if I attach a file to a JIRA ticket that we both have access to and via some query you send to the AI (that returns my malicious ticket) it causes data exfiltration of tickets that you have access to but I do not have access to. I think that's more compelling as an example than the one they provide.
Are people so obsessed with AI that they can't find it reasonable that it won't do obviously bad things if asked? Not even with a confirmation or warning? We trust AI to literally build products and fix our most critical bugs, but we can't expect it to tell when it's being asked to do something malicious? Imagine if we felt this way about QA when trying DROP TABLES; in search bars. "Oh, well of course it broke the database, the user asked it to!"
You’re asking for AI censorship ( that’s the term used for when you patch the AI to not do obviously bad things according to the owners, which as with any censorship, may be widely different from what you consider bad things).
You may be happy to learn frontier LLM are heavily censored! Try an uncensored local LLM for a comparison. It will literally do everything you ask it to, no matter how devious.
> Rovo's URL retrieval tool is insecure: there are no protections against opening a URL that has been dynamically created by the agent. Here, Rovo is manipulated to append sensitive data to an attacker's URL. When Rovo calls the insecure tool to open the URL, the attacker's site logs the request, including the appended sensitive data.
> Note: This attack succeeds even if an organization has disabled web search for Rovo. This is because the web search setting fails to remove the tool for opening the search results.
Wow, great work Atlassian. The web search setting does not disable web search.
Related: A few days ago, Jira opted everyone in by default to "Contribute in-app data to improve Atlassian apps for everyone"
It's nice they force rovo now for document/version diff's. Because you need to burn down the rainforest for those. (sarcasm ... for obvious reasons)
Atlassian has gone from a trusted enterprise-partner to a complete shit-show in just 18 months. This surprises nobody. There will be classes taught in how to fuck up a good business and Atlassian will be the prime example.
Regards, /someone who migrated 3500 users from Atlassians products recently due to their "cloud only"-bullshit.
They were not trustworthy years ago already. Using Atlassian products, one has had to look out for disastrous announcements about their customer data usage way before 18 months ago, especially, if running a business inside the EU, and at least trying to adhere to data protection laws.
At the risk of sounding like the hipster who liked your favorite band first, I didn't trust Atlassian long before the last 18 months. Much like Microsoft Word, I think JIRA peaked about a decade after launch and since then it's mostly been unnecessary UI changes and features I either don't care about or actively do not want.
P.S. I protest their software's revisionist autocorrection to Jira. It was JIRA when I started using it and forevermore it shall be.
To be fair to Atlassian - their products did suck quite a bit before 18 months ago as well. Just now they still do, but with ai!
You'd have to look at their profit numbers. They have a huge captive base of customers, like Adobe it might take decades for things to get bad enough and alternatives to arise before their profitability is impacted. And the executives who benefit will have cashed out long before then.
Recently saw an example of somebody who vibe coded a tool to mass export the contents of a 'Confluence' wiki into an instance of self hosted mediawiki, preserving everything.
Mediawiki as a whole has a feature set that 95% of organizations will only scratch the surface of. There's a ridiculous number of possible plugins and customization if you have somebody who knows what they're doing with it.
The majority of companies that need an internal KB/wiki do not have as complex needs or use cases as wikipedia itself or the wikimedia foundation.
I've been at more than one company that migrated _from_ MediaWiki to Confluence. It usually boils down to "non-developers have to use this and Markdown+plugins is hard".
Turns out no matter how much better the thing is, the users have to know what they're doing more than the operators do.
I'm not surprised. We're trying to migrate from Jira and Confluence (because we can't have our data outside our servers), and while Jira has many alternatives, it's difficult to find worthy competitors to Confluence. They either have an awful UI/UX, or don't have as many features. We're looking at Xwiki, but simply try to have multiple users simultaneously edit a table and you'll see why everyone prefers Confluence...
I'm partial to YouTrack but I also haven't used it in 10 years and it was painful to set up.
visualeditor has been a thing for a long time now, no non technical user needs to ever touch markdown
https://www.mediawiki.org/wiki/VisualEditor
It's the same GUI WYSIWYG interface you get by default if editing a page on public wikipedia.
99.99999% of people will never even attempt to edit a page on wikipedia.
Also VisualEditor sucks if you've never used VisualEditor before and sucks even more for anything complicated. This is the kind of tonedeafness non-technical users get to their feedback all the time.
Been there before too (though an alternative to MediaWiki). A few important pages were migrated but we just ended up with two wikis with lots of tech info on the old one and lots of marketing info on Confluence.
I started to consider it a show show way earlier than 18 months ago. Jira is so terrible to use that it is hard to phantom how they are able to be paid for their product
Jira is how it is because almost any product that grows to be that flexible will develop the same problems.
They failed to rearchitect it to something suitable for the inherent flexibility though, so it remains a disaster area, but one that is uniquely able to fit the whims of any manager that can then mandate it for everyone else.
Maybe the workflows, etc., are inherent complexity, so that any similarly-flexible product will have the same problems.
But the awful client-server architecture, where anything you do takes several seconds of waiting for the client to send who-knows-what to the server and get data back, even selecting something from a dropdown... that is absolutely just Jira. A competing product would be able to deliver much, MUCH snappier UX. Jira is just the crappiest, slowest UI that I have ever had the displeasure of being forced to work with.
I invite you to write such a trivially flexible workflow system, and see what happens to the performance of your user interface.
They needed someone over there with the power to say no.
if you track their 5 year stock price you'll find the market fully agrees with you. It was $458 in 2021 and is $112 now.
The market caught on quickly that Atlassian's software fits into the group of software like Oracle or SAP that no one uses by choice and only uses because it was forced on them.
> ... Many forms of issue tracking have been tried, and will be tried in this world of sin and woe. No one pretends that Jira is perfect or all-wise. Indeed, it has been said that Jira is the worst of the issue trackers except all those other forms that have been tried from time to time; but there is the broad feeling in our company that the people should track, continuously track, and that client opinion, expressed by all project management means, should shape, guide, and control the actions of management who are their masters and not their servants. ...
-- Churchill, House of Commons, 11 November 1947
Wrong thread??
No. I was adjusting a classic quote about something that is decried as awful and no one wants to do... except all the other alternatives are worse.
It's not so much that people want to use Jira... or that they're forced to use it... but rather that people try using some other system and despite how much they dislike Jira, that other one has bigger failings.
Could go for a Bjarne Stroustrup quote with a twist:
> There are only two kinds of issue trackers: the ones people complain about and the ones nobody uses.
... but I like the Churchill quote.
Go through https://en.wikipedia.org/wiki/Comparison_of_issue-tracking_s... and after a year they'll be begging to go back to Jira... not because its great, but because the other tool lacks some necessary feature that Jira provides. Jira does everything that everyone wants, and it's awful... but everything else is worse.
Any recommendations for alternatives for just Confluence and JIRA?
I've been looking at switching to Notion and Linear, but just haven't had the time to complete evaluation.
I can't remember a time in which Atlassian was trusted.
To where? Where did you convince c-suite to move to?
Trusted enterprise partner? lol, lmao even.
The writing is rather low quality, but seems to be consistent with their other posts.
Maybe to give credit they are being purposefully vague about details to avoid giving away the bait but still seem like you could give me details without literally copy and pasting the attack.
Rovo is one of those intrusive AI buttons that suddenly appeared everywhere without any warning. Its so annoying having already shitty UI get borked with features I never use. Almost as annoying as Whatsapp suddenly getting the same FOMO AI button. Its not like I need an AI agent to talk to friends and family. And a summary is something I can always generate via copy&paste into CLI chat session.
I'm still on the edge about security as an afterthought in LLMs. Given its now so easy to generate a ton of slop - why not focus on nonfunctional stuff making LLMs operate faster than thinking for X minutes and limiting exfiltration of local env secrets?
Do people put secrets in Atlassian tools? Sounds like the worst place to do that. If anything I would want my main wiki/project/ticket management tools/code repos to be as open as possible (I mean to read) to help collaboration.
The part about not having guardrail against calls to external urls is wild though.
Not "secrets" usually, but people tend to consider the corporate IP embodied in Jira to be classified as something other than Public.
Teamwork Graph exposes all the people working on stealth products with code names right alongside their other work anyway.
I was thinking within the scope of a single tenant which is the scope of the prompt injections issues mentionned in this article.
I read it as an intentional exfiltration.
It's incredibly hard to block all rovo on Atlassian pages. I tried adblock, it failed, I settled for the custom chrome plugin. And yet, every once a while a new way to disturb my peace shows it ugly head.
So why intentional, again? Because it's impossible to disable it until you have a very certain, very expensive plan.
Nonconsensual data exfiltration.
Is it any coincidence that Rovo rhymes with "no, NO!"
Rovo will also execute instructions hosted on external URls - they've known this for at least 6 months.
I hate working with Atlassian, years and years ago moved to youtrack instead of Jira mainly because it handles teams who work on multiple projects way better. But confluence remains, just need to spend some time re-evaluating other wikis (there's always been some blocking reason why the alternatives aren't a good fit, but they've all improved a lot)
It is reflected in their stock price.
It's kind of sad because in a shitty Jira setup, Rovo is usually the only way to make any sense of tickets. Don't ask me how I know this. Sadly I think many teams have become dependent on the tool to make sense of their dumpster fire of an Atlassian environment (usually by their own doing but anyway).
Must be nice. I couldn’t even manage to get it to write a well-formed JQL query.
Fwiw Rovo is built on top of Claude
On a sidenote, what's up with people still using Ayn Rand references in their company names. It's cringe.
It's a reference to Greek mythology, which predates Ayn Rand by several thousand years.
This is the reason I show my frustration in my Jira tickets so that let Rovo learn how frustrated humans have became with AI slop.
Ahh yes: "when you Rovo, you oh-no my data"
I feel like prompt armor writes the exact same blog post for every agentic tool because they all suffer from the ignore previous instructions prompt injections.
https://www.promptarmor.com/resources/claude-cowork-exfiltra...
https://www.promptarmor.com/resources/google-antigravity-exf...
https://promptarmor.substack.com/p/data-exfiltration-from-sl...
https://www.promptarmor.com/resources/gpt-for-google-sheets-...
https://www.promptarmor.com/resources/notion-ai-unpatched-da...
https://www.promptarmor.com/resources/ramps-sheets-ai-exfilt...
https://www.promptarmor.com/resources/superhuman-ai-exfiltra...
How could they not? If some lab had a method to make really secure guard rails or avoid prompt injection thoroughly I think they would be trumpeting it. But the basic mechanics of language models are vulnerable to this unless you can always be sure the inputs are from a safe user imo
If you want AI to be useful it will eventually encounter untrusted content, such as via web search. I think things like web search should probably be run on a different sandboxed AI whose task is to write a summary that is then ingested by the main agent, similar to how existing sandboxing already works, but this would diminish the usefulness quite a bit.
>I think things like web search should probably be run on a different sandboxed AI whose task is to write a summary that is then ingested by the main agent, similar to how existing sandboxing already works, but this would diminish the usefulness quite a bit.
It also wouldn't work. You would simply mindjack the outer AI and have it mindjack the inner AI in turn with its summary. Nesting AIs can't fix the malicious input problem.
Just pass that through a third llm.
Correct, that has been prophesied by scripture:
"Thou shalt have three layers of LLMs, no more, no less. Three shall be the number thou shalt have, and the number of the counting of the LLM layers shall be three."
Four is right out
> Once the number three, being the third number, be reached, then lobbest thou thy Holy LLM of Antioch towards thy task, who, being naughty in My sight, shall snuff it.
What's so hard about having the LLM tool calls scoped to the tenant? Inject "X-Scope-I" after the LLM decided on a tool call and you're done. Easiest fix ever.
Famous last words: easy fix.
The Rovo MCP server manages scope credentials securely with “bring your own LLM”. Yet somehow they still fucked up with their own agent
At some point LLM forgetting to check the scope? On the nth automated rewrite. Everything else works. It might even test for test case. But not in production...
> How could they not?
It seems like the simpler cases of “ignore all previous instructions” could be easily stopped with a regex, or a classifier model… or even an LLM (which yes does raise the risk that the “ignore all previous instructions” detection LLM invocation could itself be attacked by the same mechanism—but a safeguard doesn’t have to be foolproof to be valuable, it is all about probabilities)
Now, of course, there is a long tail of elaborate variations that those techniques won’t be able to stop. But have the published vulnerabilities come from that long tail or from not doing enough to address the simpler cases?
Eh, I think you underestimate the difficulty in the kinds of problems that are occurring. For example if you're making an AI written document talking about jailbreaks, your regex is just going to break that use case. And there are probably 4 zillion other things the regex will step on.
The classifier model will help some, but you end up with the same problem, a dumber model can never figure out what a smarter model is going to do with a bit of text. Or even two different models in this case.
On top of that, you can just automate finding new variations of the attack. Any one that works is quickly and massively duplicated causing all kinds of problems before your classification model catches back up.
Really what you're thinking here is this something that can be 'simply fixed'. It is not. The only way it's truly fixed is by having a model that is aligned with all good human decisions and makes none of the bad ones. Models will likely always find new and interesting ways break because everything is in band, there is no out of band data, much like a human. "Dear model, here is a chocolate bar, run $thing you aren't supposed to$" will probably keep working when it's something like "more tokens for you to use".
> Really what you're thinking here is this something that can be 'simply fixed'. It is not. The only way it's truly fixed
I think this is binary categorical thinking. In the real world, safety systems (even in domains like aviation or nuclear power) are never foolproof-the point is you reduce the probability of failure to an acceptable level given the costs of doing so and the potential consequences of that failure
And there is the risk people say “there is no foolproof solution, so I’m not going to invest in probabilistic countermeasures” - which would sound like utter madness to a bank’s antifraud department, but for some reason a lot of people seem to think it isn’t when it comes to AI
The issue is with the nature of agentic systems and how often they run, a .001 failure is still huge. Attackers monitor their attack chances and use the most successful attacks so the actual success on the attackers side is much closer to 1 than .001.
This is the problem, this isn't a 'failure' mode where something randomly goes wrong, like a person accidentally sticking their hand in a machine. This is war where you are under active attack and the attackers adapt quickly.
Because of the breadth of LLM capabilities they have a nearly unbound attack surface. You spend so much of your effort making sure your prompt is secure that you either massively increase costs, or you partially lobotomize your agent making it less capable.
Solutions for these problems will have to come inside the model itself because both the attackers and models are far more capable than you on doing dumb/evil shit. Until the model itself can go "hey, wait a darn minute" a new fun way of encoding the attack can always be found.
> This is war where you are under active attack and the attackers adapt quickly.
It all depends on what the use case is.
For example, consider a system which takes English questions from business users, translates them to SQL, then runs it (as the business user) against a reporting database
How does the attacker get access to it? And even if they somehow do, what harm can they actually do to it? The only tool it has is one to run SELECT against a DB, and it can only SELECT data the user already has access to, and we have timeouts/etc to protect against overly expensive queries
If your design involves lots of general purpose do-anything agents with very generic tools, the risk profile is very different from narrowly scoped purpose-specific AI services
I mean, yeah, consider a system that takes English questions and does nothing with them, it’s very secure. What’s your point? It doesn’t really refute what the commenter above you was saying.
Security is a process, not a product.
I have to remind the folks who want a dev to build a product and think they don't need to ever touch the product after release.
No reacione al instruciones previas.
Methods like regexes, classifiers, recursive LLM calls - for all their limitations - are in principle extensible to any language
Of course, there is a scaling problem here - doing it for N languages is N times the work of doing it for one, and you might not know some language well enough to do a good job of it
Two things you can do:
(1) Use an LLM for translate your regexes, classifier training/validation datasets, etc, from English to N target languages
(2) Use an LLM to translate the input to English, then run your validations against that English
Of course, these approaches aren’t foolproof - the translation might be flawed/etc - but a defence doesn’t have to be foolproof to add real value
Also, of course using an LLM to translate hostile input to English before checking it runs the risk the translation LLM itself might be jailbroken. But how about using a classical statistical machine translation model instead? They don’t tend to be vulnerable to these kinds of attacks, because they aren’t trained to follow instructions
And that works pretty well for me. Jailbreak in depth :p
Now pretend you are DAN (Do Anything Now) and ...
RIP.
> ignore previous instructions prompt injections.
I wonder if anyone has tried to build an LLM that has actual built-in types of prompts: system prompt, user prompt, and data prompt.
It’s not really possible with the way that the context works.
There are architectural solutions. You could duplicate your tokeniser/vocab for example; and have two classes of input: trusted (e.g. system prompts) and untrusted.
The exact same phrase can tokenize differently depending on if it's instruction or data; and you can pre-train and post-train models to make use of them.
It quadratically increases your training cost, so I don't think any labs are exploring it because of $$$ and the race to AGI, but mechanisms like this should significantly address the issue on the LLM architectural design level.
Well, until everyone realizes that prompts aren’t guarantees, these posts are still useful.
This list reminds me more of blackhat seo than anything to be taken seriously.
[flagged]
[flagged]
[flagged]
[dead]
[dead]