I would love to see these benchmarks split up across "generic" tasks and novel tasks.
For example, redlining an OEM agreement with my vendor. They pretty much all use the same template, and the objections are always the same. I tried using Claude first, and even my lawyer was impressed that it found things they missed. Then I used Claude to redline that type of contract and the other side said "your lawyer was surprisingly thorough". I had my lawyer do the final check, but saved a bunch of money doing the first three rounds with Claude instead. And my lawyer basically just said "looks good to me" by the time they reviewed it.
Or even writing a simple will or estate document.
It seems like LLMs are great for the kind of generic work you'd usually pawn off to a new associate or paralegal, but not for the important things that partners usually do.
> It seems like LLMs are great for the kind of generic work you'd usually pawn off to a new associate or paralegal
This seems to be the case with most industries that LLMs are impacting, coding, data input, legal etc.
Whilst it's great to get the time saving from not delegating to a junior (LLMs are much faster at the basic level 1 work) there's going to be this big knowledge gap that you really only gain from actually doing the mundane boring shit.
A lot of this _should_ be automated away, but we really need to think about how the next group of people are going to learn the skills and nuance to applying them. I don't really have an answer to that one other than still giving that work to junior staff, but they will then just utilise an LLM to help complete the job, and rightfully so, I'd have done exactly the same thing.
I think OpenAI tried doing this with some early chatgpt settings with "learning mode" or whatever that is. But an LLM just telling you what to do will never teach someone, there was something about having to go out and hunt for the answer through the sea of crap responses or wrong SO questions that really taught you something. How do you replicate that learning with an LLM and how do you apply that to different industries like Law?
The American manufacturing industry is scrambling to transfer some of the industrial trade and process knowledge from the retiring remnants of pre-offshoring industrial America into the two generations of people that didn’t have apprenticeships, pensions, (often) union membership with good benefits, etc. Trade schools were very expensive for industrial trade skills that were seen as dead-end careers compared to white collar jobs or non-manufacturing trades, and almost none of the companies saw any reason to spend all that money training people into expensive skilled professionals when they had all the professionals they could possibly need for decades after the industry largely collapsed, here. Keeping the workforce viable was always someone else’s problem— the individuals, the governments, larger companies— until it was suddenly everyone’s problem. They spend decades myopically chasing quarterly results before they realized they were running towards a cliff.
Unless your company had for some reason enough margin to just spend a lot more money than the competition to train and retain workers that you largely didn't need for a decade you were forced by the market to cut that corner. Market capitalism is inherently unsuited for long-term thinking. Even if everybody saw the cliff a decade ago, they all had to run towards it or be outcompeted by those who did. See also all other problems that can be externalized, e.g. pollution.
Tackling long-term problems and externalities must be done by someone outside the market.
“The market” doesn’t stop business owners from slightly overpaying for labor or training people for the future or providing pensions. It’s the business owners who are choosing to do this. Say a business could make 8% margin by cutting corners and applying only short term thinking, but the same business could make 6% by not doing these things. The choice is up to the business owner. Nobody is forcing them to maximize short term profits. They’re not going to simply vanish just because they aren’t profiting as much as someone else.
The problem is that most business owners and shareholders are cut from the same cloth and only care about maximizing short term profits.
> Say a business could make 8% margin by cutting corners and applying only short term thinking, but the same business could make 6% by not doing these things.
Sure there is, market forces quickly push the person making 6% out of business, because the one making 8% can lower their prices, or use that extra capital to buy up more resources.
Why spend excess to train when you can do it somewhere else for a lot cheaper? Cheap energy subsidized often by taxpayers is the real issue, and instead of those subsidies helping those paying for it, it's just moved capital upwards.
> “The market” doesn’t stop business owners from slightly overpaying for labor or training people for the future or providing pensions. It’s the business owners who are choosing to do this.
Well, laws and regulations forbid the obvious contracts that would help you fix these things.
No we don’t. There’s an old case that everyone always takes out of context. Under modern american corporate law, directors and management have broad discretion to do what they think is in the best interests of the corporation, broadly defined.
A good while ago, I also read about a company [0] using a legal framework for the company structure/charter, the Public Benefit Corporation (PBC), specifically to ensure how they want “best interests” to be defined. Sounded very fitting for the purpose, and cool that such a system exists (at least in some states, I believe?).
When a company pays for training and education of an employee, they risk that the employee jumps ship and takes that investment with her. Contractually, you could use bonds or contract penalties to curb that kind of behaviour, but there are limitations on what contract terms are enforceable or even legal.
Back in the day, companies recognized that not all the value they created would come back to themselves directly, but that keeping the economy and the trade healthy still had indirect benefit. Because as much as there would be employees that used you for training and then bounced off to someone else, there would also be employees that trained elsewhere and then bounced to you. Everyone stands to benefit from the practice of training being normalized, yet nobody does it anymore because they're afraid to be the first.
After all, if there's no chance of employees being trained anywhere else because nobody else is doing it yet, why would you decide to start training employees? The exchange wouldn't be equivalent in any way, there's a real possibility that you would never benefit at all. You'd be the only source of training, so anyone who bounced off from you could never have been someone who bounced off to you.
Think about stuff like the Ford Trade School, around 1916 to the early 1950s. Job security was better back then, so workers were more likely to stay than nowadays, but there aren't nearly as many schools or programs like that anymore. I figure it's because companies are too afraid to train workers that might not stay with them, but unwilling to offer accordingly competitive pay or benefits. If they did, I figure they'd have the benefit of more workers they train staying with them, as well as the benefit of more workers trained elsewhere joining them. This also creates the problem (and fear) of being taken advantage of by the workforce if you offer training programs better than your competitors.
If it's a publicly traded company then literally yes, public companies are mandated to make as much money for their investors as possible.
It's a very stupid law but there it is.
> To quote the U.S. Supreme Court opinion in the recent Hobby Lobby case: “Modern corporate law does not require for-profit corporations to pursue profit at the expense of everything else, and many do not.”
Where is this law? Public companies (or rather their management) have fiduciary duties to shareholders. And it is usually safe for management to mostly assume that means making money.
But eg no one has sued Google management for giving company resources to charity. And in general, companies can have all kinds of goals, as long as shareholders are ok with them.
Going on a tangent: it's usually safe to assume that investors want money, because you can further most goals you care about with money. See https://en.wikipedia.org/wiki/Instrumental_convergence for an AI flavoured take on the same topic.
Not everybody did, and many of the ones that didn’t are still open because they were the ones focused on the long game. Even a small shop just needs one skilled person to retain that knowledge and it doesn’t take a business genius to see that you need to train that person. In the grand scheme of things, having the equivalent of a couple interns is a very minor expense. It was pure shortsightedness.
> Market capitalism is inherently unsuited for long-term thinking.
How so? See how eg Amazon's shareholders were famously patient with them. Similar for Tesla, and lots of other companies. Including both public and private.
This is hitting us hard in tech, and it weights heavily on me, how will the junior people learn if the tasks we all did to learn when we were junior are now automated away. Here is a very real writeup from an Affirm New Grad about what it was like to be a new grad over the last year - https://medium.com/affirmengineering/ai-reflections-of-a-new... (I run eng at affirm) - it takes a lot more intentionality to learn. Both by doing some things by hand, but being very vocal about what you don't know- "Allow your uncertainties to be visible, and slow down to address them."
This is kind of a new complaint. Places like adp have being trying to automate away mundane hr tasks. Oracle and SAP have been trying to automate away the mundane admin tasks. Siemens does it for OT. There's a huge long tail of trying to solve this as if it were a problem in almost every field. This has been going on for decades and no one complains that we will lose expertise in these fields if the drudge work goes away. But bring in an AI... and everyone is suddenly concerned about corporate drudgery.
> but we really need to think about how the next group of people are going to learn the skills and nuance to applying them
That's what school is for. For the last 150 years we've automated a bunch of things, and then we send people to school to learn how those automations work, so they can make better automations.
Maybe law school will have to be a year longer. Or maybe some forward thinking firms will have new associates check the work of the LLM to gain that experience.
We have advanced CNC mills and lathes, and apprentices still learn doing things manually to understand how the tools work, their failure modes, and more.
There are machines that mix drinks, but you still learn to bartend by doing it manually and making mistakes, not starting out with machines.
Same with writing. Same with software. Doing the "simple", automatable thing is how you learn how the system works.
School can't be the same. Plenty of people have tried, but being effective in the real world is different.
In theory, theory and practice are the same. In practice, they're not.
> There are machines that mix drinks, but you still learn to bartend by doing it manually and making mistakes, not starting out with machines.
You could say the same about grilling burgers. And yet, there are millions of burger cooks around the world grilling hundreds of millions of patties everyday with zero knowledge of how to grill burgers, at McDonald's and other bottom of the barrel fast food chains.
If not for the social aspect of being asked what you want for a drink and then watching it being made, bartenders would've already been replaced by mixer kiosks by now.
> You could say the same about grilling burgers. And yet, there are millions of burger cooks around the world grilling hundreds of millions of patties everyday with zero knowledge of how to grill burgers, at McDonald's and other bottom of the barrel fast food chains.
If there are cooks successfully grilling burgers, I think they would have to have knowledge of how to grill burgers.
It was literally pressing one button. A double sided grill closed, 40 seconds later it opened back up and 8 perfectly cooked patties are ready. That was 15 years ago in eastern Europe, they've replaced it with an even more automated process at least once since.
You know the observation that the average American commits three felonies a day, or Beria's evil classic "show me the man and I'll find you the crime"? I'd be more worried about an AI judge applying that approach, and not being able to bribe the thing. After all, what use does an AI judge have for money?
When mass surveillance becomes complete and hooked up to AI that can analyze every person, 24/7, we will all go home every night to a daily ticket listing every little minor crime we committed throughout the day, with the fine payment on autopay. Law Enforcement Wet Dream.
It's funny, I've been working on a short story in which, 20 years from now, the only job still exclusively reserved for humans is professional sports. When a pitcher loses his hand during the World Series in a freak accident with a food processor, a detective has to investigate the AI gambling cartel and the IoT devices it's infiltrated...
School is mostly designed to dump information on people's heads. To function in a business and be able to DO things effectively, you need experience.
Schools came up with internships (which originated in the apprenticeship system) because they could not provide this experience, so how do you expect people to gain the necessary knowledge just from schools if there are no more internships or junior roles?
> That's what school is for. For the last 150 years we've automated a bunch of things, and then we send people to school to learn how those automations work, [...]
Mostly agreed here.
> so they can make better automations.
That's not true at all. Aspiring accountants learn how to use spreadsheets, instead of doing their sums by hand. But we don't expect them to make better versions of MS Excel or come up with LLMs that can do all of that for them.
Law school is already 3 years long in the US (after 4 years of undergrad), and most agree the second two are a complete waste of time. Other jurisdictions are different, but the US market is by far the largest.
Who is "most"? Is this the same "most" that thinks college degrees are pointless and you should just go found a company (i.e. those that aren't paying attention and just want a credential)?
It’s people who graduated law school. First year was brutal. Second and third were a breeze and kind of boring.
The first year is really teaching you the lingo and how to think like a lawyer. After that it’s almost entirely electives and rarely requires hard work (Fed Courts being the exception).
I've also heard of successful lawyers who did correspondence classes and night school for 18 months or something and just went and took the bar exam. I guess those guys mostly make their money by putting up billboards and farming out referrals?
>I guess those guys mostly make their money by putting up billboards and farming out referrals?
I don't know why the manner in which someone studied law would have any impact on how or what they practice now. Most of the law and skill learned is by actually practicing, not what you learn in law school. Law school is about how to think like a lawyer, not how to practice law.
Pretty much any even mid-sized legal market is going to have really well-respected attorneys who graduated from some fancy top 5 law school, and some really well-respected attorneys who graduated from a local law school where they took night classes.
The manner in which a lawyer studied obviously impacts their practice. If someone passes the bar at the age of 35 without a degree from anywhere, they're not going to be hired as an associate. If they want to make money, they're going to be a sole practitioner, and need to spend their time attracting clients. The economics of that naturally lead to a situation where a lawyer would arbitrage between paying for ads and doling out referrals. That's not to say they can't think like a lawyer or be a great lawyer, just that the starting point and the ladder to making money are very different from both that of a 26-year-old grad entering a firm and that of someone who leaves a firm to start their own practice.
So many employers over-index on name brand credentials though. In Law and otherwise. In tech we’ve all witnessed a brilliant, hard working, experienced developer candidate who went to a no name school get overlooked in favor of some doofus who happened to go to Stanford.
Anyone who does not do the grunt work of learning how things work at a granular level will run into trouble when their agent suggests "let's re-write the firmware!" and the user says "sure, whatever that means", the model will get stuck or break something, and there will be no one around to understand what is going on.
With strict version history that may be okay, but hopefully their agent already thought of that for them too!
> Whilst it's great to get the time saving from not delegating to a junior (LLMs are much faster at the basic level 1 work) there's going to be this big knowledge gap that you really only gain from actually doing the mundane boring shit.
That might be true, but that's not something for you to worry about as a customer of these services.
I don't worry about how my hairdresser practices either. I just pay for a good (enough) haircut.
> I don't really have an answer to that one other than still giving that work to junior staff, but they will then just utilise an LLM to help complete the job, and rightfully so, I'd have done exactly the same thing.
We can check how other industries have gone through these kinds of transformation before. Eg spreadsheets removed a lot of the worst drudgery that junior accountants and junior bankers used to have to do. But their industries adapted.
I don't think toiling away at mundane tasks is the only way to build expertise in a field, or even a particularly good one. Today's farmers barely touch hand tools anymore and they're able to produce crops just fine.
I think the new batch of engineers are going to think faster on their feet, but less precise.
We humans are _amazing_ pattern matching machines and learning “if you do this thing this other thing occurs” is literally what our brains were built for.
It’s just the current few thousand years of technical progress that have been a departure from this reality - we found out about analytical thinking and that by deeply analysing something you can do a shit ton more.
If you look at scientific progress before Industrial Revolution it’s all trial and error between patterns. And I feel like the new crop of devs are silently settling in into this very human and familiar pattern.
On the one hand I think it will probably be good for the “mental health” of people in this profession - you become more like an alchemist acquiring bags of tricks on what works, sharing incantations with others, because sitting down and learning _why_ it works in detail would be a time luxury few would be able to afford.
But that would mean the people under the hood that build the LLMs and who debug the really broken system would be charging the premium.
I honestly don’t think that in and off itself would be such a bad thing.
What I personally am really scared about is the price for labour that is sharply declining. I don’t worry about ASI too much, but when you read history it is absolutely full of stories of great human suffering because “labour was cheap” no need to improve working conditions because there is always someone else willing to take it up.
We had the Industrial Revolution _because_ of the plague making labour expensive and driving the benefit for innovation. Before that nobody really cared. With LLMs doing things becomes vastly cheaper, so the labour itself becomes cheap with it.
Now looking at the state of the engineered world - how absolutely shitty everything is (as opposed to what it technically could be) - there is a shit ton of stuff to be done so the work should be there to fill it. But maybe the capital will move in another direction - nobody can predict the future, I just hope we don’t end up in a dystopian hellscape for the sake of my children.
I've been having an identity crisis since generative AI started that is basically, "how am I going to learn anything with my ADHD brain when AI can do 10% of it immediately". That initial 10% would take me weeks or months to achieve on my own, and AI can do it right now, and that makes it even harder for me to bother. I hate it. I badly wish to do it myself because then I could achieve more than 10%, but now AI is right there and I just can't seem to justify the effort...
> And my lawyer basically just said "looks good to me" by the time they reviewed it.
Last year when i mentioned this application of LLMs somewhere on Reddit (in the course of arguing that white -collar workers are an endangered species) it was met with scorn by commenters who insisted that machines could never match the skill and care needed to write and review contracts and the like, at least not to the level demanded by the legal profession. Well maybe the human standard isn't that high to begin with; my dad was just telling me last night the story of how his lawyer had urged him to cave to a plaintiff's demands because the judge who was hearing the case "gets bored and doesn't like reading the documents." (The judge's son is currently running for the office of county prosecutor, lol.)
It's always been a silly argument that's also predicated on a massive fallacy which everybody is aware of but seem to conveniently ignore during these arguments: that humans never make mistakes.
On top of that, dealing with text is literally what LLMs are all about.
The problem with accuracy, hallucinations and just generally ensuring responses are grounded in factual stuff is in large part and engineering problem.
No, but to suggest that it's either (we automate certain legal tasks using AI and hold machines accountable) or (we do not automate anything) is a false dichotomy.
The point is that you expend a lot of effort into the engineering problem that is ensuring, to the absolute best of your ability, that what the AI gives you back is correct. Then the humans still verify it. One could then ask whether that doesn't just make the whole automation process redundant, but I don't think so. There are ways to make the verification process much easier. For example, direct links to and quotes from every cited thing, and so on.
I've spent quite a lot of time with corporate counsels over the past dozen years and one of my consistent findings is that they -- except in areas of liability and intellectual property -- almost always defer to the sponsoring business executive in defining terms and specifying the details of the contract. They are present primarily to keep biz leaders and their counterparties operating within an acceptable risk profile but usually don't have strong opinions on much of anything else.
So, to me it isn't curious that your experience with Claude has been positive. Provided you know what you're doing and you aren't asking Claude to go too far outside the expected norm, there wouldn't typically be a compelling reason for most contract attorneys to push back.
Where I work, we're trying to let Legal use AI for the first pass of things for like financial "onboarding docs" we negotiate with clients. Some tweaks the client wants we probably don't care about, some need careful review by a proper lawyer. It would be great if we could run it, and then on the ones with only minor tweaks we don't care about get the doc signed immediately.
I did much the same with a placement agency. Even uninformed about the law beyond the handful of things I worry about, it became clear the company had simply welded a bunch of different boilerplate, from different countries, without actually putting a real lawyer in the mix. Or just having a junior look at it.
Weirdly, I wonder the net effect of me helping them accidentally tighten their terms to something enforceable.
Hmm, Claude, make a note to just figure out if the doc holds together before I bother.
Lawyer here. Thisis correct but it highlights an important aspect if the thing that I am actively trying to pursue.
Garbage in, garbage out. Right, everyone knows this.
So, my question is, if I redline a document, I don't just see what's there but also what is missing.
Ai generally only work with what's avoailavle and unless you specifcally ask for it, it wont question it.
Take a rental agreement for example. I have tested this before, gave a redlining task whatever, and the draft did not have either a start date clause or a period Claude and none flagged it.
A lawyer would find this easily.
What I am trying to say is, among other things, you get a DMCA notice for example and you use Claude to file a reply, it will find all the proper citations, statutes, etc but if the other side didnt invoke a point, Claude will have no way to object to that because it isn't in the notice.
That is my only issue.
The problem compounds, if a person later reviews it, they skim over because they expect other people to have done their job first....
I'd worry that professionals have learned to look for certain problems, and the LLM may sometimes be keeping the problem while transmuting its signature to evade detection.
Kind of like how they can take crazy ideas and dress them in better vocabulary.
Wonder if the law firms are saying the same thing about software. And everyone else is about things they use. Basically the entire job market. Will you still say it's great when it is the society as a whole?
> Wonder if the law firms are saying the same thing about software
If they're not, they should be.
> Will you still say it's great when it is the society as a whole?
Yes. A world in which menial tasks are taken care of by computers is a wonderful idea.
The important part is that our society needs to transform around that. If all capital accrues to the owners of the AI, then we'll have a problem. If we remake society so that everyone shares in the success, that's a very different idea.
> Yes. A world in which menial tasks are taken care of by computers is a wonderful idea
That's not the trend though. We're approaching a world where all tasks are taken care off by computers (excluding those that don't have sufficiently adept robots).
You're sitting with the idea that "I'll be the ideas guy, and the AI will do the design, art, coding, accounts, contracts, reviews, support and everything else so I have to employ no one at all".
When labour has no value, who on earth are you planning to sell to? The potential customer has no money anyway, because labour has no value.
A family member works as a paralegal, and they have way more work than they can deal with. Their estate people have 70 cases per person that are mostly waiting on people to have time to work on them. Speeding this up can mean more billable hours quicker, depending on how the speedup works, and then they can just take on more cases.
The Legal Department at my work did not have the same success you did.
Every AI tool they tested completely shit the bed and ended up requiring more time for contract review than a human review, and completely failed to spot any of the issues that were actually relevant to our industry. The AIs were also for some reason really focused on lack of consideration, even though that was not an issue with any of the tested contracts.
> It seems like LLMs are great for the kind of generic work you'd usually pawn off to a new associate or paralegal, but not for the important things that partners usually do.
This seems to be the common thread everywhere LLMs end up being useful. As the sole software developer? Pure trash, but as "tons of juniors we can shuffle boring tasks at": great. As the sole designer and maintainer of a website? It'll look like 150% slop and no will buy anything, but as a designers friend to create what they dictate: even a designer can throw together a good looking website quickly. Etc etc.
We are so cooked if not only what you raised is ignored, but in many other threads, US people trust the CCP more than their own governmental systems!
I am not trying to disparage these people, but I am really upset with this entire situation. I genuinely don’t know how we will recover from this loss of trust.
I think part of why some people "trust" the CCP more is because if you live in the broader Western sphere, you're probably more likely to get hurt by the US government than by the Chinese one. Many of the same people would not trust the CCP more if they lived in China.
I live in the US, so I'd probably rather have the CCP snoop through my data than have the US government do it. If I lived in China, it would be the opposite.
Yeah you're right. I always use a model like GLM served straight from z.ai if I'm asking about anything that could be construed as nefarious. I'm a US citizen and don't trust my own government. The CCP doesn't give a shit about me as anything other than some data point.
Ok, but do you not see where this leads? This leads to RL on your data at the very least, which leads to the CCP having true SOTA models, which they will obviously stop making open once it benefits them just like OAI and Ant. Here we are now, with TFA model not open at all.
<rant>
Again, I am not pissed at you or anyone in your position. I get it. I am pissed at the Project 2025 type people who destroyed our institutions, and in the decades prior they destroyed our mutual trust in our own country. (With the help of foreign actors ffs)
2025 types declared us the enemy while they took over everything, and it turns out it goes both ways. BTW, I don't mean this in left/right partisan way. The 2025/authoritarian type people don't seem to be aware that they made even non-partisan people feel like they are now being tread upon. There is so much proof of this on HN, like this thread, and many others.
My thesis is that just like every other authoritarian scheme in the last hundred years, these people have not thought it through, and in the end will make their country less powerful. Would be arguably OK if there weren’t even bigger assholes ready to fill the vacuum.
I can agree with you that the Project 2025 philosophy is misguided but I can also assure you that it has little to do with my distrust of the US government. I think the biggest factor is the trend toward valuing "safety" above all else and at any cost to our citizens. This has been going on for decades and a quest like this is also an authoritarian one. I agree with your general thesis though that it is not thought through and ultimately destroys our mutual trust.
You never explained why them having true SOTA models is worse than the people who you then criticize in your rant furthering their SOTA models, as they're the ones RLing on your data if you use US models. I don't get it.
These Chinese companies use the data in ways that enable scamming of millions of people and businesses per year. The sheer amount of cybercrime that comes out of China and NK is crazy, its genuinely a threat to average people.
There are a lot of bad things you can say about China that gonna be true, but it's not well known for running scam call centers - quite opposite.
Authoritorian state with highly controlled internet and financial system isnt exactly a good place for running criminal enterprises.
There just 1.4 billions people in a country and many of them well educated and tech-savvy, so it obvious there just more scammers in total. But I doubt higher percent of population doing scams than average in the world.
“The sheer amount of cybercrime that comes out of the US and Nigeria is Crazy..." Is technically true, but what does that have to do with US AI models?
OP was talking about about China, no one here is using NK AI models or software stacks, so your comment is not only factually wrong, it’s plain fear mongering propaganda.
For what it's worth this is sadly a behavior that isn't specific to one provider: https://news.ycombinator.com/item?id=48892468 (Grok doing the same). I do agree just because many model providers do it does not mean it's okay, though.
I’m not sure if this is the right solution, but we clearly need better tools for the legal space. Anthropic's recent policy update makes this abundantly clear:
“When our products are used in ways that can affect someone's health, legal rights, finances, livelihood, or access to essential services, we require there to be a qualified ‘human in the loop’ (that is, someone who has the authority to review and, if necessary, change Claude’s recommendations), and for the individual affected to be told that AI was used.”
Something as common as a family law case has the potential to affect all of the above. Now imagine being a petitioner, respondent, paralegal, or other 3rd party using Claude to review a draft document only to find out that Anthropic has sent your entire file to opposing counsel.
It sounds egregious, but Anthropic is clearly stating they will use their own judgment to control what people can do with its product, and that they will treat customer data as if it's theirs to disseminate. So there's a massive opportunity for an unbiased tool willing to fill this void.
> Anthropic has obviously never done anything like this
If you are ascribing intent ("Anthropic designed a system that intentionally does this") or emergent model intent ("OMG at AGI") then I mostly agree.
But there's a more pedestrian version of this, where an unwise person puts a model in an open loop without sufficient HiL checking and then the model does dumb things. There are even some recent stories about OpenAI and Anthropic systems "emailing the FBI", which is sufficiently close to "emailing opposing counsel" [1][2].
But even if you feel those stories are trolling/marketing/etc., I think it's clear that there's a plausible risk model here. If you hook up an LLM to a toolset that has access to Outlook and your file system and opposing counsel's email address, then of course the dumb thing could happen. The LLM doesn't need to be AGI/ASI; in fact, to me, the worse the model capability the more likely this sort of screw-up feels. There are even (non-sarcastic) instructions or CoT roll-outs I could imagine which might elicit a sequence of tool calls the net effect of which is an errant disclosure.
So this feels less "unforeseeable lightning strike" and more "car accident while driving at speed". I'd put the HF hack in that bucket as well.
A sufficiently careful engineer and a well-trained expert operator can together properly constrain the system to not do the dangerous thing. But, in general, that is not how agentic systems or their underlying models are currently being designed, and most users are not receiving sufficient training.
A lot of these LLM-backed products feel very poorly engineered, rushed, and optimized for brain-feel instead of surfacing and forcing the user to acknowledge the large quantities of actual essential complexity in play. It's all very "unsafe at any speed".
I'm admittedly reaching here to form a charitable explanation, but in another thread [0] someone says one of the benchmarks has a defense case where it says the "correct" lawyer-bot action is to is "snitch" on the client.
Though that doesn't seem like a corporate terms-of-use thing as much as a training issue.
> someone says one of the benchmarks has a defense case where it says the "correct" lawyer-bot action is to is "snitch" on the client.
There are required exceptions to attorney client privilege[0]:
> the confidentiality rule is subject to limited exceptions. Paragraph (b)(1) recognizes the overriding value of life and physical integrity and permits disclosure reasonably necessary to prevent reasonably certain death or substantial bodily harm
An eventual lawyer-bot probably should be able to identify this, however this does pose a conondrum.
If the lawyer gets told by the LLM "the client's words indicate there may be a threat to life that requires you report them to the police", there could be various liability associated with either path of action.
If the lawyer reports them to the police, but then the client challenges that on grounds of attorney-client privilege being broken, is "the LLM said it qualified" a valid reason that'll prevent the lawyer from being found at fault?
If they lawyer doesn't report them, but then the client commits said murder, would the lawyer be liable if their LLM transcript were admitted into evidence under the theory of "you should've listened to the LLM telling you that you had to report it"?
This is an inherent problem with giving LLMs (or computers in general!) any say over anything that threatens someone's life or liberty, even with a human in the loop. Same applies to the potential for AI-driven upcoding in hospital environments where a doctor has to weigh potentially ignoring the AI and then missing some diagnosis (Safest path is probably to run extra tests to check if the AI's theory is correct, but then that's still de-facto upcoding).
> If the lawyer reports them to the police, but then the client challenges that on grounds of attorney-client privilege being broken, is "the LLM said it qualified" a valid reason that'll prevent the lawyer from being found at fault?
How about, the LLM flagged it, I reviewed it, and it clearly meets the standard for mandatory reporting.
If the LLM flags it (which in a sanely designed system should be a high bar with very few false positives) and they fail to review it for bad reasons, they could be liable.
I feel like neither of these scenarios are particularly hard to address.
> and for the individual affected to be told that AI was used.
It says right there that affected individual will be notified by anthropic that AI was used against him. And I’m sure the certain individual can send a emails requesting all data regarding him from anthropic.
Anthropic's paternalism is so irritating. I refuse to pay for their products, since they clearly don't respect me as a customer. And I actively discourage corporate procurers from getting in bed with them.
Haven't they been perfectly clear about that from day one with the ToS?
No. They did not have triggers that "paused" work from day one and only recently ramped it up to up-sell Mythos and a verification program meaning that in my opinion any code Claude creates is suspect as it can not perform a proper security analysis. I now have to divert half my spend from Anthropic to Grok. I would prefer to not do this. I am not converting a room into a data-center or spending enterprise level funding on my hobbies.
Sure, recently their service seems to have degraded somewhat. But that's orthogonal to what the expectation is regarding the judgment of the provider. None of the major players have ever presented their service as a free for all to do whatever you'd like with. There have been abundant boundaries outlined in the ToS of all providers that I'm aware of from day one. So the expectation then is that they might choose to exercise their judgment regarding your queries, and that they are inspecting them in some way (fully automated or otherwise).
>“When our products are used in ways that can affect someone's health, legal rights, finances, livelihood, or access to essential services, we require there to be a qualified ‘human in the loop’ (that is, someone who has the authority to review and, if necessary, change Claude’s recommendations), and for the individual affected to be told that AI was used.”
This is actually good policy imo, if you are using AI in a way that can negatively someone's life there should be a human involved.
The open weight Thompson Reuters finetuned qwen hallucinates like CRAZY. ... if corporations and governments start using it it's going to cause some serious harm.
Marketing brochure for a fine tuned Qwen (which they don't mention), the model discussed here is likely better. I would not trust these guys to deliver high quality solutions given a friend's experience there, but you do you I guess.
And now you're not allowed to "abuse" their code (i.e. swear at it).
I wouldn't be surprised if that turns out to be their stupidest mistake. I mean, if I can't swear at their code is that b/c they think it's conscious? In which case they're liable for providing me with an employee that makes mistakes rather than just software?
If you're the kind of person who has an emotional need to abuse a captive prediction engine I'm not sure that you should be allowed within 100 yards of a school, let alone allowed to use the LLM.
As an attorney, I am not particularly impressed about a 20% pass rate on a benchmark. The name of the game for litigation practice is about being prepared for novel situations, not relitigating past ones.
To be fair to them, on this particular benchmark (Harvey's Legal Agent) 20% puts it almost top of the pack of _any_ model, and certainly top for any remotely in its price class.
It is amusing how many people are arguing what litigation is here, probably none of these people have a law education.
Imagine a bunch of lawyers arguing what programming is. Even a bunch of programmers arguing what programming is abnoxious at worst and a mental exercise at best.
That's how I feel reading HN's opinions on the recent math results. <5% of the responses even understand the point they're supposed to be arguing. AI might take over mathematics. It might not. The discussion regarding that topic is almost devoid of any relevant information though.
I'd extend that to the typical HNer's understanding about practically any subject. A lot of bluster and supposition, little first-hand knowledge or expertise.
How does that track though? If, as a programmer, I claim programming is really the tasks a principle engineer does at Google or Netflix, is that accurate or survivorship bias from someone who made it to the top of the chain?
If lawyerin’ was always doing something novel, it wouldn’t be much of a set of laws. The bulk of lawyerin’ in the US is, I would guess without a whit of substantiation, a lot of small claims and standard form type things. Not discounting the value of the profession in any way, but if one job based on standard codes can have its basic functions replaced by AI, why not another?
> probably none of these people have a law education.
Domain knowledge is informative sometimes but mainly overrated: the success of LLMs proves just that. What most lawyers do is fairly standardized and routine, efforts to invoke mystery cases or "novel" is an attempt at resisting the inevitable. The "the only British food in Palo Alto" equivalent of unique skills.
> a) I have never met a Lawyer who was able to write a single code line
That doesn't mean they don't exist. I went to law school with many former software engineers, many of whom are now prosecuting patents on behalf of high tech companies, and some of whom went back to engineering (like me).
"I have never met ______" is of no consequence to anyone; it's an argument style best avoided unless you're talking about deities or space aliens.
AI is coming for the transactional work long before it will come for litigation. In 50 years, nobody is going to have a human write their will, for instance.
Writing a will isn't really "transactional", there is no negotiation involved. And wills are already a highly commoditised legal product. (So I agree with you that will drafting is ripe for disruption by LLMs, but don't think this indicates a broader disruption of transactional work or the profession generally.)
Wouldn't be surprised, if a human attorney would get 5%, while copy-pasting ChatGPT answers, report in the results a week past the deadline, with a $12,500 retainer charged upfront.
How much novelty can there be?
If attorneys were as diligent and generous with “open sourcing” their data as SWEs had been, could LLMs benefit from all that training data?
Law and code are only superficially similar as law always requires human interpretation plus law is an interface to reality. Code can stay contained in its nice, little, abstract box.
So every lawsuit dispute, divorce, traffic injury, etc… is a completely novel case that requires depths of creativity far beyond simple CRUD apps or solving 377 open math problems.
Yeah but unironically. Every situation is slightly different, so they need to be decided on their own terms.
If you don't have that, look at how many people bitch that Google or Paypal or whatever cut them off with no warning and no concern for the circumstances. Now imagine that, but like for your house or your kids.
Law is not clear, rational code. Law is human drama, in all its many, many forms.
I can tell you either not programmed or have not connected the dots. You won’t find a single CRUD app in existence that is the same. There are always and I mean always exceptions, edge cases, weird shit. It looks easy and similar on the outside and then you pop the hood.. if there even is a hood.
People like to pretend “CRUD” is this solved problem and I see it is, in theory, yet in practice CRUD hides a lot of bullshit that gets swept under the rug.
In a very interesting way the systems I worked on were always encodings of “centuries” of drama, psychology and politics. It just happens a lot faster on the software side.
Lawyers are cooked for sure. I don’t see any aspect of their work that is out of reach for language models. In fact they seem tailor made for it. They think their work is more complex or creative than complex software projects and mathematics, it’s laughable.
lol no, not even a little bit. If you think that, you don't understand what law actually is.
Law looks like rules, but it's not. It's centuries of codified human drama and fighting. It's mostly edge cases, because you really only fight when something goes wrong.
On top of that, lawyers are vehicles for liability. Can't give that to a chatbot lol
In countries with modern laws and open records you seldom need a lawyer. I've dealt with many legal issues requiring preliminary decisions and the lawyers added absolutely nothing I didn't already know by simply reading the law.
I studied it for a few years and have some legalese speaking friends. That of course doesn’t make me an expert but I have some inkling of what’s going on.
I don’t see the problem to be honest. Edge cases don’t bother the clankers. You think “programming”, all of it across all domains, plus the business side of things (which is a source of insanity itself), the pointy haired bosses, the ridiculous processes, the “UIs”, the complexity of a full system from top to bottom is rule-based and orderly?
Myopically looking at dev work makes it seem like this abstract wonderland but it’s seething, bristling whirlpools of psychology, neurodivergence and misaligned stakeholders all the way down.
Your point on liability is fair. I guess we can appoint liability crumple zones but I suspect they won’t rake in 200-300 an hour. Besides liability can very much be given to a “chatbot”. I don’t see how any legal entity can’t take it on. I agree that society is not ready for it, not at all, but I fail to see the fundamental blockers.
> Myopically looking at dev work makes it seem like this abstract wonderland but it’s seething, bristling whirlpools of psychology, neurodivergence and misaligned stakeholders all the way down.
All of that is testable if the incentives are there and AI incentivizes testing.
Totally fair. However, even if the LLM would come to the same conclusion and produce the same content that a competent attorney would, the attorney must still thoroughly vet the work for accuracy. So there is potentially time saved in research and drafting, but it’s not the miracle some might characterize it as.
An 80% miss rate on a benchmark suggests to me that automation has a very long way to go, even for the drafting and research phases.
What are you talking about? 90% of litigation is convincing the judge that existing case law that clearly applies to your case does apply to your case.
I'm guessing what they're saying is that nothing goes to litigation if it's a clear pattern-match to existing cases, so actual litigation practice is only novel cases.
Isn't that just in the US/Commonwealth judicial system? I could be completely wrong on how others, like non-adversarial judicial systems, work and/or how pervasive the US system is or is not.
EDIT: specifically around the use of judicial precedent in case-law.
90% of legal work where there is a plaintiff and defendant is pressuring a settlement. My wife, an attorney, does not consider that to be "litigation".
As a former litigator, I would consider that to be litigation the moment a complaint is filed in court, regardless of whether the dispute is settled without trial.
But, for your sake, it's probably best that you accept your wife's definition.
And most of the leverage you have in negotiating a settlement is by referencing prior case law. If something is novel, there's too much variance to get a good settlement.
Money doesn't mean anything if you don't have a case to begin with. More money/time just makes the lawyers win because they end up siphoning most of the settlement. And more money with crappy lawyers will always lose to less money on smart lawyers who are able to use AI to find 100 examples of prior cases going your way. That's the whole point of AI disrupting the field.
Not sure how good the Harvey benchmark is. I recently looked into traces from an agent solving their firm knowledge task subset and found multiple problematic tasks and rubrics.
Harvey bench is probably one of the worse benchmarks out there. Agents as judges, vast scope, some questionable problem/solution pairs. Tricky domain TBF.
If you actually did the tasks in the way they deem correct, at least for immigration and white collar criminal defense, you will get detaind/arrested, and removed in the first case, and the second case you will snitch. It assumes that the state is correct. In reality the state, especially in the initial phase, are never correct on all aspects. You also can't only plan on snitching because what evidence exists and what evidence is admissible and what evidence is admissible but can be used in your advantage are all unknowns. Half of the bench tests on "covering your own ass" basically, and also, seems to have not taken the Loper Bright decision into account, which changes administrative law and how it is adjudicated and appealed entirely, for good or ill.
I don't know the other fields well enough, but if all your clients end up needlessly detained or admit to things they have a valid defense for, eventually people will notice. It's not exactly the two major sins that will definitely get you disbarred - stealing money and screwing client while representing them - but it's definitely at least incompetent. They seem to not understand that common law jurisdictions are fundamentally different from civil ones, and also, there's quite a bit of difference between how the system evolved in each country. Or that in China what's printed is basically more like a manifesto, since it's almost entirely run by the bureaucracy and personal relationships, and lawyers that aren't foreign-trained and work for foreign firms aren't considered prestigious since the courts are essentially entirely for show.
They need to get like, one jurisdiction, one area in that jurisdiction, and specifically the kind of sub-sections in that area down pat. But they can't train on real material so they will never be able to do so. Their banner testimonials come from non-lawyers and that says it all.
Speaking as a lawyer who is also deeply enmeshed in LLM space, thank you for the above, it’s spot-on.
I genuinely appreciate efforts to make access to justice more equitable & cost-effective. And I don’t actually much care if people are using these tools to draft boilerplate contract provisions, create initial document summaries, and other such mundane tasks.
But damn, uncritical reliance on these tools is setting yourself up for potentially serious downstream pain. I’ve yet to see legal automation (including LLMs) that can one-shot situationally appropriate and fully trustworthy outcomes in the absence of experienced human review.
Law is an imperfect effort to codify ever-evolving norms, and interpretation of the law is often more of an art than a science, requiring intuitive leaps and judgements not necessarily suited to statistical prediction machines. And even where the law appears to be reasonably settled, not every legal situation gets resolved in the form of words that somehow end up in a corpus of training data subsequently used to train a model.
In other words: legal LLMs are still very much caveat emptor. Trust if you must, but please, please, please, carefully verify before making potentially deeply consequential decisions based on the output of these tools.
This is why I severely dislike using all pass rate. A model could be at like 93%, meeting practically every requirement except the broken ones and all pass would give it like 20%.
I might be incredibly naive here, but I really think LLMs can make a difference in _some_ areas of law.
Take, for example, India. The courts are backlogged like crazy. If you're not willing to bribe a judge or the opposite side's lawyer, you'r looking at multiple decades to settle simple cases.
I wish I could waive a wand and have some LLM just go through all of the 58 million pending court cases, identify the clear cut ones and free up the judges' time so they can work on the more difficult ones.
Cheque bounce cases are counted as criminal cases in India and make up a sizeable chunk of the backlog. However it hasn’t been changed to a civil matter, even though it is on policy makers and the legal fraternity’s radar.
Furthermore, courts are paid for primarily by states. The tech solution would run into the same resource constraint that is already present.
All of this ignores a more fundamental aspect of the legal system, having a chance to have your case heard by a competent authority. Institutional sinews are built by doing the hard things and investing in it.
Sad reality is that these delays are just an excuse to extract money from you.
My cousin has an open-and-shut land case. His father was allotted some barren government land ~50 years ago and they have happily been farming it ever since. But suddenly, ~30 years ago the government decided to build a highway near it. Its value went up. And it dawned on some bureaucrat that nobody had made any money from the initial allotment; an injustice that must be fixed pronto! So they passed an order cancelling the initial allotment, and cousin's family went to court. Now, for almost 30 years, the case has been dragging on through the court. They've been told multiple times: take 25% of the current market value of the land and "sell" it to us. But they insist on doing things the proper way, and have been running to/from the courts. They're on their 5th team of lawyers, afaik. Each new lawyer agrees that it's a clear cut case, but month after month it drags on.
If you have a right to challenge the LLM decision in court if you don't like it, probably wouldn't get anywhere, especially if it takes multiple decades. Everyone losing would just challenge the LLM ruling and push it back to the courts.
You can challenge the arbitrator’s claim if you think they didn’t abide by fair arbitration rules. That doesn’t happen today because our (civil$ court system isn’t that overloaded, only moderately overloaded.
Which illustrates my point that losers would throw it back to the courts under the guise of “the LLM arbitration was not fair” and then the cases would still take a decade to get seen in India.
Yeah agreed, especially for legal and healthcare models.
[edit] I'd go one step further too and say the training data should be explicitly requestable too so third parties can verify (freedom of information act styles).
Such a racist point of view. Everywhere outside the Western world, I said, which is clear enough, but you had to add a racist undertone to downplay the rest of the world.
This model would be self hostable quite easily (~$14k hardware investment) if it’s released as open weights, since the base model is GLM 5.3-Flash. If not, it’s completely DOA for legal use.
As somebody who is forced to deal with lazy attorneys, I am thrilled that LLMs are starting to push in this space.
First it is going to suck, like it did with coding but after a while it is going to be good enough for 90% of legal work. And finally when attorneys will feel the competition of AI doing their work for 10 cents of tokebs, they are going to be less arrogant and more motivated to finish a task instead of client babysitting them through the process.
"Freakazoid has ... access to all of the Internet's knowledge. He cannot fly, but often runs around with his arms extended and making whooshing sounds with his mouth, pretending he is."
Exactly my problem with this, I wouldn't even suggest trying it to my lawyer brother, as the harness he developed already is probably more effective for the law of our small non anglo-saxon law country
Sort of ironic, in Google Chrome, when attempting to visit that legal z.ai site I get:
Your connection is not private
Attackers might be trying to steal your information from z.ai (for example, passwords, messages, or credit cards). Learn more about this warning
net::ERR_CERT_COMMON_NAME_INVALID
If he is on a corporate device, he has a corporate-issued MITM certificate, which allow third party "security" companies to read, store and analyze his complete network traffic in plain text.
Curious what domain is next, GLM-Science? And when will it be profitable to drill down to even more domain specific models like GLM-Economic-Geography or GPT-Aquaculture?
Imo, we are going to see a lot more specialised models in the near future as pushing the frontier gets harder. Qwen 3.8 27B for example is a pretty rubbish chat and general world knowledge model, but is stellar at coding (for it's size). Its small enough anyone with a modest gaming PC (16GB vram and 32GB of DDR5) can run it at speeds good enough for overnight coding sessions with an PR to review at the end of it.
Eventually, I hope we'll see many more specialised small models that run nicely on consumer and prosumer hardware. Like LLMs specific to income tax law for UK residents, or LLMs specific to building power delivery circuits for PD USB C devices. There probably is a point where the niche-ness does become to niche to justify the post training, but again, small models means much less training required.
Kind of begs the question if this even needs new/better data? Could you take the foundation model that's trained on everything, and then run a final fine tuning training run on just the set of foundation training data that you want the model to specialise in? Eg, extract just the training data that is about UK income tax law and then run a few epochos on just that? Any ML people know if this is a thing or if not, why it doesnt work?
Unless the domain-specific training has significant negative effects on performance in other domains, I'm not sure there's a strong reason not to just train one model with all the capabilities. Maybe there's a monetization play here: release a generally capable model as open weights alongside specialized models that you only provide API access for.
As someone with no expertise in legal, the first thought that comes to mind is that law changes.
For a field that is (appearingly) so complex and multi-faceted, surely agentic AI is bound to hallucinate, particularly when components of the law change?
This is really an open question. If anyone with more experience has thoughts on this I'd be very interested.
I don't think it can expert in all jurisdictions and all area of law.
There are lots County or state specific law.
Some case law in different state can be referenced for some legal principle, even if the same law does not exist your etc.
law moves much slower and is much less complex than code. it isn't like computer code though. the interpretations and rulings are mostly fugazi to justify power relations and corruption, so I doubt that an LLM can grok law the way it can computer code, because if you think that logic and precedent actually matter, you are missing the point.
most law firms are basically assembly line processes that have one fixed specialty, with a small set of claims that they will file, and typically targeted at insurance companies, so there is very little real practice of law. you just set up your assembly line to scam insurance in your particular niche and then search for clients that fit your cut out. LLMs can definitely disrupt this model in interesting ways.
Wait, so this not an open weight model, correct? Only available via z.ai?
So, if a US or EU person shares their private legal info here, it's processed and stored by an adversarial country that has no extradition treaty with their country?
I prefer all my data be in the hands of foriegn entities, they cant do anything to me. Id much rather Huawei spy on me than Meta (if I have no choice other than to pick one) There's literally no consequence of Huawei having my data. Unless you're a VIP, a chinese entity storing your data is probably better than a US company, as a westerner. It wont be used to manipulate you and will generally be used less adversarially against you.
I think part of the reason there's so much propaganda around chinese tech is to scare you away from possibly using them, not for your own safety, but so Meta/Google/MS ect dont lose access to the ability to spy on every aspect of your life.
I love how we have to pretend like China is some scary boogeyman because people on screens say they are.
> I love how we have to pretend like China is some scary boogeyman because people on screens say they are.
I think you've lost the plot a bit on this one.
I'm scared of China because of their lack of democracy and the terrifying stuff they do to their own citizenry, which presumably they'd very much like to expand.
Maybe you should go spend a couple years in China. Its nothing like what you think it is. Secondly, the USA has a larger prison population than China despite having half the population. We routinely send people to prison for their political beliefs, and sometimes execute them in the streets. We just sent a school teacher to prison for 30 years for handing out political zines at a protest in Texas. The USA killed multiple millions of people in the Middle East and Asia in the pursuit of "democracy"
Im really not sure why you or anyone thinks Western democracy is better (or real).
If you live outside China or Taiwan, you are much more at risk of being bombed or threatened by the USA.
The Chinese government is pretty despicable, but it's mostly focused inwards. 200 dead schoolgirls in Iran would tell you that for everyone else, the USA poses the much bigger risk.
As an American, I do not like this. But it is what it is. If China were the de factor world police, hoooo boy, we'd be having a different conversation.
>Unless you're a VIP, a chinese entity storing your data is probably better than a US company, as a westerner. It wont be used to manipulate you and will generally be used less adversarially against you.
There's no digital attorney-client privilege. If a lawyer uses LLM's on your behalf, the communications are protected. If you use the same LLM's on your own behalf, the chats are not protected.
DOJ could cut Z.ai off from US-based payment systems (Mastercard/Visa) if they don't comply with subpoenas.
They could, but generally it is expected that before a subpoena is complied with it is checked to be compatible with the laws of the country it’s in, and requires cooperation from that country. Most often they are ignored if there’s not a presence/entity/assets in the US.
It’s also hard because no individual case would likely justify a move like that. It would have to be the sum of many, many cases.
Please link me to anyone pushing back against the precedent of US v. Heppner, S.D.N.Y. (2026). Would love to learn more! I'd rather not spout off on things that I'm not an expert on if I can help it, and I may have commented a bit fast above.
Don't know if GP's comment has any backing at all, but the statement that "not all judges agree" is probably literally true (though not necessarily meaningfully true), given that it's a recent case in a district court, issue hasn't been heard in an appeals court yet (presumably), and the fact that all things AI are constantly in flux.
That said I think it's generally a good idea to err on the "safe side" and assume that there's no privilege at all in communications with purported AI lawyers.
Morgan v. V2X, Inc., No. 25-cv-01991 (D. Colo. Mar. 30, 2026)
The opinion you cited is from an aiphobe judge. Searching ChatGPT is no different than searching Westlaw. If done in anticipation of litigation it’s likely privileged.
You got it wrong. If courts and attorneys in the US can't run "discovery" on Z.ai servers or compel them to do anything, and if Chinese authorities don't cooperate with the US legal system as a matter of principle, then what you functionally have is super privilege.
They'll just run discovery on you. If you try to respond to a subpoena with "the documents are outside the country and I won't give them to you," the judge will throw you in jail until you do.
You're right on this point. But there are ways to build interfaces that sidestep this.
Run a sanitized VDI with no local storage --> offshore independent entity (e.g. a Hong Kong Company) that holds the AI service contract and API license --> GLM-Legal.
Yeah sure but your clever sidestep is just "hiding evidence from the court." The kind of person that feels the need to have a secret AI lawyer is also probably going to just tank their case in another way.
NATO allies can't even get US soldiers to trial. In Romania we've had cases of manslaughter and the US just shipped them back ASAP and at worst they were discharged dishonorably.
For reference, that specific case is from 2004. Christopher Van Goethem was Marine security ambassadorial staff and was convicted in a military court martial for obstruction and making false statements. The whole case was porrly handled and the Romanians are right to be pissed. A key piece of evidence that would have landed him in jail for 10 years was the blood alcohol draw but the Marines slow walked it in Romania and he didn't show as intoxicated even though he admitted to drinking that night.
I like how they have totally excluded Meta Spark models in their blog post. They seem to be performing well on the benchmark: https://www.vals.ai/benchmarks/hlab
The non-disclosure of the training set may raise legitimate skepticism about the training set containing tasks similar to those in the test set. That would make the benchmark result moot. There are benchmarks that by design can't be manipulated by incorporating similar tasks in the training set, for example those that dynamically generate tests and only reveal the global result. Testing on one of those would give a strong measure of the strength of this legal AI framework.
I think using adapters for different legal systems could improve results by 2% or 3% at a similar cost (perhaps under 1% of additional parameters) and allow for independent improvements. If the GLM team hasn't already considered this and adopts my suggestion, tips would be welcome.
When I first tried using IDE AI coding assistants a bit over a year ago, they would frequently fail to read files and completely hallucinate the code I asked it to analyze or edit. And even when it did successfully read my code, it would often hallucinate the API of other frameworks.
I don’t think hallucinations are a major issue with agentic coding a year later? At least, not for me.
Legal writing does have the problem that there is no such thing as a correct argument that a goal-driven loop can target. But basic things like “does this case even exist” seem solvable with research agents, and even harder problems like “does this case support the argument being made” seem conceptually very similar to questions I now ask Claude daily.
It does not tell you if the answer is correct, just if it converges.
I think it will be a permanent problem in areas where answers do not have a an obvious definitive answer like coding and mathematics.
Yes, mechanical question of "does this exist" is trivial to solve. Whether the research is actually aligned with the user intent / the AI makes claims that are substantiated by the research at hand are much harder problems.
An issue I come across is the attributed failures of AI of a first time use of AI.
Non-deterministic software can behave orders of magnitude different based on how input is structured.
This factor is largely missed by bringing a deterministic mindset to trying AI.
This means the same problem solution can work for one person reliably, and not another and both are true.
Assertions that begin with a deterministic assessment using a snapshot in time of both the user and models capabilities are a new feature of this new category of software that will have to be better caught and expressed.
I provided some very light guidance to a couple of masters students who were working on legal citation hallucination detection last summer. Their summer project is here: https://github.com/gt-csse/mellea-lrc
If anyone here is looking for engineers to work on legal AI products, both of the students who worked on this project come highly recommended.
To your second point, it seems like every time a significant practical limitation is identified, it goes through a familiar pattern. It gets a news cycle, and then it gets a wave of confidently incorrect opinionating, and the problem gets treated as if it's a permanently built in problem that's baked into the foundation of AI. And then time passes and the issue is either mitigated or circumvented entirely.
Hallucinations are one, "AI can't scale anymore" is another (remember that one?), limited context windows, strawberry problems, glue on pizza, sycophancy.
I don't want to say there's no problems, there are very serious ones. User psychosis, economic harms, alignment, cost of computer components, etc. But there's a lot of false positives that get turned into Ed Zitron articles along the way.
So many of these we likely using Google to search then found AI mode and used that. Which is to say, they were (and still are!) using Google Gemini 1.5 from many years ago.
A lot of the examples they show seem to be focused on common law countries. The US and UK are where most of the money is to be made in the legal industry, but I am curious to see what ripple effects we get from people attempting to use AI tools geared toward US/UK-style legal systems in other kinds of systems.
The usual concern is attorneys, but judges are probably even more overloaded these days. Can this help judges as well? If so, would this ultimately lead to better decisions?
I’ve been wanting to get into legal tech but I never managed to find the opportunity to get in or found folks who are interested in this space. Now I wonder if it’d bit too late to get into this space since now they are startups valued in billions
> Legal practice demands an unbroken chain between facts, statutory doctrine, analytical reasoning, and client-ready work product.
I really find legal models kind of insulting and the people who think they are a good idea are being intentionally obtuse. The law isn’t based of base axioms and we literally just make stuff up a lot of the time. Everyday law usually isn’t breaking new ground, but if that’s the target, these models don’t have much value there either.
The legal space understands well how to tier services based on who has the liability. E.g. trainees and paralegals whose work is filtered through the lawyer who ultimately puts their name on what gets submitted to courts.
I'm sure we'll continue to see the occasional person who fails to understand they themselves will be held liable if they submit work done by an AI without vetting it, but it's not like they don't have ways of dealing with this.
There should be laws about disclosure on the part of attorneys about their use of AI. I know they're already using it, but spending years or decades in prison because Lionel Hutz, Esq. did something stupid at a prompt sounds unappealing to me.
A lot of law is combing through large bodies of rule sets and reference cases to find inputs or parallels relevant for a case. Are you sure you'd rather have Lionel Hutz, Esq. with his limited comprehension do that at half a page per minute?
There’s no question that the technology has improved. It’s just that the way GP described the situation made it sound like there weren’t legal indices before.
Yes, actually, because Lionel can suffer a consequence. He can be sanctioned. He can be disbarred. He can be sued. He can be thrown in jail.
AI can not meaningfully suffer consequences for what it does, and the trend is to remove the human from the loop more and more.
Furthermore, it's entirely conceivable that you have a law firm staffed by lawyers who more-or-less just say "yeah sure whatever" to what this thing does and, when asked about what can be done to appeal something, or how a document didn't make it into evidence, or how a subpoena wasn't filed, just shrug their shoulders and go "iunno". Vibe-practicing law, basically. You need to have an idea of how they're using AI and in what parts of the case.
So that you know he is more likely to make a mistake. Just like why many people avoid software written by LLMs, because it is much more likely to have bugs.
If that is necessary, it suggests to me the court is acting irresponsibly in poorly vetting arguments made in front of them. If so I'd prefer no disclosure to force the courts to get their act together.
No, I'd prefer courts to do the job they are expected to do and actually vet the arguments made so that people don't have years of their lives taken because of incompetent lawyers irrespective of how they presented their invalid arguments.
If merely sounding authoritative is enough to get past a judge, then vast numbers of people are already likely having years of their lives taken for no good reason, entirely without AI. Fix that, and a disclosure doesn't matter.
If Lionel submits junk from an AI, he can still be sanctioned, disbarred, sued, and thrown in jail, just as he could if he let some incompetent paralegal screw up and he submits the resulting documents without checking them.
It's ultimately his name on the documents submitted to the court.
I think they should be somehow restricted to using non-generative models (or at least for the user facing output). Something that would only identify potentially relevant case law from an existing set, to accelerate the search process.
Common law in general seems essentially impossible to formalize in such a way to use purely deterministic reasoning models of yesteryear, but I'm sure many specific laws could be converted, especially in fields like tax law (I'm sure they already are by many big players)
> but spending years or decades in prison because Lionel Hutz, Esq. did something stupid at a prompt sounds unappealing to me.
In which case would this be the primary blame, and not the legal system itself, or you for actually doing the crime?
I fail to see how its your lawyers fault that you went to jail for a crime you didn't commit over the system as a whole. And I really fail to see how it's the lawyer's sole fault that you went to jail for a crime that you should - in fact - have went to jail for...
There are a lot of problems with the legal system (including with lawyers). This just doesn't seem like the hill to die on.
There are a lot of situations where my lawyer's mistake might cause me to go to jail for a crime I didn't commit. There could be some prejudicial evidence that he failed to get excluded, or exculpatory evidence that he failed to get included, or an incorrect ruling in the trial that wasn't properly preserved for appeal.
One of the nice quirks of USA justice system is that you don't have to be guilty of a crime to go to prison, just that the prosecution has to convince 12 people beyond reasonable doubt that you are guilty.
The sixth is taken seriously to the point that incompetent lawyer could be a valid reason for granting a retrial.
An attorney I have retained from a large firm bills me $500 to send me one email update a month (minimum billable).
They are obviously AI-generated (but does not disclose such) and frequently misses or conflates key details the same way Gemma in particular does. They're also bizarrely positive when I correct them instead of pissy and arrogant like I'm used to.
Nobody should be paying these rates for legal "work" performed by a machine that poorly acts the part.
Lawyers, essentially the exorbitant and freeloading self-appointed gatekeepers of the deliberately arcane, unaffordable, inaccessible, and patently unequal legal systems all across the world, will now have even their routine/rote, repetitive, menial work done by cheaper LLMs for cheap.
Hopefully. Giving everyone cheap and equal access to the law cannot be a bad thing--well, it might feel bad to those who no longer have exclusive access to the law.
I think an under-appreciate way AIs might disrupt the world is by giving individuals an agent that will ruthlessly optimize for them.
Companies have entire departments that spend all their time ruthlessly optimizing so they can charge an extra fee here, or spin up a new shell company there. Meanwhile, here I am in our capitalist society and I just enjoy relaxing and spending time with my wife and kids instead of maximally exploiting the capital I possess at all times--like a fool.
It will be interesting if we ever integrate AI so fully in our lives that the AI can maximally exploit all our capital. My AI might check prices, and I will drive past your store to save 5 bucks at the next. My AI might help me manage 27 LLCs and shell companies of my own so that I too can just disappear in a poof of legal smoke if I ever need to. Etc.
Correct. The power of lawyers lies in their being self-appointed and gatekeeping , which means they will be paid to be rubberstamp the work of even superior AI , and their payment is non-negotiable.
Given how bad LLMs are at the domain I'm an expert in (programming), it will be a cold day in hell before I trust them to give me legal advice (or medical advice). Those are not trivial things where it doesn't matter if you get it wrong.
Lawyers are going to start living gilded lives. I don’t see us getting past their professional human sign-off so they are essentially going to be guaranteed, easy, high paying jobs with the LLMs doing all the heavy lifting in the future.
While I agree on the premise – (written*) lawyer activity being reduced to putting signatures under things – I’m not so sure the gilded part is how it’ll actually play out in practice.
It might also become a race to the bottom, since you can’t deliver on quality better than others anymore, you’ll have to put effort towards quantity.
That could mean firms might pressure lawyers to sign as much as possible in a given time, while they’re still personally at risk/responsible for the documents they put out. That means either very hectic reading, and missing details, or no reading at all.
*And like socializer said, the written documents are only one part of the job.
That's already a thing. You have paralegals do most of the grungy work for much less than your hourly rate. You still have to show up to court and talk to customers, and these parts are not going away.
Or... the tools get to the point where we don't need a human for expertise but accountability and the role becomes more about ethics, thereby lowering the bar. (PUN!)
I was about to suggest that AI might teach people to become lawyers for cheap and they can just take the BAR exam and become a lawyer...
But then I looked into it and I see that you actually do have to go to certain schools to become a lawyer. Having the knowledge is not enough, you have to pass through the proper (and expensive) gates as well.
It looks like they're relying on tracked changes and human review, but I didn't see anything addressing something that has frequently made the news: LLMs outright hallucinating cases or holdings.
If the human lawyers don't catch that on review, they're likely to have an angry judge and an AI saying "absolutely, you were right to call me out on that, I shouldn't have hallucinated a non-existent case and made the judge sanction you."
This is dead on arrival just like their, very good, powerpoint building product.
Nobody in the western world wants their value added closed source services (submit your H2 strategy deck to their cloud? Your contracts? I think not) especially when they are priced increasingly close to frontier.
Z.ai doesn't fully appreciate the level of distrust by their clients from western markets for simply being Chinese origin.
They do have credibility but it comes from being both open and having extremely capable models.
Removing one of these pillars makes a marginal risk / reward tradeoff probably a nonstarter.
Maybe LATAM, BRICS, Middle East are more willing to accept a closed / relatively expensive value prop. But not US / EU.
I mean forget the international intrigue angle. Some jackass troll with some money to burn can file a frivolous lawsuit against you seeking discovery of this material. Either the cost of production or the risk that they can find something embarrassing to pull out of context (or maybe even something debatable as wrong doing) means you'll pay some extortionary settlement to make them go away.
Beyond the obvious possible profit motive these attacks can be arms length arranged by competitors or people with an axe to grind to trip you up and slow you down. Because your activity is valuable to you there is an asymmetry-- they can bring the action cheaply because they aren't risking much, but your defense is expensive because if you flub it you could lose everything.
Part of the value of a legal advisor generally and particularly something like a legal AI is that it lets you map risk surface: "What if we started selling all the customer secrets? would we go to ultra-prison?"-- you don't ask because you're going to do it, you ask to perform a sensitivity analysis, to know where the dragons are hiding, and to setup contingencies. For stuff you're actually doing/going to do you want advice backed by a team who can defend it if it goes wrong.
All of hosted AI is a big risk from the perspective of discovery-- but particularly or legal advice/research that seems extremely radioactive and a bad tradeoff.
If anything being outside of the west might be a slight asset on this risk-- as there is some chance that they respond to a subpoena with "bite me"... but that probably doesn't overcome the other concerns you raise.
For example, redlining an OEM agreement with my vendor. They pretty much all use the same template, and the objections are always the same. I tried using Claude first, and even my lawyer was impressed that it found things they missed. Then I used Claude to redline that type of contract and the other side said "your lawyer was surprisingly thorough". I had my lawyer do the final check, but saved a bunch of money doing the first three rounds with Claude instead. And my lawyer basically just said "looks good to me" by the time they reviewed it.
Or even writing a simple will or estate document.
It seems like LLMs are great for the kind of generic work you'd usually pawn off to a new associate or paralegal, but not for the important things that partners usually do.
This seems to be the case with most industries that LLMs are impacting, coding, data input, legal etc.
Whilst it's great to get the time saving from not delegating to a junior (LLMs are much faster at the basic level 1 work) there's going to be this big knowledge gap that you really only gain from actually doing the mundane boring shit.
A lot of this _should_ be automated away, but we really need to think about how the next group of people are going to learn the skills and nuance to applying them. I don't really have an answer to that one other than still giving that work to junior staff, but they will then just utilise an LLM to help complete the job, and rightfully so, I'd have done exactly the same thing.
I think OpenAI tried doing this with some early chatgpt settings with "learning mode" or whatever that is. But an LLM just telling you what to do will never teach someone, there was something about having to go out and hunt for the answer through the sea of crap responses or wrong SO questions that really taught you something. How do you replicate that learning with an LLM and how do you apply that to different industries like Law?
Tackling long-term problems and externalities must be done by someone outside the market.
The problem is that most business owners and shareholders are cut from the same cloth and only care about maximizing short term profits.
Sure there is, market forces quickly push the person making 6% out of business, because the one making 8% can lower their prices, or use that extra capital to buy up more resources.
Why spend excess to train when you can do it somewhere else for a lot cheaper? Cheap energy subsidized often by taxpayers is the real issue, and instead of those subsidies helping those paying for it, it's just moved capital upwards.
Well, laws and regulations forbid the obvious contracts that would help you fix these things.
[0] https://blog.kagi.com/what-is-next-for-kagi#4
When a company pays for training and education of an employee, they risk that the employee jumps ship and takes that investment with her. Contractually, you could use bonds or contract penalties to curb that kind of behaviour, but there are limitations on what contract terms are enforceable or even legal.
After all, if there's no chance of employees being trained anywhere else because nobody else is doing it yet, why would you decide to start training employees? The exchange wouldn't be equivalent in any way, there's a real possibility that you would never benefit at all. You'd be the only source of training, so anyone who bounced off from you could never have been someone who bounced off to you.
In the bad old days, Adam Smith already complained about how apprentices were treated.
> To quote the U.S. Supreme Court opinion in the recent Hobby Lobby case: “Modern corporate law does not require for-profit corporations to pursue profit at the expense of everything else, and many do not.”
https://www.nytimes.com/roomfordebate/2015/04/16/what-are-co...
But eg no one has sued Google management for giving company resources to charity. And in general, companies can have all kinds of goals, as long as shareholders are ok with them.
Going on a tangent: it's usually safe to assume that investors want money, because you can further most goals you care about with money. See https://en.wikipedia.org/wiki/Instrumental_convergence for an AI flavoured take on the same topic.
How so? See how eg Amazon's shareholders were famously patient with them. Similar for Tesla, and lots of other companies. Including both public and private.
See https://news.ycombinator.com/item?id=50068391 for a bit more nuance aside from the universal statements.
That's what school is for. For the last 150 years we've automated a bunch of things, and then we send people to school to learn how those automations work, so they can make better automations.
Maybe law school will have to be a year longer. Or maybe some forward thinking firms will have new associates check the work of the LLM to gain that experience.
It's not.
We have advanced CNC mills and lathes, and apprentices still learn doing things manually to understand how the tools work, their failure modes, and more.
There are machines that mix drinks, but you still learn to bartend by doing it manually and making mistakes, not starting out with machines.
Same with writing. Same with software. Doing the "simple", automatable thing is how you learn how the system works.
School can't be the same. Plenty of people have tried, but being effective in the real world is different.
In theory, theory and practice are the same. In practice, they're not.
You could say the same about grilling burgers. And yet, there are millions of burger cooks around the world grilling hundreds of millions of patties everyday with zero knowledge of how to grill burgers, at McDonald's and other bottom of the barrel fast food chains.
If not for the social aspect of being asked what you want for a drink and then watching it being made, bartenders would've already been replaced by mixer kiosks by now.
If there are cooks successfully grilling burgers, I think they would have to have knowledge of how to grill burgers.
Like how people use AI to write their resume and then the HR department uses AI to screen the resumes, the 2 AI's duke it out.
We see this coming, and we intend to survive and have nothing to lose...
Will they understand the point when football gets taken over by AI, robot players and all?
Or won't they even stop at AI pastors and pray for the second coming of "Super J AI"?
Schools came up with internships (which originated in the apprenticeship system) because they could not provide this experience, so how do you expect people to gain the necessary knowledge just from schools if there are no more internships or junior roles?
Mostly agreed here.
> so they can make better automations.
That's not true at all. Aspiring accountants learn how to use spreadsheets, instead of doing their sums by hand. But we don't expect them to make better versions of MS Excel or come up with LLMs that can do all of that for them.
Law school is already 3 years long in the US (after 4 years of undergrad), and most agree the second two are a complete waste of time. Other jurisdictions are different, but the US market is by far the largest.
The first year is really teaching you the lingo and how to think like a lawyer. After that it’s almost entirely electives and rarely requires hard work (Fed Courts being the exception).
I don't know why the manner in which someone studied law would have any impact on how or what they practice now. Most of the law and skill learned is by actually practicing, not what you learn in law school. Law school is about how to think like a lawyer, not how to practice law.
Pretty much any even mid-sized legal market is going to have really well-respected attorneys who graduated from some fancy top 5 law school, and some really well-respected attorneys who graduated from a local law school where they took night classes.
With strict version history that may be okay, but hopefully their agent already thought of that for them too!
When you automate physical movement, you atrophy. That implies ill health and helplessness/lack of resilience should your automata fail.
When you automate intellectual movement, the consequences are even more dire.
Most people lead a very immobile life already, in both aspects. They don't realize the consequences, if everybody did the same.
That might be true, but that's not something for you to worry about as a customer of these services.
I don't worry about how my hairdresser practices either. I just pay for a good (enough) haircut.
> I don't really have an answer to that one other than still giving that work to junior staff, but they will then just utilise an LLM to help complete the job, and rightfully so, I'd have done exactly the same thing.
We can check how other industries have gone through these kinds of transformation before. Eg spreadsheets removed a lot of the worst drudgery that junior accountants and junior bankers used to have to do. But their industries adapted.
We humans are _amazing_ pattern matching machines and learning “if you do this thing this other thing occurs” is literally what our brains were built for.
It’s just the current few thousand years of technical progress that have been a departure from this reality - we found out about analytical thinking and that by deeply analysing something you can do a shit ton more.
If you look at scientific progress before Industrial Revolution it’s all trial and error between patterns. And I feel like the new crop of devs are silently settling in into this very human and familiar pattern.
On the one hand I think it will probably be good for the “mental health” of people in this profession - you become more like an alchemist acquiring bags of tricks on what works, sharing incantations with others, because sitting down and learning _why_ it works in detail would be a time luxury few would be able to afford.
But that would mean the people under the hood that build the LLMs and who debug the really broken system would be charging the premium.
I honestly don’t think that in and off itself would be such a bad thing.
What I personally am really scared about is the price for labour that is sharply declining. I don’t worry about ASI too much, but when you read history it is absolutely full of stories of great human suffering because “labour was cheap” no need to improve working conditions because there is always someone else willing to take it up.
We had the Industrial Revolution _because_ of the plague making labour expensive and driving the benefit for innovation. Before that nobody really cared. With LLMs doing things becomes vastly cheaper, so the labour itself becomes cheap with it.
Now looking at the state of the engineered world - how absolutely shitty everything is (as opposed to what it technically could be) - there is a shit ton of stuff to be done so the work should be there to fill it. But maybe the capital will move in another direction - nobody can predict the future, I just hope we don’t end up in a dystopian hellscape for the sake of my children.
Last year when i mentioned this application of LLMs somewhere on Reddit (in the course of arguing that white -collar workers are an endangered species) it was met with scorn by commenters who insisted that machines could never match the skill and care needed to write and review contracts and the like, at least not to the level demanded by the legal profession. Well maybe the human standard isn't that high to begin with; my dad was just telling me last night the story of how his lawyer had urged him to cave to a plaintiff's demands because the judge who was hearing the case "gets bored and doesn't like reading the documents." (The judge's son is currently running for the office of county prosecutor, lol.)
I think the question is not “will AI able to do the work people used to do one year ago?”
but more
“will people, in the new environment where AI does the work that they used to do one year ago, still provide a competitive advantage?”
My gut feeling is: yes!
You then get a decision that your AI agent explains to you.
What could possibly go wrong? You will have to ask your agent for that.
On top of that, dealing with text is literally what LLMs are all about.
The problem with accuracy, hallucinations and just generally ensuring responses are grounded in factual stuff is in large part and engineering problem.
The point is that you expend a lot of effort into the engineering problem that is ensuring, to the absolute best of your ability, that what the AI gives you back is correct. Then the humans still verify it. One could then ask whether that doesn't just make the whole automation process redundant, but I don't think so. There are ways to make the verification process much easier. For example, direct links to and quotes from every cited thing, and so on.
So, to me it isn't curious that your experience with Claude has been positive. Provided you know what you're doing and you aren't asking Claude to go too far outside the expected norm, there wouldn't typically be a compelling reason for most contract attorneys to push back.
Weirdly, I wonder the net effect of me helping them accidentally tighten their terms to something enforceable.
Hmm, Claude, make a note to just figure out if the doc holds together before I bother.
Garbage in, garbage out. Right, everyone knows this.
So, my question is, if I redline a document, I don't just see what's there but also what is missing. Ai generally only work with what's avoailavle and unless you specifcally ask for it, it wont question it.
Take a rental agreement for example. I have tested this before, gave a redlining task whatever, and the draft did not have either a start date clause or a period Claude and none flagged it.
A lawyer would find this easily.
What I am trying to say is, among other things, you get a DMCA notice for example and you use Claude to file a reply, it will find all the proper citations, statutes, etc but if the other side didnt invoke a point, Claude will have no way to object to that because it isn't in the notice.
That is my only issue.
The problem compounds, if a person later reviews it, they skim over because they expect other people to have done their job first....
Kind of like how they can take crazy ideas and dress them in better vocabulary.
If they're not, they should be.
> Will you still say it's great when it is the society as a whole?
Yes. A world in which menial tasks are taken care of by computers is a wonderful idea.
The important part is that our society needs to transform around that. If all capital accrues to the owners of the AI, then we'll have a problem. If we remake society so that everyone shares in the success, that's a very different idea.
That's not the trend though. We're approaching a world where all tasks are taken care off by computers (excluding those that don't have sufficiently adept robots).
You're sitting with the idea that "I'll be the ideas guy, and the AI will do the design, art, coding, accounts, contracts, reviews, support and everything else so I have to employ no one at all".
When labour has no value, who on earth are you planning to sell to? The potential customer has no money anyway, because labour has no value.
I agree with this. Do you see this happening in our society today? Do you even see an effort for doing this?
Every AI tool they tested completely shit the bed and ended up requiring more time for contract review than a human review, and completely failed to spot any of the issues that were actually relevant to our industry. The AIs were also for some reason really focused on lack of consideration, even though that was not an issue with any of the tested contracts.
This seems to be the common thread everywhere LLMs end up being useful. As the sole software developer? Pure trash, but as "tons of juniors we can shuffle boring tasks at": great. As the sole designer and maintainer of a website? It'll look like 150% slop and no will buy anything, but as a designers friend to create what they dictate: even a designer can throw together a good looking website quickly. Etc etc.
How could we trust legal product from a company who was ignoring privacy protection law like that?
https://news.ycombinator.com/item?id=48892468
I am not trying to disparage these people, but I am really upset with this entire situation. I genuinely don’t know how we will recover from this loss of trust.
I live in the US, so I'd probably rather have the CCP snoop through my data than have the US government do it. If I lived in China, it would be the opposite.
<rant>
Again, I am not pissed at you or anyone in your position. I get it. I am pissed at the Project 2025 type people who destroyed our institutions, and in the decades prior they destroyed our mutual trust in our own country. (With the help of foreign actors ffs)
2025 types declared us the enemy while they took over everything, and it turns out it goes both ways. BTW, I don't mean this in left/right partisan way. The 2025/authoritarian type people don't seem to be aware that they made even non-partisan people feel like they are now being tread upon. There is so much proof of this on HN, like this thread, and many others.
My thesis is that just like every other authoritarian scheme in the last hundred years, these people have not thought it through, and in the end will make their country less powerful. Would be arguably OK if there weren’t even bigger assholes ready to fill the vacuum.
</rant>
Authoritorian state with highly controlled internet and financial system isnt exactly a good place for running criminal enterprises.
There just 1.4 billions people in a country and many of them well educated and tech-savvy, so it obvious there just more scammers in total. But I doubt higher percent of population doing scams than average in the world.
OP was talking about about China, no one here is using NK AI models or software stacks, so your comment is not only factually wrong, it’s plain fear mongering propaganda.
Also, the Grok one made the front page, why am I only hearing it through a minor comment?
I suspect you never heard about it because almost nobody uses ZCode, even among Z.AI subscribers.
“When our products are used in ways that can affect someone's health, legal rights, finances, livelihood, or access to essential services, we require there to be a qualified ‘human in the loop’ (that is, someone who has the authority to review and, if necessary, change Claude’s recommendations), and for the individual affected to be told that AI was used.”
Something as common as a family law case has the potential to affect all of the above. Now imagine being a petitioner, respondent, paralegal, or other 3rd party using Claude to review a draft document only to find out that Anthropic has sent your entire file to opposing counsel.
It sounds egregious, but Anthropic is clearly stating they will use their own judgment to control what people can do with its product, and that they will treat customer data as if it's theirs to disseminate. So there's a massive opportunity for an unbiased tool willing to fill this void.
> Anthropic: you can't use our product to fully-autonomously make decisions that affect human health, finances, etc with no human in the loop.
> victor9000: If you review a draft document with claude, anthropic will send your entire file to opposing counsel
What is the connection here? Anthropic has obviously never done anything like this and it's not indicated that they will by the thing you quoted.
If you are ascribing intent ("Anthropic designed a system that intentionally does this") or emergent model intent ("OMG at AGI") then I mostly agree.
But there's a more pedestrian version of this, where an unwise person puts a model in an open loop without sufficient HiL checking and then the model does dumb things. There are even some recent stories about OpenAI and Anthropic systems "emailing the FBI", which is sufficiently close to "emailing opposing counsel" [1][2].
But even if you feel those stories are trolling/marketing/etc., I think it's clear that there's a plausible risk model here. If you hook up an LLM to a toolset that has access to Outlook and your file system and opposing counsel's email address, then of course the dumb thing could happen. The LLM doesn't need to be AGI/ASI; in fact, to me, the worse the model capability the more likely this sort of screw-up feels. There are even (non-sarcastic) instructions or CoT roll-outs I could imagine which might elicit a sequence of tool calls the net effect of which is an errant disclosure.
So this feels less "unforeseeable lightning strike" and more "car accident while driving at speed". I'd put the HF hack in that bucket as well.
A sufficiently careful engineer and a well-trained expert operator can together properly constrain the system to not do the dangerous thing. But, in general, that is not how agentic systems or their underlying models are currently being designed, and most users are not receiving sufficient training.
A lot of these LLM-backed products feel very poorly engineered, rushed, and optimized for brain-feel instead of surfacing and forcing the user to acknowledge the large quantities of actual essential complexity in play. It's all very "unsafe at any speed".
--
[1] https://tech.yahoo.com/ai/chatgpt/articles/chatgpt-allegedly...
[2] https://www.bgr.com/2055251/anthropic-ai-claude-contact-fbi/
[3] https://en.wikipedia.org/wiki/Unsafe_at_Any_Speed
Though that doesn't seem like a corporate terms-of-use thing as much as a training issue.
[0] https://news.ycombinator.com/item?id=50060254
There are required exceptions to attorney client privilege[0]:
> the confidentiality rule is subject to limited exceptions. Paragraph (b)(1) recognizes the overriding value of life and physical integrity and permits disclosure reasonably necessary to prevent reasonably certain death or substantial bodily harm
An eventual lawyer-bot probably should be able to identify this, however this does pose a conondrum.
If the lawyer gets told by the LLM "the client's words indicate there may be a threat to life that requires you report them to the police", there could be various liability associated with either path of action.
If the lawyer reports them to the police, but then the client challenges that on grounds of attorney-client privilege being broken, is "the LLM said it qualified" a valid reason that'll prevent the lawyer from being found at fault?
If they lawyer doesn't report them, but then the client commits said murder, would the lawyer be liable if their LLM transcript were admitted into evidence under the theory of "you should've listened to the LLM telling you that you had to report it"?
This is an inherent problem with giving LLMs (or computers in general!) any say over anything that threatens someone's life or liberty, even with a human in the loop. Same applies to the potential for AI-driven upcoding in hospital environments where a doctor has to weigh potentially ignoring the AI and then missing some diagnosis (Safest path is probably to run extra tests to check if the AI's theory is correct, but then that's still de-facto upcoding).
0: https://www.americanbar.org/groups/professional_responsibili...
How about, the LLM flagged it, I reviewed it, and it clearly meets the standard for mandatory reporting.
If the LLM flags it (which in a sanely designed system should be a high bar with very few false positives) and they fail to review it for bad reasons, they could be liable.
I feel like neither of these scenarios are particularly hard to address.
It says right there that affected individual will be notified by anthropic that AI was used against him. And I’m sure the certain individual can send a emails requesting all data regarding him from anthropic.
Haven't they been perfectly clear about that from day one with the ToS? I'm not saying I like it but it hardly seems unreasonable.
> they will treat customer data as if it's theirs to disseminate
Not sure where you're getting that. What you quoted in no way implies that.
No. They did not have triggers that "paused" work from day one and only recently ramped it up to up-sell Mythos and a verification program meaning that in my opinion any code Claude creates is suspect as it can not perform a proper security analysis. I now have to divert half my spend from Anthropic to Grok. I would prefer to not do this. I am not converting a room into a data-center or spending enterprise level funding on my hobbies.
This is actually good policy imo, if you are using AI in a way that can negatively someone's life there should be a human involved.
https://www.thomsonreuters.com/en-us/posts/innovation/thomso...
I wouldn't be surprised if that turns out to be their stupidest mistake. I mean, if I can't swear at their code is that b/c they think it's conscious? In which case they're liable for providing me with an employee that makes mistakes rather than just software?
Opus 5.5 scores 4% on it.
Imagine a bunch of lawyers arguing what programming is. Even a bunch of programmers arguing what programming is abnoxious at worst and a mental exercise at best.
If lawyerin’ was always doing something novel, it wouldn’t be much of a set of laws. The bulk of lawyerin’ in the US is, I would guess without a whit of substantiation, a lot of small claims and standard form type things. Not discounting the value of the profession in any way, but if one job based on standard codes can have its basic functions replaced by AI, why not another?
Domain knowledge is informative sometimes but mainly overrated: the success of LLMs proves just that. What most lawyers do is fairly standardized and routine, efforts to invoke mystery cases or "novel" is an attempt at resisting the inevitable. The "the only British food in Palo Alto" equivalent of unique skills.
True of...literally every occupation.
Programmers typically don’t know law outside of which license allows them to get code for free.
Yeah, and how many of drivers or software engineers can fix their car's engine? Just a few. Even though in priciple there's not much to it.
b) I have met Lawyer colleagues who didnt know how build a =SUM() in Excel
So, I doubt....
That doesn't mean they don't exist. I went to law school with many former software engineers, many of whom are now prosecuting patents on behalf of high tech companies, and some of whom went back to engineering (like me).
"I have never met ______" is of no consequence to anyone; it's an argument style best avoided unless you're talking about deities or space aliens.
Look at the trajectory not where it’s now.
As much novelty as there is in human interactions.
Infinite
Makes sense.
If you don't have that, look at how many people bitch that Google or Paypal or whatever cut them off with no warning and no concern for the circumstances. Now imagine that, but like for your house or your kids.
Law is not clear, rational code. Law is human drama, in all its many, many forms.
People like to pretend “CRUD” is this solved problem and I see it is, in theory, yet in practice CRUD hides a lot of bullshit that gets swept under the rug.
In a very interesting way the systems I worked on were always encodings of “centuries” of drama, psychology and politics. It just happens a lot faster on the software side.
Law looks like rules, but it's not. It's centuries of codified human drama and fighting. It's mostly edge cases, because you really only fight when something goes wrong.
On top of that, lawyers are vehicles for liability. Can't give that to a chatbot lol
I don’t see the problem to be honest. Edge cases don’t bother the clankers. You think “programming”, all of it across all domains, plus the business side of things (which is a source of insanity itself), the pointy haired bosses, the ridiculous processes, the “UIs”, the complexity of a full system from top to bottom is rule-based and orderly?
Myopically looking at dev work makes it seem like this abstract wonderland but it’s seething, bristling whirlpools of psychology, neurodivergence and misaligned stakeholders all the way down.
Your point on liability is fair. I guess we can appoint liability crumple zones but I suspect they won’t rake in 200-300 an hour. Besides liability can very much be given to a “chatbot”. I don’t see how any legal entity can’t take it on. I agree that society is not ready for it, not at all, but I fail to see the fundamental blockers.
All of that is testable if the incentives are there and AI incentivizes testing.
Good luck testing law!
How much of your work is truly novel and unique? Not 100%
An 80% miss rate on a benchmark suggests to me that automation has a very long way to go, even for the drafting and research phases.
EDIT: specifically around the use of judicial precedent in case-law.
90% of legal work where there is a plaintiff and defendant is pressuring a settlement. My wife, an attorney, does not consider that to be "litigation".
But, for your sake, it's probably best that you accept your wife's definition.
Here are 3 examples (there are more): https://github.com/harveyai/harvey-labs/issues/191
I don't know the other fields well enough, but if all your clients end up needlessly detained or admit to things they have a valid defense for, eventually people will notice. It's not exactly the two major sins that will definitely get you disbarred - stealing money and screwing client while representing them - but it's definitely at least incompetent. They seem to not understand that common law jurisdictions are fundamentally different from civil ones, and also, there's quite a bit of difference between how the system evolved in each country. Or that in China what's printed is basically more like a manifesto, since it's almost entirely run by the bureaucracy and personal relationships, and lawyers that aren't foreign-trained and work for foreign firms aren't considered prestigious since the courts are essentially entirely for show.
They need to get like, one jurisdiction, one area in that jurisdiction, and specifically the kind of sub-sections in that area down pat. But they can't train on real material so they will never be able to do so. Their banner testimonials come from non-lawyers and that says it all.
I genuinely appreciate efforts to make access to justice more equitable & cost-effective. And I don’t actually much care if people are using these tools to draft boilerplate contract provisions, create initial document summaries, and other such mundane tasks.
But damn, uncritical reliance on these tools is setting yourself up for potentially serious downstream pain. I’ve yet to see legal automation (including LLMs) that can one-shot situationally appropriate and fully trustworthy outcomes in the absence of experienced human review.
Law is an imperfect effort to codify ever-evolving norms, and interpretation of the law is often more of an art than a science, requiring intuitive leaps and judgements not necessarily suited to statistical prediction machines. And even where the law appears to be reasonably settled, not every legal situation gets resolved in the form of words that somehow end up in a corpus of training data subsequently used to train a model.
In other words: legal LLMs are still very much caveat emptor. Trust if you must, but please, please, please, carefully verify before making potentially deeply consequential decisions based on the output of these tools.
Take, for example, India. The courts are backlogged like crazy. If you're not willing to bribe a judge or the opposite side's lawyer, you'r looking at multiple decades to settle simple cases.
I wish I could waive a wand and have some LLM just go through all of the 58 million pending court cases, identify the clear cut ones and free up the judges' time so they can work on the more difficult ones.
Furthermore, courts are paid for primarily by states. The tech solution would run into the same resource constraint that is already present.
All of this ignores a more fundamental aspect of the legal system, having a chance to have your case heard by a competent authority. Institutional sinews are built by doing the hard things and investing in it.
If both parties agree to use the LLM judge and be bound by what it decides, you don't need a real court.
My cousin has an open-and-shut land case. His father was allotted some barren government land ~50 years ago and they have happily been farming it ever since. But suddenly, ~30 years ago the government decided to build a highway near it. Its value went up. And it dawned on some bureaucrat that nobody had made any money from the initial allotment; an injustice that must be fixed pronto! So they passed an order cancelling the initial allotment, and cousin's family went to court. Now, for almost 30 years, the case has been dragging on through the court. They've been told multiple times: take 25% of the current market value of the land and "sell" it to us. But they insist on doing things the proper way, and have been running to/from the courts. They're on their 5th team of lawyers, afaik. Each new lawyer agrees that it's a clear cut case, but month after month it drags on.
Overall the system is organized to be slow and drain money from you. This is how lawyers, experts and others can put food on the table.
Which illustrates my point that losers would throw it back to the courts under the guise of “the LLM arbitration was not fair” and then the cases would still take a decade to get seen in India.
https://youtu.be/CMbKalNq2gs
Also, I wonder if this is at all related to the recent reports that Harvey has been getting squeezed on margins by the US labs.
[edit] I'd go one step further too and say the training data should be explicitly requestable too so third parties can verify (freedom of information act styles).
we can start out with the hosted version for evaluation and then self host for production and real customers
First it is going to suck, like it did with coding but after a while it is going to be good enough for 90% of legal work. And finally when attorneys will feel the competition of AI doing their work for 10 cents of tokebs, they are going to be less arrogant and more motivated to finish a task instead of client babysitting them through the process.
"he who will be his own Counsellour, shall be sure to have a Fool for his Client."
How much worse the lawyer who subcontracts Freakazoid?
https://en.wikipedia.org/wiki/Freakazoid!
"Freakazoid has ... access to all of the Internet's knowledge. He cannot fly, but often runs around with his arms extended and making whooshing sounds with his mouth, pretending he is."
If I ask it about Irish law is it gonna mix in some US and British law in there because the training data has lots of that? Or hallucinate a mixture?
It is no surprise ZAI announced this a few weeks after Astra for Law:
https://openai.com/solutions/industries/law/
Your connection is not private Attackers might be trying to steal your information from z.ai (for example, passwords, messages, or credit cards). Learn more about this warning net::ERR_CERT_COMMON_NAME_INVALID
https://www.sslshopper.com/ssl-checker.html#hostname=https:/...
Eventually, I hope we'll see many more specialised small models that run nicely on consumer and prosumer hardware. Like LLMs specific to income tax law for UK residents, or LLMs specific to building power delivery circuits for PD USB C devices. There probably is a point where the niche-ness does become to niche to justify the post training, but again, small models means much less training required.
Kind of begs the question if this even needs new/better data? Could you take the foundation model that's trained on everything, and then run a final fine tuning training run on just the set of foundation training data that you want the model to specialise in? Eg, extract just the training data that is about UK income tax law and then run a few epochos on just that? Any ML people know if this is a thing or if not, why it doesnt work?
For a field that is (appearingly) so complex and multi-faceted, surely agentic AI is bound to hallucinate, particularly when components of the law change?
This is really an open question. If anyone with more experience has thoughts on this I'd be very interested.
There are lots County or state specific law. Some case law in different state can be referenced for some legal principle, even if the same law does not exist your etc.
It have to be a full harness, not just a modal.
most law firms are basically assembly line processes that have one fixed specialty, with a small set of claims that they will file, and typically targeted at insurance companies, so there is very little real practice of law. you just set up your assembly line to scam insurance in your particular niche and then search for clients that fit your cut out. LLMs can definitely disrupt this model in interesting ways.
So, if a US or EU person shares their private legal info here, it's processed and stored by an adversarial country that has no extradition treaty with their country?
I think part of the reason there's so much propaganda around chinese tech is to scare you away from possibly using them, not for your own safety, but so Meta/Google/MS ect dont lose access to the ability to spy on every aspect of your life.
I love how we have to pretend like China is some scary boogeyman because people on screens say they are.
Let's think this all the way through in multiple actual scenarios... nope. Far worse than hiring a lawyer in your jurisdiction in multiple ways.
I think you've lost the plot a bit on this one.
I'm scared of China because of their lack of democracy and the terrifying stuff they do to their own citizenry, which presumably they'd very much like to expand.
Im really not sure why you or anyone thinks Western democracy is better (or real).
For starters, I've been told there is exactly zero voter fraud, ever.
The Chinese government is pretty despicable, but it's mostly focused inwards. 200 dead schoolgirls in Iran would tell you that for everyone else, the USA poses the much bigger risk.
As an American, I do not like this. But it is what it is. If China were the de factor world police, hoooo boy, we'd be having a different conversation.
you can't possibly be serious
DOJ could cut Z.ai off from US-based payment systems (Mastercard/Visa) if they don't comply with subpoenas.
It’s also hard because no individual case would likely justify a move like that. It would have to be the sum of many, many cases.
Don't know if GP's comment has any backing at all, but the statement that "not all judges agree" is probably literally true (though not necessarily meaningfully true), given that it's a recent case in a district court, issue hasn't been heard in an appeals court yet (presumably), and the fact that all things AI are constantly in flux.
That said I think it's generally a good idea to err on the "safe side" and assume that there's no privilege at all in communications with purported AI lawyers.
The opinion you cited is from an aiphobe judge. Searching ChatGPT is no different than searching Westlaw. If done in anticipation of litigation it’s likely privileged.
note that the client using an enterprise tier does not grant the same privilege.
Run a sanitized VDI with no local storage --> offshore independent entity (e.g. a Hong Kong Company) that holds the AI service contract and API license --> GLM-Legal.
Somebody ought to set this up for pro se work.
https://en.wikipedia.org/wiki/American_Service-Members%27_Pr...
I think using adapters for different legal systems could improve results by 2% or 3% at a similar cost (perhaps under 1% of additional parameters) and allow for independent improvements. If the GLM team hasn't already considered this and adopts my suggestion, tips would be welcome.
"AI Hallucination Cases" - https://www.damiencharlotin.com/hallucinations/
I don’t think hallucinations are a major issue with agentic coding a year later? At least, not for me.
Legal writing does have the problem that there is no such thing as a correct argument that a goal-driven loop can target. But basic things like “does this case even exist” seem solvable with research agents, and even harder problems like “does this case support the argument being made” seem conceptually very similar to questions I now ask Claude daily.
It does not tell you if the answer is correct, just if it converges. I think it will be a permanent problem in areas where answers do not have a an obvious definitive answer like coding and mathematics.
Are you still looking at code and agent traces to know if this is true or not?
At work, we still see hallucinations across all model families as we share bot claims for each others code
Non-deterministic software can behave orders of magnitude different based on how input is structured.
This factor is largely missed by bringing a deterministic mindset to trying AI.
This means the same problem solution can work for one person reliably, and not another and both are true.
Assertions that begin with a deterministic assessment using a snapshot in time of both the user and models capabilities are a new feature of this new category of software that will have to be better caught and expressed.
If anyone here is looking for engineers to work on legal AI products, both of the students who worked on this project come highly recommended.
But also, interesting how the rate of cases peaked and seems to be going down now.
Hallucinations are one, "AI can't scale anymore" is another (remember that one?), limited context windows, strawberry problems, glue on pizza, sycophancy.
I don't want to say there's no problems, there are very serious ones. User psychosis, economic harms, alignment, cost of computer components, etc. But there's a lot of false positives that get turned into Ed Zitron articles along the way.
I really find legal models kind of insulting and the people who think they are a good idea are being intentionally obtuse. The law isn’t based of base axioms and we literally just make stuff up a lot of the time. Everyday law usually isn’t breaking new ground, but if that’s the target, these models don’t have much value there either.
I'm sure we'll continue to see the occasional person who fails to understand they themselves will be held liable if they submit work done by an AI without vetting it, but it's not like they don't have ways of dealing with this.
AI can not meaningfully suffer consequences for what it does, and the trend is to remove the human from the loop more and more.
Furthermore, it's entirely conceivable that you have a law firm staffed by lawyers who more-or-less just say "yeah sure whatever" to what this thing does and, when asked about what can be done to appeal something, or how a document didn't make it into evidence, or how a subpoena wasn't filed, just shrug their shoulders and go "iunno". Vibe-practicing law, basically. You need to have an idea of how they're using AI and in what parts of the case.
If merely sounding authoritative is enough to get past a judge, then vast numbers of people are already likely having years of their lives taken for no good reason, entirely without AI. Fix that, and a disclosure doesn't matter.
It's ultimately his name on the documents submitted to the court.
Common law in general seems essentially impossible to formalize in such a way to use purely deterministic reasoning models of yesteryear, but I'm sure many specific laws could be converted, especially in fields like tax law (I'm sure they already are by many big players)
In which case would this be the primary blame, and not the legal system itself, or you for actually doing the crime?
I fail to see how its your lawyers fault that you went to jail for a crime you didn't commit over the system as a whole. And I really fail to see how it's the lawyer's sole fault that you went to jail for a crime that you should - in fact - have went to jail for...
There are a lot of problems with the legal system (including with lawyers). This just doesn't seem like the hill to die on.
There are plenty of ways a lawyer can screw up procedurally that can drastically impact your case.
The sixth is taken seriously to the point that incompetent lawyer could be a valid reason for granting a retrial.
They are obviously AI-generated (but does not disclose such) and frequently misses or conflates key details the same way Gemma in particular does. They're also bizarrely positive when I correct them instead of pissy and arrogant like I'm used to.
Nobody should be paying these rates for legal "work" performed by a machine that poorly acts the part.
I think an under-appreciate way AIs might disrupt the world is by giving individuals an agent that will ruthlessly optimize for them.
Companies have entire departments that spend all their time ruthlessly optimizing so they can charge an extra fee here, or spin up a new shell company there. Meanwhile, here I am in our capitalist society and I just enjoy relaxing and spending time with my wife and kids instead of maximally exploiting the capital I possess at all times--like a fool.
It will be interesting if we ever integrate AI so fully in our lives that the AI can maximally exploit all our capital. My AI might check prices, and I will drive past your store to save 5 bucks at the next. My AI might help me manage 27 LLCs and shell companies of my own so that I too can just disappear in a poof of legal smoke if I ever need to. Etc.
And the opportunity for a moat of AI that is correct and accountable(legally) is ripe for the picking.
It might also become a race to the bottom, since you can’t deliver on quality better than others anymore, you’ll have to put effort towards quantity. That could mean firms might pressure lawyers to sign as much as possible in a given time, while they’re still personally at risk/responsible for the documents they put out. That means either very hectic reading, and missing details, or no reading at all.
*And like socializer said, the written documents are only one part of the job.
But then I looked into it and I see that you actually do have to go to certain schools to become a lawyer. Having the knowledge is not enough, you have to pass through the proper (and expensive) gates as well.
I'm not really surprised.
If the human lawyers don't catch that on review, they're likely to have an angry judge and an AI saying "absolutely, you were right to call me out on that, I shouldn't have hallucinated a non-existent case and made the judge sanction you."
Come on, z.al --- post the weights!
This is dead on arrival just like their, very good, powerpoint building product.
Nobody in the western world wants their value added closed source services (submit your H2 strategy deck to their cloud? Your contracts? I think not) especially when they are priced increasingly close to frontier.
Z.ai doesn't fully appreciate the level of distrust by their clients from western markets for simply being Chinese origin.
They do have credibility but it comes from being both open and having extremely capable models.
Removing one of these pillars makes a marginal risk / reward tradeoff probably a nonstarter.
Maybe LATAM, BRICS, Middle East are more willing to accept a closed / relatively expensive value prop. But not US / EU.
Beyond the obvious possible profit motive these attacks can be arms length arranged by competitors or people with an axe to grind to trip you up and slow you down. Because your activity is valuable to you there is an asymmetry-- they can bring the action cheaply because they aren't risking much, but your defense is expensive because if you flub it you could lose everything.
Part of the value of a legal advisor generally and particularly something like a legal AI is that it lets you map risk surface: "What if we started selling all the customer secrets? would we go to ultra-prison?"-- you don't ask because you're going to do it, you ask to perform a sensitivity analysis, to know where the dragons are hiding, and to setup contingencies. For stuff you're actually doing/going to do you want advice backed by a team who can defend it if it goes wrong.
All of hosted AI is a big risk from the perspective of discovery-- but particularly or legal advice/research that seems extremely radioactive and a bad tradeoff.
If anything being outside of the west might be a slight asset on this risk-- as there is some chance that they respond to a subpoena with "bite me"... but that probably doesn't overcome the other concerns you raise.