I know someone who works in law and deals particularly with an area of US benefits and healthcare law. One of their workflows for lower-level employees at their firm involves taking in documents from healthcare plans and organizations, analyzing them for certain kinds of data, and then importing that data into an internal system they use to analyze and provide guidance on plans. The internal system can contain hundreds of documents for an individual client. All of the documents have the same information (roughly) but in totally diverse formats and styles. Once it's in the system, it's easy to compare and analyze across documents and the research process is much faster.
They recently bought a Claude subscription and began using Claude to do the initial read of the documents and output JSON they can import into their internal systems. The work still must be reviewed by an attorney - Claude is nowhere near making the kinds of judgments a lawyer would make about this content - but it has increased their throughput from 2-3 documents an hour to 8-10 documents an hour by killing the busy work.
LLMs have great advantages for this kind of work - but not for decision-making. I just don't see OpenAI ever admitting that.
(I've left some details intentionally vague because this is a very specific area of law and I don't want my friends to be identified without their consent.)
I just realized how refreshing it is to read an honest take like "from 2-3 documents an hour to 8-10 documents an hour" instead of "it's doing the work of a month in 5 minutes!!!!1".
People will eventually realize that this is exactly what AI does for most workflows. The propaganda is "synthetic humans", "obsolete mathematicians" and so forth, but what we will likely see is a fantastic time saver for things we didn't like to do to begin with.
It's not a time saver in that you, as an employee, don't get more time to yourself. You get to work more. Can you walk up to your boss and tell them that you are going to start working 20 hours a week instead of 40?
I use these things daily, not against this kind of "AI" at all.
I was thinking about this today. I am definitely making things easier but.. if I was paying the actual costs of this service, could it even be close to worth it?
But wouldn't that be great if true? Why all the negativity over the prospect of leveling the playing between people lucky enough to be born with high IQs and average people? High IQ is a completely unearned gift and AI might reduce that fundamental source of unfairness in the world.
That's a factor of 3 to 5 improvement reported on here. If you keep stacking these kinds of improvements, you end up at month to minutes pretty quickly.
That doesn't take into account dependencies between tasks, where bottlenecks can happen. In any case, Amdahl's law applies regarding what kind of speedup you can get in such a case.
The AI can't make a reliable judgement, so now you have to hire 3x to 5x more to make those judgements if you have the documents coming in at 3x to 5x faster. Or you'll have one severely overworked lawyer. It's the same problem developers have with thousands of AI generated PRs. Just because you speed up one thing doesn't mean the whole system works faster, and yes a human really does need to be in the loop or you end up with even bigger problems, and lawsuits.
No, that'd only be true the judgement was the full work. If the judgement was 33% of the work and the rest was just old plain e.g. typing, that is now fully automated and instantaneous, then you automate 2/3 of the work, getting 3x in productivity.
Sorry for being so detached from the discussion and your actual point but if someone had made a dystopian sci-fi novel with the tagline following your sentence with "...or can they?", and made it all cheesy, I'd read it in one sitting.
In one part of Brave New World they bring up that they tried to do more automation, but people got restless without the drudgery. In another part, they bring up that they tried to have a society with only Alphas, but they didn't want to do the drudgery.
In the future, because of bad things we did to the atmosphere and stuff, miscarriages are the common ending of a pregnancy, instead of being the exception. One evil company finds a way to accelerate the process by using 10 women, lowering the chances of miscarriage simply by reducing the time in womb.
I thought, something like a medication helping the women develop the baby in 30 days, but seriously damages the host in a couple of days, so the fetus gets transferred from host to host. Some die, but the evil rich don't care. Your idea could also work though.
I realize that this is totally the wrong corner of the internet to post this but
I guess today I'll just go with it.
Yeah, that plot might actually work and is more believable than mine. (some women actually do feel like a baby is a parasite growing inside them - this would be a extreme version of it)
I'm too bad of a writer to be scared of giving my ideas away, but I'd explore a pupae stage, where (layman disclaimer, if it isn't abundantly clear from my incoming butchering of the science) the butterfly is basically melted down into a goop with no "brain part" or "feet part" of the goop existing until reformed, yet retain their memories. If you divided that goop up and imagined there was a gestation time reduction, before combining it for the final stage of formation, you'd have some "plausible" (loose definition here) venue for it.
> They recently bought a Claude subscription and began using Claude to do the initial read of the documents and output JSON they can import into their internal systems. The work still must be reviewed by an attorney - Claude is nowhere near making the kinds of judgments a lawyer would make about this content - but it has increased their throughput from 2-3 documents an hour to 8-10 documents an hour by killing the busy work.
At my company we're also in the process of deploying a system that does exactly that. And what's interesting is that you absolutely do not need a frontier model for that, a small VLM (vision-language model), with optionally a little bit of fine-tuning, gives you the same output quality at a fraction of the latency and cost.
I'm wondering if the bottom is going to fall out of frontier models when people start to realize this. Sure, as an experienced programmer you can tell the difference between a compact free model and the ones that are 100x bigger and cost billions to train, but 90% of busy work is dead simple: turn a PDF into json fields, or do the inverse and fill out a form. As the tools for this conversion become more widespread you can also imagine an economic shift toward efficient information brokers that make frontier models less necessary.
It's also not really clear if everyone is going to want a frontier model when the real implications sink in. Maybe we'll get sick of incomprehensible code optimizations and people wile tire of reading AI prose that feels ever-more-human. There might be a few use cases, but who is going to pay for this when providers start charging enough to be profitable.
I think this is the long-term reality of LLM tech and one of the objectives of engineers implementing solutions, finding the right fit for the job - the right model and cost to achieve high enough accuracy at the lowest price.
But that's phase two, phase one is finding localized problems to solve using LLMs and productize them. I'm reminded of cloud tech, where phase one was changing software to run in the cloud, and phase two was optimizing costs.
yeah, i think this is probably the more interesting use case. i do a bunch of boring data wrangling as a solo dev and the amount of time that disappears into "read this thing, find these 5 bits, put them somewhere else" is ridiculous. if you're going from 2-3 docs/hr to 8-10, even with someone checking the result, that's a pretty substantial improvement. i don't really care whether we call that decision making or not, i'd be happy to never do the copy/paste part again.
Are they? I've had excellent success. The confusing part of this is that there are two types of PDF. The first is a "normal" digital PDF. The second is a scanned PDF. The first can essentially be read like a document. LLMs have no issues with this. It's the second kind of PDF where the constraint becomes the vision capability, and this is very impressive with Astra. I've had no issues with either. I imagine there could be issues with unusually dense and/or misaligned text on scanned PDFs, but I have not tested this.
The bottom line, though, is that PDF OCR is usually regarded as a solved problem. LLMs won't usually do the recognition itself. It will farm it out to established tools which are very good.
If the goal is to only extract the unstructured text from the document, it is definitely solved.
Extracting a more natural structure like paragraph separation, tables, header, footers (what is referred as document intelligence) is much more complicated and not fully solved, but I would say almost.
Well, I would argue about the first part. Even if they contain "native" text that can be extracted, in most cases their order will be messed up and it is often crucial for correct parsing.
So in many cases the visual way is the only one that works correctly, the textual one is just a shortcut that may be walkable in certain cases.
All the models output mixed data (errors). The larger is the context, the bigger the probability to happen. And this is an inherent property of LLMs (the output is a statistical concatenation of text, the LLMs are an vectorial database filled statistically).
If you connect an LLM to pilot a plane with passengers, prepare the obituaries, because would happen sooner or latter, an undesired index got mixed and produced a mixed/wrong output. But here, the bigger mistake would be to let an LLM execute commands without human revision, aimed by the deceptive marketing around the A"I" (LLMs).
this is one of the largest tasks of any professional services support firm like big law, audit, tax. if you can automate this monkey work you have to cut teams at least in half to stay competitive (probably more).
Happy to see measured results that isn't breathless glazing, and I suspect this is the kind of thing happening everywhere all at once. The other thing to consider is "burnout", in quotes because I mean that the work you describe sounds kinda boring, so people would resent doing it after a while and productivity would slow right down. Automation is the solution to tedium / toil, has been long before AI.
Yes, remember that these are effectively random PDFs in various different designs and formats, some of them not editable or even OCR'd.
It took a human attorney 20-30 minutes on average to manually copy-paste data from these PDFs into a spreadsheet (while also fixing any errors they found in the document and re-checking for quality).
Now, the AI copies everything into the spreadsheet in a small amount of time, and then the human reviews it. It takes maybe ~5-7 minutes to scroll to the appropriate pages in the document, read the lines vs the spreadsheet, and make corrections. So you've gone from 2-3 items an hour to ~8-10 items an hour.
Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.
Does the human find enough bugs that they stay on guard, or just rubber stamp everything without really looking at it? It’s hard to stay vigilant when stuff looks plausible.
This is what bag scanners at airports do - the hit rate is so low and the job so boring the software projects fake contraband onto the imagery. Fail to spot the knuckledusters and expect a chat with the manager.
And it's possible to run each document through the LLM pipeline multiple times, using different models and/or prompts each time, to check for errors and inconsistencies. That will take more time and cost more, but it can reduce the error and hallucination rate significantly.
I'm doing some public court records processing for bankruptcy cases (interested mostly to seek out corruption in big national cases), and yes, the "variousness" of random PDFs is exactly the issue. Trying to get the cost for a whole case down to a minimum.
Sample is around 300 court dates, shy under 1k files.
> Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.
A better use of these Claude subscription would be to develop the app (which it can pretty much do at that point) and you could iterate to make the workflow even more efficient than your current one.
Nobody working there has the requisite experience to do this in a reasonable amount of time. These are not particularly tech-savvy folks, Claude use aside.
> These are not particularly tech-savvy folks, Claude use aside.
The difference between a tech-savyy person, and a non-tech-savyy person has always been mostly in the later's head, but this is even more true now that we have pocket assistants who can answer pretty much all of our questions in a language tuned to our level of understanding.
Yes. And it might not even be worth it, as the AI agents gets cheaper and cheaper.
Keep in mind that the task is fixed, so as the frontier of AI advances, you can switch to a cheaper trailing edge system and still get the same or even better performance for this task.
Thanks for sharing the details. Does the attorney check that the AI copied the data accurately? Or is it just assumed to be correct?
Your experience mirrors my own. AI is great for parsing data that can take up a huge amount of time. My only concern is whether or not it’s done accurately. I wouldn’t use it for anything where mistakes cause serious consequences.
"Write a python script that breaks down this PDF by X feature" would not hallucinate anything in the PDF. Certainly you could trivially double check that all text in the extracted JSON was in the text layer of the PDF.
How much experience do you have with LLMs exactly? It would be consistent with my experience if Claude stuck in a line of python that just emits a JSON literal with no justification, potentially buried in a large program where an untrained person might not notice it. I don't even trust them if the output consists of structured data paired with source images from the PDF, because I've experienced LLMs fabricating the source rectangles to match the output. I only use tools like this by asking for programs, because as you note LLMs are good at that, and the verification process consists of tool calls to legitimate PDF manipulation tools so I have some confidence everything is above board. Even then I only do this for hobbies, not anything that matters.
Lawyer here. I used to trust Claude as hallucinations are near non-existent now. However for large volume tasks such as due diligence exercises, they still happen.
We also tried Legora's tabular review, there were also numerous halucinated provisions in our due diligence exercise.
And when they do, you can train them or fire them, and they learn not to do it.
LLMs change not a whit, and there's no one to take responsibility for the failure (and thus no way to fix it).
As the new variation on the old theme has it, "A computer can never be held accountable, and so very many people are trying to get them make management decisions."
You can’t train people to never make a mistake, particularly when doing highly repetitive work like this. You must build your systems to account for that regardless.
Yes, exactly. Humans are non-deterministic as well, just in different ways. A tired human can make all sorts of errors for example, regardless of how much training they've had.
That's a recipe for disaster in my experience. I tried it (with Claude) on a simple tabular bank statement PDF, and it transposed two amounts, placinh each against the other's description. And the bot assured me the result was cotrect. The chance of a human checker catching such corruption is low.
But it gets it right like 99% of the time so human attention can be put towards catching the 1%, not entering data from one table to another and then catching that human’s mistakes.
Doing similar-ish things with Claude, it's helpful to have something to ground it.
For instance, if you can say:
"Refer to the database schema in x.sql as your source of truth for the database structure we want to import into*. Do not invent data, tables or columns that do not exist. Carefully match all output against this database schema and do not create output that doesn't exist if it does not match the schema, simply skip it."
You will end up with a far better result in my experience.
Few things are. It also doesn’t require standards. “Stamp this diff” culture is everywhere even before AI. A stamp is literally easier than anything else.
Whether that is useful measurement I suppose depends on the circumstances.
Wait what, the conversion is so lossy you use an LLM to analyse it? Or you get the claude analysis into JSON and put that into the system too? I'm not seeing what's causing the 2-3x efficiency bump.
I see problems in LLMs doing research, making drafts, etc.
I see 3 kinds of potential users.
1. Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.
2. Fresh out of law school graduates.
They think they can do fast work, draft quick replies, do fast case research, enh. They should go the traditional route of interning, learning the "art", they probably won't need it because they will learn "how the law works"
3. Established attorneys.
They usually have templates made for most things, over their career, they have established routines of making things fast, maybe it can help them but to say replacing good chunk of work or offloading to LLMs isn't probably going to go well.
There is a reason why interns are made to do grunt work. The "chain of command" is built so they learn on basic stuff, learn to make decisions, etc. Without these basic decisions under their belt, an intern can't take bigger decisions later on. They just won't know....
>Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.
I used LLMs to replace a lawyer this year to great effect. I successfully advised myself as to the right strategy and drafted an immigration petition in the correct language. Granted it wasn't 100% LLMs (I also supplemented it with "traditional" online research and wrote the letter myself using the LLM version as rough guidance), but still, I certainly would've gone to a real lawyer in 2024 for this stuff but didn't.
Professional advisory work can be divided into two types:
1. Once you know the rules of the game, the next move is immediately obvious.
2. Knowing the rules is not enough to know the next move -- that requires judgement and experience.
LLMs in the hands of someone smart and savvy can easily replace the first type of work.
If it is at all possible to replace a lawyer with an LLM, it should always be the preferred choice. Using a lawyer is not a zero-sum game, using an LLM is.
Lawyers nearly universally tend to: convince clients they need a lawyer; promote extreme views; charge legal fees (that ultimately result in financial damages to the client). There are no winners in this game, except for lawyers, who are having a lovely time. Even if you won the case, you've lost, because the pie got smaller due to lawyer fees.
You’re managing risk and what you pay for is derisking yourself. You be the judge of what that is worth to you, but it’s not zero and it’s not always your attorney’s fee either.
> Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.
In my experience, it is incredibly hard, practically impossible, to find a lawyer who will even talk to you. I think this is a valid use case to get at least some understanding what the next steps might be and if it even makes sense to seek legal representation.
Agree. Now imagine the same thing in less developed and/or less well off countries.
There was a post here a few weeks ago about a UK court being inundated with some kind of claims (unfair dismissal?). To submit a claim you do not need a lawyer, but the process is (intentionally?) vague and very complicated, so it takes a lot of effort to figure it out yourself. But AI makes that kind of stuff easy.
AI is not going to replace layers billing big corporate clients millions a month, but it will replace things that are fairly standardized processes, that don't require a lawyer, but today are just too complicated for regular people.
> To submit a claim you do not need a lawyer, but the process is (intentionally?) vague and very complicated, so it takes a lot of effort to figure it out yourself.
I'm not sure there's quite accurate. The form is really not that difficult to complete.
It's basically: Party details, type of claim, particulars, damages
Obviously knowing these details requires some legal knowledge but I actually think ET claims are easy enough to bring if you've got the time to do a little research and build your case. Almost all ET judgments are public so there isn't the usual difficulty with research as you get with law reports being restricted either.
> In my experience, it is incredibly hard, practically impossible, to find a lawyer who will even talk to you.
Really? I find them quite talkative. But I'm not sure what kinds of questions you are pestering them with?
In any case, the whole discussion reminds me of people using LLM to replace medical professionals.
For both: unless you have a doctor or a lawyer always on standby, you have to make a decision on whether to even go and seek out one of these professionals. By definition, you make that decisions without professional help. I think LLMs can help you there with the initial research to decide whether it's even worth it to contact the professionals.
Might be true today, but I still feel pareto principle applies. These points also sound like gatekeeping a bit and I hope future AI versions prove that law is also essentially something that can be made as programmable as possible for most common answers if the underlying principles and constitution are organised such.
If there is no precedence for X happening when Y , Z, P, Q, R are special combinations , the judge decides and sets precedence. But if another judge interprets it differently with similar set of circumstances it is not considered incorrect I presume especially if in different jurisdiction unless overridden by a higher court. This is my layman understanding but I think this design is wrong and essentially where human subjectivity controls the system.
Subjectivity is not totally avoidable in law but the goal of the legal system should be reduce it as much as possible along with ambiguity.
Your understanding is correct, but where your understanding goes astray is thinking that that's a bug. The subjectivity is a feature. It is not possible to specify what should legally happen for all possible circumstances the entirety of humankind faces. Like, literally not possible. We humans can't even create rigorous specifications for what our programs should do that keep up with their evolving requirements, and you want to make life-ruining decisions set in stone based on a rigorous set of predetermined rules about all possible circumstances the entirety of humanity could face any time in the future in an evolving world?
Law is intentionally not code. The world is not something you can program.
Isn't achieving a rigorous, unique specificarion what civil law tries to achieve? Not that it is a success, but the idea behind it. Sorry if this is a silly question, but this is how I understand it.
No, it doesn't. It aims to provide a fairly comprehensive framework within which judgments can be made, but it is absolutely expected that there are edge cases and ambiguities which are resolved by human judgment of the circumstances.
Since this is a thread about using ChatGPT for law, I should note that this is actually one of the most important reasons you'd rather hire a lawyer than ChatGPT to represent you. A good lawyer or firm will know the local judges and how they tend to lean on ambiguities in their area of expertise, and will be able to use that information to both advise you and tailor their arguments to appeal to the judge's sensibilities. There is an element of unfairness to this, to be sure, but you'd find much more unfairness really quickly with machine-generated judgments because of how messy and non-compressable real world circumstances are.
I do think non legal background folks can use LLMs today to sense check legal ideas, like for instance, "what are my legal rights in this situation?"
These situations usually are not ones that an individual can justify the time or money to contact an actual lawyer, but then if they do decide to contact one they will come in with better questions and more of a sense of what they are expecting.
This is similar to medical. Should you use LLM to diagnose yourself, treat yourself with prescription drugs you buy from shady gray market online sellers? No. But you can use it very well to know when it's time to go to the doctor and what to ask.
Yes. Unless you have a lawyer or doctor on standby, you have to make the decision whether to contact them. And by definition you have to make that choice without professional help.
LLMs can help with that. I don't think they are worse at this than me trying to figure this out all by myself.
You sound much like software engineers sounded when the first coding agents dropped.
"You can't trust the output, it doesn't understand bigger systems"
"Its an art, you need to learn the ropes of it to truly write good code"
Its a very dangerous line of thinking. Software engineering will never be the same, as writing code has basically vanished from the daily workflow. Not for every specialized usecase, but for many.
> There is a reason why interns are made to do grunt work. The "chain of command" is built so they learn on basic stuff, learn to make decisions, etc. Without these basic decisions under their belt, an intern can't take bigger decisions later on. They just won't know....
That might be true, but that doesn't mean you benefit from training up juniors.
There are 3x as many lawyers per capita in the US as we had in the 1970s. Credit agreements that used to be 50 pages are now 500. This increase in volume has not lead to any increase in actual value. The legal profession is a metastatic cancer on our society. Un-fuck this situation and maybe I'll think about taking your advice of not using a LLM.
I'm thinking back to some of my interactions with corporate lawyers and you know what? My bar (<- pun) for trusting Astra more than a lawyer is pretty low.
Visit a developing or corrupt country and you will realise the LLMs of today analyse better than the judges from purely precedence and literature review POV
Lawyer here (non practicing so to be clear none of this affects me):
most comments I read here don't seem to realize that different areas of law have very very different economic models and don't even mention which one they think will be affected or why, they just sort of lump it all together.
For example: It is highly unlikely llms will have any meaningful effect on high value personal injury law - I don't see a 5 million dollar case being handed to an LLM when the majority of the cost is in trial aids and not even lawyers. It may affect where and how they advertise. It may affect how they work. But it seems really unlikely to put any of them out of business any time soon by people doing it themselves.
Will it affect other areas more? Maybe. Probably? But so far I haven't seen a ton of comments that make specific enough arguments that they could really be debated or responded to effectively with a useful opinion
Worked for years for personal injury lawyers as an assistant/clerk. Work as a programmer now, tho. I agree that a LLM cannot do a lawyer's job on its own. Especially a trial lawyer's - which involves appearance and performance in a court room. But LLMs are having a massive impact on anything involving digestion of documents (medical records, pleadings, briefs, blah blah) and creation of documents (demand letters, pleadings, correspondence, etc etc) - like 90% of the work. What's happening is very similar to software engineering - the ladder is getting pulled up for entry level and paralegal work.
No. It'll be like software. Entry level employment will be affected. You wont want or need associate attorneys when you can hire a brilliant AI associate for 1/10th the price.
In commercial law, the business model is to rent the associates out to clients at rates higher than you pay (d/t your supervision). No client will pay $350/hour for your AI.
And for direct billing, it's not your interest to do things faster for the client (at risk of making a mistake).
So the real question is whether these pay-by-the-hour models will die as a result of competition from fixed-price transactions made viable via automation. I think not because the entire difficulty of legal work is that it's a complex bundle of issues until you sort them out; once sorted, there's really little to do other than execute (read: paralegals).
Corporate lawyer at big law firm here. I believe the model will be like this: we will review your 1000 agreements but with AI. AI is generally accurate but has some risks, if you agree we will charge you less. The client will agree. Instead of spending 300-400 hours to review the ageeements AI will do instantly. A lawyer will spend the time on reviewing the red flags identified and draft a report.
We already do the same with translation. Before computer translation became good (i.e pre-Deepl era), we used to translate documents (cease and desist letters, court petitions etc.) ourselves. Now we offer AI translation free of charge. If client wants reliance on the translation we charge a small bit.
Lawfirms are also under pressure for meeting internal budgets. Most clients ask for capped fees. If you charge 10k for a job to a client and the hourly costs ramp up to 20k, the management criticises the partner, the associates become unwilling to work for the partner (as their hours get written off).
So with AI law firms can bet more competitively as the cost (human attorney hours) are reduced.
But they may be willing to pay $200/hour for your AI (plus, of course, your final expertise in the matter as the lawyer), and the firm can ditch the paralegals and pocket the difference.
The point is, if you need fewer grunts doing the busywork, you can charge less and be competitive.
There's a middle area too between your options - a law firm that utilizes AI offers cheaper pricing than the current firms for the same work. Ideally competition will drive prices down but of course law doesn't have the benefit of do-overs on cases, so reputation still plays a big factor.
That's the big question, how prices evolve if LLMs increase supply. Standard economic theory says that prices must collapse. There is probably still a short time of 2-5 years where AI native firms can pocket in quite some arbitrage but I guess markets will eventually figure it out and prices will collapse.
This is not only for law. The same applies to all the other industries where LLMs can be used to automate work tasks.
But they already do! There are cases of professional service firms creating their reports using LLMs, there have even been a few prominent scandals re the fake data generated in them
It is in litigation as well. The issue is that corporate clients are demanding AI. They don’t want to pay associates to do a first draft, they want senior partner to just push the ChatGPT button and then check it.
> No. It'll be like software. Entry level employment will be affected.
Whether everyone at the labs knows it or not, this it what they are doing to entire industries, starting with software: Embrace, extend, and extinguish
They came for software first. It was the industry that was “eating the world”. Mathematicians were also affected. Because math was also “code” thanks to Lean. It was collateral damage.
Now they’re coming for lawyers. Law also has “code”. It’s text-based. It’s arms extend to every industry.
What's next is capturing all of our audio and visual inputs from cradle to grave, so that we can be truly replaced.
Honestly, if that happened in a 100% trusted environment, it might be cool. But, there is no money in that. Well, maybe Apple would do that? Private Cloud Compute is cool af.
Still, we are far more likely fully cooked than fully trustworthy personal AI agents happening at scale.
It'll be like software. The demand for both software and legal services is effectively infinite. If it becomes cheaper to file lawsuits then we'll get more lawsuits, not the same number of lawsuits at a lower price.
There will still be jobs for competent associate attorneys to prompt LLMs and review output. But frankly a lot of attorneys — especially many of those who attended "Third Tier Toilet" (TTT) law schools — were always a waste of money and should have never entered the profession in the first place.
There is a huge problem in at least family law where a tactic is to simply ddos your opponent with motions and frivolous complaints until someone runs out of money and settles. I'm watching someone finally successfully defend themselves using a Harvey like tool they built themselves. An appellate attorney they consulted with recently, told them they had never seen anyone going pro se get as far as they had in appellate court. So I'm hoping that this serves as a return to making the legal system more accessible, and not just something used to railroad people.
You can’t 10x the number of lawsuits, as the courts are already at capacity. If frivolous lawsuits become common they’ll just change the laws to require a greater damages for it to be heard.
Lawsuits are just one type of legal services. There will also be more contract reviews, more cease-and-desist letters, more patent applications, more regulatory comments, more license applications, etc. Although I do expect legislatures will eventually budget for expanding civil court systems in response to lobbying from business groups to speed up trials.
I think this is key. Lawyers make a lot of money for being experts on the sidelines of disputes with values greatly in excess of their fees. They were never paid for their busy work.
Can AI replace divorce lawyers? Marriage has no friction, but divorce is high friction. Made impossibly expensive and time consuming because of the cartel of divorce industry.
There are hundreds if not thousands of divorce lawyers. How is it a “cartel?”
Divorce is expensive because divorces are acrimonious and people in that situation would rather spend the assets fucking each other over than letting the other person get them.
Divorce is expensive because it is big bucks for the lawyers and family court cartel. They have a vested interest in not changing the laws. When laws are made fair, and the perverse incentives to divorce are lessened, divorce rates plunge. However, divorce lawyers lobby against these laws.
Kentucky became the first state to pass a law making equally shared custody the default arrangement in divorces and separations.
AI will likely replace divorce lawyers (or, at least make existing divorce lawyers much more efficient) - for non-contested divorces that are amicable. Roughly 95% of what a divorce lawyer does in that context is entirely automatable.
A template document and some software already does it without an LLM. Advising the clients on what it means and what they have to do is unfortunately something AI assistants are bad at. But this use case is not an expensive use case for family lawyers; court systems already try to make this type of settlement accessible to pro se people.
Divorce is extremely cheap if you both agree on everything. Its literally one of the options on the divorce filing. You just pay the court fees. The problem is both parties rarely agree on everything and that's when it gets expensive.
There’s an argument to be made if divorce was easier than more people might get married when they are less serious. My view though is make it harder to get married and easier to get divorced.
He clearly meant getting married has no friction, while getting out of a marriage has quite a bit. Presumably there’s friction in the marriage otherwise why would they get divorced.
huh? Many/most are no-fault jurisdictions with strict rules on dividing community property. Divorce is easy and cheap, unless it's hard to find assets.
It's not just the divorce lawyers. We have 3x as many lawyers per capita in the US as in the 1970s. The system is designed by lawyers for lawyers. They will create more bureaucracy and more procedure to consume any surplus.
Personal injury attorney. AI is entrenched of every single phase of litigation, from the busywork of getting records, research, developing case theory and themes, expert selection, expert collaboration, negotiating, drafting demands, drafting pleadings, drafting discovery, drafting discovery responses, deposition prep, motion practice, jury selection, voir dire, opening statements, witness direct outlines, witness cross outlines, closing statements, and appeals.
Yeah, this matches my friends - they use LLM's for tons of things, using Eve and such. But nobody is replacing the lawyers themselves in cases of any value.
I also have friends in the world of high-volume personal injury (IE the morgan and morgan's of the world) but they also aren't being replaced by LLMs so far.
Yeah. Let's not forget that just a year ago, those of us in tech could not conceive of developers getting replaced by AI. Things have come so far since then however that there are multiple studies showing junior developer hiring has slowed down to a crawl.
I'd love to get your thoughts on the current frontier models for law, or a link to anything you've written on the subject. Where do you think they're at?
The word “never” is a dangerous word. In my lifetime I have seen it many times where something was supposed to “never” happen only to see it happen a few years later.
I'll buy that - but, how often does one need an honest-to-goodness expert? 99% of the time most people's requirements can be satisfied with general-competence, and, while the AI systems we have today are mostly only generally competent in porting code from one language to another - they are getting consistently better. I expect we'll start to see general competence in many more areas in the next few years.
> I'll buy that - but, how often does one need an honest-to-goodness expert?
Any time you are doing anything important, where the outcome actually matters. For example, I was recently applying for a study permit and if I generated my application using an LLM instead of consulting with an immigration lawyer and writing it myself, I would not have gotten my study permit. I was recently applying for internships, and if I hadn't paid a real recruiter to review my resume and advise me, I would not have gotten my interview. And trust me, I spent at least an hour trying to get an LLM to generate what I wanted, but no matter how I begged or described what I wanted, it was no use. Good luck fine tuning an LLM to be good at any of this.
> they are getting consistently better
No, they are not getting better, and they will never replace an actual human expert, because a human expert has the actual experience and up-to-date knowledge of what they are an expert at, and also all the complementary context and the current state of affairs in that area. The LLM has none of that.
Then hopefully at some point they get to experience the consequences of having low quality thresholds, and perhaps then their quality thresholds will increase.
There are plenty of areas where a "low bar" is more than enough. "low" doesn't necessarily mean bad here, it can simply mean "not the best".
I would argue the majority (in the true sense of the word) of white-collar service work does not require an expert.
It requires someone who has reasonable qualifications in the given field, and the motivation ($) to help you with your task.
So no one needs a top-shelf $900/hour lawyer to draw up a basic low-stakes contract where both parties agree to all the terms as written. You just need someone to ensure that the contract is legally valid as written.
Lol no one cares. Most of that is smoke and mirrors anyways. Stokes was plagiarized. Most of these "breakthrough" solutions are actually present in the training data and in the prompts researchers and mathematicians input into ChatGPT themselves, prompts OpenAI then steals even when you don't agree to improve the model.
PS: I use LLMs every day, basically let Codex write most of my code (after 15+ years of professional programming), these are amazing tools, smart in some ways, but completely demented in others. People just need to fucking stop being AI doomers and AI coomers. Is this technology great? Yes. Is it worth 10 trillion? Hell fucking no. If we get to a point where RSI is a thing, then maybe. So far it's not a thing and its not even close.
Professional maths is sometimes seen as requiring IQ 200,000 creative problem solving skill. You have to imagine creative ways of solving problems which people have never solved before. Failing to do that means you can't become a professional. The subject has lots of theory to help with this, and is sprawling. LLMs can do it better now than most professionals can. It is therefore notable that mathematicians were once confident that their subject tested human ingenuity to its limit. The existential crisis of their field grew from 0 to 100 in only 6 months.
One of the things I've been noticing are lawyers getting annoyed when they realized you spent time working a matter with an LLM. In the cases I've seen, it had nothing to do with the quality of what came out of the LLM. It seems to be related to --and this is only my perspective-- a reality where now reasonably intelligent/experienced clients are coming to them with a case fairly well digested, whereas before they would have to pay the law firm tens of thousands of dollars to ingest documentation and information and do the digestion.
In a couple of instances I've come to the table better informed about the relevant law and case history than the attorneys were. In many ways I think this is a matter of people understanding that none of us are infinite human databases, so you can't know everything. Instead of rejecting LLMs (perhaps being passive-aggressive), they should embrace them as well as better-informed clients.
Having used LLMs for software development I know full well just how terrible they can be if you don't know what you are doing, don't guide the work correctly, etc. So I am not taking legal LLM analysis as ground truth, but it isn't garbage either.
Yeah geeks just stop trying! Seriously? Yeah, we’re in a bubble and narratives are ahead of reality but if you really don’t think AI is and will continue to eat knowledge work, just keep making your buggy whips.
The need for actual lawyers will persist I think from my own experience. I attempted drafting a contract with some points myself using AI, but after several edits I wasn't sure if it was correct. Sending it to an actual lawyer ended up in so many corrections I couldn't imagine the first time. One big thing was the overly excessive protective clauses which didn't make sense for reality or conflicted with another.
Its just like code I suppose, if you can read and understand and validate, you can use it to scale and otherwise it could end up being a vibe effort.
I'd correct this to "The need for actual GOOD lawyers will persist I think from my own experience."
I have worked with many lawyers in my career. The bad ones will look at a contract and tell you fifty things that can be improved. Meanwhile, you take the same contract to a really great lawyer and they tell you that's it's not worth fighting over the changes and to just keep it as is. It's kind of like how a beginner or intermediate software engineer might tell you how you can build out a k8s or auto-scaling fargate cluster for your project and a really senior engineer might look at the same thing and explain that it's not worth worrying about scalability right now.
I find that when I feed contracts to ChatGPT (which I do all the time) it will try to find 50 things wrong with it. For example, as a test, i fed chatgpt an employment offer from a US government agency. It came back with 1152 words and 14 recommended (some of them substantial) changes. Meanwhile, any half-decent employment attorney would probably laugh at me if I asked them to redline an employment offer from the State Department.
I’ve found that good lawyers are about getting things done. You could nitpick the terms of a contract, but would you rather buy the house or have a perfect contract? Once I realized that, my view of lawyers changed (for the better).
"the hierarchy of knowledge", right. A bad or insecure speaker will ramble about insubstantial minutae whilst a good one will get straight to the key points that form an argument. Basically the confidence to prioritise information.
In my recent experience, it seems like LLMs may even increase need/demand for actual lawyers (more ideas + more words = more lawyer time). I expect lawyers will generally adopt the technologies that will benefit them, rather than the technologies that would increase standardization and trust
The same has happened with modern finance. Despite more technology that has power to drive more transparent/efficient markets, we have less perfect information sharing and a larger group of middlemen capturing a growing share of profits of the economy
I think you're onto something, but it'll be more of a reshaping of practitioner distribution to match the mode of work (perhaps fewer employment lawyers, many more litigators/ip litigators, etc...)
Not trying to be dismissive here but a lawyer will always edit your proposal. I do think lawyers will persist but not as many. And they will work very differently, much like we already use claude / codex to code - a big part of the contract probably won't be read by the lawyer.
I would rather argue they SPECIFICALLY will read everything. However, they'll likely often just be like "I'd phrase this differently but that works, too".
LLMs are the first genuinely useful legal tech since the Internet. I'm pretty shocked, though, at the delta between how competent Claude is on code versus legal work. It's good for research and data organization, but terrible for drafting. I wonder if this is a structural problem with the lack of feedback loops. In law, there's no compiler to check for logical or continuity errors in your brief, and there's no unit tests to check for correctness or performance.
Even without that, I think it'll be extremely valuable to clients to allow them to answer simple questions without a lawyer, figure out the lay of the land so they can supervise their counsel, etc.
You should know - for coding they make terrible mistakes as well.
But programmers have this concept of a "code review" where another person looks at the code to look for problems.
We use this same technique with our LLMs. Most mistakes are caught by having a second LLM look at it. Doesn't even need to be a different model - just make sure it has a different context.
> Doesn't even need to be a different model - just make sure it has a different context.
I find that the number of issues discovered is noticeably higher if you do use different models though. I'm doing some very finicky things (formal semantics) and find that there's value in review panels as large as 5 different models from different families. It gets even more profitable if you set it up as a truly agentic panel where after writing their own separate reviews they get to see what others have said, and adjust their positions or defend them etc. Some models are not that good by themselves but can be surprisingly good at finding flaws in others' reasoning (Grok for example).
Programming has a long standing culture of accepting the code to be somewhat wrong, so we have various tests, linters, reviews and error handling. Also in programming there are many ways to do something and it's the end result that matters most.
Not so in other knowledge work. There's no test harness for a contract and error is non-recoverable. Likewise in finance. There are specific ways of doing things and these ways are many times set in regulations. LLMs can assist all day, sure. But replacing the human, in highly regulated, zero tolerance for error environment?
The highly regulated, zero tolerance for error environment is a huge problem because anyone doing these jobs is more like a small sample size LORA than a general model.
It doesn't matter how smart someone is, they need specialized training to be good at these jobs. Specialized training in the area the company specializes in.
There is a category error in all this that is hard to think about because of the normal discourse and ordinary language. We say people work in "finance" but no one works in just "finance". They work at a company that has a specialization within "finance", inside a hierarchy that has specialization on top of specialization.
What we really need is exactly what we don't have and aren't going to get. A type of LORA that generalizes the task specific intelligence needed from a very small sample size and that in practice makes so many less mistakes in a highly regulated, zero tolerance for error environment that it is irresponsible to not use the model.
I have worked in this type of environment for 3 years and I have made zero mistakes in 3 years. The people that make even a small number of mistakes get fired.
Any real automation in this area is going to be incredibly slow and piecemeal over a long period of time because even an amazing model would need a long time to prove itself against what the human standards for error rates are.
Even the ensemble average error rate on a large number of tasks in space would not be good enough. It needs to be an average error rate over time.
You could say the Salvatorian Clause in contracts is like exception handling:
a "catch (all)": even if some clauses in this contract are illegal,
the remaining contract stays in place.
Logically, this actually doesn't make sense strictly speaking because the sentence creates a paradox: doesn't it make clear whether it includes itself or not, and each reading ends up in trouble. There is a "tradition" in law around the world to accept the only benign reading of such clauses, which I always found funny given that in all other ways lawyers adopt the most adversarial mindset imaginable.
I'm trying to understand why I'm getting replies basically saying "they'll never be good lawyers" when all I said is that you can improve the output by getting them to check each other.
> LLMs are the first genuinely useful legal tech since the Internet
That is an incredible statement that could not be further from the truth. Large scale adoption of email, searchable document databases like Westlaw, LexisNexis, PACER, etc.. , OCR Software, electronic signatures, and tons more have had a much more defineably positive impact on the legal profession since the internet came about.
The internet came well before both of those. And none of the example I gave have lead to lawyers regularly being fined in court because of "hallucinations"
I have found that it’s useful generally speaking to get the intent of contracts and red lines, but actual drafting I agree is where I lose all confidence. My guess is that the significance of the difference between using a word like “and“ or “or“ can be so meaningful that that level of nuance can often be lost. But I know nothing I’m not in the space, I just pay too much money for lawyers.
I reckon law and medicine are really premier use case for llms since those areas are all about having vast knowledge(knowing all about obsure cases in law or about an very rare disease the average doctor wouldn't have heard about)
I've heard this argument (basically, you can just have good legal skills and adapt your practice to more areas of law) but we're still pretty far from a layperson confidently navigating court (you could say the same about code).
That's exactly it. You basically can't set up deterministic regression tests, which makes development of (legal capabilities) even more abstract and messy.
I want to take a step back and observe that you're saying expensive specialists will continue to be required if two parties want to make an agreement. That seems like something worth fixing rather than just accepting.
> I want to take a step back and observe that you're saying expensive specialists will continue to be required if two parties want to make an agreement.
A lot of contract-lawyering involves getting ahead of when things go south and the two previously agreeing parties find themselves in disagreement.
Same for SWE. AIs will churn out happy path vibe slop and fail when something unexpected happens by swallowing errors or having a bunch of terrible "fallbacks".
It's probably somewhat of a zero sum game. Both sides lawyers have AI tools which probably just keeps them on a level playing field rather than actually making anything cheaper or faster.
I spoke to a very very high up corporate lawyer specialising in tech - so knows Ai is real. And they said AI contracts are the bane of their existence. They’d rather you arrive with no idea what you want than a contract partially written by AI. They said it usually takes longer with an AI starting point than starting from scratch.
Millions if not billions of agreements are made every day without the involvement of lawyers. Lawyers are only needed for particularly complex situations and even there they are generally only "needed" in the sense that parties feel they would add value.
Yeah, and you can already get fill in the blank legal forms for a lot of basic situations, including leases, wills, employment agreements, etc.
The interesting question is whether AI can fill a niche these can’t — or, alternatively, help a layperson evaluate a contract they’re asked to sign better than they could themselves.
In my experience, those expensive specialists help me protect against things me and the counterparty cannot even imagine when making an agreement. Do you mean that this would be fixed by AI or by making simpler agreements?
When it can imagine real risks and not hallucinations, I am sure it will be better than having a human lawyer write the agreement. I am not convinced that we are there yet.
Contract law has evolved because transfering the ownership of things is fraught with difficulty. Ownership as a concept itself is notoriously hard to define.
Consider housing - when you buy a house you own that land. Except if its an apartment, you just own the internal walls. Or when the government aquires your land through eminent domain. Or mineral rights - which don't belong to you, you just own the top 5 feet of topsoil. Or if you neglect a property and some squats in it. Or if you were given a fraudulent payment, and they take back the property.
Just getting people to agree is notoriously difficult when people have different often conflicting desires.
Most people dont appreciate that the world is a complex place. Nothing is ever simple, sure concepts can be extremely simplified but the nuances and details are lost. These details are the important parts, just ask a surgeon or rocket scientist etc.
Also, unlike SW engineers, lawyers control their industry and will quickly circle the wagons to protect themselves. Paralegals are probably screwed, but lawyers will do fine. I'm considering law as my post-ai-SWE-apocolypse career path.
I’m thinking asking the same lines, but also looking into research / lab chemist so I can hang out with my best friend. If im going to change careers so drastically, I might as well enjoy the company.
Yes. But this option still exists in addition to reputation which I mentioned.
With AI models this option does not exist. I can't sue OpenAI because the contract that I got ChatGPT to draft carelessly exposes me to some type of financial loss. I can't contact the state bar and have Open AI banned from drafting up contracts or doing any other work that lawyers are licenced to perform. Even if I could it would not compare to a lawyer loosing their license to practice for life. If I tell a friend or acquaintance about my negative experience they will still likely use OpenAI for professional work and chalk up my experience to a skill issue or premature use of the technology.
Also note the use of the word "professionals". I was referring to doctors, engineers, accountants etc too.
Except it doesn't, particularly as far as lawyers are concerned. There are almost a million and a half active attorneys in the US. Just 500-600 are disbarred every year, and almost always for money-related violations, not bad advice. Public discipline is also relatively uncommon (about a quarter of 1% of attorneys every year) and usually related to issues like financial impropriety, neglect, undisclosed conflicts of interest and failure to communicate, not disagreement over the quality of advice.
If a lawyer drafts an agreement that causes a loss, a bar complaint will not help you recover your money. You'd need to sue for malpractice. This costs a lot of money and takes a lot of time. Because you're going up against a lawyer, expect the defense to be vigorous, especially, ironically, if you hired an attorney at a good firm. The full weight of the firm will back the attorney and you might find it difficult to find an attorney, as many won't want to be involved in suing a fellow professional in their market.
I don't disagree with your fundamental premise about the value of human responsibility, but these professions have been structured to protect their members and the ABA in particular basically functions like a cartel today.
Not being snarky here but I've encountered some really crappy lawyers and they seem to be doing great ($$$).
It really sucks one cannot publish something like "this guy charged me $5k, wrote my name wrong and forgot important paperwork the day we showed up to court" as that could make you liable for damages. There should be a safe harbor for that.
Assuming you are in the US, there is a safe harbour called "opinion based on disclosed facts" for that. You could also fairly argue that the conduct of a lawyer is a matter of public concern
> It really sucks one cannot publish something like "this guy charged me $5k, wrote my name wrong and forgot important paperwork the day we showed up to court" as that could make you liable for damages.
That's pretty much what bar associations are for, filing a complaint against an attorney at the bar seems like a pretty standard procedure.
>There should be a safe harbor for that.
A safe harbor would be the opposite, it would be immunity for the lawyer.
I was in a similar situation and tried a different approach.
I had start with asking for a contact with some details I provided. But then I ask the model to be an experienced corporate lawyer and ask me a series of questions to gather the details it needed and then write the contract. The result covered a lot of details that were highly relevant but were absent from the original attempt.
The key insight is that you can lean on the model to cover your unknown unknowns.
Your comment is a great example of the phenomenon where people think AI is an expert in areas that are not because they have no way to determine if the seemingly good looking result is genius, gibberish, or somewhere in between.
The point was that you can go further than "write me a contact that says x,y and z". I actually believe the process I went through with the AI was a more custom, tailored experience than any interaction I have had with actual lawyers, who'd usually apply a template. The AI asked me a bunch of questions, many of which were relevant, and led to follow up questions. I could ask my own questions, make corrections, etc. and all along, they made sense to me. This part of the experience was great.
There could still be more unknowns that I did not encounter and by definition don't know about.
And the final output could be wrong, wording, exact legal terminology etc.
Can we say working with an actual lawyer is guaranteed to be better, more correct? Lawyers are people and they also make mistakes, wake up on the wrong side of the bed, hate their job etc, just like the rest of us.
Here's another techniqe that gets you (even) more out of LLMs:
run each prompt several times and compare the various outputs. It is not uncommon for models to contradict their own advice, and also you will get additional insights not included in previous runs.
This is due to the fact that LLMs are statistical processes that rely on pseudo random numbers in chosing what to say and how to say it to a substantial degree.
I agree. But, we will need a lot fewer of them. My small company called our lawyer a lot more before the advent of AI. Mostly to clarify a few things or review simpler contracts. Now, for simple things, we do not call them at all.
I anticipate an effect on society that troubles me:
(1) Rich people will use human expertise and "the rest of us" will use AI models to get by. This could happen in law, but also in medicine (in particular in societies that - like in the US - do not have universal healthcare).
(2) A positive effect in both domains may be that access to knowledge will be broader and cheaper.
I hope I will be wrong about (1) - I would not like to live in such a two-class society.
(1) already exists. A more banal example is rich people get their kids private tutors. Poor kids lean more on public resources, wiki, the youtube and the internet. Economic forces will always dictate this kind of change.
You should never rely on just one model. Use Opus Max for review and Sol Pro for drafting, then Gemini, CoPilot for additional "junior" reviews. Once passes make no material difference, feed it again to Sol Pro for review. At the end still worth checking with actual lawyer, but this way you might get document as good as it can get. Depending on the domain of course.
Yes the probabilistic smudging machine does not make many redlines with confidence, unless A) directed or B) it's a painfully out-of-distribution argument.
The salaries are a result of the insane hours they have to do. Biglaw needs to pay those salaries to keep getting young associates in the door because burnout is a real problem in the industry. If salaries were lower and hours/stresses were lower, you would probably still find people wanting to be lawyers. A different kind of person, maybe.
(Not that I think AI will necessarily have that effect. We just don't know yet)
Salaries are about scarcity and leverage, they have little to do with working hours. Nurses pull insane hours and they still don't get paid as much as doctors.
Very few people, if any, are going to bust ass in law school or medical school if they're not going to get rich off it. Better to do literally anything else.
In Biglaw, the scarcity is driven by the hours and stress. If it was a chilled 9-5 job you would certainly have more people willing to do it, even if it paid less.
It’s a professional organization that only allows so many people to pass the bar per year so I don’t see that changing anytime soon… if anything if they need fewer lawyers they will just pass fewer people to keep their wages high
As costs for legal work drop we might expect demand for that work to increase. For example, it may be the case that legal help becomes accessible to entire segments of the population that had no access previously.
The best writer I know of was an associate attorney. He didn't have the technical background nor the in depth computer related knowledge, and relying on the information I fed him. But man the briefs he filed to the court were amazingly good. Reading them I would have been convinced his side was right if I were in the jury.
Law LLM will surely help competent lawyers in their fields with greater sources of knowledge not in their core area of expertise.
you can't in imagine the same way you couldn't imagine generating the text from your initial prompt. when will the hubris end?
some variant of "oh i am capable of generating text i know nothing about, but it helps to have a human i can rely on to tell me more about the text i know nothing about."
pick up a bachelors degree worth of books to humble yourself
And to add to your last point yes it should also be like code: where the costs of development are coming down. Legal costs should trend down if law firms are effectively leveraging LLMs to accellerate their procedures (though I won't hold my breath)
> The need for actual lawyers will persist I think from my own experience.
The outcome of a case shouldn't depend on someone's fallible ability to recall facts or convince other people or point their index finger*
Law should generally be deterministic. One's CHA stat should have no bearing on justice.
There should still be human judges, but the middleman between the judge and petitioner could easily be removed, and have generally been seen as leeches since forever anyway.
Though, like how the USA opts to remain in the Stone Age with regard to tax filing because of lobbying by tax software companies, this faction of society will flail the hardest before they admit they're obsolete.
----
* What's a lawyer's favorite programming language? Objection C.
The ability to clearly and effectively communicate, and the ability to know and recall the facts of a case is always going to be important. It'd probably be a more fair system if judges and jurors couldn't see the attorneys or even the people they're representing. Now that many cases are handled over video calls it'd be pretty easy to make that change.
A lot of communication is conveyed through more than the words themselves though and even though it's often misinterpreted people put a lot of value in that information. I suspect that even if we took away the ability to see the people involved judges and jurors would still find bias in the tone/quality of people's voices. Even if everything was reduced to text they'd find bias based on word choices, phrasing, spelling, and grammar.
Yep, just yesterday, Nike removed one leach middlemen called retailers and sell directly through their app. Turns out working out very well for them too.
An Amazing Lawyer not winning an unwinnable case = a good thing.
A Mediocre Lawyer not winning a winnable case = not a good thing.
Say a starving child steals a loaf of bread. The law says all theft is at least 1 year in prison. But the judge can see that the child is actually destitute and it was his first time. The judge could shorten the sentence to 1 week; that's fine, there should be room for interpretation and leniency.
But it should NOT depend on whether that child had a persuasive lawyer or not.
A good "AI Lawyer" would simply behave as how ALL lawyers SHOULD: Simply present the facts, the laws that apply to those, offer suggestions for the verdict, and the possible long-term consequences for each possible verdict, without injecting dramatization, exaggeration, or attempts to pull at heartstrings etc.
The typical pattern is called “deskilling”. It doesn’t usually mean a skilled profession will disappear overnight. Instead, the job might be done by less expensive folks like paralegals.
An example is in the banking industry, where making a loan used to require deep analysis of a person’s credit worthiness. Now they use an algorithm (credit scores) which means someone with less experience can do it.
If law follows the same pattern, a job done by someone making $500/hour might be done by someone making $50/hour.
i don't think that's the point anybody is making. "good lawyers" is not the profession at risk here. a top tier courtroom lawyer won't be replaced by AI.
but how much of the revenue of the average law firm comes from that, vs the day-to-day "we need this relatively routine contract reviewed" sort of work?
Yes. And it would be the continuation of a trend that’s already been going on for two decades. When I started out as an associate at a fancy Wall Street firm, we had 30 people working on reviewing documents for this matter even that level of firm today would not be able to justify doing that anymore. We’re going to move from a pyramid model to more of a diamond shaped model with many fewer associated attorneys doing the legal research and information marshaling.
> Instead, the job might be done by less expensive folks like paralegals.
So who is going to define your strategy, represent you in court, adapt the strategy to changing circumstances, negotiate with counter-parties on a mutually accepted settlement? A paralegal? An AI model? Please
Very little of corporate law is in court. The mass of things is “does this contract look good?” And “what do we need to change in our business when regulation X comes into force?”
Former attorney here. A good paralegal is worth 3-4 junior attorneys. A good paralegal with AI should probably still have a senior attorney oversee them but this will absolutely hollow out firms.
"law" and those who pratice "law" have managed to place itself largly in a monopoly. This is a huge drain on the economy. Law is expensive and irrational.
Our entire world is about reducing the friction of conducting a transaction
How to fix the lack of trust which is needed to conduct a transaction
Contracts affix a moment of trust in time, so even when trust is lost between parties the contract has guardrails for behavior and consequences for acting outside those rails
There are many transactions that are completely inaccessible to most people because the need for a contract or lawyer is too high
LLM use allows for those inaccessible transactions to decrease, and get people further along to the point where a lawyer is accessible as well, and economically viable to use
> One big thing was the overly excessive protective clauses which didn't make sense for reality or conflicted with another.
>Its just like code I suppose, if you can read and understand and validate
I share this, when I asked an attorney for a contract, they were much simpler when compared to self redacted or AI redacted contract, it was as if I were rediscovering lots of concepts and I HAD to put them in paper, even if they were defaults and were essentially no-ops.
Comparable to:
> Mkdir $PWD/path
instead of
> mkdir path
or conditions so rare that they don't merit including.
> x = 0
> if x!=0:
> raise Exception("Bit flip by solar rays or otherwise")
>The need for actual lawyers will persist I think from my own experience
At the very least because the attorney monopoly assigns a 5ish year of training as PoW and natural personhood as Identity as protection for Sybil Attacks.
If anything, the exclusive right of attorneys to represent clients in court, file motions, and enjoy professional secret, is more valuable, as it's not something AI can ever compete at. (Barring a wild recognition of machines as humans, or an overturn of the impossibility of companies to appear in court.)
Max Junestrand has consistently said Legora treats the model layer as swappable, selecting across frontier providers rather than building the product around one.
OpenAI didn't need to name Legora and Harvey in the second paragraph of the launch post.
They are pre-empting the obvious interpretation of Astra for Law: that moving this far up the legal stack puts them in direct competition with their biggest legal AI customers.
“Don't worry, they can build on us” is a pretty conspicuous message to include on launch day.
They have clearly thought about some pessimistic outcomes.
Switching from ChatGPT Enterprise to Legora at my firm was a godsend, it's so much better for legal work, even with the frequent changes to the underlying models.
They are cutting into their market. They can dress it up however they want, they might not be competing for enterprise contracts yet (hence the fluff statement), but they will.
>Max Junestrand has consistently said Legora treats the model layer as swappable, selecting across frontier providers rather than building the product around one.
Vendor-neutrality for LLMs is such a weak thesis all around, whether for providers or consumers. It weakens the product by being promiscuous and gains no material benefit at all.
LLMs are magic byte(byte) functions, it doesn't make sense to say "we have different providers for magic".
Different flavors of magic require different ritual components and in the name of all that is stable and production worthy, please don't get a necromancer to do your civil construction magic.
Vendor-neutrality helps reduce lock-in, and OpenAI and Anthropic are big enough that the reduction is valuable.
the thing is a lot of the legal work which will go through this is drafting 100 and 1 variations of draft versions of standard contracts not containing any trade secrets where the contract can be drafted with "replacement/place holder names"
the kind of work mostly done by juniors not yet through their final exam and other "non" lawyers etc.
so it's a slippery slope of "lets just use it for <this> things where it doesn't matter" and then out of laziness and convenience it creeps into all the other places (at least for drafts).
Yeah, it should be freely available, you have to be able to know the rules you're supposed to obey in order to obey them well. I've been making a free API for US law search, you can point whatever model you want at it: https://law.agentlookups.ai/
Very much a work in progress, only federal and state so far, no municipal codes yet, and no case law yet. Big hole, I know. Also working on making the search ranking work better.
Right now it's just a bunch of crawlers for the individual states. If there's interest, I could periodically stand up snapshot torrents or something. That something you'd be interested in?
Alternatively, if someone else knows an all-in-one option that exists, I wouldn't mind retiring those crawlers...
Not op, but that's a very interesting proposition. While the law and legal code are technically property of the people, I'm not aware of any single point of download for it all.
There’s no single point of download for it all because there’s thousands of autonomous entities that issue law and adjudicate cases, at least 51 of them distinct sovereign entities.
We could enforce (suggest?) a common format / api at the federal level. Especially if it’s incentivized with funding that more than justifies the cost of maintenance. Similar to how federal interstate funding is only available to states with a 21+ drinking age.
Yeah, going to the courts and municipal code seems like it's going to be a heavy lift. Many of them seem to hang off of municode, though, so maybe it's not a huge number of unique crawlers.
There is none, not for statutes and definitely not for case law; even at the appellate level where you have multiple federal circuits, then 50 states, then territories, military, tribal and a whole host of other niche courts. And the appellate court systems can be split into districts, and by lower and higher levels.
Then if you want to really get into it, The People should also be able to access trial court level, and at that point you have over 3000 distinct court systems with their own access systems, usually requiring logins and CAPTCHAs, and half of them not even having anything accessible online at all, and the other half only having recent stuff online and the rest rotting in a flooded basement.
That’s awesome! I especially like RECAP as a method for freeing things from PACER.
Seems like you all are already doing a lot of what I’ve been aiming for with mine. Are there useful ways to contribute, or have you all gotten it to a pretty good place technically, and it’s mostly a matter of spreading it at this point?
> By using the legal search index, Astra for Law can search U.S. case law, statutes, regulations, court rules, and administrative decisions across a corpus of more than 230 million URLs, with sources added daily. Our work with Free Law Project, the nonprofit behind CourtListener, brings its case-law collection covering more than 99.9% of published U.S. precedential case law (opens in a new window) into this research experience.
CourtListener already has an MCP interface and Grok is quite good at pulling from it. In my experience, Grok 4.6 is quite good at analyzing legal cases and human-written documents. Better than Opus 5. I'm not sure if it's better than Fable 5.1 on that task, b/c I'm not willing to spend my precious Fable tokens on case law searches lol.
I find the idea that people can use LLMs to exercise their rights as citizens appealing however. Many people aren't aware of the rights they have, and LLMs are pretty good at surfacing some stuff without having to pay lawyers. Having to hire a lawyer is imo actually a huge way of gatekeeping people from exercising their rights. I heard a lot of local German public institutions are currently being flooded with people arguing their case with the help of LLM that they previously weren't really realistically able to do. So I don't see it all as bad.
>are pretty good at surfacing some stuff without having to pay lawyers
They are also really good at making stuff up as evidenced by the many, many, many examples you read in the news about actual lawyers using AI to write briefs that are full of errors and hallucinations.
In civil courts, you'd likely get more sympathy from a judge if you represented yourself and admitted your lack of understanding, rather than try to appear as someone you're not because you wrote some prompts and copied the output.
Current court systems around the world are just not built to handle the flooding of cases from the citizens.
The stupidest analogy is open source projects having a hard time accepting LLM generated PRs from the masses, because review process is the bottleneck.
No idea how to fix this, to be honest. In coding world, with some mental gymnastics, I can see code not being reviewed by people anymore. In courts, things generally have more consequences, and you can’t really roll back decisions that easily.
you could probably write a cool little gotcha of an SF short story about a barren wasteland of a planet that keeps broadcasting out legalese that's revealed to just be LLM chatbot lawyers pedantically arguing with one another about xeno legal doctrine
The legal system, which is a machine/technology by itself, will be eaten out. I wonder what will replace it. Botnet law arbitrage? – Personal assistants constantly negotiating with each other to avoid permanent civil lawsuits?
the cost of making a legal argument can collapse while the cost of reaching an enforceable, legitimate decision may go higher, which will gate the "justice" system even more.
It's really not. The legal system is slow and inefficient because it's a deeply pipelined system built to maximize the throughput of the bottleneck resource: judges. Judges are constitutional officers who exercise independent authority in meat space and thus are necessarily limited in number. The rest of the design flows from that.
If you got the judge, all the parties, and all the witnesses in a conference room together until the case was resolved, you could probably handle a lawsuit in a few months. But each judge has hundreds of cases pending before them, so that would never work. Instead, you get something like how a GPU works. You do some work on a case, submit the work to the court, then work on something else for a few months while you wait around to get the results back. Then you do some more work and submit it to the court, then go do something else for a few months while you wait to get the results back. A few months of actual work gets spread out over a few years that way.
Why stop there? The bottleneck is not some immutable force of nature. The number of judges is determined by legislative action. It is within Congress's power to allocate new federal judicial seats, and likewise at the state level with the equivalent lawmaking body (for most states, at least). Why don't they do so?
Because federal district court judges have a tremendous amount of power. You both don't want 10,000 of them running around and you don't want to water down the qualification for the position--if you did do that, then that would devalue their judgments, everyone would appeal, and you'd just shift the bottleneck up to the appellate courts.
At the end of the day, litigation is conflict resolution. You don't just need a decision, you need a decision from someone authoritative enough to bully Fortune 500 CEOs into submission to accept the judgment.
You presented the concern from my adjacent comment perfectly (“LLM performance: common law vs civil law” essentially). So is AI possibly just growing the “Reverence for Professional Experience” factor that plays such a big role for legal compensation here in the US?
Creating virtual products get cheaper, so shouldn't it even decrease if everyone vibe codes their app for less than a dollar instead of hiring a dev team for 50k or ordering a white label app or spending 3 dollars to buy an existing app?
I don’t know in the USA but in France, if it’s deemed that you launched a lawsuit knowing very well it wouldn’t succeed, you are susceptible to get a 10k€ fine. Even jail in serious cases.
Yes. And you(r lawyer) can collect lawyer's fees and you can be made to pay the court fees, if you lose.
IME the American legal system is set-up to discourage litigation, though. A common tactic is to bury your opponent in the threat of heavy damages or jail-time to get them to settle for what you were originally after, which courts are perfectly happy to facilitate because it gets a potentially lengthy trial off their dockets. They'll punish (or be biased against) whichever party seems responsible for not accepting a "reasonable" settlement.
Algorithmic abuse of the system to extract payments already exists in the form of the debt collection industry.
Why the hell are they not fighting fire with fire? This is not sustainable. But it is annoying that the AI labs get to play arms dealer, selling to both sides.
They don't have a choice. If you're a firm and you work with IE a bank, the bank uses AI, their whole supply chain uses AI- your counterparty uses AI- it's a classic arms race with even more social pressure involved (since daily news keeps banging on about AI).
Genuinely wondering (aka not snarky): Has anyone found frontier models to provide useful research in the context of European civil law systems?
Your comment made me wonder if there are any halfway-acceptable model benchmarks for law tasks? Specifically I’d love to know how the frontier models’ abilities compare between common law vs. civil law systems. My guess would be that an AI in a common law context should have a clearer idea of how a specific case is interpreted/accepted by (common law) practitioners, whereas trying to rely on AI in a civil law context, like Germany, can be daunting. In a few Germany-specific recent examples, the models feel like they present only (maybe too stubbornly?) the “civil law”-based laws. All while negating much of AI’s research benefits because civil statutes are portrayed as being absolutely accurate, binding, and their enforcement (and thereby the legal reality) being uniformly applied. Am I making this interpretation up? If so, how can I prove myself wrong?
I don't know of anything that is working yet, but I know some people working on it (intentionally vague).
Roman law, on which the legal systems of Germany, most of Europe, Turkey etc. are based relies more heavily on statutes than Anglo-American case law, but cases do play a role there, too. That's why a practically useful system also needs to have access to court decisions, commentary etc. - and while the statues are in the public domain, a lot of the other knowledge is owned by specialist publishers (such as: C.H. Beck, Nomos, Mohr Siebeck, De Gruyter, Otto Schmidt, C.F. Müller, Manz, Verlag Österreich, Schulthess, Stämpfli, Dalloz, LexisNexis, Lefebvre Dalloz, Giuffrè Francis Lefebvre, CEDAM, Giappichelli, Il Mulino, Aranzadi, Tirant lo Blanch, Marcial Pons, Dykinson, Tecnos, Lefebvre, Almedina, Gestlegal, AAFDL, Kluwer, Boom juridisch, Wolters Kluwer, Larcier-Intersentia, Anthemis, Sakkoulas, Nomiki Bibliothiki, Universul Juridic, Hamangiu).
Internationally, legal systems appear to undergo a sort of convergence, which means statutes increasingly matter in the U.S. and cases increasingly matter in European law (according to a law partner friend).
So those lawyers will likely want to know about the likeliness of it, strategies to work around it, and overall mitigations.
It might be helpful if the company launching this product included such information in their blog post instead of ignoring it in a field with such a high cost of getting it wrong.
agreed, but i dont think there are many (any?) product launch blogs that cover all the fine details and legal minutia of the product.
that all typically comes in pages of terms of service, purchase contracts, SLAs, conversations with your rep, knowledge base articles, and that sort of thing. not in the initial marketing post.
Does hallucination matter for this application? We've moved beyond raw recall being that important, it seems like for law specifically all relevant facts will be cited and checked easily by humans.
> all relevant facts will be cited and checked easily by humans
I've talked to a lawyer about how they handle this. They do indeed double-check everything, since it'd be embarrassing (or worse) to send hallucinated statements to opposing council or to the court. They still find the assembly a huge time saver
But based on stories in the news on the subject, not everyone has this same level of diligence
Sure but just like generating 100x more code, someone has to review it. So you are wasting everyone in court's time (defendants, prosecutors, judges, staff) by making them parse through what is quite often a bunch of hallucinated slop. Time that could be much better spent on parties who prepared and reviews their own arguments.
The lawyers I know are very fixated on the problem of hallucinated case citations which is amusing to me as a onetime programmer, since case citations have a well-defined syntax and would be relatively easy to check programmatically.
Which makes it all the more bizarre that LLMs have this problem. Claude Code runs the code it generates through a compiler, why can’t an LLM run its product through a cite checker? I’ve seen LLMs fabricate citations.
"We have dangerous AGI that can destroy humanity."
"Also, all of your sensitive legal documents will be totally safe with us."
"Also, for some reason even though we have AGI and selling tokens is a fine business, we need to sell a new product specifically targeted at a very high margin and lucrative industry."
At the end of the day they're a VC-backed company that has to cash in on their brand reputation. Legal is even higher margin than coding with less discerning buyers.
I'm not familiar with the Vals AI Legal Research Benchmark. But their website has other frontier models' scores, and the scores OpenAI is now revealing for "Astra for Law" are slightly less than Claude and Muse:
> The top is a three-way tie: Muse Spark 1.3 Max, Claude Opus 5, and Claude Fable 5.1 all reach 55.29% all-pass accuracy, a clear ~6-point step ahead of the next model. [Astra for Law reached 54.0%]
> Under partial-credit scoring, Claude Opus 5 reaches 90.58% weighted pass rate but 55.29% under strict all-pass grading, where every rubric check must pass. The gap shows models often get most of an answer right but fail on one or two required elements. [Astra for law reached 90.0%]
How long before we start building detailed models of each judge trained on all of their legal output, and then test various legal theories against those judge models in virtual moot court? Craft each pitch to the legal idiosyncrasies of the batter. I assume that real lawyers do this routinely and could use a simulator.
I feel like this is on a different trajectory than what LLM-based tech does. This type of individualistic extrapolation is one of the things they're really bad at, in my experience.
What worries me is the step after this. If god forbid this proves successful and models accurately predict specific outcomes, people will start to ask whether the solution to AI slop lawsuits is to do the judging with AI too.
open-ai have proven they can make good / decent models but business strategy is just spray and pray.
they need to pick a lane and optimize for it. coz at their size they can't serve the application layer (a.i startups who can fine-tune models will eat their lunch)
if they gonna do a consumer play - then go ham on that.
otherwise they're gonna get caught in the dreaded middle valley.
They have a lot of compute, so I think it makes sense to spray and see what works. Anthropic is limited in that regard, and focused in coding/tech, but OpenAI don't need to do the same. They even got the lead without having to focus in coding only, which is remarkable.
They can't be seen to commit to strongly to a specific product experience, because if they are understood as a regular tech product business that has way different financial scaling considerations than "superintelligent everything-factory"
I suspect that half-assed announcements like this are a result of different people internally with conflicting incentives resulting in a split-the-baby solution.
I think the comparison is unfair. Google had/have a product that was 10X better than anything else at the point of release (search then chrome). And they have several money making products like YouTube and android that are either singular or in a duopoly.
OpenAI doesn't have any of these things. They have products that they're paying for customers when the market they're in is rapidly converging on fighting for API reasoning as part of enterprise systems and fighting a race to the bottom for fickle consumer solutions that will be eaten by open source once they have to make money.
Maybe they had a brief window for dominance of information search (or maybe it was only ever going to last as long as Google releasing all their internal research) and maybe they had a brief moment of monopoly till Anthropic got going but theyre not in the same dominance position as Google.
Enterprise is eventually get caught (if it isn't already) by the Microsoft/Google. Because with office/teams/suite they were already in every enterprise, and that just added a new tool to existing offerings.
Provisioning and contracts and data retention was just an extension to review of existing ones.
Nobody serious is going to risk sending sensible data to OpenAI/Anthropic, etc because "the benchmarks have shown +8% performance there and +2% there". Irrelevant.
So OpenAI is partnering with Latham Watkins, Freshfields is partnering with Anthropic and Kleiner Perkins is building their own. It'll be interesting to see which wins out here, I don't see how those partnerships can end well for the law firms unless they're making an assumption they'll be sucked dry of USP but the revenue split from the AI labs will make up for it. Why would I pay a premium for Latham Watkins when every other firm can get their expertise and experience in a subscription, and add their own on top?
They'll be operating under a ZDR. Labs will still get some data but I wouldn't go as far as sucked dry of USP. Firms are very aware of the value of their USP.
Can you imagine the sort of corruption that is possible here? Like if you have a lawsuit against someone or some entity that openai or their investors have business with...
In an utopian society, lawyers are an unnecessary profession. Laws should be clear and simple so the common person can be their own "lawyer". LLMs help with that goal.
> In an utopian society, lawyers are an unnecessary profession. Laws should be clear and simple so the common person can be their own "lawyer".
Not really. Modern society is complicated, and law is a technology that is a reflection of the complexity of society.
To make an analogy: you wouldn't say that "in a utopian society, engineers are an unnecessary profession. Buildings should be clear and simple enough that a common person can be their own structural engineer," because that would mean that building technology would no longer handle a lot of the problems we expect it to handle. It wouldn't be utopia, it would be primitivism.
> LLMs help with that goal.
Not really. What they'd actually do is help them produce output they don't understand and lack the competence to evaluate.
The product isn’t meant for you or me it is for lawyers. If you can’t take on personal liability for a badly written contract then you shouldn’t be using it.
I'd expect they could indemnify you against hallucinations or similar if this gets good enough for that to be a very rare occurrence? Or you could buy insurance on it that's cheaper than hiring a lawyer (not a high bar to clear). I wouldn't rely on it currently, though.
I already use them for that, they are pretty excellent at it. Much better than the terms of use generator products that used to exist. That said, nobody cares to sue your business for the most part until you're big enough to be worth it. By that time, you'll have a team of legal analyst to assist you... or agents should I say.
Then again, nobody will have money to buy anything at this rate, so in all liklihood, this is a total non-issue.
So much for caring about the spirit of the law. Now we'll start an arms race for abusing every possible letter of the law.
It's analogous to crypto. Started from some noble anti-authoritarian ideas and morphed into machine that removes any friction for capital - whoever has the most money will keep gaining the most.
It's shoved down the world's throat from some central think-tank, despite the carefully constructed illusion of "deglobalization". Someone is very hell-bent on making the whole humankind plunge into this dystopia.
A headline from Russia to consider:
The Supreme Court approves the plan to deploy AI in Russian courts: the document states that by 2030, more than 95% of judges will have to use AI regularly. The risk matrix cites AI hallucinations and opposition from the judicial community.
Now there will be even less friction to horribly abuse the law. To democratize the law we would need to move in the opposite direction - to always keep it simple and aligned with our intuitions.
The more intricate and complex legal arguments become, the more abstracted they are from their original purpose and spirit.
Hence the crypto analogy - it was also supposed to "democratize", but the opposite happaned - it only further empowered the most powerful. Imagine legal case so purposefully complex that only those with access to best models have chances to participate and win the dispute.
The most interesting part about this to me was how they bench/compare it, like in the example with Fable:
"Given the same prompt, Astra for Law returned two closely matching precedents; in the litigation example, Claude Fable 5.1 returned a holding that had been reversed on appeal, while in the transactional example it reported finding no such case."
It made me wonder if a good deal of law is about finding a way to work in statements with clear precedents without your opposition noticing and then later drawing upon them in court (as settled precedents, in your favor) after the opposition (perhaps implicitly) accepted it. That would clarify a lot about why some lawyers need to spend so much time pouring over and memorizing past cases (even ones that are only tangentially related); because anything they miss could be used as a potential trojan horse by the opponent.
If this is true that must mean there are a good deal of cases settled using precedent "gotchas" where both sides knew that without the "load-bearing" precedent the outcome would've definitely been the opposite. (i.e precedents almost always trump even valid arguments)
Any lawyers here who have used AI agents heavily for their work? From what I've heard, they're currently very good at searching, analyzing and drafting documents like contracts and patents, but some say they suck at interpreting the law.
I’m sure there are a wide variety of experiences out there, but here’s my perspective as a former biglaw associate and current solo litigator:
I have had some success using frontier models from the last 6ish months, but only when I can break up my work into discrete and verifiable tasks. For example, I had ~15k pages of discovery I needed to dig through for a summary judgment motion. Instead of just asking Claude to find the best evidence, I asked it first to run a clean, high quality OCR pass (it was almost entirely PDFs). Then I had it generate embeddings and write some reusable python scripts to make keyword and semantic searching easy for agents. While I was writing the brief, I would routinely ask my agent (Claude Code) to use both keyword and semantic searching to find the best evidence supporting whatever assertion I was trying to make. I trusted it because there were traces I could follow.
In other cases/situations, I’ve tried just giving a model access to all the docs and saying “write a brief arguing X,” but it’s always terrible at this. It writes briefs with lots of evocative jargon and rhetorical flourish, but a low signal-to-noise ratio.
Again, I’m sure others’ experiences differ based on workflow, legal area, etc.
Agreed. Six months ago, it was basically a gloried grammarly.
But lately, I’ve been taking hints from the “company brain” models, where it develops a running model of the case, and assesses each new piece as it comes in and updates the file.
I’ve also been using “Ralph Wiggum”-type models where you pass letter or contract drafts back and forth between agents with different goals (rules compliance, grammar, conciseness, ai slop detector, an opposing counsel critic, etc.). After a few rounds, it’s not perfect — but I start with a very good first draft in my hands.
They are excellent, especially the latest models. That said, (a) I wouldn't feel safe filing something without a real lawyer looking at it; (b) it can't (easily? legally?) do oral arguments for you; and (c) a lot can happen in the hallways outside the courtroom to move a case forward that the AI can't easily do.
In my experience it basically doesn't even try to interpret the law. It just summarises the publicly available law/guidance out there and, if there is a question about how to interpret some provision, might set out the arguments for each interpretation. It doesn't really take a position. It is pretty good at drafting though. (Legora)
Interesting! In other unrelated domains, models seem more willing to take a position. It may be nuanced, but they do tend to take a stance. I think it's good that it leaves the interpretation to humans, but I wonder if this is also some sort of a guardrail to minimize liability...
What I'm trying to say which doesnt seem obvious is that all models are x% correct at benchmarks until they get saturated, and then new benchmarks get made.
Yes, it is. If they had built a specialised tool for a practice area, they could have improved it significantly- but they don't want to be THAT involved.
Recently was involved in legal matters requiring lawyers - I used OpenAI extensively for research, advice etc - I have to be honest, much of it was completely useless - the real world outcome with lawyers in the room was very different from what OpenAI was spitting out.
"Due to a spontaneous loss of alignment, JustitAI 2.3 pleads the government guilty of war crimes, launches ballistic missiles at several bunkers and tropical islands"
Just like breaking crypto in the age of cloud is more about cost than time, this will lead to legal attacks based on the same principle. The biggest wallet wins.
This was already always the case. If anything, making this more accessible will reduce the barrier to entry for whether or not it's worth your time to take on a case. Instead of 50 lawyers spending 100s of hours on a case, you can have 1 or 2 lawyers + Astra working on it and if there's a case you can add more real lawyers.
Many lawyers are owners/partners compared with software engineers who are more like cogs in the machine. They also bill hourly/contingency per case compared to engineers who are salaried. If a partner in a firm thinks they can take on more cases because of AI assistance then they will because that's just more money in their pockets.
With regard to certain legal questions this has always been the case. AT&T Fought the US Government for 20 years and eventually won because the government gave up. Without some kind of national anti-SLAPP law we're all one irritated oligarch away from having our lives financially ruined.
I am curious what level of trust established law firms treat LLMs with.
I wouldn't trust any output from Astra unless they have not nerfed instances ringfenced for that task.
Tried again a simple coding task with Astra maxed out. I did it only partially, left memory leak and hanging pointer. Sol spotted are errors immediately. I put findings of Sol to Astra and it only done partial fix with the rest claiming will remain unresolved. Sol fixed it in a minute.
I then tried Astra on farily long legal document to produce an excerpt. The excerpt had wrong conclusion and logical errors. When pointed out that the source text has the claims other way round, Astra agreed and then when asked to produce the excerpt again, it again made the same errors.
An important take-away from this is the harness is really important. The graphs show that the same model with a better harness performs many percentage points better. OpenAI are getting into the "selling the harness" game in a big way. Why?
What's interesting about that is while not everybody can train or run a model, anybody can build a harness. You and I can build harnesses.
It seems strange that OpenAI would move into a field where any developer can compete with them. I think that tells us a lot about the economics of training and selling inference.
I don't think it says what you imply. I think they're just trying to vertically integrate.
Harnesses are going to be controlled at companies eventually, just like you might not have a choice of OS. They want to make sure they are the complete package.
What we contributed to Reddit and Arxiv voluntarily before, what we contribute as traces right now are all being used to build business verticals by OpenAI and Anthropic. All these business verticals are being used for is more trace collection which would only strengthen these models and render most humans useless because the ceiling for 1000 swarms of Agents to learn is much higher than an average human.
These companies clearly can identify the relevant traces for these business verticals which only strengthens the argument that NS was built on someone else's traces.
Will the legal fees reduce after this? How about training AI models with laws of other countries especially the democratic ones? This is where sovereign AI models are required otherwise they will start hallucinating wrong laws of different countries.
I remember OpenAI was talking about sovereign AI models (OpenAI for Countries), it is time to train Astra India with Indian Constitution.
I guess the answer to the second question is "if there's a business case", and the answer to the first is the same, so probably no. Life has made me a tad cynical... :)
This seems like a very similar set of tools to Anthropic. But I’ve enjoyed using the various frontier models to criticize each other.
Using recursive loops, the output has gone from a high school level intern to a 2nd year lawyer in about a year. It still doesn’t beat the experts, but so much legal work is (legally significant) pedantry, not legal philosophy.
AI will not kill off lawyers, or reduce the amount of litigation. It will increase volume and velocity.
Can someone explain to me the economics of AI and how it intersects with billing by the hour?
I think there would be a strong incentive not to use tools that speed up your work because you'd effectively be able to bill less time?
I'm sure there are some firms out there with more work than people, but still wouldn't it be more effective to hire another human who can then bill at a high rate for many hours?
Lawyers are client-facing. Client wants lawyers to use AI. There are justifiable reasons to use AI (in measured doses). Hence, firms will use AI. They already have all kinds of strategems to increase their billing which AI slots into.
I find the watermarking dynamic to be really interesting in the legal space, as more large model providers provide increasingly powerful legal capabilities, and adoption (presumably) also increases. Attorneys aren’t the same as developers as their work can be traced back to them, and there are personal bar licenses and reputations at stake. I wonder if knowing the likelihood that AI generated something helps or hurts in that respect.
I also see a lot of watermark removal services popping up as a result.
Congratz to all the laywers that trained it and told OpenAI their business cases by using ChatGPT in their practice. You took the first step to make yourself obsolete.
Snarky comment I know, but I believe that is some truth to it. We SOftware Engineers are equally "being used", especially the Open Source contributors.
I personally only use open weights model for my business stuff. I just refuse to feed the machine with my secrets.
However, I think there is no way back. My accountant happily told me that he is using Claude to organize his cases. To do so, Claude reads his emails, customer correspondance etc. So yeah. ANthropic might know now my financial details, even though i NEVER ever used Claude myself...
It's just fucked. And the worst thing is that my accountant didn't mind. Time to find a new one I guess. Someone with a typewriter.
P.S. I am not Anti-AI. Really not. I have an AI startup and I use AI everyday. Its a love-hate relationship. AI usage has serious risk, and one is that we all train the AIs to put ourselves out of a job.
I have always felt like LLMs are uniquely suitable for legal work. I really have trouble believing we will have hardly any legal assistants and paralegals going forward when these LLMs are so incredible at spotting issues with arguments, figuring out citations, and doing semantic search.
The most interesting use case in my mind is skipping law suits. Obviously you need lawyers in court. But lawyers are people you are basically paying to fight for you.
Instead, if you resolve your dispute outside of court, you don’t need a lawyer. If both parties use ChatGPT to find the relevant laws or read contracts, they could come to an agreement without expensive legal fees.
It’s a bad day to be a lawslop company. When you’re reliant on other companies to do all of the AI part of your AI product, they can just train on your traffic and eventually eat you.
There is a lot of stuff here that I don't understand, but the concept of law firms giving user reviews is quite funny to me. Those reviews are going to be the most non-legally binding reviews ever written lol.
"Felt like a significant step toward legal-focused AI."
"Showed strength across key aspects of legal research."
In a certain emergent country, government employees that work for the law courts, are ignorantly uploading private data (audio, WhatsApp messages, and personal documents) of citizens, at scale, to OpenAi using their free accounts, without checking the terms.
Since the cost of building software is now cheap, there is nothing stopping them from building everything imaginable. They'll soon have an app store with every app built by them and they'll say its for security reasons. Nothing is stopping this coming monopoly
At a certain point why would they sell anything other than services and products their eventual (actual) AGI/ASI builds in literally every market.
When opportunity cost isn't a thing anymore because it reaches every corner of the planet simultaneously faster and builds better than any human can.
There's no reason to let others build on top of AI, except if the AI determines that it needs capitalism to continue because it's paperclip goal is to maximize shareholder value.
I've had top SV lawfirms whos partner charged our company $2000/hr and still couldn't get the right docs in the signature packet. and another getting share counts wrong during raise.
frustrating that law firms have no liability for these mistakes
no, but they are responsible for all work being done under them. What is the point of paying a partner $2000/hr if the work of their subordinates is wrong?
Most interactions with big tech firms involve 4-5 people so a basic phone call is $5k-10k. It shouldn't be unreasonable to expect after paying $80k for a financing round that they issue the right docs to the right people.
LLMs bleed context from the conversation/attachments into the output. This has always been a problem with no real solution other than some crafty iteration/loops.
Context compaction? I noticed llm seem to forget partially or completely the original task when context compaction happens. The problem is more serious with local llm with low context size.
I wonder if this is willful sabotage on the part of the model. In other words, if you ask the model to craft a defense for a morally questionable case, will the model execute the defense in good faith? Or will it apply a training or system prompt bias in subtle ways?
I do believe entry-level paralegals will be made obsolete. In the grand scheme, perhaps it reduces overall cost of legal assistance, which is a net benefit for society.
Interesting to see the callout to companies like harvey in the post itself as consumers rather than competitors? I guess openai isn't quite willing to step into those customer relations themselves?
We need a term for the dark pattern of zooming into just that part of the y-axis where the two closely competing benchmarks sit, to make the top one appear maximally better.
If the laws are too complex for the layman to understand, the laws aren't made for the layman.
AI is a blessing in law. I recently used it to win a dispute and didn't have to pay the legal moatbuilders thousands to exercise my rights in the process.
For law, then medicine, mathematics, physics, etc. I believe LASSI Local Artificial Super Specialized Intelligence is the future, just before it becomes GODD General Omniscient Distributed Daemon
Legal domain is a challenge for most firms, even ours. This is a true game changer even if it looks a bit slop style in the output, the immediate uplift is quite massive.
The domain remains hard due to the lack of availability of high quality LLM-ready data providers in legal space.
surprised the word RAG hasn’t come ho in this thread (except for a likely-LLM-generated-and-therefore-downvoted comment).
>with settings, tools, and context
call me crazy, but I think that this kind of suite, which training-uber-alles people generally dismiss as trivially replicable ‘wrapper’ is actually the differentiating factor for LLM adoption today and moreso into the future.
I’m not dismissing the near-all-out impact of training, but from a competitive busines or industry-structure lens, we’re looking at the three or four big players competing their utmost ultimately, if unintentionally, to turn foundation model access into commodity.
To the capabilities-maximalist minded (typical among engineers - my former life so I’m familiar don’t lack guilt in committing that) folks who will say “Oh the foundation model megacorps will just build out any wrapper whenever one of their third party wrapper plays demonstrates enough adoption, my rejoinder:
Apple did not rebuild an Uber-like app and cut Uber out.
We’ll see how this OpenAI legal services industry wrapper plays out, but I suspect
1) the third party legal wrapper plays will run to other foundation models not doing a legal wrapper, and
2) 3rd party wrappers will do a better job of it since it’s their all-out focus, unlike OpenAI’s whose priorities are necessarily more generalist.
Yes, we all remember the breakout startup failure-arguing quote “Google has entered your space.” That worked for several high profile applications. I believe more of those bets died on the vine than broke-out succeeded however, we only remember the biggest ones that persisted.
If legal services AI turns into one of the Mail or Maps-scale applications of the AI industry, while that would be a fair strategic action counter to the thesis I’ve laid out, the thesis itself would still tolerate it. It’s a question of short-fat tail vs mid-to-long-tail application scope & attractiveness. For example, I think it’s clear that coding is one of these short-fat-tail applications, and the low-no code plays are absolutely having their lunch eaten to acqui-hire ‘death’. I just doubt that the same will persistently transpire facing all professional service wrapper plays.
Agreed. We've basically phased-out the phrase "RAG" for "harness" even though a lot of the discussion is still the data integration rather than agent behaviour (sometimes an MCP or some kind of graph). The argument boils down to "will big company win everything" and the thermodynamics of it tend towards no.
Everyone involved is about to get a swift and thorough introduction to the world corporate contract lawyers. But don't worry, they're extremely sympathetic and understanding when you flub a multi-million dollar corporate contract and will happily refund that money.
People that are saying OpenAI is screwed because a lack of profit, I'm not of that opinion. They are encroaching on every industry they can. They have name brand recognition, a huge user base and are showing they can be a valuable tool to all types of businesses.
As much as I hate to see it. They are now threatening industries like Engineers, Game Developers, Accountants, 3D modelers, 3D animators, Video Production, Audio Production, Therapist, Tax Auditors, Journalists, Authors, Artists, Mathematicians, Product managers, Every type of analyst and pretty much any other job that can be done behind a computer screen.
I used ChatGPT the other day to resink a CPU with termal paste, replace a PSU in my PC and snake my kitchen sink from the wall. I didn't exactly need it's help but wanted someone looking over my shoulder so to speak. It can help you repair all types of stuff, but as far as county / building codes and such, that probably isn't far off. It seems to understand things quite well.
By that point money ceases to have value, because the value of money comes from the motivation it gives people to work. If AI does everything, then money is useless (unless AI like money for some reason).
There will still be a need to distribute and exchange ressources, goods and services.
Real estate, food, energy, mobility.
That will be done with money. Or violence. Either way, a scary future to people when labor doesn't provide any value. Elon promises abundance, but what can he do against greed?
Yeah could get interesting. Things like legal, programming, math, etc. are going to be relatively cheap / free. Meanwhile something like natural resources, building things, yardwork, etc. may go up in value. Or perhaps we end up with a relatively "spiky" economy since 90% of people are just doing physical jobs (the only human jobs with value), and the AI owner class of course gains all the value from intellectual products (with value diminishing towards cost of electricity). Thinking as human value may be roughly as valuable as it ever was (which is to say nearly non-existant other than the ability to solve an immediate physical problem). Might also be that physical strength and stamina are the highest value mating traits, since they will be able to protect and provide. Physically weak intellectual types will have little value since they can't out-think the machines, and they also can't provide for a family by doing valuable physical tasks. IQ overall begins to drop as mating favors intelligence less and less, as it can actually be an impedance to being productive.
For a short time in history, the ability to obsessively focus on intellectually interesting abstract concepts was highly lucrative, but just as quickly we returned to the laws of nature: those who can lift and move succeed, those who can only think are automated out of existance.
We go from a species increasingly seeing themselves as "brains with bodies" to "amazing bodies with weak brains". The limitation of robots and AI in the physical realm, power hungry, mobility limited are contrasted with human values: energy efficient, highly mobile and dexterous, extremely good strength/speed/size ratios. The brain on the other hand, while energy efficient, is completely outclassed and seen as we see our swimming/jumping abilities: a novelty for sports, but nothing we seriously consider a defining human trait.
The smartest humans can fill weekends with novelty pursuits like building circuits, games, programs, etc. But they are about as useful as whittling and hobby woodcraft, something to pass time, but ultimately of no economic value.
there are always be next frontier of problems which require creativity, unless AI become supersmart completely make humans redundant in all cognitive functions.
Most HNers are clueless that if you have Top Talent + Capital you already have an insurmountable moat. OpenAI, SpaceX, Anthropic all have that and none of the regular guys can compete against them (if they choose to attack that industry)
I really hope the bar associations continues to hold lawyers to high standards but I have feeling they may not be ready to handle fallout of AI slop-law.
Wouldn’t this and similar efforts to centralize bureaucracy make AI the new gatekeeper? Without reliable transparent models we’re just trusting OpenAI instead of a hundred top legal firms.
how about "just turn off your company if you're so worried about AI instead of immediately suggesting we use it to judge people's lives and livelihoods"?
this essay details how law firms became sweatshops from 80s. They charge hundreds of dollars to do make busy work by the junior most staff. LLM will kill the goldengoose of the law industry.
Will these models eventually replace all knowledge work, leaving lawyers, doctors, product managers, software developers, and others out of a job?
If the benefits were shared across humanity, that could bring us closer to utopia. My worry is that we’ll instead end up with a handful of even wealthier billionaires and millions of people out of work.
> Will these models eventually replace all knowledge work, leaving lawyers, doctors, product managers, software developers, and others out of a job?
Effectively yes, in the current forms. Those professions will likely evolve, but the traditional forms (ie writing code by hand, writing law filings by hand etc) are all dead.
in that case, legal cases would just come down to who has more compute lol. Many times cases win on their merits, but we've also seen evidence where overwhelming legal pressure can influence cases.
May I just note that there are other jurisdictions on this planet that are less money-biased than than the US one but will be disrupted by law-LLMs as well?
It seems much more probable to me that these LLMs will make good things worse than that they will make bad things better.
If we have enough energy and raw materials to keep building, yes, it will be an utopia made real. But if there is energy scarcity, then other two outcomes can arise.
Doubt it. When someone can go to Astra MD for 75% of what they used to go to the doctor for, then the remaining doctors only have 25% as many visits. When doctors only have 25% as many visits, they have to compete on price and they make less per visit.
Same argument for plumbers. Everyone always jokes about what a good time it is to be a plumber. But what happens when all the software engineers turn to plumbing? Suddenly it's not such a good time to be a plumber anymore.
What happens when the AI makes a mistake with a diagnosis? I can hear it now 'Oh how perceptive you are, yes that pain in your knee could have been a torn meniscus'.
I mean that's what all of these execs are openly telling everyone: they want you out of work, they want their ai to be the one to bring the world to it's knees, they want to surveille every second of your day, they want killer drones to use, they want to lay all of your cities to rubble and build "paradises" on top of them like in gaza.
They also openly tell you what they are afraid of btw: collective worker power. something that is massively lacking in our industry, although i feel like it would be one of the easiest industries to unionize in terms of # of workers.
Interesting interview I just watched about how powerful and dangerous these "wishes" or "prophecies" are especially in the hands of the ultra-wealthy: https://www.youtube.com/watch?v=eR7grHa1NR0
Perhaps, but then someone would do the same but bill you for 7 hours, someone else would undercut them again, until the price reaches a lower equilibrium.
the same way every legal rule is enforced / checked: people look and if it seems iffy they examine/complain, and if a problem is found people get fined/jail.
The grifts continue... imagine something as consequential as Law being advertised as being solved by a statistical word generation engine that regularly gets basic things wrong. Anyone who isn't a lawyer won't know any better but you draft a single document of any appreciable detail and send it to an actual lawyer and it's littered with problems.
I've always said this will be when we get the real Butlerian Jihad, when the AI firms start trying to liquidate the legal profession.
If you automate lawyers out of a job, you can absolutely automate lawmakers out of jobs next. (Not that this would be a bad thing? Maybe pervasive agents for everyone can be the gateway drug to a "this time it's different!" workable direct democracy)
They recently bought a Claude subscription and began using Claude to do the initial read of the documents and output JSON they can import into their internal systems. The work still must be reviewed by an attorney - Claude is nowhere near making the kinds of judgments a lawyer would make about this content - but it has increased their throughput from 2-3 documents an hour to 8-10 documents an hour by killing the busy work.
LLMs have great advantages for this kind of work - but not for decision-making. I just don't see OpenAI ever admitting that.
(I've left some details intentionally vague because this is a very specific area of law and I don't want my friends to be identified without their consent.)
I use these things daily, not against this kind of "AI" at all.
If you’re an employee, you are selling a quantity of your time, not your results. Of course you will have to work the same amount of time.
Your issue is with employment as a general concept, wherein the employer reaps any efficiency gains, not the worker.
That's a factor of 3 to 5 improvement reported on here. If you keep stacking these kinds of improvements, you end up at month to minutes pretty quickly.
Surely there's a middle ground / this is faster than reading all papers front to back.
In one part of Brave New World they bring up that they tried to do more automation, but people got restless without the drudgery. In another part, they bring up that they tried to have a society with only Alphas, but they didn't want to do the drudgery.
I wonder why they never tried combining both?
I thought, something like a medication helping the women develop the baby in 30 days, but seriously damages the host in a couple of days, so the fetus gets transferred from host to host. Some die, but the evil rich don't care. Your idea could also work though.
I realize that this is totally the wrong corner of the internet to post this but I guess today I'll just go with it.
As a premise for science fiction, it sounds like it might be interesting.
At my company we're also in the process of deploying a system that does exactly that. And what's interesting is that you absolutely do not need a frontier model for that, a small VLM (vision-language model), with optionally a little bit of fine-tuning, gives you the same output quality at a fraction of the latency and cost.
But between 10 and 20 seconds to analyze a document with only a bunch of pages just isn't good UX.
It's also not really clear if everyone is going to want a frontier model when the real implications sink in. Maybe we'll get sick of incomprehensible code optimizations and people wile tire of reading AI prose that feels ever-more-human. There might be a few use cases, but who is going to pay for this when providers start charging enough to be profitable.
But that's phase two, phase one is finding localized problems to solve using LLMs and productize them. I'm reminded of cloud tech, where phase one was changing software to run in the cloud, and phase two was optimizing costs.
The bottom line, though, is that PDF OCR is usually regarded as a solved problem. LLMs won't usually do the recognition itself. It will farm it out to established tools which are very good.
If the goal is to only extract the unstructured text from the document, it is definitely solved. Extracting a more natural structure like paragraph separation, tables, header, footers (what is referred as document intelligence) is much more complicated and not fully solved, but I would say almost.
So in many cases the visual way is the only one that works correctly, the textual one is just a shortcut that may be walkable in certain cases.
If you connect an LLM to pilot a plane with passengers, prepare the obituaries, because would happen sooner or latter, an undesired index got mixed and produced a mixed/wrong output. But here, the bigger mistake would be to let an LLM execute commands without human revision, aimed by the deceptive marketing around the A"I" (LLMs).
Disclaimer: I used to work in legaltech, but not those two companies.
You’ve accurately stated that AI isn’t as rigorous as a trained attorney. Doesn’t that mean that every single datapoint must be confirmed by a human?
How is that quicker than just using a human to read the content and make the call? Data entry savings?
It took a human attorney 20-30 minutes on average to manually copy-paste data from these PDFs into a spreadsheet (while also fixing any errors they found in the document and re-checking for quality).
Now, the AI copies everything into the spreadsheet in a small amount of time, and then the human reviews it. It takes maybe ~5-7 minutes to scroll to the appropriate pages in the document, read the lines vs the spreadsheet, and make corrections. So you've gone from 2-3 items an hour to ~8-10 items an hour.
Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.
And Claude should write down the mistake in a sealed envelope, so it doesn't make into the database.
A review that doesn't find the mistake counts as invalid.
Sample is around 300 court dates, shy under 1k files.
At best I'm building a claude skills file.
A better use of these Claude subscription would be to develop the app (which it can pretty much do at that point) and you could iterate to make the workflow even more efficient than your current one.
The difference between a tech-savyy person, and a non-tech-savyy person has always been mostly in the later's head, but this is even more true now that we have pocket assistants who can answer pretty much all of our questions in a language tuned to our level of understanding.
Keep in mind that the task is fixed, so as the frontier of AI advances, you can switch to a cheaper trailing edge system and still get the same or even better performance for this task.
Your experience mirrors my own. AI is great for parsing data that can take up a huge amount of time. My only concern is whether or not it’s done accurately. I wouldn’t use it for anything where mistakes cause serious consequences.
"Write a python script that breaks down this PDF by X feature" would not hallucinate anything in the PDF. Certainly you could trivially double check that all text in the extracted JSON was in the text layer of the PDF.
LLMs change not a whit, and there's no one to take responsibility for the failure (and thus no way to fix it).
As the new variation on the old theme has it, "A computer can never be held accountable, and so very many people are trying to get them make management decisions."
LLM output is nondeterministic and humans take responsibility for the failure the same way they take responsibility of a photocopy is too dark.
You mean, they notice it's too dark right after making it, change the settings, do it again, and give you the good copy?
Because yes, that's my experience of humans.
But they do learn and improve.
The models don't (yet).
It might be that the models have been improving in this sense faster than a human child improves.
For instance, if you can say:
"Refer to the database schema in x.sql as your source of truth for the database structure we want to import into*. Do not invent data, tables or columns that do not exist. Carefully match all output against this database schema and do not create output that doesn't exist if it does not match the schema, simply skip it."
You will end up with a far better result in my experience.
Gotta treat it like a child.
"Sorry for that, Your Honour, but we gave that case prep work to a child."
But now it's comparing already filled columns on a spreadsheet, not copy-pasting every single thing from an (often uncopyable) PDF.
... with a PDF, right?
> not copy-pasting every single thing from an (often uncopyable) PDF
Obviously the PDF is copyable, else your bot would not be copying it.
Whether that is useful measurement I suppose depends on the circumstances.
I see problems in LLMs doing research, making drafts, etc.
I see 3 kinds of potential users.
1. Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.
2. Fresh out of law school graduates.
They think they can do fast work, draft quick replies, do fast case research, enh. They should go the traditional route of interning, learning the "art", they probably won't need it because they will learn "how the law works"
3. Established attorneys. They usually have templates made for most things, over their career, they have established routines of making things fast, maybe it can help them but to say replacing good chunk of work or offloading to LLMs isn't probably going to go well.
There is a reason why interns are made to do grunt work. The "chain of command" is built so they learn on basic stuff, learn to make decisions, etc. Without these basic decisions under their belt, an intern can't take bigger decisions later on. They just won't know....
I used LLMs to replace a lawyer this year to great effect. I successfully advised myself as to the right strategy and drafted an immigration petition in the correct language. Granted it wasn't 100% LLMs (I also supplemented it with "traditional" online research and wrote the letter myself using the LLM version as rough guidance), but still, I certainly would've gone to a real lawyer in 2024 for this stuff but didn't.
Professional advisory work can be divided into two types:
1. Once you know the rules of the game, the next move is immediately obvious.
2. Knowing the rules is not enough to know the next move -- that requires judgement and experience.
LLMs in the hands of someone smart and savvy can easily replace the first type of work.
Lawyers nearly universally tend to: convince clients they need a lawyer; promote extreme views; charge legal fees (that ultimately result in financial damages to the client). There are no winners in this game, except for lawyers, who are having a lovely time. Even if you won the case, you've lost, because the pie got smaller due to lawyer fees.
In my experience, it is incredibly hard, practically impossible, to find a lawyer who will even talk to you. I think this is a valid use case to get at least some understanding what the next steps might be and if it even makes sense to seek legal representation.
There was a post here a few weeks ago about a UK court being inundated with some kind of claims (unfair dismissal?). To submit a claim you do not need a lawyer, but the process is (intentionally?) vague and very complicated, so it takes a lot of effort to figure it out yourself. But AI makes that kind of stuff easy.
AI is not going to replace layers billing big corporate clients millions a month, but it will replace things that are fairly standardized processes, that don't require a lawyer, but today are just too complicated for regular people.
I'm not sure there's quite accurate. The form is really not that difficult to complete.
It's basically: Party details, type of claim, particulars, damages
Obviously knowing these details requires some legal knowledge but I actually think ET claims are easy enough to bring if you've got the time to do a little research and build your case. Almost all ET judgments are public so there isn't the usual difficulty with research as you get with law reports being restricted either.
https://assets.publishing.service.gov.uk/media/65bcbd214a666...
On the margin, AI will drop these guys' billable hours.
Really? I find them quite talkative. But I'm not sure what kinds of questions you are pestering them with?
In any case, the whole discussion reminds me of people using LLM to replace medical professionals.
For both: unless you have a doctor or a lawyer always on standby, you have to make a decision on whether to even go and seek out one of these professionals. By definition, you make that decisions without professional help. I think LLMs can help you there with the initial research to decide whether it's even worth it to contact the professionals.
If there is no precedence for X happening when Y , Z, P, Q, R are special combinations , the judge decides and sets precedence. But if another judge interprets it differently with similar set of circumstances it is not considered incorrect I presume especially if in different jurisdiction unless overridden by a higher court. This is my layman understanding but I think this design is wrong and essentially where human subjectivity controls the system.
Subjectivity is not totally avoidable in law but the goal of the legal system should be reduce it as much as possible along with ambiguity.
Law is intentionally not code. The world is not something you can program.
Since this is a thread about using ChatGPT for law, I should note that this is actually one of the most important reasons you'd rather hire a lawyer than ChatGPT to represent you. A good lawyer or firm will know the local judges and how they tend to lean on ambiguities in their area of expertise, and will be able to use that information to both advise you and tailor their arguments to appeal to the judge's sensibilities. There is an element of unfairness to this, to be sure, but you'd find much more unfairness really quickly with machine-generated judgments because of how messy and non-compressable real world circumstances are.
These situations usually are not ones that an individual can justify the time or money to contact an actual lawyer, but then if they do decide to contact one they will come in with better questions and more of a sense of what they are expecting.
This is similar to medical. Should you use LLM to diagnose yourself, treat yourself with prescription drugs you buy from shady gray market online sellers? No. But you can use it very well to know when it's time to go to the doctor and what to ask.
LLMs can help with that. I don't think they are worse at this than me trying to figure this out all by myself.
"You can't trust the output, it doesn't understand bigger systems"
"Its an art, you need to learn the ropes of it to truly write good code"
Its a very dangerous line of thinking. Software engineering will never be the same, as writing code has basically vanished from the daily workflow. Not for every specialized usecase, but for many.
That might be true, but that doesn't mean you benefit from training up juniors.
3 kinds of users that don't scare the shit out of you.
My buddy works with lots of folks that have gone all in on Harvey. There are other kids of users.
That very first image uses a different inflation indices for each of the two lines in the graph.
They also only post pictures without linking to sources.
You havent given any proofs or even comments that the work is the same level of quality or accuracy.
The statements in the post are opinions, there is no actual PROOF they are true, and thats my point.
most comments I read here don't seem to realize that different areas of law have very very different economic models and don't even mention which one they think will be affected or why, they just sort of lump it all together.
For example: It is highly unlikely llms will have any meaningful effect on high value personal injury law - I don't see a 5 million dollar case being handed to an LLM when the majority of the cost is in trial aids and not even lawyers. It may affect where and how they advertise. It may affect how they work. But it seems really unlikely to put any of them out of business any time soon by people doing it themselves.
Will it affect other areas more? Maybe. Probably? But so far I haven't seen a ton of comments that make specific enough arguments that they could really be debated or responded to effectively with a useful opinion
No. It'll be like software. Entry level employment will be affected. You wont want or need associate attorneys when you can hire a brilliant AI associate for 1/10th the price.
In commercial law, the business model is to rent the associates out to clients at rates higher than you pay (d/t your supervision). No client will pay $350/hour for your AI.
And for direct billing, it's not your interest to do things faster for the client (at risk of making a mistake).
So the real question is whether these pay-by-the-hour models will die as a result of competition from fixed-price transactions made viable via automation. I think not because the entire difficulty of legal work is that it's a complex bundle of issues until you sort them out; once sorted, there's really little to do other than execute (read: paralegals).
But they may be willing to pay $200/hour for your AI (plus, of course, your final expertise in the matter as the lawyer), and the firm can ditch the paralegals and pocket the difference.
The point is, if you need fewer grunts doing the busywork, you can charge less and be competitive.
But they already do! There are cases of professional service firms creating their reports using LLMs, there have even been a few prominent scandals re the fake data generated in them
Whether everyone at the labs knows it or not, this it what they are doing to entire industries, starting with software: Embrace, extend, and extinguish
https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...
What a time to be alive!!?
Now they’re coming for lawyers. Law also has “code”. It’s text-based. It’s arms extend to every industry.
Who knows who’s next.
Honestly, if that happened in a 100% trusted environment, it might be cool. But, there is no money in that. Well, maybe Apple would do that? Private Cloud Compute is cool af.
Still, we are far more likely fully cooked than fully trustworthy personal AI agents happening at scale.
There will still be jobs for competent associate attorneys to prompt LLMs and review output. But frankly a lot of attorneys — especially many of those who attended "Third Tier Toilet" (TTT) law schools — were always a waste of money and should have never entered the profession in the first place.
Divorce is expensive because divorces are acrimonious and people in that situation would rather spend the assets fucking each other over than letting the other person get them.
On the other hand most couples with kids are not married, so the separation doesn't require a lawyer...
Kentucky became the first state to pass a law making equally shared custody the default arrangement in divorces and separations.
https://www.wsj.com/us-news/law/the-equal-custody-experiment...: Divorce Plunged in Kentucky. Equal Custody for Fathers Is a Big Reason Why.
huh? Many/most are no-fault jurisdictions with strict rules on dividing community property. Divorce is easy and cheap, unless it's hard to find assets.
So like a group of lawyers among the "divorce industry" who set prices for divorce and stuff? or wtf do you mean?
I also have friends in the world of high-volume personal injury (IE the morgan and morgan's of the world) but they also aren't being replaced by LLMs so far.
Any time you are doing anything important, where the outcome actually matters. For example, I was recently applying for a study permit and if I generated my application using an LLM instead of consulting with an immigration lawyer and writing it myself, I would not have gotten my study permit. I was recently applying for internships, and if I hadn't paid a real recruiter to review my resume and advise me, I would not have gotten my interview. And trust me, I spent at least an hour trying to get an LLM to generate what I wanted, but no matter how I begged or described what I wanted, it was no use. Good luck fine tuning an LLM to be good at any of this.
> they are getting consistently better
No, they are not getting better, and they will never replace an actual human expert, because a human expert has the actual experience and up-to-date knowledge of what they are an expert at, and also all the complementary context and the current state of affairs in that area. The LLM has none of that.
There are plenty of areas where a "low bar" is more than enough. "low" doesn't necessarily mean bad here, it can simply mean "not the best".
I would argue the majority (in the true sense of the word) of white-collar service work does not require an expert.
It requires someone who has reasonable qualifications in the given field, and the motivation ($) to help you with your task.
So no one needs a top-shelf $900/hour lawyer to draw up a basic low-stakes contract where both parties agree to all the terms as written. You just need someone to ensure that the contract is legally valid as written.
PS: I use LLMs every day, basically let Codex write most of my code (after 15+ years of professional programming), these are amazing tools, smart in some ways, but completely demented in others. People just need to fucking stop being AI doomers and AI coomers. Is this technology great? Yes. Is it worth 10 trillion? Hell fucking no. If we get to a point where RSI is a thing, then maybe. So far it's not a thing and its not even close.
Lol tell me you don't know any practicing mathematicians without telling me you don't know any practicing mathematicians lol.
In a couple of instances I've come to the table better informed about the relevant law and case history than the attorneys were. In many ways I think this is a matter of people understanding that none of us are infinite human databases, so you can't know everything. Instead of rejecting LLMs (perhaps being passive-aggressive), they should embrace them as well as better-informed clients.
Having used LLMs for software development I know full well just how terrible they can be if you don't know what you are doing, don't guide the work correctly, etc. So I am not taking legal LLM analysis as ground truth, but it isn't garbage either.
Its teh same reason claude for finance hasn't turned the finance world upside down.
This is getting tiresome seriously. Why wont these geeks learn some lessons?
Its just like code I suppose, if you can read and understand and validate, you can use it to scale and otherwise it could end up being a vibe effort.
I have worked with many lawyers in my career. The bad ones will look at a contract and tell you fifty things that can be improved. Meanwhile, you take the same contract to a really great lawyer and they tell you that's it's not worth fighting over the changes and to just keep it as is. It's kind of like how a beginner or intermediate software engineer might tell you how you can build out a k8s or auto-scaling fargate cluster for your project and a really senior engineer might look at the same thing and explain that it's not worth worrying about scalability right now.
I find that when I feed contracts to ChatGPT (which I do all the time) it will try to find 50 things wrong with it. For example, as a test, i fed chatgpt an employment offer from a US government agency. It came back with 1152 words and 14 recommended (some of them substantial) changes. Meanwhile, any half-decent employment attorney would probably laugh at me if I asked them to redline an employment offer from the State Department.
That might not be due to the contents but rather due to the fact that they won't negotiate I'd presume
The same has happened with modern finance. Despite more technology that has power to drive more transparent/efficient markets, we have less perfect information sharing and a larger group of middlemen capturing a growing share of profits of the economy
Even without that, I think it'll be extremely valuable to clients to allow them to answer simple questions without a lawyer, figure out the lay of the land so they can supervise their counsel, etc.
But programmers have this concept of a "code review" where another person looks at the code to look for problems.
We use this same technique with our LLMs. Most mistakes are caught by having a second LLM look at it. Doesn't even need to be a different model - just make sure it has a different context.
I find that the number of issues discovered is noticeably higher if you do use different models though. I'm doing some very finicky things (formal semantics) and find that there's value in review panels as large as 5 different models from different families. It gets even more profitable if you set it up as a truly agentic panel where after writing their own separate reviews they get to see what others have said, and adjust their positions or defend them etc. Some models are not that good by themselves but can be surprisingly good at finding flaws in others' reasoning (Grok for example).
Not so in other knowledge work. There's no test harness for a contract and error is non-recoverable. Likewise in finance. There are specific ways of doing things and these ways are many times set in regulations. LLMs can assist all day, sure. But replacing the human, in highly regulated, zero tolerance for error environment?
It doesn't matter how smart someone is, they need specialized training to be good at these jobs. Specialized training in the area the company specializes in.
There is a category error in all this that is hard to think about because of the normal discourse and ordinary language. We say people work in "finance" but no one works in just "finance". They work at a company that has a specialization within "finance", inside a hierarchy that has specialization on top of specialization.
What we really need is exactly what we don't have and aren't going to get. A type of LORA that generalizes the task specific intelligence needed from a very small sample size and that in practice makes so many less mistakes in a highly regulated, zero tolerance for error environment that it is irresponsible to not use the model.
I have worked in this type of environment for 3 years and I have made zero mistakes in 3 years. The people that make even a small number of mistakes get fired.
Any real automation in this area is going to be incredibly slow and piecemeal over a long period of time because even an amazing model would need a long time to prove itself against what the human standards for error rates are.
Even the ensemble average error rate on a large number of tasks in space would not be good enough. It needs to be an average error rate over time.
Logically, this actually doesn't make sense strictly speaking because the sentence creates a paradox: doesn't it make clear whether it includes itself or not, and each reading ends up in trouble. There is a "tradition" in law around the world to accept the only benign reading of such clauses, which I always found funny given that in all other ways lawyers adopt the most adversarial mindset imaginable.
>> You should know - for coding they make terrible mistakes as well.
>> But programmers have this concept of a "code review" where another person looks at the code to look for problems.
More to the point, programmers have this concept of a complier - which delivers 100% success in catching uncompilable errors.
Lawyers don't.
Defensive much?
That is an incredible statement that could not be further from the truth. Large scale adoption of email, searchable document databases like Westlaw, LexisNexis, PACER, etc.. , OCR Software, electronic signatures, and tons more have had a much more defineably positive impact on the legal profession since the internet came about.
A lot of contract-lawyering involves getting ahead of when things go south and the two previously agreeing parties find themselves in disagreement.
If the expensive person is still better at getting what _you_ want from a contract then it’s always worth doing that.
There’s a reason lawyers are so expensive to begin with - they do very complicated stuff with very expensive consequences if done badly.
The interesting question is whether AI can fill a niche these can’t — or, alternatively, help a layperson evaluate a contract they’re asked to sign better than they could themselves.
Contract law has evolved because transfering the ownership of things is fraught with difficulty. Ownership as a concept itself is notoriously hard to define.
Consider housing - when you buy a house you own that land. Except if its an apartment, you just own the internal walls. Or when the government aquires your land through eminent domain. Or mineral rights - which don't belong to you, you just own the top 5 feet of topsoil. Or if you neglect a property and some squats in it. Or if you were given a fraudulent payment, and they take back the property.
Just getting people to agree is notoriously difficult when people have different often conflicting desires.
Do you know how difficult and costly it is to sue an attorney for malpractice?
With AI models this option does not exist. I can't sue OpenAI because the contract that I got ChatGPT to draft carelessly exposes me to some type of financial loss. I can't contact the state bar and have Open AI banned from drafting up contracts or doing any other work that lawyers are licenced to perform. Even if I could it would not compare to a lawyer loosing their license to practice for life. If I tell a friend or acquaintance about my negative experience they will still likely use OpenAI for professional work and chalk up my experience to a skill issue or premature use of the technology.
Also note the use of the word "professionals". I was referring to doctors, engineers, accountants etc too.
> This risk functions as a guarantee.
Except it doesn't, particularly as far as lawyers are concerned. There are almost a million and a half active attorneys in the US. Just 500-600 are disbarred every year, and almost always for money-related violations, not bad advice. Public discipline is also relatively uncommon (about a quarter of 1% of attorneys every year) and usually related to issues like financial impropriety, neglect, undisclosed conflicts of interest and failure to communicate, not disagreement over the quality of advice.
If a lawyer drafts an agreement that causes a loss, a bar complaint will not help you recover your money. You'd need to sue for malpractice. This costs a lot of money and takes a lot of time. Because you're going up against a lawyer, expect the defense to be vigorous, especially, ironically, if you hired an attorney at a good firm. The full weight of the firm will back the attorney and you might find it difficult to find an attorney, as many won't want to be involved in suing a fellow professional in their market.
I don't disagree with your fundamental premise about the value of human responsibility, but these professions have been structured to protect their members and the ABA in particular basically functions like a cartel today.
Not being snarky here but I've encountered some really crappy lawyers and they seem to be doing great ($$$).
It really sucks one cannot publish something like "this guy charged me $5k, wrote my name wrong and forgot important paperwork the day we showed up to court" as that could make you liable for damages. There should be a safe harbor for that.
Also for those things you could sue for damages or lodge a complaint for malpractice.
That's pretty much what bar associations are for, filing a complaint against an attorney at the bar seems like a pretty standard procedure.
>There should be a safe harbor for that.
A safe harbor would be the opposite, it would be immunity for the lawyer.
I had start with asking for a contact with some details I provided. But then I ask the model to be an experienced corporate lawyer and ask me a series of questions to gather the details it needed and then write the contract. The result covered a lot of details that were highly relevant but were absent from the original attempt.
The key insight is that you can lean on the model to cover your unknown unknowns.
There could still be more unknowns that I did not encounter and by definition don't know about.
And the final output could be wrong, wording, exact legal terminology etc.
Can we say working with an actual lawyer is guaranteed to be better, more correct? Lawyers are people and they also make mistakes, wake up on the wrong side of the bed, hate their job etc, just like the rest of us.
This is due to the fact that LLMs are statistical processes that rely on pseudo random numbers in chosing what to say and how to say it to a substantial degree.
Another one suckered by the plausibility engines.
I agree. But, we will need a lot fewer of them. My small company called our lawyer a lot more before the advent of AI. Mostly to clarify a few things or review simpler contracts. Now, for simple things, we do not call them at all.
(1) Rich people will use human expertise and "the rest of us" will use AI models to get by. This could happen in law, but also in medicine (in particular in societies that - like in the US - do not have universal healthcare).
(2) A positive effect in both domains may be that access to knowledge will be broader and cheaper.
I hope I will be wrong about (1) - I would not like to live in such a two-class society.
(Fable and Astra are crap at these tasks)
I'm currently negotiating with someone who I feel is clearly using AI and this is EXACTLY what has been happening
But will their glamorous salaries persist? That is the question that matters.
AI doesn't need to wipe out lawyers. If they just depress salaries enough, virtually nobody is going to want to be a lawyer anymore.
(Not that I think AI will necessarily have that effect. We just don't know yet)
Very few people, if any, are going to bust ass in law school or medical school if they're not going to get rich off it. Better to do literally anything else.
This doesn’t replace lawyers, but paralegals surely will be affected. A good enough model could shrink the number of paralegals needed in a firm.
Law LLM will surely help competent lawyers in their fields with greater sources of knowledge not in their core area of expertise.
some variant of "oh i am capable of generating text i know nothing about, but it helps to have a human i can rely on to tell me more about the text i know nothing about."
pick up a bachelors degree worth of books to humble yourself
The outcome of a case shouldn't depend on someone's fallible ability to recall facts or convince other people or point their index finger*
Law should generally be deterministic. One's CHA stat should have no bearing on justice.
There should still be human judges, but the middleman between the judge and petitioner could easily be removed, and have generally been seen as leeches since forever anyway.
Though, like how the USA opts to remain in the Stone Age with regard to tax filing because of lobbying by tax software companies, this faction of society will flail the hardest before they admit they're obsolete.
----
* What's a lawyer's favorite programming language? Objection C.
A lot of communication is conveyed through more than the words themselves though and even though it's often misinterpreted people put a lot of value in that information. I suspect that even if we took away the ability to see the people involved judges and jurors would still find bias in the tone/quality of people's voices. Even if everything was reduced to text they'd find bias based on word choices, phrasing, spelling, and grammar.
A Mediocre Lawyer not winning a winnable case = not a good thing.
Say a starving child steals a loaf of bread. The law says all theft is at least 1 year in prison. But the judge can see that the child is actually destitute and it was his first time. The judge could shorten the sentence to 1 week; that's fine, there should be room for interpretation and leniency.
But it should NOT depend on whether that child had a persuasive lawyer or not.
A good "AI Lawyer" would simply behave as how ALL lawyers SHOULD: Simply present the facts, the laws that apply to those, offer suggestions for the verdict, and the possible long-term consequences for each possible verdict, without injecting dramatization, exaggeration, or attempts to pull at heartstrings etc.
Lawyers can practice, can represent you in court, etc. that's not going away soon.
An example is in the banking industry, where making a loan used to require deep analysis of a person’s credit worthiness. Now they use an algorithm (credit scores) which means someone with less experience can do it.
If law follows the same pattern, a job done by someone making $500/hour might be done by someone making $50/hour.
but how much of the revenue of the average law firm comes from that, vs the day-to-day "we need this relatively routine contract reviewed" sort of work?
So who is going to define your strategy, represent you in court, adapt the strategy to changing circumstances, negotiate with counter-parties on a mutually accepted settlement? A paralegal? An AI model? Please
How to fix the lack of trust which is needed to conduct a transaction
Contracts affix a moment of trust in time, so even when trust is lost between parties the contract has guardrails for behavior and consequences for acting outside those rails
There are many transactions that are completely inaccessible to most people because the need for a contract or lawyer is too high
LLM use allows for those inaccessible transactions to decrease, and get people further along to the point where a lawyer is accessible as well, and economically viable to use
>Its just like code I suppose, if you can read and understand and validate
I share this, when I asked an attorney for a contract, they were much simpler when compared to self redacted or AI redacted contract, it was as if I were rediscovering lots of concepts and I HAD to put them in paper, even if they were defaults and were essentially no-ops.
Comparable to:
> Mkdir $PWD/path
instead of
> mkdir path
or conditions so rare that they don't merit including.
> x = 0
> if x!=0:
> raise Exception("Bit flip by solar rays or otherwise")
At the very least because the attorney monopoly assigns a 5ish year of training as PoW and natural personhood as Identity as protection for Sybil Attacks.
If anything, the exclusive right of attorneys to represent clients in court, file motions, and enjoy professional secret, is more valuable, as it's not something AI can ever compete at. (Barring a wild recognition of machines as humans, or an overturn of the impossibility of companies to appear in court.)
> API customers including Harvey and Legora will be able to build on Astra for Law, bringing this intelligence into their own products and workflows.
In other words: "no, no, we're not eating our children to prep for the IPO. Don't worry."
OpenAI didn't need to name Legora and Harvey in the second paragraph of the launch post.
They are pre-empting the obvious interpretation of Astra for Law: that moving this far up the legal stack puts them in direct competition with their biggest legal AI customers.
“Don't worry, they can build on us” is a pretty conspicuous message to include on launch day.
They have clearly thought about some pessimistic outcomes.
Vendor-neutrality for LLMs is such a weak thesis all around, whether for providers or consumers. It weakens the product by being promiscuous and gains no material benefit at all.
LLMs are magic byte(byte) functions, it doesn't make sense to say "we have different providers for magic".
Vendor-neutrality helps reduce lock-in, and OpenAI and Anthropic are big enough that the reduction is valuable.
This is everything OpenAI have to say about privacy in this announcement. No guarantees. No promises. Just a pinky-swear promise.
Anyone trusting them–or a lawyer who relies on them–for legal work deserves what they get.
the kind of work mostly done by juniors not yet through their final exam and other "non" lawyers etc.
so it's a slippery slope of "lets just use it for <this> things where it doesn't matter" and then out of laziness and convenience it creeps into all the other places (at least for drafts).
There’s another case making headlines every week.
I get the feeling a lot of them won’t care about this stuff.
Very much a work in progress, only federal and state so far, no municipal codes yet, and no case law yet. Big hole, I know. Also working on making the search ranking work better.
Alternatively, if someone else knows an all-in-one option that exists, I wouldn't mind retiring those crawlers...
If you decide to give it a download, let me know if you have any issues with it/suggestions for improvement.
Then if you want to really get into it, The People should also be able to access trial court level, and at that point you have over 3000 distinct court systems with their own access systems, usually requiring logins and CAPTCHAs, and half of them not even having anything accessible online at all, and the other half only having recent stuff online and the rest rotting in a flooded basement.
Seems like you all are already doing a lot of what I’ve been aiming for with mine. Are there useful ways to contribute, or have you all gotten it to a pretty good place technically, and it’s mostly a matter of spreading it at this point?
- https://okfn.de/en/projekte/bundesgit
- https://github.com/bundestag/gesetze
Found that to be very interesting
[0]https://www.technologyreview.com/2026/06/04/1138391/courts-c...
They are also really good at making stuff up as evidenced by the many, many, many examples you read in the news about actual lawyers using AI to write briefs that are full of errors and hallucinations.
In civil courts, you'd likely get more sympathy from a judge if you represented yourself and admitted your lack of understanding, rather than try to appear as someone you're not because you wrote some prompts and copied the output.
The stupidest analogy is open source projects having a hard time accepting LLM generated PRs from the masses, because review process is the bottleneck.
No idea how to fix this, to be honest. In coding world, with some mental gymnastics, I can see code not being reviewed by people anymore. In courts, things generally have more consequences, and you can’t really roll back decisions that easily.
https://www.ftrain.com/nanolaw
the cost of making a legal argument can collapse while the cost of reaching an enforceable, legitimate decision may go higher, which will gate the "justice" system even more.
If you got the judge, all the parties, and all the witnesses in a conference room together until the case was resolved, you could probably handle a lawsuit in a few months. But each judge has hundreds of cases pending before them, so that would never work. Instead, you get something like how a GPU works. You do some work on a case, submit the work to the court, then work on something else for a few months while you wait around to get the results back. Then you do some more work and submit it to the court, then go do something else for a few months while you wait to get the results back. A few months of actual work gets spread out over a few years that way.
At the end of the day, litigation is conflict resolution. You don't just need a decision, you need a decision from someone authoritative enough to bully Fortune 500 CEOs into submission to accept the judgment.
Creating virtual products get cheaper, so shouldn't it even decrease if everyone vibe codes their app for less than a dollar instead of hiring a dev team for 50k or ordering a white label app or spending 3 dollars to buy an existing app?
You dropped this /s
I don’t know in the USA but in France, if it’s deemed that you launched a lawsuit knowing very well it wouldn’t succeed, you are susceptible to get a 10k€ fine. Even jail in serious cases.
IME the American legal system is set-up to discourage litigation, though. A common tactic is to bury your opponent in the threat of heavy damages or jail-time to get them to settle for what you were originally after, which courts are perfectly happy to facilitate because it gets a potentially lengthy trial off their dockets. They'll punish (or be biased against) whichever party seems responsible for not accepting a "reasonable" settlement.
Algorithmic abuse of the system to extract payments already exists in the form of the debt collection industry.
https://artificialanalysis.ai/models/gpt-6-astra?omniscience...
Your comment made me wonder if there are any halfway-acceptable model benchmarks for law tasks? Specifically I’d love to know how the frontier models’ abilities compare between common law vs. civil law systems. My guess would be that an AI in a common law context should have a clearer idea of how a specific case is interpreted/accepted by (common law) practitioners, whereas trying to rely on AI in a civil law context, like Germany, can be daunting. In a few Germany-specific recent examples, the models feel like they present only (maybe too stubbornly?) the “civil law”-based laws. All while negating much of AI’s research benefits because civil statutes are portrayed as being absolutely accurate, binding, and their enforcement (and thereby the legal reality) being uniformly applied. Am I making this interpretation up? If so, how can I prove myself wrong?
Roman law, on which the legal systems of Germany, most of Europe, Turkey etc. are based relies more heavily on statutes than Anglo-American case law, but cases do play a role there, too. That's why a practically useful system also needs to have access to court decisions, commentary etc. - and while the statues are in the public domain, a lot of the other knowledge is owned by specialist publishers (such as: C.H. Beck, Nomos, Mohr Siebeck, De Gruyter, Otto Schmidt, C.F. Müller, Manz, Verlag Österreich, Schulthess, Stämpfli, Dalloz, LexisNexis, Lefebvre Dalloz, Giuffrè Francis Lefebvre, CEDAM, Giappichelli, Il Mulino, Aranzadi, Tirant lo Blanch, Marcial Pons, Dykinson, Tecnos, Lefebvre, Almedina, Gestlegal, AAFDL, Kluwer, Boom juridisch, Wolters Kluwer, Larcier-Intersentia, Anthemis, Sakkoulas, Nomiki Bibliothiki, Universul Juridic, Hamangiu).
Internationally, legal systems appear to undergo a sort of convergence, which means statutes increasingly matter in the U.S. and cases increasingly matter in European law (according to a law partner friend).
It might be helpful if the company launching this product included such information in their blog post instead of ignoring it in a field with such a high cost of getting it wrong.
that all typically comes in pages of terms of service, purchase contracts, SLAs, conversations with your rep, knowledge base articles, and that sort of thing. not in the initial marketing post.
probably available by the "contact legal sales" link at the top of the post https://openai.com/business/contact-sales-legal/
Don't be too sure about that. [0]
0: https://www.damiencharlotin.com/hallucinations/
I've talked to a lawyer about how they handle this. They do indeed double-check everything, since it'd be embarrassing (or worse) to send hallucinated statements to opposing council or to the court. They still find the assembly a huge time saver
But based on stories in the news on the subject, not everyone has this same level of diligence
You can get an effectively-zero hallucination rate with the right setup already.
"Also, all of your sensitive legal documents will be totally safe with us."
"Also, for some reason even though we have AGI and selling tokens is a fine business, we need to sell a new product specifically targeted at a very high margin and lucrative industry."
> The top is a three-way tie: Muse Spark 1.3 Max, Claude Opus 5, and Claude Fable 5.1 all reach 55.29% all-pass accuracy, a clear ~6-point step ahead of the next model. [Astra for Law reached 54.0%]
> Under partial-credit scoring, Claude Opus 5 reaches 90.58% weighted pass rate but 55.29% under strict all-pass grading, where every rubric check must pass. The gap shows models often get most of an answer right but fail on one or two required elements. [Astra for law reached 90.0%]
https://www.vals.ai/benchmarks/legal_research
they need to pick a lane and optimize for it. coz at their size they can't serve the application layer (a.i startups who can fine-tune models will eat their lunch)
if they gonna do a consumer play - then go ham on that.
otherwise they're gonna get caught in the dreaded middle valley.
I suspect that half-assed announcements like this are a result of different people internally with conflicting incentives resulting in a split-the-baby solution.
You can afford to play silly buggers when you have dumpster trucks of money backing up to your door every day see also: Meta.
OpenAI doesn't have any of these things. They have products that they're paying for customers when the market they're in is rapidly converging on fighting for API reasoning as part of enterprise systems and fighting a race to the bottom for fickle consumer solutions that will be eaten by open source once they have to make money.
Maybe they had a brief window for dominance of information search (or maybe it was only ever going to last as long as Google releasing all their internal research) and maybe they had a brief moment of monopoly till Anthropic got going but theyre not in the same dominance position as Google.
Provisioning and contracts and data retention was just an extension to review of existing ones.
Nobody serious is going to risk sending sensible data to OpenAI/Anthropic, etc because "the benchmarks have shown +8% performance there and +2% there". Irrelevant.
Not really. Modern society is complicated, and law is a technology that is a reflection of the complexity of society.
To make an analogy: you wouldn't say that "in a utopian society, engineers are an unnecessary profession. Buildings should be clear and simple enough that a common person can be their own structural engineer," because that would mean that building technology would no longer handle a lot of the problems we expect it to handle. It wouldn't be utopia, it would be primitivism.
> LLMs help with that goal.
Not really. What they'd actually do is help them produce output they don't understand and lack the competence to evaluate.
As in translating the legalese nondeterminism into neuralese one? That does bring us closer to utopian society for sure.
You. Don't take legal advice from a word calculator.
That will be a decacorn product or more.
https://commonpaper.com/standards
I don't think you can. What I get from this article is that this is not a product they're going to sell to average consumers.
Then again, nobody will have money to buy anything at this rate, so in all liklihood, this is a total non-issue.
And yes, even contracts drafted for millions of $ have oversights and unlawful or unenforceable terms.
It's analogous to crypto. Started from some noble anti-authoritarian ideas and morphed into machine that removes any friction for capital - whoever has the most money will keep gaining the most.
A headline from Russia to consider:
The Supreme Court approves the plan to deploy AI in Russian courts: the document states that by 2030, more than 95% of judges will have to use AI regularly. The risk matrix cites AI hallucinations and opposition from the judicial community.
https://www.rbc.ru/technology_and_media/15/09/2026/6aa8e0342...
Hence the crypto analogy - it was also supposed to "democratize", but the opposite happaned - it only further empowered the most powerful. Imagine legal case so purposefully complex that only those with access to best models have chances to participate and win the dispute.
If this is true that must mean there are a good deal of cases settled using precedent "gotchas" where both sides knew that without the "load-bearing" precedent the outcome would've definitely been the opposite. (i.e precedents almost always trump even valid arguments)
I have had some success using frontier models from the last 6ish months, but only when I can break up my work into discrete and verifiable tasks. For example, I had ~15k pages of discovery I needed to dig through for a summary judgment motion. Instead of just asking Claude to find the best evidence, I asked it first to run a clean, high quality OCR pass (it was almost entirely PDFs). Then I had it generate embeddings and write some reusable python scripts to make keyword and semantic searching easy for agents. While I was writing the brief, I would routinely ask my agent (Claude Code) to use both keyword and semantic searching to find the best evidence supporting whatever assertion I was trying to make. I trusted it because there were traces I could follow.
In other cases/situations, I’ve tried just giving a model access to all the docs and saying “write a brief arguing X,” but it’s always terrible at this. It writes briefs with lots of evocative jargon and rhetorical flourish, but a low signal-to-noise ratio.
Again, I’m sure others’ experiences differ based on workflow, legal area, etc.
Are they separate searches, or some sort of weighted search?
But lately, I’ve been taking hints from the “company brain” models, where it develops a running model of the case, and assesses each new piece as it comes in and updates the file.
I’ve also been using “Ralph Wiggum”-type models where you pass letter or contract drafts back and forth between agents with different goals (rules compliance, grammar, conciseness, ai slop detector, an opposing counsel critic, etc.). After a few rounds, it’s not perfect — but I start with a very good first draft in my hands.
Interesting! In other unrelated domains, models seem more willing to take a position. It may be nuanced, but they do tend to take a stance. I think it's good that it leaves the interpretation to humans, but I wonder if this is also some sort of a guardrail to minimize liability...
How is this a product that you are selling?
<my own consternation, stated>
this vertigoruntime is brand new and his submissions absolutely dominate the front page recently https://news.ycombinator.com/submitted?id=vertigoruntime
the about link https://vertigo.kuber.studio is obvious AI llm spam
isn't that kind of poor?
It’s cleaner.
https://www.reuters.com/legal/litigation/lawyer-state-farm-f...
So no, I don’t know what they mean.
https://www.reuters.com/legal/litigation/appeals-court-warns...
It's everywhere and not hard to find if you make even the most modest effort.
Honestly, this word has lost all meaning, outside of perhaps “any use of AI”.
Well at least I know where team “lawslop” is coming from. Thanks, I guess?
People confuse slop with "bad", but slop isn't bad per se, it only becomes bad when real effort was required.
> broadly : a product of little or no value
> food waste (such as garbage) fed to animals
> excreted body waste
followed a month later by
"Anthropic's Claude inadvertently repeals the 19th amendment"
"Due to a spontaneous loss of alignment, JustitAI 2.3 pleads the government guilty of war crimes, launches ballistic missiles at several bunkers and tropical islands"
This was already always the case. If anything, making this more accessible will reduce the barrier to entry for whether or not it's worth your time to take on a case. Instead of 50 lawyers spending 100s of hours on a case, you can have 1 or 2 lawyers + Astra working on it and if there's a case you can add more real lawyers.
1) Lawyers are not as naive as software engineers and will fight being replaces by new laws.
2) If they are replaced, OpenAI will take a cut commensurate with the amount in dispute (OAI, please credit me for the idea in the IPO brochure).
I am curious what level of trust established law firms treat LLMs with.
Tried again a simple coding task with Astra maxed out. I did it only partially, left memory leak and hanging pointer. Sol spotted are errors immediately. I put findings of Sol to Astra and it only done partial fix with the rest claiming will remain unresolved. Sol fixed it in a minute.
I then tried Astra on farily long legal document to produce an excerpt. The excerpt had wrong conclusion and logical errors. When pointed out that the source text has the claims other way round, Astra agreed and then when asked to produce the excerpt again, it again made the same errors.
Absolutely awful model.
What's interesting about that is while not everybody can train or run a model, anybody can build a harness. You and I can build harnesses.
It seems strange that OpenAI would move into a field where any developer can compete with them. I think that tells us a lot about the economics of training and selling inference.
Harnesses are going to be controlled at companies eventually, just like you might not have a choice of OS. They want to make sure they are the complete package.
I remember OpenAI was talking about sovereign AI models (OpenAI for Countries), it is time to train Astra India with Indian Constitution.
Using recursive loops, the output has gone from a high school level intern to a 2nd year lawyer in about a year. It still doesn’t beat the experts, but so much legal work is (legally significant) pedantry, not legal philosophy.
AI will not kill off lawyers, or reduce the amount of litigation. It will increase volume and velocity.
I think there would be a strong incentive not to use tools that speed up your work because you'd effectively be able to bill less time?
I'm sure there are some firms out there with more work than people, but still wouldn't it be more effective to hire another human who can then bill at a high rate for many hours?
I also see a lot of watermark removal services popping up as a result.
Snarky comment I know, but I believe that is some truth to it. We SOftware Engineers are equally "being used", especially the Open Source contributors.
I personally only use open weights model for my business stuff. I just refuse to feed the machine with my secrets. However, I think there is no way back. My accountant happily told me that he is using Claude to organize his cases. To do so, Claude reads his emails, customer correspondance etc. So yeah. ANthropic might know now my financial details, even though i NEVER ever used Claude myself...
It's just fucked. And the worst thing is that my accountant didn't mind. Time to find a new one I guess. Someone with a typewriter.
P.S. I am not Anti-AI. Really not. I have an AI startup and I use AI everyday. Its a love-hate relationship. AI usage has serious risk, and one is that we all train the AIs to put ourselves out of a job.
Instead, if you resolve your dispute outside of court, you don’t need a lawyer. If both parties use ChatGPT to find the relevant laws or read contracts, they could come to an agreement without expensive legal fees.
"Felt like a significant step toward legal-focused AI."
"Showed strength across key aspects of legal research."
A hefty asteroid will. The whole situation today is so Tower Of Babel 2.0...
When opportunity cost isn't a thing anymore because it reaches every corner of the planet simultaneously faster and builds better than any human can.
There's no reason to let others build on top of AI, except if the AI determines that it needs capitalism to continue because it's paperclip goal is to maximize shareholder value.
frustrating that law firms have no liability for these mistakes
I welcome ai law
Most interactions with big tech firms involve 4-5 people so a basic phone call is $5k-10k. It shouldn't be unreasonable to expect after paying $80k for a financing round that they issue the right docs to the right people.
Our counsel made a few edits where it clearly drafted in favor of the customer instead of us.
Would have been prudent to highlight that this is for (presumably) US law
Is the play here a set of specialized harnesses using their best general model?
AI is a blessing in law. I recently used it to win a dispute and didn't have to pay the legal moatbuilders thousands to exercise my rights in the process.
The domain remains hard due to the lack of availability of high quality LLM-ready data providers in legal space.
Qwen 3.8 Max and Opus 4.8 score highest.
>with settings, tools, and context
call me crazy, but I think that this kind of suite, which training-uber-alles people generally dismiss as trivially replicable ‘wrapper’ is actually the differentiating factor for LLM adoption today and moreso into the future.
I’m not dismissing the near-all-out impact of training, but from a competitive busines or industry-structure lens, we’re looking at the three or four big players competing their utmost ultimately, if unintentionally, to turn foundation model access into commodity.
To the capabilities-maximalist minded (typical among engineers - my former life so I’m familiar don’t lack guilt in committing that) folks who will say “Oh the foundation model megacorps will just build out any wrapper whenever one of their third party wrapper plays demonstrates enough adoption, my rejoinder:
Apple did not rebuild an Uber-like app and cut Uber out.
We’ll see how this OpenAI legal services industry wrapper plays out, but I suspect 1) the third party legal wrapper plays will run to other foundation models not doing a legal wrapper, and 2) 3rd party wrappers will do a better job of it since it’s their all-out focus, unlike OpenAI’s whose priorities are necessarily more generalist.
Yes, we all remember the breakout startup failure-arguing quote “Google has entered your space.” That worked for several high profile applications. I believe more of those bets died on the vine than broke-out succeeded however, we only remember the biggest ones that persisted.
If legal services AI turns into one of the Mail or Maps-scale applications of the AI industry, while that would be a fair strategic action counter to the thesis I’ve laid out, the thesis itself would still tolerate it. It’s a question of short-fat tail vs mid-to-long-tail application scope & attractiveness. For example, I think it’s clear that coding is one of these short-fat-tail applications, and the low-no code plays are absolutely having their lunch eaten to acqui-hire ‘death’. I just doubt that the same will persistently transpire facing all professional service wrapper plays.
He almost singlehandedly killed LSUs football program until people were able to convince him to talk to a real lawyer.
As much as I hate to see it. They are now threatening industries like Engineers, Game Developers, Accountants, 3D modelers, 3D animators, Video Production, Audio Production, Therapist, Tax Auditors, Journalists, Authors, Artists, Mathematicians, Product managers, Every type of analyst and pretty much any other job that can be done behind a computer screen.
We have big problems for humanity.
Real estate, food, energy, mobility.
That will be done with money. Or violence. Either way, a scary future to people when labor doesn't provide any value. Elon promises abundance, but what can he do against greed?
For a short time in history, the ability to obsessively focus on intellectually interesting abstract concepts was highly lucrative, but just as quickly we returned to the laws of nature: those who can lift and move succeed, those who can only think are automated out of existance.
We go from a species increasingly seeing themselves as "brains with bodies" to "amazing bodies with weak brains". The limitation of robots and AI in the physical realm, power hungry, mobility limited are contrasted with human values: energy efficient, highly mobile and dexterous, extremely good strength/speed/size ratios. The brain on the other hand, while energy efficient, is completely outclassed and seen as we see our swimming/jumping abilities: a novelty for sports, but nothing we seriously consider a defining human trait.
The smartest humans can fill weekends with novelty pursuits like building circuits, games, programs, etc. But they are about as useful as whittling and hobby woodcraft, something to pass time, but ultimately of no economic value.
The biggest problem is that we're conditioned by a paradigm that frames these as problems.
https://aeon.co/essays/what-made-law-into-a-white-collar-swe...
If the benefits were shared across humanity, that could bring us closer to utopia. My worry is that we’ll instead end up with a handful of even wealthier billionaires and millions of people out of work.
Effectively yes, in the current forms. Those professions will likely evolve, but the traditional forms (ie writing code by hand, writing law filings by hand etc) are all dead.
there will still be writing initial and incremental prompts by hands, until and if LLMs surpass humans in all intellectual functions.
this is more likely to democratize the legal system by reducing the cost of a good legal team
It seems much more probable to me that these LLMs will make good things worse than that they will make bad things better.
As it stands, it seems far more likely to result in a wonderful life for a few, and an absolute catastrophe for most.
Same argument for plumbers. Everyone always jokes about what a good time it is to be a plumber. But what happens when all the software engineers turn to plumbing? Suddenly it's not such a good time to be a plumber anymore.
They also openly tell you what they are afraid of btw: collective worker power. something that is massively lacking in our industry, although i feel like it would be one of the easiest industries to unionize in terms of # of workers.
Interesting interview I just watched about how powerful and dangerous these "wishes" or "prophecies" are especially in the hands of the ultra-wealthy: https://www.youtube.com/watch?v=eR7grHa1NR0
In the real world, lawyers submit detailed bills and their clients examine them. If you don’t, that’s on you.
If you automate lawyers out of a job, you can absolutely automate lawmakers out of jobs next. (Not that this would be a bad thing? Maybe pervasive agents for everyone can be the gateway drug to a "this time it's different!" workable direct democracy)