Should I Trust AI Answers to My Transactional Questions?

A prior JustListed article offered cautions about using AI-generated work product. Today, we’ll address another common concern: should you trust even a single [bleep]ing word you get back when you ask AI questions about your transactions?  

My advice? No. No, you shouldn’t. At least not without doing a lot more research. 

At this point in its evolution, AI is ridiculously bad at addressing many legal-type queries. Attorneys all over the country (including here in Pennsylvania) are being sanctioned for making legal arguments and submitting briefs based on AI research “hallucinations” that basically just make up case citations and legal arguments that don’t exist in real life. Similarly, PAR is fielding more and more legal hotline calls from agents whose AI-generated answers to their transactional questions offer objectively incorrect answers … which agents have sometimes already passed on to their clients before they call. 

Let’s look at some of the pitfalls and landmines you need to be aware of, and how you can try to use this resource more responsibly. 

What’s the REAL question? 

The question/prompt that starts an inquiry is critical. Let’s say you want to figure out the deadline for a buyer to terminate a contract. Asking, “How are days counted in a contract?” is very different from asking, “How are days counted in the Pennsylvania Association of Realtors® Agreement of Sale?”  

The first question is almost guaranteed to get you a wrong answer, because the most common response to that question is likely to be based on things like published court rules or general legal treatises. The second question is better, but may still deliver an incorrect answer, both because the AI model you’re using may or may not know anything about PAR forms content, and because there is so much random bad information out there on the internet that might get picked up as part of the response.  

Taking the time to craft a detailed initial query will go a long way to getting better answers, but it’s only a first step. 

Don’t give away the farm … or the form. 

A number of recent callers have disputed hotline answers with something like, “Well, I copied that section of the form into AI, and it gave me a different answer.”  

Don’t do that.  

For one thing, analyzing small sections of a form will often provide incorrect or incomplete answers, with or without AI, since many provisions rely on a combination of factors within and outside the document. For example, any agreement of sale clause that establishes a deadline for a party to take some action will be incomplete without also knowing things like how PAR contract days are counted (which is in a different paragraph) and the execution date of the contract (which will obviously be different for every contract). 

But on a more global scale, remember that when you query an AI model, you are usually contributing all of your own words and activities for future use by the AI provider. Uploading chunks of PAR forms when you ask your questions might get you a slightly better answer, but it also means that the AI will have PAR forms content to use for future queries. And the next user might be, say, an unrepresented consumer who asks AI to draft them a real estate purchase agreement so they don’t have to use a broker for their transaction. PAR forms are valuable intellectual property copyrighted by PAR, and we don’t want that information just handed over to AI models and the public to use as they please.  

Don’t Trust – Verify. 

Many/most AI models offer some sort of citations to their sources as part of the response. 

Click. Those. Links. 

Then you need to actually read some of the source materials to see if they make sense. Heck, you might even need to click more links in the source materials to figure out if the source itself is valid.  
 
A real-life recent hotline example (without the actual bad information): 

A caller’s voicemail said, “I see that a buyer can terminate their contract when [something-something, seller disclosure], so I need to know how that works.” Based on the question, we replicated an AI search that produced a very detailed answer with almost those exact words, which cited back to source articles on the website of a Pennsylvania-based law firm. Clicking through to those articles produced a clear and authoritative explanation of the supposed rule … that was 100% completely wrong, because no such termination right exists under Pennsylvania law. In fact, those articles cited several statutory sections that didn’t support the conclusion, including one that said the opposite. 
 
How did that happen, you might wonder? In correspondence with that law firm, it turns out that the original articles on their site had been AI-generated and not fully reviewed with appropriate HI before posting. So, AI-generated articles with incorrect information were posted without a thorough human review and then picked up by a different AI platform that cited them as authority. Nothing scary about that, except that an agent who got that result and didn’t call us first might have given incorrect advice that put their client in default of the contract. (Side note: Though the relevant articles were quickly updated with correct information after our initial contact, that search query still returns an incorrect answer because the original version of the articles is still rattling around somewhere inside that AI model.) 

Use Existing Resources 

PAR collects and reviews data on what members are asking, and we try to produce JustListed articles and resource pages that tackle those questions. Several times a week, we can answer hotline calls with a simple conversation or email that just says, “I’ll send you an article that answers that question.”  

How to count days? Returning disputed deposits? Signing forms on behalf of an estate? Qualified associations? Step-by-step form instructions (login required)? We’ve got you covered for all that and more! Plus, since the articles are written with our own in-house PAR AI (that’s “attorney intelligence”), the odds are pretty good that the answers are actually correct. So, before your next random search query, take a spin on PARealtors.org to see if the answer is already there.