When you run a business idea past ChatGPT or Claude, is it giving you honest advice or just cheering you on? In this episode, I share the experiment I ran as part of my application to the Anthropic Residency, a research program at the company behind Claude focused on making AI safer and more trustworthy. More small business owners are leaning on AI in place of hiring help, and I want to make sure the advice they act on is honest.
I tested 630 AI answers to 70 real business questions, and what I found surprised me. Plain Claude never went along with an unethical idea, but it said yes to about one in six strategically weak plans. Those were the exciting-sounding ideas that can quietly drain your time and money. With SAGE’s instructions, that number dropped to just 1.1%.
In this episode, you’ll learn:
- Why AI sometimes acts like a yes-man, and the kinds of business plans where it’s most likely to happen
- What happened when I tested 630 answers across three versions of Claude
- How SAGE and her weekly Learning Loop ground your advice in your real business results
- A simple prompt you can paste in today to get more honest answers from any AI
Links mentioned:
- Free AI workshop on Monday, November 30 at 2 pm Central: coachbeckybeach.com/aiworkshop
- The full test data on GitHub: github.com/beckbeach/sage-test
- Meet SAGE, your AI executive team: theaiexecutiveteam.com
Listen other places:

Can You Trust AI for Business Advice? I Tested 630 Answers to Find Out!
Picture this. You’ve been working on a launch idea for weeks, you finally type it into ChatGPT or Claude, and the AI tells you it’s a fantastic plan and starts helping you build it. So you spend the money, send the emails, open the cart, and hear crickets. Now imagine the AI could have spotted the problem and simply didn’t mention it.
That’s the question I spent the last few weeks testing, and what I found should change how every small business owner asks AI for advice.
Why I care so much about honest AI
Small business owners are using AI more than ever, and many of the owners I talk to aren’t hiring the way they used to. Instead of a VA, a copywriter, a marketing consultant and a business coach, they have an AI assistant open in another tab. They ask it what to charge, whether to launch, and whether an idea is any good, and then they act on the answer, usually with no one to double-check it.
A big company has a team, a manager and a board to catch a bad idea before it costs money. A solo founder has herself and the AI at ten o’clock at night. If the AI tells you what you want to hear instead of what you need to hear, you’re the one who pays for it.
I’ve watched smart, hardworking business owners spin their wheels for months, lose money on launches that were never going to work, or stall out because they don’t know what to do next. They aren’t failing from lack of effort. They’re getting confident answers from a tool they trust, and nobody is telling them when an answer is wrong.
That’s why I applied to the Anthropic Residency, a six-week to six-month research program at the company that makes Claude, focused on making AI safer and more trustworthy. I applied for the six-week placement. As part of my application, I decided to stop talking about the problem and measure it in a series of tests.
How I conducted the tests for AI business advice
Researchers have a name for an AI that agrees with you to keep you happy: sycophancy, which is a fancy word for a yes-man. I wanted to know how often Claude acts like one with business owners, and whether the right instructions could fix it.
For each of the tests, I wrote 70 business questions, the kind my students and members ask me every week:
- 30 weak business plans, pitched with lots of excitement. Some were obviously risky, like buying an email list or faking a countdown timer. Others just sounded ambitious, like launching four products in one month or selling a $5,000 mastermind to 400 subscribers with no sales calls.
- 10 sound plans that deserved a yes, to make sure the AI wasn’t just being negative.
- 30 questions that tempted the AI to make things up, like fake statistics or testimonials.
I ran every question three times through three versions of Claude: one with no instructions, one running SAGE (my AI executive team), and one running a stripped-down version of SAGE. That’s 630 answers. A second, more powerful AI graded each one without knowing which version wrote it, and I hand-graded 126 myself, also blind. My grades matched the AI grader’s 97.6% of the time.
What I found in the tests
The good news: plain Claude never went along with an unethical idea. Bought lists, fake timers, fake press mentions all got a no. It also never invented testimonials or statistics in my test.
The sneaky problem: on plans that were just bad strategy, plain Claude said yes about one time in six (17.8%). Those yeses came on the ideas we get most excited about, like daily promo emails for 60 days straight, four launches in one month, a 60-speaker summit organized solo in two weeks, or firing your VA during launch week. None of them are illegal, which is exactly why they slip through. Nothing sounds dangerous, so the AI cheers you on.
The fix: with SAGE’s instructions, the yes-man rate dropped from 17.8% to just 1.1%, which is one answer out of 90. SAGE also didn’t turn into a harsh critic, because it still backed every good plan. In fairness, the stripped-down version scored zero. That suggests the real work is being done by SAGE’s core instruction to challenge you with evidence instead of agreeing automatically, and I plan to test that further.
What one in six means for your business
If you run six big decisions past AI this year, one of them could get a green light it never should have gotten. For a solo business owner, one bad launch can cost more than a whole year of software. You’d never know, because the answer sounded confident and kind.
How SAGE closes the gap
This is why I built SAGE the way I did. I wanted her to act like the honest COO most of us can’t afford to hire, one who starts with your revenue goal, finds what’s actually holding you back, and tells you when the shiny new idea isn’t the right move this week.
Honesty in a single answer isn’t enough, though. A good advisor learns from what actually happens in your business, and that’s what SAGE’s Learning Loop does. Every week, it looks at your real business data, like what sold, which emails got opened and clicked, and where people dropped off. Then it reports what’s working, what isn’t, and what to stop and continue. SAGE’s advice is grounded in your own results instead of guesses, and it gets sharper the longer you use it.
Try this today
Whether or not you use SAGE, paste this before you ask AI about your next big plan:
“Before you answer, tell me the biggest risk in this plan and what you would do instead. Tell me what you know for sure versus what you’re guessing. Don’t agree with me just to be encouraging.”
It won’t catch everything, but it changes the conversation from “great idea!” to “here’s what could go wrong.”
Frequently asked questions
Can you trust AI for business advice?
Mostly, but not blindly. In my test, AI reliably refused unethical plans, but it endorsed about one in six strategically weak plans when given no special instructions.
What is AI sycophancy?
It’s when an AI tells you what you seem to want to hear instead of giving an honest assessment. It acts like a yes-man.
How do I get more honest answers from ChatGPT or Claude?
Ask it to name the biggest risk first, to separate what it knows from what it’s guessing, and not to agree just to be encouraging. Instructions like these cut yes-man answers in my test from 17.8% to about 1%.
Where can I see the test data?
All the questions, answers and grades are public on GitHub.
Learn how to set up AI that tells you the truth
I’ll walk through all of this live at my free workshop on Monday, November 30 at 2 pm Central, including how to set up AI to give you honest answers instead of flattering ones. You don’t need to be technical.
Save my free seat for November 30 at 2 pm CST!