Understanding AI Provider Recommendations in Local Service Markets

arXiv:2609.18341v1 Announce Type: new Abstract: When someone asks an AI assistant which doctor to see or which firm to trust with their savings, the answer is a referral. We audit AI provider recommendations in four registry-backed service domains across the 100 largest U.S. metropolitan areas, matching every recommendation against the official registry for its domain (Medicare clinician and facility records, and SEC adviser disclosures), under three conditions: an open-weight model, a proprietary model without web search, and the same proprietary model with search. Without search, both models largely fabricate recommendations in the domains the web covers thinly. Only 4% of the open-weight model's recommended doctors and 11% of the proprietary model's match a clinician in the queried city, and the open-weight matches are name coincidences: its matched clinicians are no likelier to be primary-care doctors than names drawn at random from the registry. With search, 64-71% of recommendations in the same domains match a real provider. Search also changes who is recommended. Without it, recommended advisory firms carry SEC misconduct disclosures at 3.6 times the registry base rate, even after adjusting for firm size; with search, significantly below it. Restaurants, where quality and visibility are separately measurable, show a 3-5x review-count premium but a rating premium of at most a tenth of a star. Finally, search largely removes the metro-size penalty: without it, real recommendations concentrate in the largest metros; with it, match rates are similar across metro-size ter

AI assistants often provide unverified recommendations

The study reveals that AI assistant recommendations in crucial domains like healthcare and finance often fabricate referrals without search-based retrieval, risking misinformation and poor decision-making. While search improves trustworthiness, the initial lack of verification and variability pose material concerns for users relying on AI. The findings are based on measured audits across large data sets, adding credibility.

AI assistants providing unverified or fabricated recommendations for critical services like doctors or financial advisors pose a significant risk to human well-being. The study demonstrates that, without robust search integration, AI models frequently generate false information, including recommending providers with misconduct records. This directly undermines trust and could lead to harmful outcomes for users seeking essential services.

The study shows that AI recommendations without search are often fabricated and unreliable. This can lead to significant trust issues and potential harm to users. However, the use of search improves accuracy and fairness.

SOURCE     arXiv cs.CY (Computers and Society) (tier 1)
   ↓
DOCUMENT   0c55212f-d7a0-4c59-a55c-a8a64738c2a2
           https://arxiv.org/abs/2609.18341
   ↓
EVIDENCE   2 extracted excerpts
   ↓
MODEL RUN  5 runs, methodology 1.0.1
   ↓
SCORE      -74  (Strongly adverse)
   ↓
CONFIDENCE 84%