We Built AI Lead Extraction and Lost 40 Hours of Data

We set out to solve a problem that burns through sales teams everywhere: manual lead collection from Google Maps. According to Salesforce's State of Sales Report, sales reps spend only 28% of their time actually selling, with the rest consumed by data entry and administrative tasks. Google Maps prospecting sits squarely in that waste category — hours of copying business names, addresses, and phone numbers into spreadsheets.

I wanted to automate the entire process using Claude's reasoning capabilities combined with Apify's web scraping infrastructure. The plan seemed straightforward: scrape Google Maps results, feed them to Claude for filtering and enrichment, then output clean prospect lists ready for outreach.

Our first integration worked beautifully for two weeks. Apify would scrape Google Maps searches for specific business types and locations. Claude would analyze each result, filtering out irrelevant businesses and enriching the data with industry classifications and contact likelihood scores. The output was clean, targeted prospect lists that our sales team could immediately use for cold outreach.

Then we lost everything.

The system had been processing a particularly large dataset — 2,400 businesses across three metropolitan areas. Apify completed the scraping phase successfully. Claude had analyzed roughly 1,800 businesses when our pipeline hit a critical error. We had configured the system to write final results directly to our CRM as the last step. When the CRM integration failed due to a rate limit we hadn't anticipated, the entire pipeline crashed.

Every piece of analysis Claude had completed vanished. The reasoning model had spent tokens classifying businesses, scoring contact potential, and writing enrichment summaries. All of that intelligence disappeared because we treated the CRM write as a blocking operation.

We learned this lesson again building our Outbound Prospecting Agent. Our Contact Intelligence Agent wrote enrichment data to HubSpot as its final step. When HubSpot returned a 403 — missing API scopes on the customer's account — the entire pipeline returned an error and threw away all the intelligence it had already generated. The Researcher had completed, the Analyst had scored, the Briefer had written the report — all lost because a supplementary CRM write failed.

Lesson 1: Make external writes non-blocking. Now every pipeline we build treats CRM updates, webhook calls, and third-party integrations as optional final steps. If the HubSpot update fails, the pipeline still returns the full intelligence brief with a flag: hubspot_updated: false. The data you paid tokens to generate never gets thrown away because of an integration hiccup.

We rebuilt the Google Maps system with this principle. Claude's analysis gets saved to local storage first. Only after we have the complete enriched dataset do we attempt CRM writes. If those fail, we still have the processed leads.

Lesson 2: Batch processing beats real-time for large datasets. Our original system tried to process each Google Maps result immediately as Apify scraped it. This created a fragile dependency chain where any single failure could cascade through the entire operation.

The rebuilt version separates scraping from analysis completely. Apify runs first, collecting all raw business data. Only when scraping finishes completely does Claude begin its filtering and enrichment work. This isolation means scraping errors don't affect analysis, and analysis errors don't corrupt the raw data.

Lesson 3: Geographic filtering happens before AI analysis. We initially fed every scraped business to Claude for evaluation, including obvious mismatches like residential addresses or businesses clearly outside our target criteria. This wasted tokens on analysis that basic filtering could eliminate.

Now we apply geographic and basic business type filters in Apify before sending anything to Claude. A restaurant search in downtown Chicago doesn't need AI to eliminate results from suburban residential areas. Simple rule-based filtering handles the obvious cases, leaving Claude to focus on nuanced business classification and contact scoring where reasoning models add real value.

Test integration failures deliberately. We should have simulated CRM rate limits, API timeouts, and authentication failures during development. Most pipeline failures happen at integration boundaries, not in the core logic. Building failure scenarios into testing would have caught the blocking write issue before we lost production data.

Implement progressive data persistence. Instead of saving results only at the end, we'd checkpoint progress every 100 processed records. This creates recovery points that let you resume from partial completion rather than starting over completely.

Add manual override capabilities. When automated filtering makes mistakes — which it will — you need a way to manually include or exclude specific businesses without rebuilding the entire dataset. We'd build simple override flags that let sales teams adjust the AI's decisions without touching the underlying automation.

The current system processes thousands of Google Maps results reliably, but it took losing 40 hours of AI analysis to learn how external dependencies can destroy otherwise solid automation. Our Outbound Prospecting Agent guide incorporates these lessons into a more resilient approach to lead generation that treats every external integration as potentially unreliable.