Scraping political ad libraries: how we built it, why Selenium failed, and why Camoufox worked

A few months back, I needed to pull political ad records out of Facebook's public Ad Library for a research project. To keep within privacy guidelines, I am not going to discuss the specific targets, campaigns, or what the records contained. This post is just about the engineering side: how the pipeline works, why standard browser automation broke down almost immediately, how Camoufox solved the bot-detection wall, and the trade-offs of building your own scraper instead of paying for a commercial service.

Scraping Facebook is tricky. The interface relies on heavy client-side rendering, dynamic popups, and aggressive anti-bot defenses. Here is an honest breakdown of how the pipeline works, where standard tools failed, how we handled edge cases, and the compromises we had to accept.

---

How the scraper works step by step

The scraper moves through six stages from raw search terms to finished spreadsheets:

The script starts with a keyword list loaded from a local file. One early surprise: political research relies heavily on hashtags like #Vote4Central or #INElection. Most standard configuration parsers assume any line starting with a hash symbol is a comment and discard it. I wrote a custom parser that recognizes hashtags as active queries while ignoring actual comments, with explicit UTF-8 decoding to keep regional language characters from crashing the process.

From there, the script queries the Meta Graph API for those keywords. If an access token expires, hits a rate limit, or fails, the code shifts automatically to an internal web search fallback that queries the ad library directly through browser requests.

Next comes deduplication. A political page often runs dozens of identical ads for the same campaign with slight geographic variations. Scraping every single copy burns hours for zero new information. The script groups the returned ads by advertiser page ID, counts how many ads that page published, and picks one representative ad that already has a published disclaimer attached.

Once that representative ad is selected, the scraper launches a browser session to load the snapshot URL. Because the ad library relies on dynamic scripts, static HTTP fetchers only see an empty shell.

Inside the browser, the scraper navigates to the snapshot, expands the disclaimer panel, and extracts the campaign details. If any expected details are missing, it triggers an external fallback routine.

Every finished ad is recorded immediately in a JSON checkpoint file and synced to an Excel spreadsheet. If the connection drops or someone stops the script, it picks up right where it left off on the next run.

---

Why Selenium and plain Firefox failed, and why Camoufox worked

The main technical challenge was finding a browser setup that could open Facebook ad snapshots without getting stopped at the door.

Why Selenium failed

We started with Selenium because it is familiar and quick to set up. Facebook caught it almost immediately. Selenium flags navigator.webdriver = true in the browser environment. Even when you try to hide that flag with startup arguments, modern bot detection checks deeper properties such as window dimensions, missing hardware plugins, and automated event timing.

With Selenium, Facebook consistently served a blank white screen, an endless loading circle, or an immediate login prompt that stopped us from viewing public snapshot URLs.

Why native headless Firefox failed

Next, we tested native Firefox in headless mode through Playwright. Playwright handles asynchronous events much better than Selenium, but vanilla headless Firefox still ran into a wall.

Headless browsers leak obvious signatures. They report generic display resolutions, lack hardware audio devices, use restricted default font lists, and lack standard WebGL graphics rendering. Facebook's client scripts detect those headless traits. Whenever our headless Firefox instance loaded an ad snapshot, the React interface failed to render the inner details.

Why Camoufox worked

That led us to Camoufox.

Camoufox is fundamentally different from a Python script that injects JavaScript wrappers before a page loads. It is a custom build of the Firefox browser engine itself, modified in C++. Because the patches exist inside the engine:

• It spoofs hardware traits like canvas rendering, WebGL output, screen resolutions, and operating system font lists at the native level.
• It bypasses low-level fingerprinting checks without leaving behind user-script artifacts that bot detection can inspect.
• It integrates directly with Playwright, so we keep reliable click handling, auto-scrolling, and element waits.
• It supports persistent profiles. We logged in once, saved the session state, and reused that authenticated profile across runs.

Once we moved to Camoufox, the blank screens and bot blocks disappeared. Ad snapshots loaded reliably, just as they would for a regular user browsing on desktop.

---

Data extraction and fallback methods

Facebook does not display campaign details in static text tags. Because the interface is built with React, sections appear and disappear based on user interactions. We split extraction into two phases:

Primary extraction through the disclaimer panel

The browser navigates to the ad snapshot URL. Frequently, Facebook loads the ad inside a search preview modal instead of taking you directly to the full page. When that happens, the script locates the "See ad details" button and clicks it.

Once the detail page finishes loading, the script waits for the accordion heading labeled "About the disclaimer" or "Information from the advertiser". It clicks that button and pauses briefly for the panel to expand. The scraper then pulls the rendered HTML, strips out script and style tags, and extracts the structured campaign information.

Fallback extraction through advertiser websites

Not every ad contains a complete disclaimer panel inside Facebook. Some advertisers provide only the bare minimum text when registering their disclaimer.

When campaign details were missing after the primary pass, the scraper initiated an external fallback:

1. It checked the ad creative cards for an outbound website link.
2. It filtered out generic social media domains like Twitter, YouTube, or Facebook itself.
3. It sent a lightweight web request directly to that advertiser website.
4. To gather complete information, it also checked common subpages like /about and /contact.
5. It combined the details from the external site with whatever was extracted from Facebook.

This fallback prevented records from ending up with empty fields whenever Facebook's own panel lacked the full picture.

---

Handling broken, missing, or dropped links

When dealing with thousands of political ads, you run into messy data. Some links are broken, some ads were taken down, and some ran without disclaimers. If a scraper assumes every link works cleanly, it will eventually freeze.

When an ad lacked disclaimers or bylines from the start, we skipped opening the browser tab entirely. The script marked the ad as skipped in the checkpoint file and moved on. This saved minutes of idle browser time.

For malformed URLs missing an ad ID, the parser checked the link structure beforehand. If the ID was missing, the link was dropped cleanly before navigation.

When Camoufox opened a snapshot but found no disclaimer panel, it saved the URL to output/missing_disclaimer_links.txt for manual inspection later, keeping the automated queue running.

Finally, we added a circuit breaker. If the scraper hit 50 empty results in a row, it assumed an IP throttle or bot flag and halted automatically, preventing corrupted output.

Because every completed or skipped ad was stored in the checkpoint file, we could stop and restart the script at any time without reprocessing existing ads.

---

Compromises and project trade-offs

Every technical decision comes with trade-offs. To make this pipeline stable, we made several compromises:

First, we traded speed for stealth. Real browsing is not instant. To avoid triggering automated rate limits, we added random pauses of 2.5 to 5.5 seconds between ads, along with rendering wait times. As a result, processing a few hundred ads took several hours.

Second, running full browser instances requires substantial memory and CPU compared to plain HTTP requests. Camoufox is lightweight for a browser, but running it continuously still requires a machine with decent RAM.

Third, the scraper depends on Facebook's frontend structure. If Facebook renames buttons like "See ad details" or restructures the disclaimer accordion, the locators have to be updated manually.

Fourth, we ran through a single IP address without a paid pool of residential proxies. That kept our costs at zero and made setup simple, but it also capped our throughput to a single sequential browser tab.

---

How our scraper compares to paid commercial tools

Before building our own pipeline, we looked at commercial scrapers and cloud extraction services. While paid tools have their place, building our own setup gave us several distinct advantages, alongside a few obvious trade-offs.

Zero recurring costs

Commercial scrapers usually bill per thousand requests, per gigabyte of proxy bandwidth, or through high monthly subscriptions. When scraping tens of thousands of political ads, those costs climb rapidly into hundreds of dollars. Our scraper runs locally on consumer hardware and costs nothing to operate.

Total code control versus a black box

With a paid service, you send a target URL into an external API and hope their backend returns what you need. If Facebook changes a button label, a modal dialog, or an accordion layout, the third-party scraper breaks. You then have to submit a support ticket and wait days or weeks for their team to update their code, all while paying for failed requests. With our own repository, when Facebook updates an interface element, we inspect the change, edit the locator in Python, and get back to scraping within ten minutes.

Custom multi-step fallbacks

Commercial Facebook scrapers generally stop at the boundary of Facebook itself. If an advertiser filed a minimal disclaimer with missing contact details, a paid scraper returns blank values and moves on. Our custom scraper links the primary pass and external fallback together. It pulls the advertiser's outbound link from the ad creative, visits their external site, checks contact subpages, and fuses the records. Commercial scrapers do not offer this kind of custom logic out of the box.

Privacy and data control

Using a cloud scraping vendor requires routing your search keywords, target campaigns, and extracted data through third-party servers. Running our own script locally ensures that our keyword lists, session tokens, and output spreadsheets never leave our local drive.

Where paid scrapers still win

The main advantage of commercial tools is raw speed and infrastructure. Paid platforms maintain massive pools of rotating residential proxies and distributed server clusters. They can fire off hundreds of browser tabs simultaneously from different IP addresses without worrying about rate limits. Because our scraper runs on a single desktop connection, we have to proceed sequentially and wait between requests.

---

Improvements we can make in the future

While the scraper works reliably today, there are several practical upgrades we could make to improve speed, resilience, and convenience:

Adding residential proxy rotation

The biggest bottleneck is running from a single home IP address. If we plug in a rotating proxy service, we can remove the sequential constraint and launch three to five Camoufox browser workers in parallel. That would cut total runtimes from hours down to minutes without increasing the risk of IP blocks.

Intercepting internal network responses directly

Right now, the scraper waits for the DOM to render, clicks the "About the disclaimer" accordion button, and parses HTML text. Camoufox can also intercept network traffic directly. By monitoring background GraphQL calls that Facebook's frontend makes when loading an ad, we could grab the raw JSON response payload as it arrives. That would make extraction instantaneous and remove any dependency on button clicks or accordion animations.

Running in headless containers on a small server

Keeping a personal laptop running with open browser windows for hours is inconvenient. Packaging the script into a Docker container and deploying it on a low-cost virtual private server would allow long keyword runs to finish unattended in the background.

Smarter layout detection

Right now, the code searches for specific text strings like "See ad details" and "About the disclaimer". If Facebook conducts an A/B test with different wording, those locators can miss. Adding fuzzy text matching or relative DOM path checks would make the parser more tolerant to minor interface experiments.

---

Final thoughts

Moving from standard Selenium and vanilla headless Firefox to Camoufox gave us a reliable way to view public political ad archives without getting blocked. While commercial scraping tools offer massive proxy infrastructure, building our own tool gave us complete control over custom fallbacks, protected our data privacy, and eliminated ongoing subscription costs. Adding proxy rotation and network response interception will be the natural next steps to take the pipeline even further.