Web Scrapping Engineer - Permanent - Remote
70000 Annual
Web Scrapping Engineer - Permanent - RemoteOverview
As a Senior Python Scraping Engineer, you will design, build, and operate large-scale, resilient web scraping systems, with a strong focus on Google and Google-like search environments.
This role sits at the intersection of data engineering, reverse engineering, and systems reliability, and is critical to delivering accurate, timely, and trustworthy data at scale.
You will tackle complex scraping challenges where naïve approaches fail-such as dynamic rendering, anti-bot defences, frequent markup changes, and high-volume data collection.
This position is ideal for someone excited about modern development practices, including AI-assisted and agentic coding, and who wants to deepen their expertise in this evolving space.
Responsibilities- Design and maintain large-scale Python scraping systems that operate reliably against highly protected websites
- Build scrapers for dynamic, JavaScript-heavy pages using browser automation and hybrid approaches
- Continuously adapt to changes in page structure, request flows, and anti-automation mechanisms
- Engineer robust extraction pipelines that prioritise data accuracy, consistency, and observability
- Implement Proxy, fingerprinting, and request-routing strategies to maximise stability and throughput
- Monitor scraping health, detect failures early, and debug production issues efficiently
- Optimise performance, cost, and latency across scraping infrastructure
- Collaborate with data engineers, data scientists, and product teams to ensure scraped data is usable and trusted
- Document scraping logic and maintain operational runbooks for long-term sustainability
- Strong professional experience with Python, including building production-grade scraping systems
- Deep hands-on experience scraping Google or similarly protected platforms
- Solid understanding of HTTP, TLS, cookies, headers, redirects, and browser networking behaviour
- Experience with browser automation tools such as Playwright, Selenium, Puppeteer, or equivalents
- Strong knowledge of HTML parsing, DOM traversal, and data extraction techniques
- Experience handling rate limiting, CAPTCHA, IP rotation, and bot detection systems
- Comfort working with asynchronous and concurrent scraping architectures
- Experience operating scrapers at scale in cloud environments
- Strong debugging skills and the ability to reason through complex failure scenarios
- Experience with headless browser fingerprinting and anti-bot evasion techniques
- Familiarity with containerisation and orchestration tools such as Docker and Kubernetes
- Experience with distributed task systems and job queues
- Background in data quality monitoring and anomaly detection
- Experience working with search, advertising, or competitive intelligence datasets
To apply for this role please submit your CV or contact Dillon Blackburn (see below)
Tenth Revolution Group are the go-to recruiter for Data & AI roles in the UK offering more opportunities across the country than any other recruitment agency. We're the proud sponsor and supporter of SQLBits, Power Platform World Tour, and the London Fabric User Group. We are the global leaders in Data & AI recruitment.