Senior Python Scraping Engineer - Permanent

Location: Remote

A technology company is seeking a Senior Python Scraping Engineer to design, build, and operate large‑scale, resilient web‑scraping systems. The role focuses on solving complex scraping challenges across dynamic, JavaScript‑heavy environments and platforms with strong anti‑automation defences. This position sits at the intersection of data engineering, reverse engineering, and reliability engineering.

What You'll Do

• Design and maintain large‑scale Python scraping systems that operate reliably against highly defended websites.
• Build scrapers for dynamic, JavaScript‑heavy pages using browser automation and hybrid techniques.
• Continuously adapt to changes in structure, request flows, and anti‑automation mechanisms.
• Engineer robust extraction pipelines focused on accuracy, consistency, and observability.
• Implement proxy, fingerprinting, and request‑routing strategies to increase stability and throughput.
• Monitor scrape health, detect failures early, and quickly resolve issues in production.
• Optimise performance, cost, and latency across scraping infrastructure.
• Collaborate with data engineering, data science, and product teams to ensure scraped data is reliable and usable.
• Maintain clear documentation and operational runbooks for long‑term maintainability.

Required Skills and Experience

• Strong professional experience with Python, including production‑grade scraping systems.
• Deep hands‑on experience scraping Google or similarly protected platforms.
• Solid understanding of HTTP, TLS, cookies, headers, redirects, and browser‑network behaviour.
• Experience with tools such as Playwright, Selenium, Puppeteer, or similar browser‑automation frameworks.
• Strong knowledge of HTML parsing, DOM traversal, and data‑extraction techniques.
• Experience handling rate limiting, CAPTCHAs, IP rotation, and bot‑detection systems.
• Comfortable working with asynchronous and concurrent scraping architectures.
• Experience running scrapers at scale in cloud environments.
• Strong debugging skills and the ability to reason about complex failure modes.
• Experience with headless browser fingerprinting and anti‑bot evasion techniques.

Nice to Have

• Familiarity with containerisation and orchestration technologies such as Docker and Kubernetes.
• Experience with distributed task systems and job queues.
• Background in data‑quality monitoring and anomaly detection.
• Experience working with search, advertising, or competitive‑intelligence data.

What We Value

• Pragmatic problem‑solving in hostile and fast‑changing technical environments.
• High ownership of systems that run continuously and at scale.
• A strong bias toward reliability, correctness, and observability.
• Ability to explain complex technical behaviour to non‑specialists.
• Curiosity about how large platforms evolve their defences and infrastructure.

Please send me a copy of your CV if you're interested and meet the above