Applications are closed for this role. It was originally posted 2026-05-27. It’s no longer accepting applicants — see roles Smart Working Solutions is still hiring for →, or browse the live openings below.
About this role
- Design, build, and maintain large-scale Python-based scraping systems targeting highly protected websites (including Google-like environments).
- Architect resilient extraction systems capable of handling dynamic, JavaScript-heavy pages using browser automation and hybrid approaches.
- Continuously adapt systems to frequent changes in page structures, request flows, and anti-bot mechanisms.
- Build robust, production-grade data extraction pipelines with strong emphasis on accuracy, observability, and fault tolerance.
- Implement advanced strategies such as proxy rotation, fingerprinting, session management, and request routing to ensure stability at scale.
- Monitor system health, proactively detect anomalies, and debug complex production failures across distributed systems.
- Optimise scraping infrastructure for performance, cost efficiency, and reliability at scale.
- Collaborate with data engineering and product teams to ensure scraped data is structured, validated, and trusted.
- Operate and improve systems running continuously in cloud environments.
- Document system architecture, scraping logic, and operational procedures for long-term maintainability.
- Contribute to improving resilience, automation, and adaptability in adversarial environments.
- 7+ years of professional software engineering experience, with a strong focus on backend systems, data engineering, or distributed systems.
- Proven experience building and operating large-scale production web scraping systems.
- Deep hands-on experience scraping Google or similarly complex, heavily protected / anti-bot environments.
- Strong expertise in Python (or comparable production languages such as Go, Rust, or JavaScript).
- Strong understanding of HTTP internals: headers, cookies, TLS, redirects, sessions, and browser networking behaviour.
- Experience with browser automation frameworks such as Playwright, Selenium, Puppeteer, or equivalent.
- Strong knowledge of HTML parsing, DOM traversal, and high-performance data extraction techniques.
- Proven experience handling anti-bot systems, including rate limiting, CAPTCHAs, IP rotation, and fingerprinting.
- Experience designing asynchronous and concurrent systems for high-throughput workloads.
- Strong debugging skills across distributed, failure-prone production systems.
- Experience running cloud-based systems at scale with strong operational ownership.
- Comfortable working in highly adversarial and fast-changing technical environments.
- Experience with Docker and Kubernetes in production environments.
- Exposure to distributed task queues or large-scale job orchestration systems.
- Experience with monitoring, anomaly detection, or data quality validation systems.
- Background in search intelligence, advertising tech, or competitive intelligence platforms.
- Experience building systems that operate against high-defence web environments at scale.
- Familiarity with observability tooling and production-grade system monitoring.
- Exposure to AI-assisted development workflows or agentic coding tools.
- Fixed Shifts: 12:00 PM - 9:30 PM IST (Summer) | 1:00 PM - 10:30 PM IST (Winter)
- No Weekend Work: Real work-life balance, not just words
- Day 1 Benefits: Laptop and full medical insurance provided
- Support That Matters: Mentorship, community, and forums where ideas are shared
- True Belonging: A long-term career where your contributions are valued
Tech stack
PythonJavaScriptRustDockerKubernetesGo
Still hiring at Smart Working Solutions
AI Agent Engineer (Remote, Full-Time) [AS311]
Engineering
Cloud Engineer - Frontend-Focused(Remote, Full-Time) [AS315]
Engineering
Cloud Engineer - Frontend-leaning (Remote, Full-Time) [AS315]
Engineering
See all live roles at Smart Working Solutions →Similar roles at other companies
A
Ground Software Engineer, Space
Engineering
$166K–$220K
C
Full-Stack Developer - Python/JavaScript/CSS/REST/Docker
Engineering
$130K–$270K
A
Lead Site Reliability Engineer
Engineering
$179K–$226K
A
Staff+ Software Security Engineer
Engineering
$405K–$485K
See all Engineering roles →