DESCRIPTION: We are seeking an expert Data Engineer to build and maintain a robust data scraping and ingestion pipeline. The goal is to aggregate daily sales and listing data from several global auction sites and specialized dealer marketplaces to populate our internal database with structured market data.
KEY DELIVERABLES:
- Custom Scrapers: Develop resilient scrapers for multiple major auction houses and dealer listing sources.
- Data Normalization: Extract and clean fragmented data (Sale Date, Price, Vehicle Specifications, Mileage, etc.) and map it into a unified PostgreSQL schema.
- Database Ingestion: Build a reliable pipeline to ingest this normalized data into our Supabase/PostgreSQL database daily.
- Automation & Maintenance: Set up automated workflows (cron jobs) to ensure the database stays current and implement strategies to handle anti-scraping measures (proxy rotation, etc.).
REQUIRED SKILLS:
- Languages: Proficiency in Python (Scrapy, Selenium, Playwright) or Node.js (Puppeteer).
- AI development for efficiency: Know how to use modern AI dev tools to speed up development drastically.
- Database: Strong experience with PostgreSQL and schema design for large datasets.
- Engineering: Proven track record of building production-grade scraping pipelines and bypassing sophisticated bot detection.
- DevOps: Experience with cloud-based job scheduling and data warehousing.
SCOPE
- Initially build followed by ongoing maintenance & optimization).
- This can turn into a full-time job if you do a good job.
LOCATION: remote.
WHY WORK WITH US: You will work directly with the technical founder to take full ownership of the data extraction and ingestion layer. We have a clear roadmap and need a specialist who prioritizes data integrity and pipeline uptime.
HOW TO APPLY: Send us:
- Portfolio and/or links to relevant work.
- Your experience with AI coding tools.
- Hourly rate expectation
- When you can start
- Weekly availability for the next 1–2 months.