ScrapeLens API Documentation

Overview

ScrapeLens is a powerful web scraping API that enables you to extract data from any website with ease. Our service handles complex challenges like JavaScript rendering and proxy rotation automatically.

Authentication

ScrapeLens uses API key authentication to secure scraping requests. Include your API key in the ApiKey header of each request. You can find and manage your API keys in the ScrapeLens Dashboard. Keep your API key secure and never expose it in client-side code or public repositories.

Authorization Header

ApiKey: sk-1234567890abcdef...

Example Request

curl -X POST https://api.scrapelens.com \
  -H "ApiKey: sk-1234567890abcdef..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.scrapingcourse.com/javascript-rendering"
  }'

Quick Start

Scraping a web page with ScrapeLens is a simple process. Send a POST request to the root endpoint, specifying the url you want to scrape in the request body. JavaScript rendering and rotating proxies are handled automatically for you.

POST https://api.scrapelens.com

Basic Scrape Request

curl -X POST https://api.scrapelens.com \
  -H "ApiKey: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.scrapingcourse.com/javascript-rendering"
  }'

Field	Required	Description
url	Required	The URL to scrape
script	Optional	Specify a request script to interact with the page before capturing the document's content. Check out the Script Operations section to learn more.
output	Optional	Description of the data to extract from the page. You can learn more about data extraction in the Data Extraction section.

If the output field is omitted, the full HTML content of the page will be returned in the response.

Script Operations

Script operations allow you to interact with web pages before extracting data. This is essential for handling dynamic content, loading more data, or navigating through multi-step processes.

Wait

Pause execution for a specified duration. Useful for waiting for animations, AJAX requests, or other time-based events to complete.

Wait Example

{
  "url": "https://example.com",
  "script": [
    {
      "wait": "5s"
    }
  ]
}

Duration Format	Example	Description
seconds	"5s"	Wait for 5 seconds
minutes	"2m"	Wait for 2 minutes
milliseconds	"500ms"	Wait for 500 milliseconds

Wait For Element

Wait until a specific element appears on the page. This is ideal for handling dynamically loaded content, AJAX responses, or elements that appear after user interactions.

Wait For Element Example

{
  "url": "https://example.com",
  "script": [
    {
      "wait_for": ".content-loaded",
      "selector_type": "css",
      "timeout": "60s"
    }
  ]
}

Field	Required	Description
wait_for	Required	CSS selector or XPath expression of the element to wait for
selector_type	Optional	Type of selector: "css" (default) or "xpath"
timeout	Optional	Maximum time to wait (default: "30s")

Click

Click on a specific element to trigger interactions like loading more content, opening modals, submitting forms, or navigating to other pages.

Click Example

{
  "url": "https://example.com",
  "script": [
    {
      "click": ".load-more-button",
      "selector_type": "css"
    },
    {
      "wait": "2s"
    }
  ]
}

Field	Required	Description
click	Required	CSS selector or XPath expression of the element to click
selector_type	Optional	Type of selector: "css" (default) or "xpath"

Complex Script Example

Combine multiple script operations to handle complex interactions. This example loads a page, clicks a button to load more content, waits for the content to appear, then extracts data.

Complex Script Example

{
  "url": "https://example.com/products",
  "script": [
    {
      "wait_for": ".product-grid",
      "selector_type": "css",
      "timeout": "10s"
    },
    {
      "click": ".load-more",
      "selector_type": "css"
    },
    {
      "wait": "3s"
    },
    {
      "wait_for": ".product-item:nth-child(20)",
      "selector_type": "css",
      "timeout": "15s"
    }
  ],
  "output": {
    "products": [{
      "selector": ".product-item",
      "output": {
        "name": "h3",
        "price": ".price",
        "rating": ".rating"
      }
    }]
  }
}

Data Extraction

The output field defines what data to extract from the page and how to structure it. ScrapeLens supports various extraction patterns from simple text extraction to complex nested data structures.

Simple Text Extraction

Extract text content from single elements using CSS selectors or XPath expressions.

Simple Text Extraction

{
  "url": "https://example.com",
  "output": {
    "title": "h1",
    "description": ".description",
    "author": ".author"
  }
}

Response

{
  "title": "Page Title",
  "description": "Page description text",
  "author": "John Doe"
}

Multiple Elements Extraction

Extract text from multiple elements matching a selector into an array.

Multiple Elements

{
  "url": "https://example.com",
  "output": {
    "headings": ["h2"],
    "links": ["a"],
    "tags": [".tag"]
  }
}

Response

{
  "headings": [
    "First Heading",
    "Second Heading",
    "Third Heading"
  ],
  "links": [
    "Link 1",
    "Link 2",
    "Link 3"
  ],
  "tags": [
    "tag1",
    "tag2",
    "tag3"
  ]
}

Nested Object Extraction

Extract complex nested data structures from repeated elements like product listings, articles, or search results.

Nested Object Extraction

{
  "url": "https://example.com/products",
  "output": {
    "products": [{
      "selector": ".product-item",
      "output": {
        "name": "h3",
        "price": ".price",
        "description": ".description",
        "rating": ".rating",
        "availability": ".stock"
      }
    }]
  }
}

Response

{
  "products": [
    {
      "name": "Product 1",
      "price": "$29.99",
      "description": "Product description",
      "rating": "4.5",
      "availability": "In Stock"
    },
    {
      "name": "Product 2",
      "price": "$49.99",
      "description": "Another product",
      "rating": "4.8",
      "availability": "In Stock"
    }
  ]
}

Attribute Extraction

Extract HTML attributes like href, src, alt, data attributes, or any custom attributes.

Attribute Extraction

{
  "url": "https://example.com",
  "output": {
    "links": [{
      "selector": "a",
      "output": {
        "text": "text",
        "url": {"href": "href"},
        "title": {"title": "title"}
      }
    }],
    "images": [{
      "selector": "img",
      "output": {
        "src": {"src": "src"},
        "alt": {"alt": "alt"},
        "title": {"title": "title"}
      }
    }]
  }
}

Response

{
  "links": [
    {
      "text": "Link Text",
      "url": "https://example.com/page",
      "title": "Link Title"
    }
  ],
  "images": [
    {
      "src": "https://example.com/image.jpg",
      "alt": "Image Description",
      "title": "Image Title"
    }
  ]
}

Complex Nested Extraction

Combine multiple extraction patterns for complex data structures. This example extracts articles with nested metadata and links.

Complex Nested Example

{
  "url": "https://example.com/blog",
  "output": {
    "title": "h1",
    "articles": [{
      "selector": ".article",
      "output": {
        "title": "h2",
        "summary": ".summary",
        "author": {
          "name": ".author-name",
          "bio": ".author-bio",
          "avatar": {"selector": ".author-avatar", "src": "src"}
        },
        "metadata": {
          "date": ".date",
          "category": ".category",
          "tags": [".tag"]
        },
        "content": {
          "text": ".content",
          "read_more": {"selector": ".read-more", "href": "href"}
        }
      }
    }]
  }
}

                    
                        Extraction Tips
                    
                    Use specific selectors to avoid extracting unwanted content
Combine script operations with extraction for dynamic content
Test selectors in browser dev tools before using them
Use CSS selectors (:nth-child, :first-child, :last-child) for precise targeting

Error Handling

ScrapeLens uses standard HTTP status codes and JSON error responses to communicate the success or failure of API requests. Understanding these responses will help you build robust applications that handle errors gracefully.

200

Success

Request completed successfully

401

Unauthorized

Invalid or missing authentication

402

Payment Required

Insufficient credits

429

Rate Limited

Too many requests

Error Response Format

All error responses follow a consistent JSON format with an error message describing what went wrong.

Error Response Example

{
  "error": "unauthenticated"
}

Common Error Types

Error Message	Status Code	Cause	Solution
unauthenticated	401	Missing or invalid API key	Check API key format and headers
insufficient credits	402	Not enough credits for request	Purchase more credits via dashboard
invalid request	400	Malformed request body or parameters	Validate JSON and required fields
timeout	408	Request exceeded time limit	Optimize page or increase timeout

Advanced Examples

Real-world examples showing how to solve common web scraping challenges using ScrapeLens. These examples demonstrate combining script operations with data extraction for complex scenarios.

E-commerce Product Listing

Scrape product information from an e-commerce site with pagination and dynamic content loading.

E-commerce Product Scraper

{
  "url": "https://example-shop.com/products",
  "script": [
    {
      "wait_for": ".product-grid",
      "selector_type": "css",
      "timeout": "10s"
    },
    {
      "click": ".load-more-button",
      "selector_type": "css"
    },
    {
      "wait": "3s"
    },
    {
      "wait_for": ".product-item:nth-child(30)",
      "selector_type": "css",
      "timeout": "15s"
    }
  ],
  "output": {
    "products": [{
      "selector": ".product-item",
      "output": {
        "name": "h3.product-title",
        "price": {
          "selector": ".price",
          "output": {
            "current": ".current-price",
            "original": ".original-price",
            "currency": {"selector": ".price", "output": {"data-currency": "data-currency"}}
          }
        },
        "rating": {
          "selector": ".rating",
          "output": {
            "stars": ".rating-stars",
            "count": ".rating-count"
          }
        },
        "availability": ".stock-status",
        "image": {"selector": ".product-image", "src": "src"},
        "url": {"selector": ".product-link", "href": "href"},
        "badge": ".product-badge"
      }
    }]
  }
}

News Article Scraper

Extract article content including title, author, metadata, and comments from a news website.

News Article Extraction

{
  "url": "https://example-news.com/article/123",
  "script": [
    {
      "wait_for": ".article-content",
      "selector_type": "css",
      "timeout": "10s"
    },
    {
      "click": ".show-more-comments",
      "selector_type": "css"
    },
    {
      "wait_for": ".comment-item:nth-child(5)",
      "selector_type": "css",
      "timeout": "8s"
    }
  ],
  "output": {
    "article": {
      "title": "h1.article-title",
      "subtitle": ".article-subtitle",
      "author": {
        "name": ".author-name",
        "bio": ".author-bio",
        "avatar": {"selector": ".author-avatar", "src": "src"}
      },
      "metadata": {
        "publish_date": ".publish-date",
        "read_time": ".read-time",
        "category": ".category",
        "tags": [".tag"]
      },
      "content": {
        "body": ".article-body",
        "summary": ".article-summary"
      },
      "social": {
        "shares": ".share-count",
        "likes": ".like-count"
      },
      "comments": [{
        "selector": ".comment-item",
        "output": {
          "author": ".comment-author",
          "text": ".comment-text",
          "timestamp": ".comment-time",
          "likes": ".comment-likes"
        }
      }]
    }
  }
}

Real Estate Listings

Scrape real estate listings with filters, maps, and detailed property information.

Real Estate Property Scraper

{
  "url": "https://example-realestate.com/listings",
  "script": [
    {
      "wait_for": ".property-listings",
      "selector_type": "css",
      "timeout": "10s"
    },
    {
      "click": ".filter-toggle",
      "selector_type": "css"
    },
    {
      "wait": "1s"
    },
    {
      "click": ".price-filter-high",
      "selector_type": "css"
    },
    {
      "click": ".apply-filters",
      "selector_type": "css"
    },
    {
      "wait_for": ".property-card",
      "selector_type": "css",
      "timeout": "15s"
    }
  ],
  "output": {
    "properties": [{
      "selector": ".property-card",
      "output": {
        "title": ".property-title",
        "price": {
          "selector": ".property-price",
          "output": {
            "amount": ".price-amount",
            "period": ".price-period"
          }
        },
        "location": {
          "address": ".property-address",
          "city": ".property-city",
          "state": ".property-state",
          "zip_code": ".property-zip"
        },
        "details": {
          "bedrooms": ".bedrooms",
          "bathrooms": ".bathrooms",
          "sqft": ".square-feet",
          "lot_size": ".lot-size"
        },
        "features": [".property-feature"],
        "description": ".property-description",
        "agent": {
          "name": ".agent-name",
          "phone": ".agent-phone",
          "email": ".agent-email"
        },
        "images": [{
          "selector": ".property-image-gallery img",
          "output": {
            "src": {"src": "src"},
            "alt": {"alt": "alt"}
          }
        }],
        "listing_url": {"selector": ".property-link", "href": "href"},
        "listed_date": ".listing-date",
        "status": ".property-status"
      }
    }],
    "search_info": {
      "total_results": ".results-count",
      "page_info": ".page-info"
    }
  }
}

Job Board Scraper

Extract job postings with company information, requirements, and application details.

Job Listings Extraction

{
  "url": "https://example-jobs.com/search?q=developer&location=remote",
  "script": [
    {
      "wait_for": ".job-listings",
      "selector_type": "css",
      "timeout": "8s"
    },
    {
      "click": ".load-more-jobs",
      "selector_type": "css"
    },
    {
      "wait": "2s"
    }
  ],
  "output": {
    "jobs": [{
      "selector": ".job-card",
      "output": {
        "title": ".job-title",
        "company": {
          "name": ".company-name",
          "logo": {"selector": ".company-logo", "src": "src"},
          "industry": ".company-industry",
          "size": ".company-size"
        },
        "location": {
          "city": ".job-city",
          "state": ".job-state",
          "country": ".job-country",
          "remote_type": ".remote-status"
        },
        "employment": {
          "type": ".employment-type",
          "salary": {
            "range": ".salary-range",
            "currency": ".salary-currency"
          }
        },
        "requirements": {
          "experience": ".experience-level",
          "skills": [".skill-tag"],
          "education": ".education-requirement"
        },
        "description": ".job-description",
        "benefits": [".benefit-item"],
        "posted_date": ".posting-date",
        "application_deadline": ".application-deadline",
        "apply_url": {"selector": ".apply-button", "href": "href"},
        "job_id": {"selector": ".job-card", "output": {"data-job-id": "data-job-id"}}
      }
    }],
    "search_metadata": {
      "total_jobs": ".total-jobs-count",
      "search_query": ".search-query-display",
      "filters_applied": [".active-filter"]
    }
  }
}

                    
                        
                        Pro Tips for Advanced Scraping
                    
                    Start Simple: Begin with basic extraction, then add script operations incrementally
Test Selectors: Use browser dev tools to test selectors before implementation
Handle Timing: Use wait operations strategically to ensure dynamic content loads
Error Handling: Implement retry logic for timeout and rate limiting errors
Respect Robots: Check robots.txt and implement reasonable request rates
Monitor Credits: Track usage to optimize costs and avoid interruptions
Structure Data: Plan your output structure before writing extraction rules

                

Pricing

Simple, transparent pricing based on API credits. No monthly fees, no hidden costs. Pay only for what you use with our flexible credit system.

                        How It Works
                    10,000
credits for $1
20
credits per request

Free Tier

New users get 1,000 free credits (50 requests) to test our service. No credit card required.

Get Started with Free Credits