Web Scraping with Python: Complete Proxy Setup Guide 2026.
How to set up proxies in Python for web scraping using requests, Scrapy, and Playwright. Code examples with Bright Data, Smartproxy, and IPRoyal.
Published: 2026-04-14 | Updated: 2026-04-14
Why You Need Proxies for Python Scraping
Python is the most popular language for web scraping, with libraries like requests, BeautifulSoup, Scrapy, and Playwright. But any scraping project that sends more than a few hundred requests will get blocked without proxies. IP bans, CAPTCHAs, and rate limits make proxy rotation essential. This guide shows you how to integrate proxy services into every major Python scraping library with working code examples.
Python + Requests: Basic Proxy Setup
The simplest proxy integration uses Python's requests library. Set the `proxies` parameter with your proxy credentials: ```python import requests proxy = "http://user:pass@gate.smartproxy.com:10001" response = requests.get("https://example.com", proxies={"http": proxy, "https": proxy}, timeout=30) ``` For rotating proxies, most providers offer a gateway that automatically assigns a new IP per request. Bright Data's endpoint: `brd-customer-{id}-zone-residential:password@brd.superproxy.io:22225`. Smartproxy: `user:pass@gate.smartproxy.com:10001`. IPRoyal: `user:pass@geo.iproyal.com:12321`.
Scrapy: Middleware-Based Rotation
Scrapy's middleware system makes proxy rotation clean and maintainable. Create a custom downloader middleware that rotates proxies on each request: ```python # middlewares.py class ProxyMiddleware: def process_request(self, request, spider): request.meta["proxy"] = "http://user:pass@gate.smartproxy.com:10001" ``` For production scrapers, use the `scrapy-rotating-proxies` package or implement your own middleware that tracks failed IPs and removes them from rotation. Bright Data's Super Proxy handles rotation server-side, so you only need one endpoint URL.
Playwright (Python): Browser-Based Scraping with Proxies
For JavaScript-heavy sites, Playwright provides headless browser automation with built-in proxy support: ```python from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch( proxy={"server": "http://gate.smartproxy.com:10001", "username": "user", "password": "pass"}) page = browser.new_page() page.goto("https://example.com") ``` For maximum anti-detection, use Bright Data's Scraping Browser instead of local Playwright. Connect via WebSocket — your Playwright code stays the same, but anti-detection and proxy rotation happen server-side.
Choosing Your Proxy Provider for Python
**IPRoyal ($1.75/GB):** Best for learning and small projects. Non-expiring traffic means no waste. Basic rotating proxy, works with requests and Scrapy. **Smartproxy ($4.50/GB):** Best for mid-size projects. Fast, reliable, good documentation with Python examples. Their eCommerce API returns structured JSON — no parsing needed. **Bright Data ($8.40/GB):** Best for production scrapers targeting protected sites. Web Unlocker handles everything automatically. Scraping Browser replaces local Playwright. Python SDK available (`pip install brightdata`). All three providers offer Python-compatible HTTP/SOCKS5 proxies that work with any library.
Recommended Providers
FAQ
- How do I use a proxy in Python requests? +
- Pass a proxies dictionary to requests.get(): `requests.get(url, proxies={"http": "http://user:pass@proxy:port", "https": "http://user:pass@proxy:port"})`. For rotating proxies, use a gateway endpoint from Bright Data, Smartproxy, or IPRoyal that assigns new IPs automatically.
- What is the best Python library for web scraping? +
- For simple pages: requests + BeautifulSoup. For large-scale crawling: Scrapy. For JavaScript-heavy sites: Playwright. All three work with proxy services. For the hardest targets, Bright Data's Python SDK provides integrated scraping with built-in proxy rotation.
- How much do proxies cost for Python scraping? +
- From $1.75/GB (IPRoyal) to $8.40/GB (Bright Data). A typical scraping project uses 1-10 GB/month. That's $2-84/month for basic projects, or $100-500/month for production data collection at scale.