Web scraping is getting harder. Major websites implement rate limiting and IP bans. If you send too many requests from the same IP, you get blocked instantly. The solution? Residential Proxy Rotation.
In this tutorial, we will show you how to use the requests library in Python to rotate residential IPs automatically using the UnlimitedResidentialProxy gateway.
Why Rotate Proxies?
Static datacenter IPs are easily detected. By rotating residential IPs (IPs assigned to real devices), you mimic human behavior. Each request comes from a different location, making your scraper virtually undetectable.
Step 1: Setup your Proxy String
Rotation is handled on the gateway level. You don't need complex Python logic to manage lists of IPs.
import requests
# ---------------------------------------------------------
# CONFIGURATION
# ---------------------------------------------------------
# Login to your dashboard to get your assigned Gateway Host
PROXY_HOST = "<YOUR_GATEWAY_HOST>"
PROXY_PORT = "7777"
PROXY_USER = "USERNAME"
PROXY_PASS = "PASSWORD"
# Construct the Proxy String
# Note: We support HTTP, HTTPS, and SOCKS5 on the same port
proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
proxies = {
"http": proxy_url,
"https": proxy_url,
}
target_url = "https://httpbin.org/ip"
# Make 5 requests to show rotation
for i in range(5):
try:
# Timeout is crucial when scraping to avoid hanging
response = requests.get(target_url, proxies=proxies, timeout=10)
# Verify the IP has changed
print(f"Request {i+1}: IP is {response.json()['origin']}")
except Exception as e:
print(f"Error: {e}")Understanding the Gateway
When you connect to our gateway, we assign you a new Residential IP from our pool of 10 Million+ IPs for every request (or sticky session if configured). This ensures:
- Zero Maintenance: No need to validate or refresh proxy lists manually.
- High Anonymity: Real ISP/Mobile connections.
- Geo-Targeting: Add
-country-usto your username to target specific regions.
Need Unlimited Bandwidth?
Most providers charge per GB. We don't. Scrape as much as you want with our concurrent-based plans starting at just $79.