Python程序中如何使用亿牛云爬虫代理

-3 · 2019-3-25 17:13

1.先在亿牛云客服处获取爬虫代理信息
2.添加代理信息
#! -*- encoding:utf-8 -*-

      import requests
      import random

      # 要访问的目标页面
      targetUrl = "http://httpbin.org/ip"

      # 要访问的目标HTTPS页面
      # targetUrl = "https://httpbin.org/ip"

      # 代理服务器
      proxyHost = "t.16yun.cn"
      proxyPort = "31111"

      # 代理隧道验证信息
      proxyUser = "username"
      proxyPass = "password"

      proxyMeta = "http://%(user)s:%(pass)s@%(host)s:%(port)s" % {
         "host" : proxyHost,
         "port" : proxyPort,
         "user" : proxyUser,
         "pass" : proxyPass,
      }

      # 设置 http和https访问都是用HTTP代理
      proxies = {
         "http"  : proxyMeta,
         "https" : proxyMeta,
      }

      #  设置IP切换头
      tunnel = random.randint(1,10000)
      headers = {"Proxy-Tunnel": str(tunnel)}

      resp = requests.get(targetUrl, proxies=proxies, headers=headers)

      print resp.status_code
      print resp.text

3 注意：亿牛云的爬虫代理每秒请求次数是有限制，可以在程序中对请求速度进行设置