日本免费高清视频-国产福利视频导航-黄色在线播放国产-天天操天天操天天操天天操|www.shdianci.com

學無先后，達者為師

網站首頁 編程語言 正文

Python無法用requests獲取網頁源碼的解決方法_python

作者：henanlion ? 更新時間： 2022-08-31 編程語言

最近在抓取http://skell.sketchengine.eu網頁時，發現用requests無法獲得網頁的全部內容，所以我就用selenium先模擬瀏覽器打開網頁，再獲取網頁的源代碼，通過BeautifulSoup解析后拿到網頁中的例句，為了能讓循環持續進行，我們在循環體中加了refresh()，這樣當瀏覽器得到新網址時通過刷新再更新網頁內容，注意為了更好地獲取網頁內容，設定刷新后停留2秒，這樣可以降低抓不到網頁內容的機率。為了減少被封的可能，我們還加入了Chrome，請看以下代碼：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from bs4 import BeautifulSoup
import time,re
 
path = Service("D:\\MyDrivers\\chromedriver.exe")#
# 配置不顯示瀏覽器
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('User-Agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36')
 
# 創建Chrome實例 。
 
driver = webdriver.Chrome(service=path,options=chrome_options)
lst=["happy","help","evening","great","think","adapt"]
 
for word in lst:
    url="https://skell.sketchengine.eu/#result?lang=en&query="+word+"&f=concordance"
    driver.get(url)
    # 刷新網頁獲取新數據
    driver.refresh()
    time.sleep(2)
    # page_source——》獲得頁面源碼
    resp=driver.page_source
    # 解析源碼
    soup=BeautifulSoup(resp,"html.parser")
    table = soup.find_all("td")
    with open("eps.txt",'a+',encoding='utf-8') as f:
        f.write(f"\n{word}的例子\n")
    for i in table[0:6]:
        text=i.text
        #替換多余的空格
        new=re.sub("\s+"," ",text)
        #寫入txt文本
        with open("eps.txt",'a+',encoding='utf-8') as f:
            f.write(re.sub(r"^(\d+\.)",r"\n\1",new))
driver.close()

1. 為了加快訪問速度，我們設置不顯示瀏覽器，通過chrome.options實現

2. 最近通過re正則表達式來清理格式。

3. 我們設置table[0:6]來獲取前三個句子的內容，最后顯示結果如下。

happy的例子
1. This happy mood lasted roughly until last autumn.?
2. The lodging was neither convenient nor happy .?
3. One big happy family "fighting communism".?
help的例子
1. Applying hot moist towels may help relieve discomfort.?
2. The intense light helps reproduce colors more effectively.?
3. My survival route are self help books.?
evening的例子
1. The evening feast costs another $10.?
2. My evening hunt was pretty flat overall.?
3. The area nightclubs were active during evenings .?
great的例子
1. The three countries represented here are three great democracies.?
2. Our three different tour guides were great .?
3. Your receptionist "crew" is great !?
think的例子
1. I said yes immediately without thinking everything through.?
2. This book was shocking yet thought provoking.?
3. He thought "disgusting" was more appropriate.?
adapt的例子
1. The novel has been adapted several times.?
2. There are many ways plants can adapt .?
3. They must adapt quickly to changing deadlines.?

補充：經過代碼的優化以后，例句的爬取更加快捷，代碼如下：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from bs4 import BeautifulSoup
import time,re
import os
 
# 配置模擬瀏覽器的位置
path = Service("D:\\MyDrivers\\chromedriver.exe")#
# 配置不顯示瀏覽器
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('User-Agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36')
 
# 創建Chrome實例 。
 
def get_wordlist():
    wordlist=[]
    with open("wordlist.txt",'r',encoding='utf-8') as f:
        lines=f.readlines()
        for line in lines:
            word=line.strip()
            wordlist.append(word)
    return wordlist
 
def main(lst):
    driver = webdriver.Chrome(service=path,options=chrome_options)
    for word in lst:
        url="https://skell.sketchengine.eu/#result?lang=en&query="+word+"&f=concordance"
        driver.get(url) 
        driver.refresh()
        time.sleep(2)
        # page_source——》頁面源碼
        resp=driver.page_source
        # 解析源碼
        soup=BeautifulSoup(resp,"html.parser")
        table = soup.find_all("td")
        with open("examples.txt",'a+',encoding='utf-8') as f:
            f.writelines(f"\n{word}的例子\n")
        for i in table[0:6]:
            text=i.text
            new=re.sub("\s+"," ",text)
            with open("eps.txt",'a+',encoding='utf-8') as f:
                f.write(new)
#                 f.writelines(re.sub("(\.\s)(\d+\.)","\1\n\2",new))
 
if __name__=="__main__":
    lst=get_wordlist()
    main(lst)
    os.startfile("examples.txt")

總結

原文鏈接：https://blog.csdn.net/henanlion/article/details/122757040

欄目分類
最近更新