seleniumを使って、yahoo newsのwebサイトからURLを抽出して
IDのみ取得したいです。
実現したい内容
❶からURLを取得してから、❷IDのみ取得したいです。
❶URL取得
['https://news.yahoo.co.jp/pickup/6435551']
['https://news.yahoo.co.jp/pickup/6435550']
['https://news.yahoo.co.jp/pickup/6435548']
['https://news.yahoo.co.jp/pickup/6435547']
❷ID取得
['6435551'、'6435550'、'6435548'、'6435547']
現在は下記のコードで❶までのURLを取得できますが、
❷のID取得できません。
実現の結果のprint(new_list)のように[]空白として出力されます。
スライス方法が間違っていいますでしょうか。
もし分かる方がいましたら、教えていただけると幸いです。
お手数ですが、よろしくお願い致します。
現在の結果
print(data_list)
※一部省略
['https://www.yahoo.co.jp/']
['https://support.yahoo-net.jp/PccNews/s/']
['https://news.yahoo.co.jp/']
['https://news.yahoo.co.jp/ranking/access/news']
['https://news.yahoo.co.jp/paidnews?source=pc-common-glonav']
['https://news.yahoo.co.jp/']
['https://news.yahoo.co.jp/categories/domestic']
['https://news.yahoo.co.jp/categories/world']
['https://news.yahoo.co.jp/categories/business']
['https://news.yahoo.co.jp/categories/entertainment']
['https://news.yahoo.co.jp/categories/sports']
['https://news.yahoo.co.jp/categories/it']
['https://news.yahoo.co.jp/categories/science']
['https://news.yahoo.co.jp/categories/life']
['https://news.yahoo.co.jp/categories/local']
['https://news.yahoo.co.jp/topics/top-picks']
['https://news.yahoo.co.jp/topics']
['https://news.yahoo.co.jp/topics/top-picks']
['https://news.yahoo.co.jp/topics/top-picks']
['https://news.yahoo.co.jp/topics/domestic']
['https://news.yahoo.co.jp/topics/world']
['https://news.yahoo.co.jp/topics/business']
['https://news.yahoo.co.jp/topics/entertainment']
['https://news.yahoo.co.jp/topics/sports']
['https://news.yahoo.co.jp/topics/it']
['https://news.yahoo.co.jp/topics/science']
['https://news.yahoo.co.jp/topics/local']
['https://news.yahoo.co.jp/pickup/6435551']
['https://news.yahoo.co.jp/pickup/6435550']
['https://news.yahoo.co.jp/pickup/6435548']
['https://news.yahoo.co.jp/pickup/6435547']
['https://news.yahoo.co.jp/pickup/6435546']
['https://news.yahoo.co.jp/pickup/6435545']
['https://news.yahoo.co.jp/pickup/6435544']
['https://news.yahoo.co.jp/pickup/6435542']
['https://news.yahoo.co.jp/pickup/6435543']
['https://news.yahoo.co.jp/pickup/6435541']
['https://news.yahoo.co.jp/pickup/6435539']
['https://news.yahoo.co.jp/pickup/6435538']
['https://news.yahoo.co.jp/pickup/6435537']
['https://news.yahoo.co.jp/pickup/6435535']
['https://news.yahoo.co.jp/pickup/6435534']
['https://news.yahoo.co.jp/pickup/6435532']
['https://news.yahoo.co.jp/pickup/6435533']
['https://news.yahoo.co.jp/pickup/6435531']
['https://news.yahoo.co.jp/pickup/6435530']
['https://news.yahoo.co.jp/pickup/6435528']
['https://news.yahoo.co.jp/pickup/6435527']
['https://news.yahoo.co.jp/pickup/6435524']
['https://news.yahoo.co.jp/pickup/6435525']
['https://news.yahoo.co.jp/pickup/6435526']
['https://news.yahoo.co.jp/pickup/6435521']
['https://news.yahoo.co.jp/topics/top-picks?page=2']
['https://www.facebook.com/yjnews']
['https://twitter.com/YahooNewsTopics']
['https://news.yahoo.co.jp/newshack/']
['https://news.yahoo.co.jp/']
['https://news.yahoo.co.jp/flash']
['https://news.yahoo.co.jp/live']
['https://news.yahoo.co.jp/info/commercial-transactions']
['https://support.yahoo-net.jp/voc/s/news']
['https://support.yahoo-net.jp/PccNews/s/']
print(l_in)
※一部省略
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
['https://news.yahoo.co.jp/pickup/6435552']
['https://news.yahoo.co.jp/pickup/6435551']
['https://news.yahoo.co.jp/pickup/6435550']
['https://news.yahoo.co.jp/pickup/6435548']
['https://news.yahoo.co.jp/pickup/6435547']
['https://news.yahoo.co.jp/pickup/6435546']
['https://news.yahoo.co.jp/pickup/6435545']
['https://news.yahoo.co.jp/pickup/6435544']
['https://news.yahoo.co.jp/pickup/6435542']
['https://news.yahoo.co.jp/pickup/6435543']
['https://news.yahoo.co.jp/pickup/6435541']
['https://news.yahoo.co.jp/pickup/6435539']
['https://news.yahoo.co.jp/pickup/6435538']
['https://news.yahoo.co.jp/pickup/6435537']
['https://news.yahoo.co.jp/pickup/6435535']
['https://news.yahoo.co.jp/pickup/6435534']
['https://news.yahoo.co.jp/pickup/6435532']
['https://news.yahoo.co.jp/pickup/6435533']
['https://news.yahoo.co.jp/pickup/6435531']
['https://news.yahoo.co.jp/pickup/6435530']
['https://news.yahoo.co.jp/pickup/6435528']
['https://news.yahoo.co.jp/pickup/6435527']
['https://news.yahoo.co.jp/pickup/6435524']
['https://news.yahoo.co.jp/pickup/6435525']
['https://news.yahoo.co.jp/pickup/6435526']
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
print(new_list)
※一部省略
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
[]
コード
from selenium.webdriver.common.keys import Keys
import time
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.select import Select
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
option = Options()
#ログイン情報を維持するための設定
# 参考→https://rabbitfoot.xyz/selenium-chrome-profile
PROFILE_PATH ="C:\\Users\\test\\AppData\\Local\\Google\\Chrome\\User Data\\" # 変更
option.add_argument('--user-data-dir=' + PROFILE_PATH)
option.add_argument('--profile-directory=Default')
#Getting Default Adapter failed error message
option.add_experimental_option('excludeSwitches', ['enable-logging'])
# ブラウザを開く設定
driver = webdriver.Chrome(executable_path=ChromeDriverManager().install() ,options=option)
#news.yahoo URL
URL= "https://news.yahoo.co.jp/"
# URLを開く。
driver.get(URL)
#待機時間
time.sleep(3)
while True:
try:
#ボタンクリック top25まで
button = driver.find_element_by_xpath('//*[@id="uamods-topics"]/div/div/div/div/p[1]/a')
button.click()
#待機時間
time.sleep(3)
except NoSuchElementException:
break
#待機時間
time.sleep(3)
#全てのURLを抽出
elements = driver.find_elements_by_xpath("//a[@href]")
for element in elements:
list=element.get_attribute("href")
#URLをリストとして作成
data_list = list.splitlines()
#print(data_list)
#URLに含んでいるpickupのURLのみ 出力
l_in = [s for s in data_list if 'pickup' in s]
#print(l_in)
#IDを取得
new_list=l_in[32:]
print(new_list)