{"id":2224,"date":"2023-10-15T08:38:08","date_gmt":"2023-10-15T03:08:08","guid":{"rendered":"https:\/\/python-programs.com\/?p=2224"},"modified":"2023-11-10T11:41:13","modified_gmt":"2023-11-10T06:11:13","slug":"how-to-code-a-scraping-bot-with-selenium-and-python","status":"publish","type":"post","link":"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/","title":{"rendered":"How to Code a Scraping Bot with Selenium and Python"},"content":{"rendered":"<p>Selenium is a powerful tool for controlling web browsers through programs and performing browser automation. Selenium is also used in python for scraping the data. It is also useful for interacting with the page before collecting the data, this is the case that we will discuss in this article.<\/p>\n<p>In this article, we will be scraping the investing.com to extract the historical data of dollar exchange rates against one or more currencies.<\/p>\n<p>There are other tools in python by which we can extract the financial information. However, here we want to explore how selenium helps with data extraction.<\/p>\n<h2>The Website we are going to Scrape:<\/h2>\n<p>Understanding of the website is the initial step before moving on to further things.<\/p>\n<p>Website consists of historical data for the exchange rate of dollars against euros.<\/p>\n<p>In this page, we will find a table in which we can set the date range which we want.<\/p>\n<p>That is the thing which we will be using.<\/p>\n<p>We only want the currencies exchange rate against the dollar. If that\u2019s not the case then replace the \u201cusd\u201d in the URL.<\/p>\n<h3>The Scraper\u2019s Code:<\/h3>\n<p>The initial step is starting with the imports from the selenium, the Sleep function to pause the code for some time and the pandas to manipulate the data whenever necessary.<\/p>\n<p><img loading=\"lazy\" class=\"alignnone size-full wp-image-2227\" src=\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python.png\" alt=\"How to Code a Scraping Bot with Selenium and Python\" width=\"1306\" height=\"285\" srcset=\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python.png 1306w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-300x65.png 300w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-1024x223.png 1024w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-768x168.png 768w\" sizes=\"(max-width: 1306px) 100vw, 1306px\" \/><\/p>\n<p>Now, we will write the scraping function. The function will consists of:<\/p>\n<ul>\n<li>A list of currency codes.<\/li>\n<li>A start date.<\/li>\n<li>An End date.<\/li>\n<li>A boolean function to export the data into .csv file. We will be using False as a default.<\/li>\n<\/ul>\n<p>We want to make a scraper that scrapes the data about the multiple currencies. We also have to initialise the empty list to store the scraped data.<\/p>\n<p><img loading=\"lazy\" class=\"alignnone size-full wp-image-2228\" src=\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-1.png\" alt=\"How to Code a Scraping Bot with Selenium and Python 1\" width=\"1305\" height=\"130\" srcset=\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-1.png 1305w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-1-300x30.png 300w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-1-1024x102.png 1024w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-1-768x77.png 768w\" sizes=\"(max-width: 1305px) 100vw, 1305px\" \/><\/p>\n<p>As we can see that the function has the list of currencies and our plan is to iterate over this list and get the data.<\/p>\n<p>For each currency we will create a URL, instantiate the driver object, and we will get the page by using it.<\/p>\n<p>Then the window function will be maximized but it will only be visible when we will keep the option.headless as False.<\/p>\n<p>Otherwise, all the work will be done by the selenium without even showing you.<\/p>\n<p><img loading=\"lazy\" class=\"alignnone size-full wp-image-2229\" src=\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-2.png\" alt=\"How to Code a Scraping Bot with Selenium and Python 2\" width=\"1261\" height=\"190\" srcset=\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-2.png 1261w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-2-300x45.png 300w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-2-1024x154.png 1024w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-2-768x116.png 768w\" sizes=\"(max-width: 1261px) 100vw, 1261px\" \/><\/p>\n<p>Now, we want to get the data for any time period.<\/p>\n<p>Selenium provides some awesome functionalities for getting connected to the website.<\/p>\n<p>We will click on the date and fill the start date and end dates with the dates we want and then we will hit apply.<\/p>\n<p>We will use <strong><em>WebDriverWait, ExpectedConditions<\/em><\/strong><em>, <\/em>and <strong><em>By <\/em><\/strong>to make sure that the driver will wait for the elements we want to interact with.<\/p>\n<p>The waiting time is 20 seconds, but it is to you whichever the way you want to set it.<\/p>\n<p>We have to select the date button and it\u2019s XPath.<\/p>\n<p>The same process will be followed by the <strong><em>start_bar, end_bar, and apply_button<\/em><\/strong>.<\/p>\n<p>The start_date field will take in the date from which we want the data.<\/p>\n<p>End_bar will select the date till which we want the data.<\/p>\n<p>When we will be done with this, then the apply_button will come into work.<\/p>\n<p><img loading=\"lazy\" class=\"alignnone size-full wp-image-2230\" src=\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-3.png\" alt=\"How to Code a Scraping Bot with Selenium and Python 3\" width=\"1251\" height=\"438\" srcset=\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-3.png 1251w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-3-300x105.png 300w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-3-1024x359.png 1024w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-3-768x269.png 768w\" sizes=\"(max-width: 1251px) 100vw, 1251px\" \/><\/p>\n<p>Now, we will use the pandas.read_html file to get all the content of the page. The source code of the page will be revealed and then finally we will quit the driver.<\/p>\n<p><img loading=\"lazy\" class=\"alignnone size-full wp-image-2231\" src=\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-4.png\" alt=\"How to Code a Scraping Bot with Selenium and Python 4\" width=\"1061\" height=\"126\" srcset=\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-4.png 1061w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-4-300x36.png 300w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-4-1024x122.png 1024w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-4-768x91.png 768w\" sizes=\"(max-width: 1061px) 100vw, 1061px\" \/><\/p>\n<h3>How to handle Exceptions In Selenium:<\/h3>\n<p>The collecting data process is done. But selenium is sometimes a little unstable and fail to perform the function we are performing here.<\/p>\n<p>To prevent this we have to put the code in the try and except block so that every time it faces any problem the except block will be executed.<\/p>\n<p>So, the code will be like:<\/p>\n<pre>for currency in currencies:\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 while True:\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 try:\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # Opening the connection and grabbing the page\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 my_url = f'https:\/\/br.investing.com\/currencies\/usd-{currency.lower()}-historical-data'\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 option = Options()\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 option.headless = False\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 driver = webdriver.Chrome(options=option)\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 driver.get(my_url)\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 driver.maximize_window()\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # Clicking on the date button\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 date_button = WebDriverWait(driver, 20).until(\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 EC.element_to_be_clickable((By.XPATH,\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \"\/html\/body\/div[5]\/section\/div[8]\/div[3]\/div\/div[2]\/span\")))\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 date_button.click()\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # Sending the start date\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 start_bar = WebDriverWait(driver, 20).until(\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 EC.element_to_be_clickable((By.XPATH,\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \"\/html\/body\/div[7]\/div[1]\/input[1]\")))\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 start_bar.clear()\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 start_bar.send_keys(start)\r\n\r\n\r\n\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # Sending the end date\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 end_bar = WebDriverWait(driver, 20).until(\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 EC.element_to_be_clickable((By.XPATH,\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \"\/html\/body\/div[7]\/div[1]\/input[2]\")))\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 end_bar.clear()\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 end_bar.send_keys(end)\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # Clicking on the apply button\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 apply_button = WebDriverWait(driver,20).until(\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0 EC.element_to_be_clickable((By.XPATH,\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0 \"\/html\/body\/div[7]\/div[5]\/a\")))\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 apply_button.click()\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 sleep(5)\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # Getting the tables on the page and quiting\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 dataframes = pd.read_html(driver.page_source)\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 driver.quit()\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 print(f'{currency} scraped.')\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 break\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 except:\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 driver.quit()\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 print(f'Failed to scrape {currency}. Trying again in 30 seconds.')\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 sleep(30)\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Continue<\/pre>\n<p>For each DataFrame in this dataframes list, we will check if the name matches, Now we will append this dataframe to the list we assigned in the beginning.<\/p>\n<p>Then we will need to export a csv file. This will be the last step and then we will be over with the extraction.<\/p>\n<p><img loading=\"lazy\" class=\"alignnone size-full wp-image-2226\" src=\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-5.png\" alt=\"How to Code a Scraping Bot with Selenium and Python 5\" width=\"842\" height=\"261\" srcset=\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-5.png 842w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-5-300x93.png 300w, https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python-5-768x238.png 768w\" sizes=\"(max-width: 842px) 100vw, 842px\" \/><\/p>\n<h3>Wrapping up:<\/h3>\n<p>This is all about extracting the data from the website.So far this code gets the historical data of the exchange rate of a list of currencies against the dollar and returns a list of DataFrames and several <em>.csv<\/em> files.<\/p>\n<p>https:\/\/www.investing.com\/currencies\/usd-eur-historical-data<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Selenium is a powerful tool for controlling web browsers through programs and performing browser automation. Selenium is also used in python for scraping the data. It is also useful for interacting with the page before collecting the data, this is the case that we will discuss in this article. In this article, we will be &hellip;<\/p>\n<p class=\"read-more\"> <a class=\"\" href=\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/\"> <span class=\"screen-reader-text\">How to Code a Scraping Bot with Selenium and Python<\/span> Read More &raquo;<\/a><\/p>\n","protected":false},"author":1,"featured_media":2225,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[5],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v18.9 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How to Code a Scraping Bot with Selenium and Python - Python Programs<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Code a Scraping Bot with Selenium and Python - Python Programs\" \/>\n<meta property=\"og:description\" content=\"Selenium is a powerful tool for controlling web browsers through programs and performing browser automation. Selenium is also used in python for scraping the data. It is also useful for interacting with the page before collecting the data, this is the case that we will discuss in this article. In this article, we will be &hellip; How to Code a Scraping Bot with Selenium and Python Read More &raquo;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/\" \/>\n<meta property=\"og:site_name\" content=\"Python Programs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/btechgeeks\" \/>\n<meta property=\"article:published_time\" content=\"2023-10-15T03:08:08+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-11-10T06:11:13+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1280\" \/>\n\t<meta property=\"og:image:height\" content=\"710\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@btech_geeks\" \/>\n<meta name=\"twitter:site\" content=\"@btech_geeks\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"veer\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https:\/\/python-programs.com\/#organization\",\"name\":\"BTech Geeks\",\"url\":\"https:\/\/python-programs.com\/\",\"sameAs\":[\"https:\/\/www.instagram.com\/btechgeeks\/\",\"https:\/\/www.linkedin.com\/in\/btechgeeks\",\"https:\/\/in.pinterest.com\/btechgeek\/\",\"https:\/\/www.youtube.com\/channel\/UC9MlCqdJ3lKqz2p5114SDIg\",\"https:\/\/www.facebook.com\/btechgeeks\",\"https:\/\/twitter.com\/btech_geeks\"],\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/python-programs.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/python-programs.com\/wp-content\/uploads\/2020\/11\/BTechGeeks.png\",\"contentUrl\":\"https:\/\/python-programs.com\/wp-content\/uploads\/2020\/11\/BTechGeeks.png\",\"width\":350,\"height\":70,\"caption\":\"BTech Geeks\"},\"image\":{\"@id\":\"https:\/\/python-programs.com\/#\/schema\/logo\/image\/\"}},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/python-programs.com\/#website\",\"url\":\"https:\/\/python-programs.com\/\",\"name\":\"Python Programs\",\"description\":\"Python Programs with Examples, How To Guides on Python\",\"publisher\":{\"@id\":\"https:\/\/python-programs.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/python-programs.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#primaryimage\",\"url\":\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python.jpg\",\"contentUrl\":\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python.jpg\",\"width\":1280,\"height\":710,\"caption\":\"How to Code a Scraping Bot with Selenium and Python\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#webpage\",\"url\":\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/\",\"name\":\"How to Code a Scraping Bot with Selenium and Python - Python Programs\",\"isPartOf\":{\"@id\":\"https:\/\/python-programs.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#primaryimage\"},\"datePublished\":\"2023-10-15T03:08:08+00:00\",\"dateModified\":\"2023-11-10T06:11:13+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/python-programs.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to Code a Scraping Bot with Selenium and Python\"}]},{\"@type\":\"Article\",\"@id\":\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#webpage\"},\"author\":{\"@id\":\"https:\/\/python-programs.com\/#\/schema\/person\/9f9e30fd3f415217a11ac0d939213b7f\"},\"headline\":\"How to Code a Scraping Bot with Selenium and Python\",\"datePublished\":\"2023-10-15T03:08:08+00:00\",\"dateModified\":\"2023-11-10T06:11:13+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#webpage\"},\"wordCount\":732,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/python-programs.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python.jpg\",\"articleSection\":[\"Python\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#respond\"]}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/python-programs.com\/#\/schema\/person\/9f9e30fd3f415217a11ac0d939213b7f\",\"name\":\"veer\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/python-programs.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/f1f6915d5328abaea9a64249313d1c55?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/f1f6915d5328abaea9a64249313d1c55?s=96&d=mm&r=g\",\"caption\":\"veer\"},\"sameAs\":[\"https:\/\/python-programs.com\"],\"url\":\"https:\/\/python-programs.com\/author\/veer\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to Code a Scraping Bot with Selenium and Python - Python Programs","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/","og_locale":"en_US","og_type":"article","og_title":"How to Code a Scraping Bot with Selenium and Python - Python Programs","og_description":"Selenium is a powerful tool for controlling web browsers through programs and performing browser automation. Selenium is also used in python for scraping the data. It is also useful for interacting with the page before collecting the data, this is the case that we will discuss in this article. In this article, we will be &hellip; How to Code a Scraping Bot with Selenium and Python Read More &raquo;","og_url":"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/","og_site_name":"Python Programs","article_publisher":"https:\/\/www.facebook.com\/btechgeeks","article_published_time":"2023-10-15T03:08:08+00:00","article_modified_time":"2023-11-10T06:11:13+00:00","og_image":[{"width":1280,"height":710,"url":"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_creator":"@btech_geeks","twitter_site":"@btech_geeks","twitter_misc":{"Written by":"veer","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Organization","@id":"https:\/\/python-programs.com\/#organization","name":"BTech Geeks","url":"https:\/\/python-programs.com\/","sameAs":["https:\/\/www.instagram.com\/btechgeeks\/","https:\/\/www.linkedin.com\/in\/btechgeeks","https:\/\/in.pinterest.com\/btechgeek\/","https:\/\/www.youtube.com\/channel\/UC9MlCqdJ3lKqz2p5114SDIg","https:\/\/www.facebook.com\/btechgeeks","https:\/\/twitter.com\/btech_geeks"],"logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/python-programs.com\/#\/schema\/logo\/image\/","url":"https:\/\/python-programs.com\/wp-content\/uploads\/2020\/11\/BTechGeeks.png","contentUrl":"https:\/\/python-programs.com\/wp-content\/uploads\/2020\/11\/BTechGeeks.png","width":350,"height":70,"caption":"BTech Geeks"},"image":{"@id":"https:\/\/python-programs.com\/#\/schema\/logo\/image\/"}},{"@type":"WebSite","@id":"https:\/\/python-programs.com\/#website","url":"https:\/\/python-programs.com\/","name":"Python Programs","description":"Python Programs with Examples, How To Guides on Python","publisher":{"@id":"https:\/\/python-programs.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/python-programs.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#primaryimage","url":"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python.jpg","contentUrl":"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python.jpg","width":1280,"height":710,"caption":"How to Code a Scraping Bot with Selenium and Python"},{"@type":"WebPage","@id":"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#webpage","url":"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/","name":"How to Code a Scraping Bot with Selenium and Python - Python Programs","isPartOf":{"@id":"https:\/\/python-programs.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#primaryimage"},"datePublished":"2023-10-15T03:08:08+00:00","dateModified":"2023-11-10T06:11:13+00:00","breadcrumb":{"@id":"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/python-programs.com\/"},{"@type":"ListItem","position":2,"name":"How to Code a Scraping Bot with Selenium and Python"}]},{"@type":"Article","@id":"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#article","isPartOf":{"@id":"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#webpage"},"author":{"@id":"https:\/\/python-programs.com\/#\/schema\/person\/9f9e30fd3f415217a11ac0d939213b7f"},"headline":"How to Code a Scraping Bot with Selenium and Python","datePublished":"2023-10-15T03:08:08+00:00","dateModified":"2023-11-10T06:11:13+00:00","mainEntityOfPage":{"@id":"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#webpage"},"wordCount":732,"commentCount":0,"publisher":{"@id":"https:\/\/python-programs.com\/#organization"},"image":{"@id":"https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#primaryimage"},"thumbnailUrl":"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python.jpg","articleSection":["Python"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/python-programs.com\/how-to-code-a-scraping-bot-with-selenium-and-python\/#respond"]}]},{"@type":"Person","@id":"https:\/\/python-programs.com\/#\/schema\/person\/9f9e30fd3f415217a11ac0d939213b7f","name":"veer","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/python-programs.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/f1f6915d5328abaea9a64249313d1c55?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/f1f6915d5328abaea9a64249313d1c55?s=96&d=mm&r=g","caption":"veer"},"sameAs":["https:\/\/python-programs.com"],"url":"https:\/\/python-programs.com\/author\/veer\/"}]}},"jetpack_featured_media_url":"https:\/\/python-programs.com\/wp-content\/uploads\/2021\/04\/How-to-Code-a-Scraping-Bot-with-Selenium-and-Python.jpg","_links":{"self":[{"href":"https:\/\/python-programs.com\/wp-json\/wp\/v2\/posts\/2224"}],"collection":[{"href":"https:\/\/python-programs.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/python-programs.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/python-programs.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/python-programs.com\/wp-json\/wp\/v2\/comments?post=2224"}],"version-history":[{"count":1,"href":"https:\/\/python-programs.com\/wp-json\/wp\/v2\/posts\/2224\/revisions"}],"predecessor-version":[{"id":2232,"href":"https:\/\/python-programs.com\/wp-json\/wp\/v2\/posts\/2224\/revisions\/2232"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/python-programs.com\/wp-json\/wp\/v2\/media\/2225"}],"wp:attachment":[{"href":"https:\/\/python-programs.com\/wp-json\/wp\/v2\/media?parent=2224"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/python-programs.com\/wp-json\/wp\/v2\/categories?post=2224"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/python-programs.com\/wp-json\/wp\/v2\/tags?post=2224"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}