A beginner’s guide on scraping sites using Selenium

Introduction

There are multiple technologies available to any aspiring web scraper. Depending on the nature of the specific task at hand, different tools have their advantages. In this guide, we will focus on Selenium, a commonly used python library that automates browser testing.

Since Selenium runs a browser instance, it is particularly suited to scraping sites with JavaScript and other interactive features such as buttons. For simpler sites, I highly recommend using a library like BeautifulSoup to simplify the process.

Below, I shall walk through a recent project where a Selenium web scraper was built to scrape job vacancies from a…

Shawn Tan

Consultant passionate about data

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store