{"id":32246,"date":"2022-09-07T12:56:22","date_gmt":"2022-09-07T10:56:22","guid":{"rendered":"https:\/\/www.wbscodingschool.com\/?p=32246"},"modified":"2025-05-12T15:33:52","modified_gmt":"2025-05-12T15:33:52","slug":"web-scraping-with-python-for-beginners-how-to-get-started","status":"publish","type":"post","link":"https:\/\/www.wbscodingschool.com\/blog\/web-scraping-with-python-for-beginners-how-to-get-started\/","title":{"rendered":"Web Scraping With Python For Beginners: How To Get Started"},"content":{"rendered":"\n

Web scraping is the process of using programming tools to extract data from a website. It is also a skill that has exploded in relevance in the past couple of decades, as the amount of information amassed on the internet has offered more and more information to businesses, researchers, marketers, and journalists, among many others.<\/p>\n\n\n\n

In this article we will look at some of the basic functions offered by Python for web scraping, as well as how to retrieve data via APIs. This is not an exhaustive exploration of the topic (such a thing would take much more than just one article!), only a gentle introduction that should allow you, as a beginner, to get started with getting data from the internet in practice.<\/p>\n\n\n\n

Before starting, please be mindful of what data you scrape, from where, and what you wish to do with it. Some websites contain private data, others are subject to copyright. If you wish to retrieve data online beyond the step-by-step examples in this article, please check if you have permission<\/a>.<\/p>\n\n\n\n

While this article is for the beginner, we will assume a minimum level of literacy with Python. You should already know what variables, loops and functions are, for example.<\/p>\n\n\n\n

The contents of this article will be organised as follows:<\/p>\n\n\n\n

CONTENTS<\/strong><\/h3>\n\n\n\n

First Things First: What Libraries Will You Need?<\/a><\/em><\/p>\n\n\n\n

Understanding The Code Of A Web Page<\/a><\/em><\/p>\n\n\n\n

Getting Started With Requests<\/a><\/em><\/p>\n\n\n\n

Put Beautiful Soup To Use<\/a><\/em><\/p>\n\n\n\n

Web Scraping With APIs<\/a><\/em><\/p>\n\n\n\n

Retrieving Dynamic Data About Flights<\/a><\/em><\/p>\n\n\n\n

Conclusion: The Next Steps<\/a><\/em><\/p>\n\n\n\n

With no further ado then, let\u2019s get you scraping the web!<\/p>\n\n\n\n

<\/p>\n\n\n\n

First Things First: What Libraries Will You Need?<\/em><\/h2>\n\n\n\n
\"Web
Adobe Stock \/ Ryan<\/figcaption><\/figure>\n\n\n\n

As is often the case with a general-use language like Python, performing specialised tasks will require libraries designed specifically for those purposes.<\/p>\n\n\n\n

When it comes to scraping data from the internet, there are two different libraries that you should familiarise yourself with. The first is Requests<\/strong>, while the second is Beautiful Soup<\/strong>. We will introduce you to both in greater detail over the course of the article.<\/p>\n\n\n\n

We will also be making a marginal use of Pandas<\/strong>, the popular Python library for analysing and manipulating data. You will need Pandas again if you intend to plot, visualize or manipulate the data you scrape, but for the purposes of this particular guide, we will only be using one of its methods.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Understanding The Code Of A Web Page<\/em><\/h2>\n\n\n\n

For those who have been learning Python (and programming in general) for purposes other than web development, let\u2019s start from the absolute basics.<\/p>\n\n\n\n

The majority of webpages is composed of code written in three languages: HTML, CSS, and JavaScript. Right now we are interested in getting information primarily from HTML.<\/p>\n\n\n\n

The good news is that the HTML code for any given website is incredibly easy to find.<\/strong><\/p>\n\n\n\n

Simply open a webpage, right click with your mouse anywhere within it, and you will open a scroll down menu with a number of options. At the bottom of that menu you will find an option called Inspect<\/em>. Click on it, and a whole new section will open (either on the right hand side or the bottom of your browser), showing you the code that composes the website.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

This newly-opened \u2018section\u2019 is called is called the Developer Tools, or DevTools<\/strong>. It is an incredibly powerful resource to explore, analyse and even edit the webpage you are on. Today we will look at only one of the many things that we can do with it: retrieving its HTML code and the information it contains.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The HTML code of a website is the long script that you will see in the DevTools you just opened (right beneath the title \u2018elements\u2019)<\/strong>.<\/p>\n\n\n\n

Pro tip: <\/em><\/strong>If you want to see the code that generates a particular<\/strong> part of the website (for example the specific paragraph of text that you are reading, or an image, or a button), simply highlight that part, right click on it, and click inspect. The section will open at the pertinent line of code, highlighting it for you to see!<\/em><\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

This code that you have revealed contains literally all of the information that is in the page you are visiting. In its raw form, however, it is difficult to read, and even more difficult to use.<\/p>\n\n\n\n

This is where Requests comes to our aid.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Getting Started With Requests<\/em><\/h2>\n\n\n\n

It makes sense to start with Requests, as this library involves some basic concepts which we will use again later on.<\/p>\n\n\n\n

Requests is one of a number of libraries (alongside httplib, urllib, httplib2, and treq) that can be used to do exactly what this guide is about: retrieve information from web pages<\/strong>, or what in the business is known as web scraping.<\/strong><\/p>\n\n\n\n

How is this done? Open your console and let\u2019s do this together!<\/p>\n\n\n\n

First, of course, we import the Requests library.<\/p>\n\n\n\n

import requests<\/code><\/p>\n\n\n\n

Then we use the \u2018get\u2019 method from that same library on the URL of a website to get the contents of that website, that is to say, its HTML code. We will store those contents in an object called response<\/em>, as this is, after all, the response by the website to our request.<\/p>\n\n\n\n

response = requests.get('https:\/\/www.imdb.com\/title\/tt0034583\/fullcredits\/?ref_=tt_ql_cl')<\/code><\/p>\n\n\n\n

The response object isn\u2019t a simple thing.<\/strong> It contains the retrieved data, as well as a variety of attributes and methods to retrieve it. This means that whenever we want to do anything with it, we can\u2019t just treat it like any old variable \u2013 rather, we need to specify which part of it we want to work with<\/strong>. For example:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

You can get the same results by writing print(response.content)<\/em> Just bear in mind that we can\u2019t simply write print(response)<\/em>. This is because the response object in its entirety is more than just readable text, and would therefore return an error if you tried to print it.<\/p>\n\n\n\n

You should also know that Request methods can also be written in this syntax:<\/p>\n\n\n\n

\u00a0<\/em>requests.request(method, url, **kwargs).<\/code><\/p>\n\n\n\n

This means that our line response = requests.get(‚https:\/\/www.imdb.com\/title\/tt0034583\/fullcredits\/?ref_=tt_ql_cl)<\/em> could also have been written like this:<\/p>\n\n\n\n

response = requests.request('get', 'https:\/\/www.imdb.com\/title\/tt0034583\/fullcredits\/?ref_=tt_ql_cl')<\/code><\/p>\n\n\n\n

Now that we have converted all of the contents of a website into a response object, we can do all sorts of things with it.<\/p>\n\n\n\n

For a complete list of all the methods that you can use with Requests, check out the official documentation<\/a>.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Put Beautiful Soup To Use<\/em><\/h2>\n\n\n\n

Once you have an HTML response object on your Python console, it\u2019s time to work with it. The library Beautiful Soup exists to let you pull data out of HTML and XML files, and is perfectly suitable for our purposes. So how do we use it?<\/p>\n\n\n\n

Once again, let\u2019s do this step by step. First of all, import the library Beautiful Soup into your Python script.<\/p>\n\n\n\n

from bs4 import BeautifulSoup<\/code><\/p>\n\n\n\n

We will then use the method html.parser, which is Python\u2019s in-built parser, to turn our response object into something that Python can work with. We will use it on the HTML code we have scraped and put the result into a new variable, which here we will call soup<\/em>.<\/p>\n\n\n\n

soup = BeautifulSoup(response.text, 'html.parser')<\/code><\/p>\n\n\n\n

What can we do with this \u2018soup\u2019? Let\u2019s have a look.<\/p>\n\n\n\n

HTML code is composed of tags<\/em>, which are pairs of letters or words that wrap up each item of content on a web page, and which are contained by the symbols <> and <\/>. Thus, a paragraph of text on a webpage will be indicated by the letter p<\/em> and written like so:<\/p>\n\n\n\n

<p>This is a paragraph.<\/p><\/code><\/p>\n\n\n\n

With this in mind, we can explore the contents of a web page simply by asking Python to show us its appropriate HTML tags:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

You can explore tags in greater depth by concatenating them in the same command.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

A more practical way to explore this code is to use methods built specifically for that purpose. For example:<\/p>\n\n\n\n

soup.find(\"p\")<\/code><\/p>\n\n\n\n

This will only return the first element of its type that it finds, so in this case the first paragraph.<\/p>\n\n\n\n

soup.find_all(\"p\")<\/code><\/p>\n\n\n\n

This will return all<\/em> the elements of this type, so in this case all paragraphs.<\/p>\n\n\n\n

Another option instead of .find_all() is to use .select(), which will locate all the elements of a CSS class, and \u2013 importantly \u2013 return them as a list<\/em>.<\/p>\n\n\n\n

Note that .select() requires an uncommon syntax to be used. It lets you conveniently find tags within other tags, but you must call them both within the same inverted commas (as though they were both parts of a string), and surprisingly, you don\u2019t have to separate them with a comma. Meanwhile, different attributes that are part of the same tag (for example, a CSS class) are separated by full stops!<\/p>\n\n\n\n

This means that for an HTML code that looks like this…<\/p>\n\n\n\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <div class=\"parent\"><\/code><\/p>\n\n\n\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0 <h3 itemprop=\"name\"><\/code><\/p>\n\n\n\n

<a href=\"\/title\/tt0034583\/?ref_=ttfc_fc_tt<\/a>\"<\/code><\/p>\n\n\n\n

itemprop='url'>Casablanca<\/a><\/code><\/p>\n\n\n\n

\u2026the way one asks Python to find all the <a> beneath the <div> tags that contain a class called \u201cparent\u201d is this:<\/p>\n\n\n\n

soup.select(\"div.parent a\")<\/code><\/p>\n\n\n\n

See what\u2019s happening there? \u2018Parent\u2019 is a class<\/em> within the tag called \u2018div\u2019, so we call that div.parent. At the same time, a<\/em> is the tag within the higher tag div<\/em>, so we divide that from \u2018title.movies\u2019 with just a blank space, and we also wrap up both the div<\/em> and the a<\/em> within the same inverted commas.<\/p>\n\n\n\n

This can be confusing for the beginner at first, but you\u2019ll get used to it with practice!<\/p>\n\n\n\n

A particularly common and useful method is get_text,\u00a0 which \u2013 as the name suggests \u2013 will return the parts of a webpage that are actually meant to be read.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

You can combine all of these methods with regular Python loops to explore and manipulate the HTML code app your leisure.<\/p>\n\n\n\n

for\u00a0p\u00a0in\u00a0soup:<\/code><\/p>\n\n\n\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0print(soup.p.get_text())\u00a0<\/code><\/p>\n\n\n\n

It then becomes relatively simple to write Python functions that let you repeat particular tasks on different pages of the same website, as long as those pages are structurally similar. A function that lets me print the names of all the actors in the IMDB cast page of Casablanca<\/em> should do the same for the cast page of The Godfather<\/em>.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Web Scraping With APIs<\/em><\/h2>\n\n\n\n

We\u2019ve looked at some of the basic methods to take data from the internet and put it into your Python console so you can work with it. Let\u2019s now look at something just a little more complex.<\/p>\n\n\n\n

Web scraping is generally easier the more static the data you are working with is. A function that takes the name of a city and returns its latitude and longitude based on its Wikipedia page is relatively easy to build, for example, because neither the coordinates nor the website displaying them will change. To whit, you just get the function to create a soup based on its url as an f_string…<\/p>\n\n\n\n

city = \"Berlin\"<\/code><\/p>\n\n\n\n

html = requests.get(f\"https:\/\/en.wikipedia.org\/wiki\/{city}\")<\/code><\/p>\n\n\n\n

soup = BeautifulSoup(html.text, 'html.parser')<\/code><\/p>\n\n\n\n

…then you identify which tags within the HTML code contain the coordinates, extract them with .find() or .select(), and get the function to return those. In this case the relevant tag is called span<\/em>, and its CSS classes are latitude<\/em> and longitude<\/em> respectively.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Therefore:<\/p>\n\n\n\n

latitude\u00a0=\u00a0soup.select(\"span.latitude\")<\/code><\/p>\n\n\n\n

longitude\u00a0=\u00a0soup.select(\"span.longitude\")<\/code><\/p>\n\n\n\n

print(f\"For\u00a0{city},\u00a0the\u00a0longitude\u00a0is\u00a0\"\u00a0+\u00a0longitude[0].get_text()<\/code><\/p>\n\n\n\n

+\u00a0\"\u00a0and\u00a0the\u00a0latitude\u00a0is\u00a0\"\u00a0+\u00a0latitude[0].get_text())<\/code><\/p>\n\n\n\n

Done and done!<\/p>\n\n\n\n

However, things get a lot trickier when you want to use Python to handle data that is constantly changing.<\/p>\n\n\n\n

Suppose, for instance, that you wanted to create a function that checks the results for matches played in a particular football league. Now you are dealing with results that are changing all the time, and the web pages showing these results will change accordingly. How can we do this?<\/p>\n\n\n\n

Skilled data scientists are able to write complex, flexible code that can investigate dynamic queries using a combination of different web resources. But beginners won\u2019t have to go that far.<\/p>\n\n\n\n

For the most popular queries related to dynamic data \u2013 including things like the results of football games \u2013 there already exist online resources that do the work of storing and updating that data regularly. These resources aren\u2019t always fully-fledged websites \u2013 they may be something as simple as APIs.<\/p>\n\n\n\n

An API, or Application Programming Interface, is simply the code that websites use to request data from each other. APIs are made up of routes<\/em>, and actually consist of pretty simple code. You can see an example of an API simply by looking at the URL of any given website \u2013 that URL is, precisely, the API that tells your browser where to go!<\/p>\n\n\n\n

For example, here is the API for a YouTube video:<\/p>\n\n\n\n

https:\/\/www.youtube.com\/watch?v=dQw4w9WgXcQ<\/code><\/p>\n\n\n\n

And here\u2019s what this is doing:<\/p>\n\n\n\n