There are times when we want to extract data from a website. In most cases, you are provided with an API, but that’s not always plausible. So, when a website does not provide an API, the only way to get the data from the website is to scrape it off yourself.
In this tutorial, we are going to build a simple scraper using PHP to extract data from Wikipedia (I’d highly recommend you to use Wikipedia API over this. This is just for the tutorial purpose.) and another scraper that extracts Anna University result.
There are various external libraries such as Simple HTML DOM and cURL, which I highly recommend you to check out after this tutorial, but we are going to proceed with simple raw PHP.
Before you proceed, remember that Web Scraping is definitely a gray area. So, make sure you are not breaking terms and conditions of the site you are scraping the data off. This tutorial is just for educational purpose.
Simple Wikipedia Scraper
We are going to do this in 3 simple steps:
Step 1: Get the content
First, we have to pull the content off the Wikipedia page. For this, we use
function, which is used to retrieve the content of a file using its URL. It takes in
a URL of a file and returns its content.
The above code returns the content of the Wikipedia page as a string which the browser interprets as something like this:
Step 2: Parse the string as DOM
DOM stands for
Document Object Model.
It defines the logical structure of documents and the way a document is accessed
and manipulated. In simple words, it generates a tree like structure to access the
data in it. It is used by the browser to parse HTML into displayable content.
$url can be converted to DOM using
class in PHP.
DOMDocument's constructor takes in
XML version and
encoding as parameters.
Don’t worry if you don’t know what they are, just use the line below:
$dom = new DOMDocument('1.0', 'UTF-8');
loadHTML() method of
DOMDocument takes in the content and creates a DOM.
Note: If you’re new to PHP classes, they are very similar to the ones in C++ and Java.
The only difference you have to know to continue this tutorial is that PHP uses
-> instead of
. to access class members.
Step 3: Scrape the data
This is the part where you have to use your brain. Since every website is not built
the same way, you have to know your way through the website’s structure. To do this,
just try going through the website’s source to determine the appropriate
to crack into the data.
if you go through
source, you can see that the title has an ID of
The content inside a
DOMElement is stored in
Next, We’re going to extract the first paragraph of the Wikipedia page. Going
through the source again, we can see that the first paragraph is within a
In the example, the first paragraph is returned by
item(0). Like how we did in
the title, we extract the text using
That’s it! We have successfully scraped data off the Wikipedia page.
Try going through the documentation to get a clearer picture. If you find the documentation difficult, I’d recommend you to go with some library. If you have any problem following the tutorial, please do post it in the comments below.
The part 2 of this tutorial continues on a real world example of scraping the Anna University website for result into an excel file. If you liked this, please do check this out: Web Scraping 101: Anna University Result Scraper