Skip to main content
Time to complete: ~25 minutes
  • Python 3.8 or higher
  • Basic knowledge of Python programming
  • Familiarity with web scraping concepts
  • Basic understanding of CSS selectors

​
Our Goal

For this demo, we’ll scrape commercial properties for sale in Inner West, Sydney for the following information:
Property Information:
  • Title
  • Address
  • Price information
  • Attributes (Floor Area, etc.)
  • Type
  • Description
  • Websites
  • Photos
Agency Information:
  • Agency Name
  • Agency Address
  • Agency Phone Number
  • Agent Names
  • Agent Phone Numbers
All of this data lives on the detail page of each listing. To find those detail pages, we start from the search results page, which we’ll call our start_url:
property-card-highlighted.png

​
Our Scraper

From the , we’ll collect the URLs of all the :
details-page.png
The flow is simple: search page → detail pages → data.
You can replace the starting URL with any other search page on the website, such as:Or even a search results page with specific filters applied, such as:
search-results.png
We have to visit the detail page regardless, since the search page doesn’t carry every field we need. Given that, it’s cleaner to give each step one job: the search page is for discovering URLs, and the detail page is for extracting data. Splitting a property’s fields across two scrapers means two places to update when the site changes, and two places for bugs to hide.

​
Pagination

The search page only shows 10 properties at a time, out of 278 total. To get the rest, we need to keep clicking the next page button until we’ve visited every page. This is called , and our scraper needs to handle it so no properties are missed.
listing-page.png
next-page-button.png

The next page button

last-page.png

Last page of results

TL;DR: We loop through all the search pages to collect the URLs of every detail page, then loop through those detail pages to extract the information we want.
With the plan in place, let’s build it.

​
Step by step guide

1

Create the project

  1. Open your editor. We’ll use VS Code, but any editor works.
  2. Create your project folder. You can also use the terminal:
  3. Open the project folder in VS Code:
    File > Open folder > Select the folder you just created
  4. Open a new terminal:
    Terminal > New Terminal
    Or use the shortcut:
    for Windows
    for macOS
  5. Create a new virtual environment:
  6. Activate your virtual environment:
  7. Create a new Python file:
    File > New File
Your project directory will look like this:
2

Install dependencies and import packages

We’ll use three Python libraries:
  • requests for making HTTP requests to the website
  • BeautifulSoup for parsing HTML and extracting data
  • pandas for organizing the scraped data into a DataFrame and saving it to a CSV file
Install them:
Then add these imports at the top of scraper.py:
3

Declare the URL variables

Two variables here:
  • base_url: used to complete partial/relative URLs, for example:
    base_url + /for-sale/inner-west-nsw/?page=2
    https://www.realcommercial.com.au/for-sale/inner-west-nsw/?page=2
  • start_url: the search page we’ll use to collect all the detail URLs.
4

Make a test request

Let’s make a test request to the start_url and see what comes back:
HTTP 429 means “Too Many Requests.” The fix: copy your browser’s request headers into the script.
request-headers.png
The user-agent and cookie are the ones that matter most. Retry the request:
And we’re in.
This is a short-term fix. For long-running scrapers, you’ll want to rotate proxies and user-agents and add strategic delays. See the “Going to production” section at the end.
5

Collect all the detail page URLs

Time to write the logic for pulling detail URLs off the search page:
anchor-highlighted.pngWhat’s happening:
  1. We make a request to the search page.
  2. We parse the response into a BeautifulSoup object.
  3. Detail URLs live in anchor tags with the class Address_link_Hqm3u (you can confirm this via right-click → Inspect on any property address). We collect all 10 per page.
  4. We look for the next page link and call the function recursively until there are no more pages.
This gives us the URLs for all 278 detail pages. Next stop: the detail pages.
6

Extract the property JSON from the detail page

Let’s look at a detail page and figure out where our data lives. Starting with the price, you’ll see the selector is:
But when you try to find it with BeautifulSoup:
you get:
Why? The price isn’t in the initial HTML. It’s rendered by JavaScript. So how do we get it?
  1. Right-click → View Page Source
  2. Search for the price. There’s only one instance, buried in a <script> tag.
price-on-html.png
price-in-inspect.png
That script tag holds a big JSON blob called REA.pageData, and it contains every field we need. Here’s the function that digs it out:
What’s happening:
  1. We find the script tag containing REA.pageData.
  2. We use regex to extract everything between the curly braces:
The parsed page_data object has a lot of noise. It has three top-level keys, but we only care about one:
So we grab listing and pass it to normalize_property (covered in the next step), which flattens it and strips the fields we don’t need. The payoff: no CSS selector hunting, just clean JSON.Now wrap the whole thing in a function that takes a detail URL and returns the property dict:
7

Normalize the property JSON

On to normalize_property:
What’s happening:
  1. The full listing object is massive. We only keep the fields listed in fields_to_keep.
  2. Then we flatten and normalize each one.
The result is a clean, flat dict like this:
8

Write the main function

With all the pieces in place, here’s the main block that ties them together:
  1. We collect all 278 URLs using collect_detail_urls.
  2. We loop through them and append each property dict to properties.
  3. We convert the list into a DataFrame and save it as a CSV.
  4. For this demo, we only scrape the first 30 URLs to avoid hammering the server. Remove the [:30] slice to scrape everything.
Python dicts are unordered, so columns in the final output may appear in a random order. The order_columns function fixes that:
The extra columns come from attributes, which vary per property. We slot them in the middle.
Sample Output

​
The complete script

scraper.py

​
Going to production

The script above is happy-path code built for a demo. Before running it at scale or on a schedule, there are a few things worth adding:
  • Rotate proxies and user-agents. A single IP hitting the site repeatedly will get throttled or blocked. Rotate both to spread the load.
  • Add strategic delays. The sleep(3) between requests is a floor, not a ceiling. Consider randomised delays and exponential backoff on errors.
  • Handle failures gracefully. Detail pages can 404, REA.pageData may be missing on some listings, and requests can time out. Wrap requests in try/except and log failures instead of crashing the run.
  • Respect the site. Check robots.txt and the site’s terms of service, and keep your request volume reasonable.