Is Web Scraping or Crawling Legal in Singapore?

Crawling and scraping websites can be a quick way for researchers, tech start-ups, and other businesses to rapidly gather data on a company or market by analysing publicly available data on websites. This article explains:

• What is web crawling and scraping
• Issues caused by web crawling and scraping
• Whether web crawling and scraping is legal in Singapore
• What should you do to minimise risk when scraping a site
• Potential means to protect one’s own website from being crawled

What is Web Crawling and Scraping?

A web crawler is a program that accesses and downloads websites in an automated manner. Typically, the program is given a set of starting webpages, and it will use hyperlinks on those webpages to access and download other pages on the website. This process is repeated until the entire website is downloaded.

While a human user can manually access and download these same pages, the advantage of using a web crawler is its speed – a web crawler can potentially access more than 200 pages per second.

A web scraper, on the other hand, is a bot that automatically extracts specific useful information or data from webpages downloaded by the crawler. This information could include product prices, financial research, news content, and even market sentiment.

These tools are often used together to automatically index and generate meaningful data from websites. Search engines such as Google use such bots to create a searchable index of websites on the internet. Bots may also be used by businesses to gather information on competitors, or by researchers as a means of gathering data.

What are the Issues Caused by Web Crawling and Scraping?

Many website owners would want their site to be crawled and scraped by search engines. This is so that their site would be indexed and made searchable via the search engine, and allow them to benefit from increased traffic. However, there are also potential downsides involved.

First, the speed at which web crawler bots can access sites is much faster than is humanly possible. As a result, this can cause intense strain on the server of the website being crawled, potentially causing a severe slowdown of the website or even crashing it completely.

The web scraping could also be done by businesses to quickly collect data from the websites of competing businesses and thereby generate insights as to the operations, pricing or other market information of the business whose website has been scraped.

In either case, the website owner could wish to prevent others from crawling or scraping his website and may do so by enforcing his legal rights through civil and/or criminal law.

Is Web Scraping and Crawling Legal in Singapore?

There is no specific law in Singapore that directly addresses whether web crawling or scraping is legal. However, crawling and scraping could possibly attract civil liability under existing contract law and copyright law, and even criminal liability under the Computer Misuse Act.

Breach of website terms of use

When accessing a website, the user generally agrees to access it in accordance with the website’s “terms of use” as stated on the site, and this forms a legally binding agreement.

If the terms of use prohibit crawling and scraping, doing so on the website would be a breach of this agreement. The website owner may then sue the bot operator for breach of contract, and claim monetary compensation for any loss suffered in relation to any downtime or slowdown the website may have incurred.

For example, when property listing platform 99.co first started operations, it used a web scraper to scrape rental listings from competing platform PropertyGuru for listing on its own platform. PropertyGuru regarded this as potential breaches of its website’s Terms of Service and Acceptable Use Policy and infringement of its copyright in the listings, and requested 99.co to stop doing so.

Both platforms eventually settled the matter out of court in 2015, with 99.co signing an agreement not to substantially reproduce the content in PropertyGuru’s website without its consent.

What Should You Do to Minimise Risk When Scraping a Site?

Liability under the website’s terms of use only arises if the website’s terms of use state that crawling or scraping is prohibited. It is therefore important to first check the website’s terms of use before crawling or scraping the website. Not all websites have an anti-crawling and scraping clause within their terms of use, and not all sites have terms of use at all.

If it is prohibited, you may nonetheless seek consent from the site owner by contacting them. For example, a company might provide permission to non-profit researchers or as part of a business collaboration provided there is an undertaking that the crawler bot will be limited to a reasonable speed that will not cause the website to noticeably slow down.

Copyright infringement

When a web crawler downloads pages from a website, this constitutes copying as data and information on the website is being copied onto the web crawler user’s computer. However, whether this unauthorised copying infringes copyright law depends on the content of the website that has been copied.

Generally, copyright law allows the owner of the copyrighted material to prevent copying of that content. However, copyright protects only content which was produced using a level of human creativity or intellect. This means that if the information being copied is unsorted or automatically sorted data, there may not be any infringement of copyright.

If the site crawled includes news or opinion articles, images of products, or data arranged in a creative and non-automated way, copying this would likely constitute an infringement of the website owner’s copyright in that content.

That said, even if the process of web crawling or scraping may involve the copying of copyrighted content, such copying may nonetheless constitute “fair dealing”, in which case there would not be an infringement. In determining if there was fair dealing, a court deciding the issue might consider whether the copying was for profit, how much was copied, and whether it is possible to acquire a copy within a reasonable time for a fee.

Case study: is there copyright infringement when a website’s content is copied?

Although there appears to be no local cases on the extent to which crawled or scraped webpages would be protected by copyright, and whether the copying of such webpages may nonetheless constitute fair dealing, the later events in the case between 99.co and PropertyGuru may still offer a useful case study on this issue.

This was because the parties went to court after 99.co started offering a Posting Assistant Service for property agents to repost their PropertyGuru listings on 99.co. Here, there was no web scraping involved because 99.co hired independent contractors to manually copy the property agents’ listings on PropertyGuru for reposting on 99.co.

However, the High Court found that PropertyGuru did not own the copyright in the listing photos that had been copied over to 99co’s website. This was even though the photos on PropertyGuru bore the platform’s watermark, and had also been edited in ways such as being resized and having their light balance altered or their edges softened.

The court found that such editing and watermark placement did not substantially alter the photos that had been originally provided by the property agents to PropertyGuru. The fact that PropertyGuru had stated in its website’s Terms of Service that it owned the copyright in the edited photos did not make a difference either, because the court found that PropertyGuru did not legally own the copyright in such photos in the first place.

Although these events did not involve web scraping, it is likely that the court would have come to these same conclusions if PropertyGuru’s listings had been scraped instead of manually copied to 99.co’s website.

What can you do to minimise the risk of copyright infringement?

To limit the likelihood of copyright infringement, you could attempt to limit the web scraping and crawling to raw data, since such information may not be protected by copyright at all.

If the purpose of the crawling requires that articles or images be downloaded, you should first seek the permission of the website owner as written articles, photographs or illustrations involve human creativity or intellect and would therefore be protected by copyright.

If you are suing or being sued for copyright infringement involving a crawling or scraping bot, we recommend engaging an intellectual property lawyer to advise you as this is an uncertain area of law.

Criminal offence under the Computer Misuse Act

Web crawling or scraping may be illegal under section 3 of the Computer Misuse Act (CMA) which criminalises unauthorised access to computer material. This means using a computer to access a program or data without permission. Crawling and scraping is especially likely to be considered an offence if the crawler bot accesses parts of the site that individuals normally would not be able to access, such as by using hidden hyperlinks or finding other backdoors.

If you are found guilty of obtaining unauthorised access to computer material, this offence is punishable by a fine of up to $5,000, or up to 2 years’ imprisonment, or both. For subsequent convictions, the maximum punishment increases to a fine of up to $10,000, or up to 3 years’ imprisonment, or both.

In addition, if any damage is caused, such as the web crawler or scraping bot causing the website to crash, the punishment increases to a fine of up to $50,000, or up to 7 years’ imprisonment, or both.

Section 7 of the CMA on the other hand, criminalises the unauthorised obstruction of the use of a computer. This means interrupting or slowing down a computer system or server without permission.

The offence was targeted at criminalising “denial-of-service” attacks but has also been applied to the situation of an individual spamming an email server with thousands of emails to cause a slowdown.

Similarly, due to the rapid speed at which web crawler bots can access sites, using such a bot could slow down access to the website owner’s site or even cause it to crash. If so, using a web-crawler could result in criminal liability.

If you are found guilty of unauthorised obstruction of the use of a computer, this offence is punishable by a fine of up to $10,000, or up to 3 years’ imprisonment, or both. For subsequent convictions, the maximum punishment increases to a fine of up to $20,000, or up to 5 years’ imprisonment, or both.

If any damage is caused, the punishment increases to a fine of up to $50,000, or up to 7 years’ imprisonment, or both.

What can you do to minimise criminal liability under the CMA?

Avoiding possible criminal liability under section 3 of the CMA for unauthorised access to computer material would involve ensuring that the crawling or scraping is not against the website’s terms of use, or that you receive permission from the website owner first such that you have authority to access the site.

On the other hand, avoiding liability under section 7 of the CMA for unauthorised obstruction of use of a computer would involve ensuring that the crawler bot does not overload the site. This can be done by limiting the page access rate of the bot closer to the speed a human could reasonably achieve – such as accessing a new page only once every 10 seconds.

How Can You Protect Your Website From Being Crawled or Scraped?

Through your website’s terms of use

The first means is by stating in your website’s terms of use that use of any bots to crawl or scrape the site is prohibited. This acts as a deterrent against individuals using crawling and scraping bots on your website and some may seek your permission before using these bots.

If you detect unauthorised scraping or crawling and can identify the operator of a web crawler or scraper, you may be able to claim monetary compensation from that person or business resulting from any website downtime or loss to your business.

Through bot permissions

In addition to prohibiting crawling in scraping in the terms of use, you may also include a “robots.txt” file on your website to set out, in code, the permissions given to bots crawling the site. This allows you to specify which pages may be accessed by crawlers. Google and other search engine crawler bots are programmed to check this file to determine which pages they are denied access to.

However, the crawler bot needs to be programmed to check and obey the permissions stated in “robots.txt”. While it is considered good practice for a crawler bot to obey these permissions, it is possible for a researcher or competitor to use a bot that ignores the permissions set out in the “robots.txt” file.

Thus, having a “robots.txt” file is not a foolproof method of preventing bots from crawling your site.

Through CAPTCHA tests

Another technological means of preventing bots from crawling your website is by implementing “CAPTCHA” tests on the site. This prevents a user from continuing to access your site unless they pass a test which a bot may have difficulties with, such as deciphering distorted text or identifying elements from an image.

If the bot is unable to pass the test, it would not be able to continue to crawl your website.

Through making a police report

If a crawler bot has gained unauthorised access to parts of your site, or caused your website to crash, you may make a police report as an offence may have been committed under the CMA.

It may, however, be difficult to accurately trace and identify the perpetrator, especially if he had taken steps to mask his online identity and address.

—

As the application of law to new technologies, such as the use of bots and AI for web crawling and scraping, is a fast-evolving area in Singapore, it may be helpful to consult a technology lawyer who is familiar with the most recent legislation and decisions made in Singapore and in other jurisdictions relating to these issues.

For example, if you intend to use a web crawler and scraping bot for your business, a lawyer will be able to advise on legal best practices to minimise both your civil and criminal liability.

On the other hand, if you are seeking to protect your website, a lawyer can advise on appropriate clauses to be included in your website’s terms of use to deter unauthorised web crawling and scraping, and possible legal actions to take in response to unauthorised crawling and scraping.

You can get in touch with experienced technology lawyers here.