Is Web Scraping or Crawling Legal in Singapore?

Last updated on November 27, 2020

Hands on computer keyboard

Crawling and scraping websites can be a quick way for researchers, tech start-ups, and other businesses to rapidly gather data on a company or market by analysing publicly available data on websites. This article explains:

What is Web Crawling and Scraping?

A web crawler is a program that accesses and downloads websites in an automated manner. Typically, the program is given a set of starting webpages, and it will use hyperlinks on those webpages to access and download other pages on the website. This process is repeated until the entire website is downloaded.

While a human user can manually access and download these same pages, the advantage of using a web crawler is its speed – a web crawler can potentially access more than 200 pages per second.

A web scraper, on the other hand, is a bot that automatically extracts specific useful information or data from webpages downloaded by the crawler. This information could include product prices, financial research, news content, and even market sentiment.

These tools are often used together to automatically index and generate meaningful data from websites. Search engines such as Google use such bots to create a searchable index of websites on the internet. Bots may also be used by businesses to gather information on competitors, or by researchers as a means of gathering data.

What are the Issues Caused by Web Crawling and Scraping?

Many website owners would want their site to be crawled and scraped by search engines. This is so that their site would be indexed and made searchable via the search engine, and allow them to benefit from increased traffic. However, there are also potential downsides involved.

First, the speed at which web crawler bots can access sites is much faster than is humanly possible. As a result, this can cause intense strain on the server of the website being crawled, potentially causing a severe slowdown of the website or even crashing it completely.

The web scraping could also be done by businesses to quickly collect data from the websites of competing businesses and thereby generate insights as to the operations, pricing or other market information of the business whose website has been scraped.

In either case, the website owner could wish to prevent others from crawling or scraping his website and may do so by enforcing his legal rights through civil and/or criminal law.

Is Web Scraping and Crawling Legal in Singapore?

There is no specific law in Singapore that directly addresses whether web crawling or scraping is legal. However, crawling and scraping could possibly attract civil liability under existing contract law and copyright law, and even criminal liability under the Computer Misuse Act.

Breach of website terms of use

When accessing a website, the user generally agrees to access it in accordance with the website’s “terms of use” as stated on the site, and this forms a legally binding agreement.

If the terms of use prohibit crawling and scraping, doing so on the website would be a breach of this agreement. The website owner may then sue the bot operator for breach of contract, and claim monetary compensation for any loss suffered in relation to any downtime or slowdown the website may have incurred.

For example, when property listing platform 99.co first started operations, it used a web scraper to scrape rental listings from competing platform PropertyGuru for listing on its own platform. PropertyGuru regarded this as potential breaches of its website’s Terms of Service and Acceptable Use Policy and infringement of its copyright in the listings, and requested 99.co to stop doing so.

Both platforms eventually settled the matter out of court in 2015, with 99.co signing an agreement not to substantially reproduce the content in PropertyGuru’s website without its consent.

What Should You Do to Minimise Risk When Scraping a Site?

Liability under the website’s terms of use only arises if the website’s terms of use state that crawling or scraping is prohibited. It is therefore important to first check the website’s terms of use before crawling or scraping the website. Not all websites have an anti-crawling and scraping clause within their terms of use, and not all sites have terms of use at all.

If it is prohibited, you may nonetheless seek consent from the site owner by contacting them. For example, a company might provide permission to non-profit researchers or as part of a business collaboration provided there is an undertaking that the crawler bot will be limited to a reasonable speed that will not cause the website to noticeably slow down.

Copyright infringement

When a web crawler downloads pages from a website, this constitutes copying as data and information on the website is being copied onto the web crawler user’s computer. However, whether this unauthorised copying infringes copyright law depends on the content of the website that has been copied.

Generally, copyright law allows the owner of the copyrighted material to prevent copying of that content. However, copyright protects only content which was produced using a level of human creativity or intellect. This means that if the information being copied is unsorted or automatically sorted data, there may not be any infringement of copyright.

If the site crawled includes news or opinion articles, images of products, or data arranged in a creative and non-automated way, copying this would likely constitute an infringement of the website owner’s copyright in that content.

That said, even if the process of web crawling or scraping may involve the copying of copyrighted content, such copying may nonetheless constitute “fair dealing”, in which case there would not be an infringement. In determining if there was fair dealing, a court deciding the issue might consider whether the copying was for profit, how much was copied, and whether it is possible to acquire a copy within a reasonable time for a fee.

Case study: is there copyright infringement when a website’s content is copied?

Although there appears to be no local cases on the extent to which crawled or scraped webpages would be protected by copyright, and whether the copying of such webpages may nonetheless constitute fair dealing, the later events in the case between 99.co and PropertyGuru may still offer a useful case study on this issue.

This was because the parties went to court after 99.co started offering a Posting Assistant Service for property agents to repost their PropertyGuru listings on 99.co. Here, there was no web scraping involved because 99.co hired independent contractors to manually copy the property agents’ listings on PropertyGuru for reposting on 99.co.

However, the High Court found that PropertyGuru did not own the copyright in the listing photos that had been copied over to 99co’s website. This was even though the photos on PropertyGuru bore the platform’s watermark, and had also been edited in ways such as being resized and having their light balance altered or their edges softened.

The court found that such editing and watermark placement did not substantially alter the photos that had been originally provided by the property agents to PropertyGuru. The fact that PropertyGuru had stated in its website’s Terms of Service that it owned the copyright in the edited photos did not make a difference either, because the court found that PropertyGuru did not legally own the copyright in such photos in the first place.

Although these events did not involve web scraping, it is likely that the court would have come to these same conclusions if PropertyGuru’s listings had been scraped instead of manually copied to 99.co’s website.

What can you do to minimise the risk of copyright infringement?

To limit the likelihood of copyright infringement, you could attempt to limit the web scraping and crawling to raw data, since such information may not be protected by copyright at all.

If the purpose of the crawling requires that articles or images be downloaded, you should first seek the permission of the website owner as written articles, photographs or illustrations involve human creativity or intellect and would therefore be protected by copyright.

If you are suing or being sued for copyright infringement involving a crawling or scraping bot, we recommend engaging an intellectual property lawyer to advise you as this is an uncertain area of law.

Criminal offence under the Computer Misuse Act

Web crawling or scraping may be illegal under section 3 of the Computer Misuse Act (CMA) which criminalises unauthorised access to computer material. This means using a computer to access a program or data without permission. Crawling and scraping is especially likely to be considered an offence if the crawler bot accesses parts of the site that individuals normally would not be able to access, such as by using hidden hyperlinks or finding other backdoors.

If you are found guilty of obtaining unauthorised access to computer material, this offence is punishable by a fine of up to $5,000, or up to 2 years’ imprisonment, or both. For subsequent convictions, the maximum punishment increases to a fine of up to $10,000, or up to 3 years’ imprisonment, or both.

In addition, if any damage is caused, such as the web crawler or scraping bot causing the website to crash, the punishment increases to a fine of up to $50,000, or up to 7 years’ imprisonment, or both.

Section 7 of the CMA on the other hand, criminalises the unauthorised obstruction of the use of a computer. This means interrupting or slowing down a computer system or server without permission.

The offence was targeted at criminalising “denial-of-service” attacks but has also been applied to the situation of an individual spamming an email server with thousands of emails to cause a slowdown.

Similarly, due to the rapid speed at which web crawler bots can access sites, using such a bot could slow down access to the website owner’s site or even cause it to crash. If so, using a web-crawler could result in criminal liability.

If you are found guilty of unauthorised obstruction of the use of a computer, this offence is punishable by a fine of up to $10,000, or up to 3 years’ imprisonment, or both. For subsequent convictions, the maximum punishment increases to a fine of up to $20,000, or up to 5 years’ imprisonment, or both.

If any damage is caused, the punishment increases to a fine of up to $50,000, or up to 7 years’ imprisonment, or both.

What can you do to minimise criminal liability under the CMA?

Avoiding possible criminal liability under section 3 of the CMA for unauthorised access to computer material would involve ensuring that the crawling or scraping is not against the website’s terms of use, or that you receive permission from the website owner first such that you have authority to access the site.

On the other hand, avoiding liability under section 7 of the CMA for unauthorised obstruction of use of a computer would involve ensuring that the crawler bot does not overload the site. This can be done by limiting the page access rate of the bot closer to the speed a human could reasonably achieve – such as accessing a new page only once every 10 seconds.

How Can You Protect Your Website From Being Crawled or Scraped?

Through your website’s terms of use

The first means is by stating in your website’s terms of use that use of any bots to crawl or scrape the site is prohibited. This acts as a deterrent against individuals using crawling and scraping bots on your website and some may seek your permission before using these bots.

If you detect unauthorised scraping or crawling and can identify the operator of a web crawler or scraper, you may be able to claim monetary compensation from that person or business resulting from any website downtime or loss to your business.

Through bot permissions

In addition to prohibiting crawling in scraping in the terms of use, you may also include a “robots.txt” file on your website to set out, in code, the permissions given to bots crawling the site. This allows you to specify which pages may be accessed by crawlers. Google and other search engine crawler bots are programmed to check this file to determine which pages they are denied access to.

However, the crawler bot needs to be programmed to check and obey the permissions stated in “robots.txt”. While it is considered good practice for a crawler bot to obey these permissions, it is possible for a researcher or competitor to use a bot that ignores the permissions set out in the “robots.txt” file.

Thus, having a “robots.txt” file is not a foolproof method of preventing bots from crawling your site.

Through CAPTCHA tests

Another technological means of preventing bots from crawling your website is by implementing “CAPTCHA” tests on the site. This prevents a user from continuing to access your site unless they pass a test which a bot may have difficulties with, such as deciphering distorted text or identifying elements from an image.

If the bot is unable to pass the test, it would not be able to continue to crawl your website.

Through making a police report

If a crawler bot has gained unauthorised access to parts of your site, or caused your website to crash, you may make a police report as an offence may have been committed under the CMA.

It may, however, be difficult to accurately trace and identify the perpetrator, especially if he had taken steps to mask his online identity and address.

As the application of law to new technologies, such as the use of bots and AI for web crawling and scraping, is a fast-evolving area in Singapore, it may be helpful to consult a technology lawyer who is familiar with the most recent legislation and decisions made in Singapore and in other jurisdictions relating to these issues.

For example, if you intend to use a web crawler and scraping bot for your business, a lawyer will be able to advise on legal best practices to minimise both your civil and criminal liability.

On the other hand, if you are seeking to protect your website, a lawyer can advise on appropriate clauses to be included in your website’s terms of use to deter unauthorised web crawling and scraping, and possible legal actions to take in response to unauthorised crawling and scraping.

You can get in touch with experienced technology lawyers here.

Appointment and Removal of Company Officers and Other Key Personnel
  1. Appointing Company Directors in Singapore: Eligibility, Process etc.
  2. Managing Director vs CEO in Singapore: Roles and Obligations
  3. Guide to Directors' Remuneration in Singapore
  4. Directors' Duties in Singapore
  5. Shadow Directors: Who are They and What Duties Do They Owe to the Company?
  6. How to Remove a Director from a Company in Singapore
  7. Removal and Resignation of Company Auditor in Singapore
  8. Appointing a Company Secretary: Roles and Responsibilities
  9. Appointing an Authorised Representative for Foreign Companies in Singapore
  10. Process Agents in Singapore
Holding Meetings
  1. What are Annual General Meetings (AGMs) in Singapore?
  2. How to Hold Extraordinary General Meetings (EGMs) in Singapore
  3. How to Hold a Board Meeting in Singapore
Shareholder Matters
  1. 2 Ways to Remove a Singapore Company Shareholder ASAP
  2. Guide to Paid-Up Capital in Singapore (Is $1 Enough?)
  3. Preparing a Register of Shareholders for a Singapore Company
  4. How to Issue Shares in a Singapore Private Company
  5. Guide to Transferring Shares in a Singapore Private Company
  6. Your Guide to Share Certificates in Singapore: Usage and How to Prepare
  7. Shareholder Rights in Singapore Private Companies
  8. Shareholder Roles and Obligations in Singapore Companies
  9. Dividend Payments Guide for Singapore Business Owners
  10. Share Transmission: What Happens If a Shareholder Dies in Singapore?
  11. How to Reduce the Share Capital of Your Singapore Company
  12. Buy-Sell Agreements: How to Write & Fund Them in Singapore
  13. Oppression of Minority Shareholders
Compliance
  1. Essential Regulatory Compliance Guide for Singapore Companies
  2. Dormant Companies and Their Filing Obligations in Singapore
  3. Anti-Money Laundering Regulations and Your Business: What You Need to Know
  4. Price-Fixing, Bid-Rigging and Other Anti-Competitive Practices to Avoid
  5. Legally Conducting Lucky Draws for Singapore Businesses
  6. Restaurant Inspection and Food Safety Rules in Singapore
Company Management
  1. Does Your Company Need a Legal Team (In-House Counsel)?
  2. Acqui-Hiring of Singapore Companies: How Does It Work?
  3. How to Change the Name of Your Singapore Company
  4. Can Directors be Liable for Company Debts in Singapore?
  5. Company Loans to Directors/Shareholders in Singapore
  6. 3 Types of Insurance Every Singapore Business Needs
  7. Creating and Registering Charges in Singapore: Guide for Companies
  8. Guide to Effective Business Continuity Planning in Singapore
  9. Business Asset Sale & Disposal in Singapore: How Do They Work?
  10. Business Partnership Disputes in Singapore: How to Resolve
  11. How to Commence a Derivative Action on Behalf of a Company in Singapore
  12. Business Will: How to Pass on Your Business to Your Successors in Singapore
Company Documents
  1. Record-Keeping Requirements for Singapore Companies
  2. Company Constitutions in Singapore and How to Draft One
  3. Company Memorandum and Articles of Association
  4. Company Resolutions: What are They?
  5. Board Resolutions in Singapore
  6. Minutes of Company Meeting in Singapore: How to Record
  7. How to Set Up a Register of Controllers
  8. How to Set Up a Register of Nominee Directors
  9. Guide to Filing Financial Statements for Singapore Business Owners
  10. Filing Annual Returns For Your Business
Tax, Accounting and Audit Matters
  1. Singapore Corporate Tax: How to Pay, Tax Rate, Exemptions
  2. Start-Up Tax Exemption Guide for New Singapore Companies
  3. GST Registration: Requirements and Procedure in Singapore
  4. What is Withholding Tax and When to Pay It in Singapore
  5. Singapore Influencers: Here's How to Calculate Your Income Tax
  6. Tax Investigation of Tax-Evading Business Owners in Singapore
  7. Small Business Accounting Services in Singapore
  8. Company Audits in Singapore: Requirements and Exemptions
Data Protection
  1. Suspect a PDPA Data Breach? Here's What to Do Next
  2. Must You Notify PDPC About a Data Breach in Your Business?
  3. Summary: Your Organisation's 10 Main PDPA Obligations
  4. Essential PDPA Compliance Guide for Singapore Businesses
  5. PDPA Consent Requirements: How Can Your Business Comply?
  6. Is It Legal for Businesses to Ask for Your NRIC in Singapore?
  7. Here's a 7-Step Plan for Companies to Prevent Unauthorised Disclosure When Processing and Sending Personal Data
  8. Cloud Storage of Personal Data: Your Business’ Data Protection Obligations
  9. Drafting a Comprehensive Privacy Policy For Your Singapore Website
  10. GDPR Compliance in Singapore: Is it Required and How to Comply
  11. Appointing a Data Protection Officer For Your Business: All You Need to Know
  12. How Can Companies Dispose of Documents Containing Personal Data?
  13. Check the Do-Not-Call Registry Before Marketing to Singapore Phone Numbers
  14. How to Legally Install CCTVs for Home/Business Use in Singapore
  15. Is Web Scraping or Crawling Legal in Singapore?
  16. Legal Options If Employees Breach Confidentiality in Singapore
Marketing
  1. Your Guide to E-commerce Website Terms of Service in Singapore
  2. Dealing with Defamation of Your Business: Can You Sue?
  3. Sending Email Newsletters That Comply With Singapore Law
  4. A legal guide to drafting a social media policy for your company
  5. Your Guide to a Media Release Form in Singapore
  6. Your Guide to an Influencer Marketing Agreement in Singapore
  7. Outdoor Advertising: How to Legally Display Public Ads in Singapore
Franchising
  1. Starting a Franchise in Singapore: What Franchisors Should Look Out For
  2. Running a Franchise in Singapore: What To Look Out for as a Franchisee
Debt Restructuring
  1. What is Judicial Management and How It Works in Singapore
  2. Schemes of Arrangement: How They Work and How to Apply
  3. Informal Debt Restructuring and Workout in Singapore
Ending a Business
  1. Claw-Back of Assets From Unfair Preference and Undervalued Transactions
  2. Should You Save or Close Your Zombie Company in Singapore?
  3. Voluntary Suspension of Business in Singapore: How to Handle
  4. Winding Up a Singapore Company: Grounds and Procedure
  5. Closing Your Singapore Business: What You Need to Settle
  6. Striking Off a Company
  7. Can a Company that Struck Itself Off the Register Later Apply to Restore Itself?
  8. Dissolution of partnerships in Singapore
  9. What Should a Creditor Do When a Company Becomes Insolvent?
  10. How to File a Proof of Debt Against a Company in Liquidation
  11. Validation of Payments Made by Companies Being Wound Up