Crawling/Scrapping a website

Let's discuss about Crawling:
Here, we’re going to discuss all the steps to do a web crawling using any language or technology. Crawler/Scrapper/Spider/Bot/ multiple synonyms for same stuff which is basically meant to copy content from any site.
Q.1 Do you think crawling is legal?
Yes, it’s legal until unless you’re not copying data without a website admin’s consent and their permission. (Seek with your local judicial terms & rules before proceeding)
Q.2 I and my company don’t belong to software field, how it can be helpful for my business then?
It can help you out in creating a comparison site, where yours as well as your similar product can be compared easily.

1. Online presence can be tracked- That’s also an important aspect of web scraping where business profiles and reviews on the websites can be scrapped. This can be used to see the performance of the product, the user behavior and reaction.
2. Custom Analysis and curation- This one is basically for the new websites/ channels wherein the scrapped data can be helpful for the channels in knowing the viewer behavior.
3. Online Reputation - In this world of digitalization companies are bullish about the spent on the online reputation management. Thus the web scrapping is essential here as well.
4. Detect fraudulent reviews - It has become a common practice for people to read online opinions and reviews for different purposes. Thus it’s important to figure out the Opinion Spamming: It refers to "illegal" activities example writing fake reviews on the portals. It is also called shilling, which tries to mislead readers. Thus the web scrapping can be helpful crawling the reviews and detecting which one to block, to be verified, or streamline the experience.
5. To provide better targeted ads to your customers- The scrapping not only gives you numbers but also the sentiments and behavioral analytic thus you know the audience types and the choice of ads they would want to see.
6. Business specific scrapping – Taking doctors for example: you can scrape health physicians or doctors from their clinic websites to provide a catalog of available doctors as per specialization and region or any other specification.

7. To gather public opinion- Monitor specific company pages from social networks to gather updates for what people are saying about certain companies and their products. Data collection is always useful for the product’s growth.

8. Search engine results for SEO tracking- By scraping organic search results you can quickly find out your SEO competitors for a particular search term. You can determine the title tags and the keywords they are targeting. Thus you get an idea of which keywords are driving traffic to a website, which content categories are attracting links and user engagement, what kind of resources will it take to rank your site.
9. Price competitiveness- It tracks the stock availability and prices of products in one of the most frequent ways and sends notifications whenever there is a change in competitors' prices or in the market. In ecommerce, Retailers or marketplaces use web scraping not only to monitor their competitor prices but also to improve their product attributes. To stay on top of their direct competitors, nowadays e-commerce sites have started closely monitoring their counterparts
10. Scrape leads- This is another important use for the sales driven organization wherein lead generation is done. Sales teams are always hungry for data and with the help of the web scrapping technique you can scrap leads from directories such as Yelp, Sulekha, Just Dial, Yellow Pages etc. and then contact them to make a sales introduction.
11. For events organization – You can scrape events from thousands of event websites in the US to create an application that consolidates all of the events together.
12. Job scraping sites: Job sites are also using scrapping to list all the data in one place. They scrape different company websites or jobs sites to create a central job board website and have a list of companies that are currently hiring to contact.

For more details, please visit my article. This link has all type of code set as well.
In other thread you'll find all types of Crawler implementation using different major technology. Let’s check out an example of using proxy in crawler to escape from anti robot algorithm and cross browser data as well:

use WWW::Mechanize;
use Try::Tiny;
my $source_file=shift; open (INPUT_FILE, "<$source_file") || die "Can't open $source_file: $!\n";
my @sources = ; my $crawler = WWW::Mechanize->new(); 
foreach (@sources) { 
    try { $crawler->get($_);
        # hunt for IP:PORT combination 
        my @ips= $crawler->text() =~ /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5})/g; 
        foreach (@ips){ 
            print "$_\n"; 
        } 
    } catch { warn "[!] Error, who cares\n";}}