Using regular expressions to pull out the raw data can be a bit intimidating for the uninitiated and a bit messy as a script can contain a lot of them. At the same time, if you're already familiar with regular expressions, and scrape your project is relatively small, they can be a great solution. Other approaches or hierarchical vocabularies intended to represent the content domain deals with the development.
There are some companies (including our own) specific for commercial applications are offered to screen scraping. Applications vary widely, but for medium to large projects, they are often a good solution. Each has its own learning curve, take the time to learn a new application must plan on the ins and outs.
What is the best way to retrieve data? Different approaches here, as well as tips on when you would have any use some of the pros and cons:
Benefits:
- If you already have a regular expression and be familiar with at least one programming language, it can be a quick solution.
- Regular expression that the content of such small changes will not break them in the "vagueness" to achieve a reasonable amount.
Regular expressions are supported in most modern programming languages. Heck, even VBScript is a regular expression engine. It's also good because the various regular expression implementations is not significantly different in their syntax.
Disadvantages:
They do not have much experience with them can be complicated to. Learning regular expressions is not like Perl to Java.
- They are often confusing to analyze.
- The process of data discovery (where data from different web pages you want to get on page crossing) remains to be addressed, and very complex as you can use cookies or similar need.
Benefits:
- Data models typically built example, if you are extracting information from websites about cars already extraction engine, model, and rewarding, it easily to existing data structures has been able to identify (such as inserting data into your database the right places).
- Relatively low long term maintenance.
Disadvantages:
- To work with relative to such an engine is complex.
- Are expensive to build these types of engines.
In cases where data is highly structured (meaning there is clearly marked on the various fields to identify), the regular expression makes more sense to go with a screen scraping application.
Screen scraping software
Benefits:
- The abstract complex things away. Something about regular expressions, HTTP, or cookies without knowing the screen scraping applications can do anything very sophisticated things.
- Setting up the site had to be drastically scaled reduces the amount of time.
- Support for a commercial company. If you run into problems while using a commercial application, screen scraping, chances are that there are support forums and help lines where you can get help.
Disadvantages:
- The learning curve. Each application has its own way to go about things in the screen scraping.
- A possible cost.
- An individual approach.
When the screen scraping applications use this approach to ease of use, price, fitness, and dealing with a wide range of very different scenarios. Chances are however that if you do not mind paying a little bit, you find yourself using one can save a considerable amount of time. If you have a single page, a quick scraping with regular expressions that you can use virtually any language die. More about anything, though maybe, you designed for screen scraping to consider investing in the application.
We currently have a project engaged in extracting the newspaper ads work. About the data in the ads as you can get is. However, we had to find the data processing. we decided to use the screen scraper and it's just great to deal with. The basic process that the various pages of the screen scraper site cross dates then inserted into a database.
Gungun Vghl writes article on Product Description Writing, OCR Data Conversion, Website Data Mining, Web Screen Scraping, Web Data Mining, Web Data Extraction etc.