Literature Review
Machine learning usually refers to the changes in systems that perform tasks associated with artificial intelligence (AI). Such tasks involve recognition, diagnosis, planning, robot control, prediction, etc. The “changes” might be either enhancements to already performing systems or ab initio synthesis of new systems. To be slightly more specific, we show the architecture of a typical AI
“Why should machines have to learn? Why not design machines to perform as desired in the first place?” There are several reasons why machine learning is important. Of course, we have already mentioned that the achievement of learning in machines might help us understand how animals and humans learn. But there are important engineering reasons as well. Some of these are:
• Some tasks cannot be defined well except by example; that is, we might be able to specify input/output pairs but not a concise relationship between inputs and desired outputs. We would like machines to be able to adjust their internal structure to produce correct outputs for a large number of sample inputs and thus suitably constrain their input/output function to approximate the relationship implicit in the examples.
• It is possible that hidden among large piles of data are important relationships and correlations. Machine learning methods can often be used to extract these relationships (data mining).
Machine Learning is, undoubtedly, one of the most exciting subsets of Artificial Intelligence. It completes the task of learning from data with specific inputs to the machine. It’s important to understand what makes Machine Learning work and, thus, how it can be used in the future. The Machine Learning process starts with inputting training data into the selected algorithm. Training data being known or unknown is used to develop the final Machine Learning algorithm. The type of training data input does impact the algorithm, and that concept will be covered further momentarily. New input data is fed into the machine learning algorithm to test whether the algorithm works correctly. The prediction and results are then checked against each other. If the prediction and results don’t match, the algorithm is re-trained multiple times until the data scientist gets the desired outcome. This enables the machine learning algorithm to continually learn on its own and produce the optimal answer, gradually increasing in accuracy over time
We trace the origin and short history of machine learning and its most important milestones. 18th century —statistical methods: Several vital concepts in machine learning derive from probability theory and statistics, and they root back to the 18th century. In 1763, English statistician Thomas Bayes set out a mathematical theorem for probability, which came to be known as Bayes Theorem that a central concept of machine learning. 1950 — The Turing Test: mathematician Alan Turing’s papers in the 1940s were full of ideas on machine intelligence. “Can machines think?”, paving way for automata theory. In 1950, he suggested a test for machine intelligence, later known as the Turing Test, in which a machine is said to be “intelligent” if it could convince a human. 1952 — Game of Checkers: In 1952, researcher Arthur Samuel created an early learning machine, capable of learning to play checkers. It is an annotated guides to learn to distinguish right moves from bad. 1956 — The Dartmouth Workshop: The term ‘artificial intelligence’ was born during the Dartmouth Workshop in 1956, The workshop scientists, including computer scientist John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude Shannon. 1957 — The Perceptron: Noted American psychologist Frank Rosenblatt’s Perceptron was an early attempt to create a neural network with the use of a rotary resistor (potentiometer) driven by an electric motor. The machine could create an output. 1967 — Nearest neighbor algorithm: The Nearest Neighbor (NN) rule in pattern recognition, which enabled article written by T. Cover and P. Hart in 1967. The algorithm gave an idea of solution to traveling sales problem. 1973 — The Light hill report and the AI winter: James Light hill in 1973, presented a very pessimistic forecast in the development of core aspects in AI research”. 1979 — Stanford Cart: The students at Stanford University invented a robot called the Cart, radio-linked to a large mainframe computer, which can navigate obstacles in a room on its own. The invention was state of the art at the time. 1981 — Explanation Based Learning (EBL): Gerald Dejong introduced the concept of Explanation Based Learning (EBL), which analyses data and creates a general rule it can follow by discarding unimportant data. 1985 — Net Talk: Francis Crick Professor Terry Sejnowski invented NetTalk, NETtalk, a program that learns to pronounce written English text by being shown text as input and matching phonetic transcriptions for comparison. This has shed light on human learning. 1986 — Parallel Distributed Processing and neural network models: David Rumelhart and James McClelland published Parallel Distributed Processing, which advanced the use of neural network models for machine learning
1992 — playing backgammon: Researcher Gerald Tesauro created a program based on an artificial neural network, which was capable of playing backgammon with abilities that matched top human players. 1997 — deep Blue: IBM’s Deep Blue became the first computer chess-playing system to beat a reigning world chess champion. Deep Blue used the computing power in the 1990s to perform large-scale searches of potential moves and select the best move. 2006 — Deep Learning: Geoffrey Hinton created the term “deep learning” to explain new algorithms that help computers distinguish objects and text in images and videos. 2010 — Kinect: Microsoft developed the motion-sensing input device named Kinect that can track 20 human characteristics at a rate of 30 times per second. It allowed people to interact with the computer through movements and gestures. 2011 — Watson and Google Brain: IBM’s Watson won a game of the US quiz show Jeopardy against two of its champions. In the same year, Google Brain was developed its deep neural network which could discover and categorize objects in the way a cat does. 2012 — ImageNet Classification and computer vision: The year saw the publication of an influential research paper by Alex Krizhevsky, Geoffrey Hinton, and Ilya Sutskever, describing a model that can dramatically reduce the error rate in image recognition systems. Meanwhile, Google’s X Lab developed a machine learning algorithm capable of autonomously browsing YouTube videos to identify the videos that contain cats. 2014 — Deep Face: Facebook developed a software algorithm Deep Face, which can recognize and verify individuals on photos with an accuracy of a human. 2015 — Amazon Machine Learning: AWS’s Andy Jassy launched their Machine Learning managed services that analyze users’ historical data to look for patterns and deploy predictive models. In the same year, Microsoft created the Distributed Machine Learning Toolkit, which enables the efficient distribution of machine learning problems across multiple computers. 2016 — AlphaGo: AlphaGo, created by researchers at Google Deep Mind to play the ancient Chinese game of Go, won four out of five matches against Lee Sedol, who has been the world’s top Go player for over a decade. 2017 — Libratus and Deepstack: Researchers at Carnegie Mellon
https://www.ijfmr.com/papers/2023/1/1367.pdf
CONCEPT LEARNING AS SEARCH
Concept learning can be viewed as the task of searching through a large space of hypotheses implicitly defined by the hypothesis representation. The goal of this search is to find the hypothesis that best fits the training examples. It is important to note that by selecting a hypothesis representation, the designer of the learning algorithm implicitly defines the space of all hypotheses that the program can ever represent and therefore can ever learn.
Consider, for example, the instances X and hypotheses H in the EnjoySport learning task. Given that the attribute Sky has three possible values, and that AirTemp, Humidity, Wind, Water, and Forecast each have two possible values, the instance space X contains exactly 3 .2 2 .2 2 .2 = 96 distinct instances.
A similar calculation shows that there are 5.4-4 -4 -4.4 = 5 120 syntactically distinct hypotheses within H. Notice, however, that every hypothesis containing one or more “IZI” symbols represents the empty set of instances; that is, it classifies every instance as negative. Therefore, the number of semantically distinct hypotheses is only 1 + (4.3.3.3.3.3) = 973.
Our EnjoySport example is a very simple learning task, with a relatively small, finite hypothesis space. Most practical learning tasks involve much larger, sometimes infinite, hypothesis spaces. If we view learning as a search problem, then it is natural that our study of learning algorithms will exa~the different strategies for searching the hypothesis space. We will be particula ly interested in algorithms capable of efficiently searching very large or infinite hypothesis spaces, to find the hypotheses that best fit the training data.
Lycos (Mauldin & Leavitt 1994), Alta Vista, and similar Web search engines have become essential as tools for locating information on the ever-growing World Wide Web. Underlying these systems are statistical methods for indexing plain text documents. However, the bulk of the Web consists of HyperText Markup Language (HTML) documents, which exhibit two kinds of structure not present in general text documents: 1. They have an internal structure consisting
ning bold and italic text. Others have richer semantic import, such as headlines and anchors, the text segments which serve as hyperlinks to other documents.
nes a set of roles to which text in a document can be assigned. Some of these roles relate to formatting, such as those de
The primary focus of the field since the 1950s has been on text and text documents. Web pages, email, scholarly papers, books, and news stories are just a few of the many examples of documents. All of these documents have some amount of structure, such as the title, author, date, and abstract information associated with the content of papers that appear in scientific journals. The elements of this structure are called attributes, or fields, when referring to database records. The important distinction between a document and a typical database record, such as a bank account record or a flight reservation, is that most of the information in the document is in the form of text, which is relatively unstructured
Enterprise search involves finding the required information in the huge variety of computer files scattered across a corporate intranet. Web pages are certainly a part of that distributed information store, but most information will be found in sources such as email, reports, presentations, spreadsheets, and structured data in corporate databases. Desktop search is the personal version of enterprise search, where the information sources are the files stored on an individual computer, including email messages and web pages that have recently been browsed. Peer-to-peer search involves finding information in networks of nodes or computers without any centralized control. This type of search began as a file sharing tool for music but can be used in any community based on shared interests, or even shared locality in the case of mobile devices. Search and related information retrieval techniques are used for advertising, for intelligence analysis, for scientific discovery, for health care, for customer support, for real estate, and so on. Any application that involves acollection3 of text or other unstructured information will need to organize and search that information
Search based on a user query (sometimes called ad hoc search because the range of possible queries is huge and not prespecified) is not the only text-based task that is studied in information retrieval. Other tasks include filtering, classification, and question answering. Filtering or tracking involves detecting stories of interest based on a person’s interests and providing an alert using email or some other mechanism. Classification or categorization uses a defined set of labels or classes
of typed text segments marked by meta-linguistic tags (markup). HTML
Significance of the Study
MACHINE LEARNING IN SEARCH ENGINES
. Introduction to Search Engines Search Engine[1] is a service that allows internet users to search for contents via the world wide web(www). Search engine is a software program that searches for sites based on the words that you designate as search terms. They look through their own databases of information in order to find what it is that you are looking for. There are mainly three components for search engines. They are:
• web crawler
• database
• search interfaces Web crawlers are also known as spiders or bots.
It is a software component that traverses the web to gather information. All the information on the web are stored in a database. It consists of huge web resources. Search interface acts as an interface between user and the database. It helps the users to search through the database. B Classification of Search Engines There are many search engines on web based on the usage and features, users can use them. Every search engine has many web pages stored on their database but search engines with large number of pages on web are not top search engines. Search engines which will provide accurate information based on requested keyword will be the top search engines. Search engines are classified as follows:
• Crawler based search engines[3]
• Human powered directories
• Meta search engines[4]
• Hybrid search engines
• Specialty search engines
B.1 Crawler based Search Engines Crawler based search engines[3] such as Google create their listings automatically. They crawl or spider the web, then people search through what they have found. If you change your webpages, crawler based search engine will find these changes and that can affect how you are listed.
Three elements in crawler based search engines are:
• Crawler or spider
• index or catalog
• search engine software Crawler or spider visits webpages and reads it and index or catalog is like a giant book containing a copy of every webpage that crawler or spider finds.
If a webpage changes, then this book is updated with a new one.
Human Powered Directories
A human powered directory such as the open directory depends on humans for its listings. In this type of search engine, site owner submits a short description of the site to the directory along with category it is to be listed. Submitted site is then manually reviewed and added in the appropriate category or rejected for listing. Keywords entered in a search box will be matched with the description of the sites. This means the changes made to the content of web pages are not taken into consideration as it is only the description that matters. A good site with good content is more likely to be reviewed for free compared to a site with poor content. B.3 Meta Search Engines Meta search engines[4] gives results based on a combination of results from other search engine databases. It uses complex algorithms and virtual databases. A search engine that queries other search engines and then combines the results that are received from all. In effect, the user is not using just one search engine but a combination of many search engines at once to optimize web searching. For example, Dog pile is a meta search engine.
Directories as secondary mechanism. For example, google may take the description of a webpage from human powered directories and show in the search results. As human powered directories are disappearing, hybrid types are becoming more and more crawler based search engines. But still there are manual filtering of search result happens to remove the copied and spammy sites. When a site is being identified for spammy activities, the website owner needs to take corrective action and resubmit the site to search engines. The experts do manual review of the submitted site before including it again in the search results. In this manner though the crawlers control the processes, the control is manual to monitor and show the search results naturally. B.5 Speciality Search Engines Speciality search engines search a specially created database which is limited to a particular subject. A speciality search engine, sometimes called a topican or vertical search engine, searches a specially-created database limited to a particular subject. Specialty search engines fall into two main categories: service
• subject-specific Specialty service search engines provide services that are often not available from larger general search engines. Subject-specific search engines search a database tailored to a particular subject. Depending on your area of interest and the type of information you are seeking, specialty search engines can provide more relevant results more quickly than a general purpose search engine such as Google or Yahoo.
Specialty search engines are also an excellent source for typical research. Because of this it would be wise to also submit your blog or website to some of the speciality search engines that cater for your niche. C Search Engine Working While you should always create website content geared to your customers rather than search engines, it is important to understand how a search engine works. Most search engines build an index based on crawling, which is the process through which engines like Google, Yahoo and others find new pages to index. Mechanisms known as bots or spiders crawl the web looking for new pages. The bots typically start with a list of website. URL’s determined from previous crawls. When they detects new links on these pages, through tags like HREF and SRC, they add theses to the list of sites to index. Then, search engine use their algorithms to provide you with a ranked list from their index of what pages you should be most interested in based on the search terms you used. Then, the engine will return a list of web results ranked using its specific algorithm. On Google, other elements like personalized and universal results may also change your page ranking. In personalized results, the search engine utilizes additional information it knows about the user to return results that are directly catered to their interests. Universal search results combine video, images and Google news to create a bigger picture result, which can mean greater competition from other websites for the same keywords. Search engine optimization is a set of rules that can be followed by website owners to optimize their websites for search engines and thus improve their search engine ranking. In addition, it is a great way to increase the quality of your website by making it userfriendly, faster and easier to navigate. Steps in search engine optimization are as follows:
• Website analysis
• Client requirements
• Keyword research
• Content writing
• Website optimization
• SEO submission
• Link building
• Reporting
Introduction to Machine Learning
Machine Learning is a branch of artificial intelligence(AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed. Some applications of machine learning are: email spam and filtering, online fraud detection, product recommendations. There are mainly three types of learning. They are as follows:
- supervised learning
• unsupervised learning
• reinforcement learning
E Applications of Machine Learning in Search Engines Machine Learning can be applied in various areas realted to search engines.
They are:
• Pattern Detection
• Identifying new signals
• Custom signals based on specific query
• Image search to understand photos
• Identifying similarities between words in a search query
• Improve ad quality
• Query understanding
• URL/Document understanding
• Search features[2]
• Crawling[1]
• User classification
• Search Ranking
• Synonyms Identification/Query Expansion
• Intent Disambiguation Image search to understand photos: Users can upload photos on google image search and get information about the image, similar looking images, etc. There is a lot of data in the form of images on the internet. Hence, search engines can use machine learning on the huge number of images and power their feature of image searching.
Query Understanding:
Machine Learning is used for understanding the search queries typed by the users. Query classification is one of the problem that is solved by using machine learning. Search engines run different classifiers on the search query. They are: • Navigational search queries • Informational search queries • Transactional search queries URL/Document Understanding: This includes everything that is done to understand a URL(Uniform Resource Locator).
For example, spam detection, page classification, etc. Intent Disambiguation: Consider an example, when you search for eagles, is it eagles the band or Philadelphia eagles or the bird or all of them together. Machine Learning is applied in these types of scenarios. Improve Ad quality: A lot of revenue of the search engines comes from the advertisements that they display on their websites. Suggesting the advertisements that are relevant to the users increases the chance of the user actually purchasing the advertised product or service, which would, in turn be beneficial for the company providing the product or service and the search engine that advertised the product. Both will take money. Machine Learning is used by the search engines to identify the correct target audience for the showing of the various advertisements. Depending on the queries the user asks on the search engine, relevant advertisements are shown to him/her. Identifying the meaning of words based on their usage: The number of words in the English language is constantly growing. Around ten years back, no one knew words and phrases like “selfie”, “it’s lit”, “muggle”, etc. When a word or phrase is very n
There are some who regard data mining as synonymous with machine learning. There is no question that some data mining appropriately uses algorithms from machine learning. Machine-learning practitioners use the data as a training set, to train an algorithm of one of the many types used by machine-learning practitioners, such as Bayes nets, support-vector machines, decision trees, hidden Markov models, and many others. There are situations where using data in this way makes sense. The typical case where machine learning is a good approach is when we have little idea of what we are looking for in the data.
For example, it is rather unclear what it is about movies that makes certain movie-goers like or dislike it. Thus, in answering the “Netflix challenge” to devise an algorithm that predicts the ratings of movies by users, based on a sample of their responses, machine learning algorithms have proved quite successful. We shall discuss a simple form of this type of algorithm in
On the other hand, machine learning has not proved successful in situations where we can describe the goals of the mining more directly. An interesting case in point is the attempt by WhizBang! Labs1 to use machine learning to locate people’s resumes on the Web. It was not able to do better than algorithms designed by hand to look for some of the obvious words and phrases that appear in the typical resume. Since everyone who has looked at or written a resume has a pretty good idea of what resumes contain, there was no mystery about what makes a Web page a resume. Thus, there was no advantage to
In several applications of data mining, we shall be faced with the problem of categorizing documents (sequences of words) by their topic. Typically, topics are identified by finding the special words that characterize documents about that topic. For instance, articles about baseball would tend to have many occurrences of words like “ball,” “bat,” “pitch,”, “run,” and so on. Once we have classified documents to determine they are about baseball, it is not hard to notice that words such as these appear unusually frequently. However, until we have made the classification, it is not possible to identify these words as characteristic.
Importance of Words in Documents In several applications of data mining, we shall be faced with the problem of categorizing documents (sequences of words) by their topic. Typically, topics are identified by finding the special words that characterize documents about that topic. For instance, articles about baseball would tend to have many occurrences of words like “ball,” “bat,” “pitch,”, “run,” and so on. Once we have classified documents to determine they are about baseball, it is not hard to notice that words such as these appear unusually frequently. However, until we have made the classification, it is not possible to identify these words as characteristic. Thus, classification often starts by looking at documents, and finding the significant words in those documents. Our first guess might be that the words appearing most frequently in a document are the most significant.
However, that intuition is exactly opposite of the truth. The most frequent words will most surely be the common words such as “the” or “and,” which help build ideas but do not carry any significance themselves. In fact, the several hundred most common words in English (called stop words) are often removed from documents before any attempt to classify them. In fact, the indicators of the topic are relatively rare words. However, not all rare words are equally useful as indicators.
There are certain words, for example “notwithstanding” or “albeit,” that appear rarely in a collection of documents, yet do not tell us anything useful. On the other hand, a word like “chukker” is probably equally rare, but tips us off that the document is about the sport of polo. The difference between rare words that tell us something and those that do not has to do with the concentration of the useful words in just a few documents. That is, the presence of a word like “albeit” in a document does not make it terribly more likely that it will appear multiple times. However, if an article mentions “chukker” once, it is likely to tell us what happened in the “first chukker,” then the “second chukker,” and so on. That is, the word is likely to be repeated if it appears at all. The formal measure of how concentrated into relatively few documents are the occurrences of a given word is called TF.IDF (Term Frequency times Inverse Document Frequency). It is normally computed as follows. Suppose we have a collection of N documents. Define fij to be the frequency (number of occurrences) of term (word) i in document j. Then, define the term frequency TFij to be: TFij = fij maxk fkj
That is, the term frequency of term i in document j is fij normalized by dividing it by the maximum number of occurrences of any term (perhaps excluding stop words) in the same document. Thus, the most frequent term in document j gets a TF of 1, and other terms get fractions as their term frequency for this document.
Thus, classification often starts by looking at documents and finding the significant words in those documents. Our first guess might be that the words appearing most frequently in a document are the most significant. However, that intuition is exactly opposite of the truth. The most frequent words will most surely be the common words such as “the” or “and,” which help build ideas but do not carry any significance themselves. In fact, the several hundred most common words in English (called stop words) are often removed from documents before any attempt to classify them. In fact, the indicators of the topic are relatively rare words. However, not all rare words are equally useful as indicators. There are certain words, for example “notwithstanding” or “albeit,” that appear rarely in a collection of documents, yet do not tell us anything useful. On the other hand, a word like “chukker” is probably equally rare, but tips us off that the document is about the sport of polo. The difference between rare words that tell us something and those that do not has to do with the concentration of the useful words in just a few documents.
That is, the presence of a word like “albeit” in a document does not make it terribly more likely that it will appear multiple times. However, if an article mentions “chukker” once, it is likely to tell us what happened in the “first chukker,” then the “second chukker,” and so on. That is, the word is likely to be repeated if it appears at all. The formal measure of how concentrated into relatively few documents are the occurrences of a given word is called TF.IDF (Term Frequency times Inverse Document Frequency). It is normally computed as follows. Suppose we have a collection of N documents. Define fij to be the frequency (number of occurrences) of term (word) i in document j.
Then, define the term frequency TFij to be: TFij = fij maxk fkj That is, the term frequency of term i in document j is fij normalized by dividing it by the maximum number of occurrences of any term (perhaps excluding stop words) in the same document. Thus, the most frequent term in document j gets a TF of 1, and other terms get fractions as their term frequency for this document. The IDF for a term is defined as follows. Suppose term i appears in ni
1.3.3
Indexes
An index is a data structure that makes it efficient to retrieve objects given the value of one or more elements of those objects. The most common situation is one where the objects are records, and the index is on one of the fields of that record. Given a value v for that field, the index lets us retrieve all the records with value v in that field. For example, we could have a file of (name, address, phone) triples, and an index on the phone field. Given a phone number, the index allows us to find quickly the record or records with that phone number.
Until the mid-1990s, metadata was a term used primarily by communities involved with the management and interoperability of geospatial data and with data management and systems design and maintenance in general. For these communities, metadata referred to a suite of industry or disciplinary standards as well as additional internal and external documentation and other data necessary for the identification, representation, interoperability, technical management, performance, and use of data contained in an information system.
Background: Text Mining and Information Extraction “Text mining” is used to describe the application of data mining techniques to automated discovery of useful or interesting knowledge from unstructured text [20]. Several techniques have been proposed for text mining including conceptual structure, association rule mining, episode rule mining, decision trees, and rule induction methods. In addition, Information Retrieval (IR) techniques have widely used the “bag-of-words” model [2] for tasks such as document matching, ranking, and clustering. The related task of information extraction aims to find specific data in natural-language text. DARPA’s Message Understanding Conferences (MUC) have concentrated on IE by evaluating the performance of participating IE systems based on blind test sets of text documents [13]. The data to be extracted is typically given by a template which specifies a list of slots to be filled with substrings taken from the document. Figure 2 shows a (shortened) document and its filled template for an information extraction task in the job-posting domain. This template includes slots that are filled by strings taken directly from the document. Several slots may have multiple fillers for the job-posting domain as in programming languages, platforms, applications, and areas. We have developed machine learning techniques to automatically construct information extractors for job postings, such as those listed in the USENET newsgroup misc.jobs.offered [6]. By extracting information from a corpus of such textual job postings, a structured, searchable database of jobs can be automatically constructed; thus making the data in online text more easily accessible. IE has been shown to be useful in a variety of other applications, e.g. seminar announcements, restaurant guides, university web pages, apartment rental ads, and news articles on corporate acquisitions [5, 9, 23].
https://www.cs.utexas.edu/~ml/papers/discotex-melm-03.pdf
Text mining is a burgeoning new field that attempts to glean meaningful information from natural language text. It may be loosely characterized as the process of analyzing text to extract information that is useful for particular purposes. Compared with the kind of data stored in databases, text is unstructured, amorphous, and difficult to deal with algorithmically. Nevertheless, in modern culture, text is the most common vehicle for the formal exchange of information. The field of text mining usually deals with texts whose function is the communication of factual information or opinions, and the motivation for trying to extract information from such text automatically is compelling—even if success is only partial. Four years ago, Hearst [Hearst, 1999] wrote that the nascent field of “text data mining” had “a name and a fair amount of hype, but as yet almost no practitioners.” It seems that even the name is unclear: the phrase “text mining” appears 17 times as often as “text data mining” on the Web, according to a popular search engine (and “data mining” occurs 500 times as often). Moreover, the meaning of either phrase is by no means clear: Hearst defines data mining, information access, and corpus-based computational linguistics and discusses the relationship of these to text data mining—but does not define that term. The literature on data mining is far more extensive, and also more focused: there are numerous textbooks and critical reviews that trace its development from roots in machine learning and statistics. Text mining emerged at an unfortunate time in history. Data mining was able to ride the back of the high technology extravaganza throughout the 1990s, and became firmly established as a widely-used practical technology—though the dot com crash may have hit it harder than other areas [Franklin, 2002]. Text mining, in contrast, emerged just before the market crash—the first workshops were held at the International Machine Learning Conference in July 1999 and the International Joint Conference on Artificial Intelligence in August 1999—and missed the opportunity to gain a solid foothold during the boom years. The phrase “text mining” is generally used to denote any system that analyzes large quantities of natural language text and detects lexical or linguistic usage patterns in an attempt to extract probably useful (although only probably correct) information [Sebastiani, 2002]. In discussing a topic that lacks a generally accepted definition in a practical Handbook such as this, I have chosen to cast the net widely and take a liberal viewpoint of what should be included, rather than attempting a clear-cut characterization that will inevitably restrict the scope of what is covered. The remainder of this section discusses the relationship between text mining and data mining, and between text mining and natural language processing, to air important issues concerning the meaning of the term. The article’s major section follows: an introduction to the great variety of tasks that involve mining plain text. We then examine the additional leverage that can be obtained when mining semi-structured text such as pages of the World-Wide Web, which opens up a range
Text mining and natural language processing Text mining appears to embrace the whole of automatic natural language processing and, arguably, far more besides—for example, analysis of linkage structures such as citations in the academic literature and hyperlinks in the Web literature, both useful sources of information that lie outside the traditional domain of natural language processing. But, in fact, most text mining efforts consciously shun the deeper, cognitive, aspects of classic natural language processing in favor of shallower techniques more akin to those used in practical information retrieval. The reason is best understood in the context of the historical development of the subject of natural language processing. The field’s roots lie in automatic translation projects in the late 1940s and early 1950s, whose aficionados assumed that strategies based on word-for-word translation would provide decent and useful rough translations that could easily be honed into something more accurate using techniques based on elementary syntactic analysis. But the sole outcome of these 3 high-profile, heavily-funded projects was the sobering realization that natural language, even at an illiterate child’s level, is an astonishingly sophisticated medium that does not succumb to simplistic techniques. It depends crucially on what we regard as “common-sense” knowledge, which despite—or, more likely, because of—its everyday nature is exceptionally hard to encode and utilize in algorithmic form [Lenat, 1995]. As a result of these embarrassing and much-publicized failures, researchers withdrew into “toy worlds”—notably the “blocks world” of geometric objects, shapes, colors, and stacking operations—whose semantics are clear and possible to encode explicitly. But it gradually became apparent that success in toy worlds, though initially impressive, does not translate into success on realistic pieces of text. Toy-world techniques deal well with artificially-constructed sentences of what one might call the “Dick and Jane” variety after the well-known series of eponymous children’s stories. But they fail dismally when confronted with real text, whether painstakingly constructed and edited (like this article) or produced under real-time constraints (like informal conversation). Meanwhile, researchers in other areas simply had to deal with real text, with all its vagaries, idiosyncrasies, and errors. Compression schemes, for example, must work well with all documents, whatever their contents, and avoid catastrophic failure even when processing outrageously deviant files (such as binary files, or completely random input). Information retrieval systems must index documents of all types and allow them to be located effectively whatever their subject matter or linguistic correctness. Key-phrase extraction and text summarization algorithms have to do a decent job on any text file.
Practical, working systems in these areas are topic independent, and most are language-independent. They operate by treating the input as though it were data, not language. Text mining is an outgrowth of this “real text” mindset. Accepting that it is probably not much, what can be done with unrestricted input? Can the ability to process huge amounts of text compensate for relatively simple techniques? Natural language processing, dominated in its infancy by unrealistic ambitions and swinging in childhood to the other extreme of unrealistically artificial worlds and trivial amounts of text, has matured and now embraces both viewpoints: relatively shallow processing of unrestricted text and relatively deep processing of domain-specific material. It is interesting that data mining also evolved out of a history of difficult relations between disciplines, in this case machine learning—rooted in experimental computer science, with ad hoc evaluation methodologies—and statistics—well-grounded theoretically, but based on a tradition of testing explicitly-stated hypotheses rather than seeking new information. Early machine learning researchers knew or cared little of statistics; early researchers on structured statistical hypotheses remained ignorant of parallel work in machine learning. The result was that similar techniques (for example, decision-tree building and nearest-neighbor learners) arose in parallel from the two disciplines, and only later did a balanced rapprochement emerge.
Document retrieval
Given a corpus of documents and a user’s information need expressed as some sort of query, document retrieval is the task of identifying and returning the most relevant documents. Traditional libraries provide catalogues (whether physical card catalogues or computerized information systems) that allow users to identify documents based on surrogates consisting of metadata—salient features of the document such as author, title, subject classification, subject headings, keywords. Metadata is a kind of highly structured (and therefore actionable) document summary, and successful methodologies have been developed for manually extracting metadata and for identifying relevant documents based on it, methodologies that are widely taught in library school (e.g. [Mann, 1993]). Automatic extraction of metadata (e.g. subjects, language, author, key-phrases; see below) is a prime application of text mining techniques. However, contemporary automatic document retrieval techniques bypass the metadata creation stage and work on the full text of the documents directly [Salton and McGill, 1983].
The basic idea is to index every individual word in the document collection. Effectively, documents are represented as a “bag of words”—that is, the set of words that they contain, along with a count of how often each one appears in the document.
Despite the fact that this representation discards the sequential information given by the word order, it underlies many remarkably effective and popular document retrieval techniques. There are some practical problems: how to define a “word,” what to do with numbers; these are invariably solved by simple ad hoc heuristics. Many practical systems discard common words or “stop words”, primarily for efficiency reasons, although suitable compression techniques obviate the need for this [Witten et al., 1999]. A query is expressed as a set, or perhaps a Boolean combination, of words and phrases, and the index is consulted for each word in the query to determine which documents satisfy the query. A well-developed technology of relevance ranking allows the salience of each term to be assessed relative to the document collection as a whole, and also relative to each document that contains it. These measures are combined to give an overall ranking of the relevance of each document to the query, and documents are presented in relevance order. Web search engines are no doubt the most widely-used of document retrieval systems. However, search queries are typically restricted to just a few words or phrases—usually one or two.
In contrast, queries made by professionals to advanced document retrieval systems are often far more complex and specific.
Information retrieval Information retrieval might be regarded as an extension to document retrieval where the documents that are returned are processed to condense or extract the particular information sought by the user. Thus document retrieval could be followed by a text summarization stage that focuses on the query posed by the user, or an information extraction stage using techniques described below. In practice, however, standard textbooks (e.g. [Baeza-Yates and Ribiero-Neto, 1999]) use the term simply for plain document retrieval. Of course, the granularity of documents may be adjusted so that each individual subsection or paragraph comprises a unit in its own right, in an attempt to focus results on individual nuggets of information rather than lengthy documents.