萨师煊国际大数据分析与研究中心.ppt
《萨师煊国际大数据分析与研究中心.ppt》由会员分享,可在线阅读,更多相关《萨师煊国际大数据分析与研究中心.ppt(40页珍藏版)》请在三一文库上搜索。
1、Weiyi Meng 孟卫一 Department of Computer Science State University of New York at Binghamton July 9, 2012,Large-Scale Distributed Information Retrieval on the Web,萨师煊国际大数据分析与研究中心 Summer Research Camp Seminar,About SUNY Binghamton,Founded in 1946 after WWII. Located in Binghamton a city in Southern Tier
2、of New York State About 15,000 students (3,000 grad students) IBM was founded in Binghamton One of the 4 University Centers of SUNY system: SUNY at Stony Brook, SUNY at Buffalo, SUNY at Albany. For more information, see http:/www2.binghamton.edu/features/premier/index.html,What is Information Retrie
3、val?,Information retrieval (IR) is a computer science discipline for finding unstructured data (usually text documents) that satisfy an information need from within large collections that are stored on computers. In this seminar, we are going to extend this definition to include both unstructured an
4、d structured data.,What is Distributed Information Retrieval (DIR)?,It is a special branch of information retrieval where the data of the IR system are stored in multiple distributed locations/collections. In the Web environment, DIR deals with data that are distributed across many websites or web s
5、ervers. Related terms for DIR: metasearch engine, federated search, web DB integration system,The Scale How Large?,It can be as large as the number of data sources on the Web. A 2007 survey (Madhavan et al. 2007) indicates there were about 50 million searchable Web data sources in 2007. 25 million f
6、or un- or less structured data (web pages, weibo, ) 25 million for structured data (web databases),Where do Web data reside?,Iceberg Structure: A small fraction is on the Surface Web with mostly static web pages that are crawlable by following hyperlinks. Publicly indexable portion: 40-60 billion pa
7、ges Most are in the Deep Web with both structured data and less structured text documents hidden behind numerous search interfaces. About 1 trillion pages/records,Two paradigms to provide integrated access to Web data,Crawling-based: Gather Web data from various Web servers and/or search engines and
8、 build a search index for the gathered data. Surface Web crawling Deep Web crawling Metasearching-based (DIR-based): Integrate existing search engines into federated systems. Metasearching text documents Metasearching structured data by domain,Advantages of each approach,Crawling-based: Complete con
9、trol on crawled data: Can add metadata Can link data from different sources in advance Can create an archive gradually Complete control on retrieving techniques and ranking functions Fast response time,Metasearching-based: Capabilities of search engines can be leveraged Natural clustering of the dat
10、a by individual search engines can be utilized Three-level query evaluation process (SE selection, SE retrieval, result merging) can lead to better effectiveness More likely to obtain fresher results,Disadvantages of each approach,Crawling-based: Deep Web crawling difficult Often incomplete Many sit
11、es not crawlable Lose semantics/structure of the data Cannot leverage search engines capabilities Crawling delay leads to less up-to-date results Copyright and privacy issues,Metasearching-based: Performance depends on the quality of used search engines May cause search engines to crash Access could
12、 be blocked by search engines No direct control of the data Slower response time,Conclusions?,Both technologies (crawling-based and metasearching-based) have unique values and they should co-exist. They actually complement each other! Question: Is there an effective way to combine both technologies
13、into a single platform?,Our seminar will focus on the metasearching (DIR)-based approach.,Two types of metasearching systems,Because structured and unstructured data have very different characteristics, they are often handled separately with different technologies. Metasearching systems for text doc
14、uments (metasearch engines or DIR systems). Metasearching systems for structured data, each for a given domain (Web database integration systems). We will first introduce large-scale metasearch engines and then introduce large-scale Web database integration systems. Due to limited time, we will focu
15、s on challenges and remaining challenges, not on current solutions.,Large-Scale Metasearch Engines (MSE),user user interface query dispatcher result merger search search search engine 1 engine 2 engine n . . . . . . text text text source 1 source 2 source n,query,result,A simple MSE architecture,Wha
16、t is a large-scale MSE?,A large-scale metasearch engine needs to satisfy ALL of the following requirements: It is a metasearch engine. It is connected to a large number of (thousands or more) component search engines. The component search engines are special-purpose search engines Covering a specifi
17、c domain: news, sports, medicine, Covering a specific organization: RenDa, IBM, ACM, Why the third requirement? To retain the advantages on freshness and searching the deep Web.,Technical challenges with large-scale MSE,Scalable and accurate search engine selection Most search engines are useless fo
18、r a given user query. Best 10 results, 10,000 search engines at least 9990 useless. Using useless search engines is bad Unnecessary network traffic Waste resources of local search engines Incur higher cost at the metasearch engine Lead to poor effectiveness How to identify the most appropriate searc
19、h engines for any given query accurately and in a timely manner? How to summarize a search engine content (representative)? How to collect the representative? How to use the representatives to perform selection?,Technical challenges (cont.),Automatic search engine inclusion into metasearch engine Au
20、tomatic connection to search engines (automatic connection wrapper generation) Submit queries and receive result pages via program Automatic search result records (SRR) extraction (automatic extraction wrapper generation) Automatic wrapper maintenance Search engines may change the connection paramet
21、ers and and result presentation any time,Technical challenges (cont.),Effective and efficient result merging Autonomous component search engines likely employ different matching techniques between queries and documents (index techniques, weighting schemes, similarity functions, link-based ranking, e
22、tc) Local scores and ranks are generally not comparable How to re-rank the results returned from different search engines into a single ranked list such that high effectiveness can be achieved in a speedy manner?,Large-scale MSE architecture, ,Search Engine m,Search Engine Selector,Query Dispatcher,
23、Result Merger,Result Collector and Extractor,Search Engine 1,Search Engine Representatives,User query,World Wide Web,Web,Search Engine Discovery,SE List,SE Incorporation,Automatic connection and result extraction,Metasearch Engine Construction Module,Query Processing Module.,Result,Search engine Rep
24、resentatives Generation,Two Recent Books (Monographs),W. Meng and C. Yu. Advanced Metasearch Engine Technology. Morgan & Claypool Publishers, December 2010. http:/ Table of content: Introduction Metasearch engine architecture Search engine selection Search engine incorporation Result merging Summary
- 配套讲稿:
如PPT文件的首页显示word图标,表示该PPT已包含配套word讲稿。双击word图标可打开word文档。
- 特殊限制:
部分文档作品中含有的国旗、国徽等图片,仅作为作品整体效果示例展示,禁止商用。设计者仅对作品中独创性部分享有著作权。
- 关 键 词:
- 萨师煊 国际 数据 分析 研究中心
链接地址:https://www.31doc.com/p-3497232.html