c++ - store a high amount of HTML files -


I am working on an academic project (a search engine ), its main work search The engines are: 

  1 / - crawling 2 / -storing 3 / -deption 4 / -pap ranking   
 All sites that crawl my search engine locally Which means that this is a  intranet search engine .  
 After storing files found by the crawler, these files need service quickly for caching purpose.  
  So I wonder what is the fastest way to store and retrieve these files?   
 The first idea to use IPT or SSH, but these protocols are connections based protocols, time to connect, search the file and make it long.  
 I have already read about the anatomy of Google, I have seen that they use a data repository, I want to do but I do not know how.  
 Notes: I am using Linux / Debian, and the search engine back-end has been coded using C / C ++. Storing personal files is very easy -  wget -       http://www.example.com  
 Be careful of the pages that are generated by the code (crawlable) of example.com, where you can access the content (or from where) The content is different depending on it.  
 One more thing to consider is that you may not really want to store all the pages manually, but actually move on to the site that actually include the pages - in this way, You only need to make a reference cache, which word is included, not on the whole page. Since many pages will contain very repetitive content, you only store unique words in your database and a list of pages that contain that word (if you filter the words that occur on almost every page, like "if You can reduce the amount of data that you need to store, calculate the number of each word on each page, and then compare it, and then, "", "and", "this", "to", "do", etc. Different pages, to find meaningless pages for searching.

Comments

Post a Comment

Popular posts from this blog

c# - ASP.NET MVC - Attaching an entity of type 'MODELNAME' failed because another entity of the same type already has the same primary key value -

In brief, the wrapper module is inserted during posting and the status of an entry is changed to 'modified' . Before changing the state, the state is set to 'separate', but calling attachments () causes the same error to be thrown away. I am using EF 6. Model / wrapper class public class avi model {public A is a {get; Set; } Public listing & lt; B & gt; B {received; Set; } Public cc {receipt; Set; }} Administrator edit public functioning (int? id) {if (id == null) {new HttpStatusCodeResult (HttpStatusCode.BadRequest); } If (! Conveyor Access (id.Value)) new HTTPTitus code result (HTTP status code. Forbidden); Var aViewModel = new AViewModel (); aViewModel.A = db.As.Find (id); If (aViewModel.Receipt == zero) {return HttpNotFound (); } aViewModel.b = db.Bs.Where (x => x.aid == id.Value) .Oolist (); AViewModel.Vendor = db.Cs.Where (x => x.cid == aViewModel.a.cID) .FirstOrDefault (); See Return (aViewModel); } [HTPFost] [Valid AntitiferousTeacon...
Read more

jasper reports - How to center align barcode using jasperreports and barcode4j -

I use iReport 5.5.0 . How to align a barcode component to the center The relevant piece of my jrxml is below & Lt; JR: Code 128 xmlns: JR = "http://jasperreports.sourceforge.net/jasperreports/components" xsi: schemaLocation = "http://jasperreports.sourceforge.net/jasperreports/components http: //jasperreports.sourceforge Purge / xsd / components.xsd "textPosition =" bottom "& gt; & Lt; Jr: codeExpression & gt; & Lt; [CDATA [$ F {number}]] & gt; & Lt; / Jr: codeExpression & gt; & Lt; / Jr: Code 128 & gt; & Lt; / componentElement & gt; "Text" itemprop = "text"> Create a style after this, it is called "barcode", like: & lt; Style name = "barcode" fontName = "Helvetica" fONTSIZE = "10" hAlign = "center" valign = "center" /> and set the property of your barcode element to "barcode".
Read more

django - CommandError: You must set settings.ALLOWED_HOSTS if DEBUG is False -

I use Django 1.6.5 with settings in my program txsite: DEBUG = true I set the lie to debug, but when I get the runner , I get the following error: CommandError: You have to set the settings. OLOWOWHESTS If DEBUG is False I ALLOWED_HOSTS = ['127.0.0.1', 'localhost'] but I get the same error. Does anyone know this address? Try ALLOWED_HOSTS = ['*'] Less secure If you are not on a firewall or public LAN, but what I use and it works. Edit: Interestingly enough, I have to add some of my 1.8 projects, even if DEBUG = True . Why so very uncertain
Read more