Download as PDF, PPTX
News Article Summarization
- 1. News Article Summarization Ryan Endacottand Krit Pattamadit
- 2. Project Goal ● Toprovide a concise and informative summary of a given news article ● Reduce information overload ● Quickly determine which articles are worth reading
- 3. Corpora ● ● ● ● Brown Corpus newsarticles New York Times articles Quora answers Can be used on any text
- 4. Summarization ● Extraction vsAbstraction ● Steps: ○ Preprocess and tokenize article ○ Transform into intermediate representation for analysis ○ Generate summary analysis
- 5. Sentence Extraction ● Analyzeand classify the importance of each sentence in order to form a summary with existing sentences ● Author already conveyed main ideas, we just have to pick them out ● Abstraction is hard and hasn’t given good results in the past
- 6. Preprocessing and Interpretation ●Goal: tokenize input into sentences and words for evaluation ● NLTK sent_tokenizer and word_tokenizer
- 7. How to choosesentences ● Goal: Choose the most important n sentences to be the summary ● Number of Proper Nouns ● Distance from start of article ● TextRank
- 8. TextRank ● Unsupervised graph-basedranking algorithm for text ● Based on Google PageRank ● Takes into account: ○ Word frequency ○ Keywords ○ Relationship between sentences
- 9. TextRank continued ● ● ● ● Connected, undirectedgraph of the text Each sentence is a vertex Edges are similarity between sentences Weighted by word occurrence and frequency
- 10. Results ● Simple summariesof main article ideas ● Large reduction in quantity of text while conveying main points and ideas ● Examples…
- 11. Beer Running I ranand drank beer every day for 3 years. It started the streak on my 29th birthday and my goal was to run at least one mile and drink at least one beer per day for a year. I wanted to do something big for the final year of my 20s. ● Tim Cigelske, Beer Runner ● http://qr.ae/Gk4bO ● 415 words -> 52 words
- 12. Being CEO ofa Startup The exponential productivity from great people will always amaze you. You will not settle for things anymore because you will see what is possible when you hold out for the best and push to find people that are the best. You'll become addicted to finding the hardest challenges because there's a direct relationship between how difficult something is and the euphoria of a feeling when you do the impossible. ● Paul DeJoe, founder of Ecquire. com ● http://qr.ae/GkvvP ● 1198 words -> 71 words
- 13. Biden on ChinaAir Dispute BEIJING --- Chinese leaders pushed back at visiting Vice President Joseph R. Biden Jr. on Wednesday over what they assert is their right to control a wide swath of airspace in the bitterly contested East China Sea. But the Chinese also indicated that they had not decided how aggressively to enforce their so-called air defense identification zone, which has ignited tensions with Japan. Shuttling from one feuding neighbor to the other, Mr. Biden arrived here from Tokyo to urge China’s president, Xi Jinping, to show restraint in the zone, which Mr. Biden said the United States regarded as illegitimate and a provocation. ● New York Times ● http://www.nytimes. com/2013/12/05/world/a sia/biden-arrives-inchina-seeking-restraintfrom-beijing.html? smid=pl-share ● 1090 words -> 102 words
- 14. Problems & FurtherDevelopment ● Didn’t complete web crawler, currently just takes raw text ● Works best on articles with one central idea ● Possible improvements include: ○ ○ ○ ○ Adding title to the graph (with extra weight?) Word stemming or lemmatization Sentence position Sentence length
- 15.
- 16. Thanks! Questions? Comments? Ideasfor improvement?