Diseño y construcción de una plataforma declasificación y calificación de post para una red de blogs basada en textmining ...
Objetivo General    El objetivo general del trabajo es el apoyo al procesamiento manual degrandes volúmenes de publicacion...
Objetivos Específicos1. Entender a fondo la problemática y el contexto de la empresa junto   con los conocimientos necesar...
Objetivos Específicos5. Diseñar e implementar un prototipo que permita al usuario ingresar   información en forma adecuada...
El Problema
betazeta
7,5 millones  Visitas Mensuales
User Generated Content
Volumen
Google Categories
Spam
Filtro de contenido
Filtro de contenidoCategorizar
Background  Teórico
Data Mining
Limpieza del texto
Stemming
Stop Words
Diccionario
Latent Dirichlet Allocation
LDALatent Dirichlet Allocation
Lorem Ipsum is simply dummy text of the printing and typesetting industry.Lorem Ipsum has been the industrys standard dumm...
Lorem Ipsum is simply dummy text of the printing and typesetting industry.A    Lorem Ipsum has been the industrys standard...
Lorem Ipsum is simply dummy text of the printing and typesetting industry.A    Lorem Ipsum has been the industrys standard...
Lorem Ipsum is simply dummy text of the printing and typesetting industry.A    Lorem Ipsum has been the industrys standard...
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem IpsumA    has been the industrys standard...
A B C
ALDA      B      C
Descripción de la  plataforma
Entrenamiento
EntrenamientoData Histórica
EntrenamientoData Histórica   Limpieza
EntrenamientoData Histórica   Limpieza   Entrenamiento
EntrenamientoData Histórica   Limpieza   Entrenamiento  Scrapping
EntrenamientoData Histórica   Limpieza       Entrenamiento                 Stop Words                 Frecuencia 1        ...
EntrenamientoData Histórica   Limpieza   Entrenamiento                                  LDA                            Alm...
Categorización
Categorización Texto Plano
Categorización Texto Plano     Limpieza
Categorización Texto Plano     Limpieza   Clasificación
Categorización Texto Plano     Limpieza     Clasificación                 Stop Words
Categorización Texto Plano     Limpieza        Clasificación                 Stop Words         Modelo LDA                ...
Python
PythonDjango
Python         DjangoModelo Vista Controlador
Python          DjangoModelo Vista Controlador      MySql
Python           Django  Modelo Vista Controlador        MySqlWeb Service
Resultados
Entrenamiento4.000       Post
1000 FayerWayer 1000 WayerLess   1000 Belelu   1000 Ferplei
5   Temáticas detectadas
chile colo equipo copa partido barcelona    futbol universidad jugador partidos  seleccion ex jugadores alexis goles clubt...
93%   Ferplei
Validación350     Post
FayerWayer y Wayerless             Suma   Frecuencia   TFIDF   NB    NBMPrecission   88%      85%        83%     91%   92%...
Ferplei             Suma   Frecuencia   TFIDF   NB     NBMPrecission   96%      94%        91%     100%   100%Recall      ...
Entrenamiento14.400           Post
Validación1.600         Post
Video Juegos            Futbol Musica, Fiestas y Panoramas        Telefonia Movil         Automoviles      Pareja y Vida S...
Complejidad
Trabajo Futuro
Mejorar limpieza del texto     Sistema multimodelo       Predicción de BlogMejoras del modelo en el tiempo     Incorporar ...
Muchas Gracias
Presentacion tema memoria v1
Presentacion tema memoria v1
Presentacion tema memoria v1
Presentacion tema memoria v1
Presentacion tema memoria v1
Presentacion tema memoria v1
Presentacion tema memoria v1
Presentacion tema memoria v1
Presentacion tema memoria v1
Presentacion tema memoria v1
Upcoming SlideShare
Loading in …5
×

Presentacion tema memoria v1

602
-1

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
602
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Presentacion tema memoria v1

  1. 1. Diseño y construcción de una plataforma declasificación y calificación de post para una red de blogs basada en textmining para Betazeta Networks S.A. Camilo López A.
  2. 2. Objetivo General El objetivo general del trabajo es el apoyo al procesamiento manual degrandes volúmenes de publicaciones en la red de blogs de Betazeta mediante eldiseño e implementación de un prototipo para la categorización automática de estos datos utilizando text mining.
  3. 3. Objetivos Específicos1. Entender a fondo la problemática y el contexto de la empresa junto con los conocimientos necesarios respecto a text mining, modelos y metodologías necesarias.2. Selección de los datos históricos, las consultas sobre éstos y los modelos que permitan realizar predicciones exitosas de categorización.3. Establecer métodos y métricas para la evaluación de la solución propuesta.4. Utilizando el conocimiento adquirido en los objetivos anteriores, diseñar el proceso de categorización automático de posts.
  4. 4. Objetivos Específicos5. Diseñar e implementar un prototipo que permita al usuario ingresar información en forma adecuada para su análisis y a la empresa procesarla, filtrarla y publicarla en base a criterios del negocio.6. Implementación de la metodología de evaluación.
  5. 5. El Problema
  6. 6. betazeta
  7. 7. 7,5 millones Visitas Mensuales
  8. 8. User Generated Content
  9. 9. Volumen
  10. 10. Google Categories
  11. 11. Spam
  12. 12. Filtro de contenido
  13. 13. Filtro de contenidoCategorizar
  14. 14. Background Teórico
  15. 15. Data Mining
  16. 16. Limpieza del texto
  17. 17. Stemming
  18. 18. Stop Words
  19. 19. Diccionario
  20. 20. Latent Dirichlet Allocation
  21. 21. LDALatent Dirichlet Allocation
  22. 22. Lorem Ipsum is simply dummy text of the printing and typesetting industry.Lorem Ipsum has been the industrys standard dummy text ever since the 1500s,when an unknown printer took a galley of type and scrambled it to make a typespecimen book. It has survived not only five centuries, but also the leap intoelectronic typesetting, remaining essentially unchanged. It was popularised inthe 1960s with the release of Letraset sheets containing Lorem Ipsum passages,and more recently with desktop publishing software like Aldus PageMakerincluding versions of Lorem Ipsum.It is a long established fact that a reader will be distracted by the readablecontent of a page when looking at its layout. The point of using Lorem Ipsum isthat it has a more-or-less normal distribution of letters, as opposed to usingContent here, content here, making it look like readable English. Many desktoppublishing packages and web page editors now use Lorem Ipsum as their defaultmodel text, and a search for lorem ipsum will uncover many web sites still intheir infancy. Various versions have evolved over the years, sometimes byaccident, sometimes on purpose (injected humour and the like).Contrary to popular belief, Lorem Ipsum is not simply random text. It has rootsin a piece of classical Latin literature from 45 BC, making it over 2000 years old.Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia,looked up one of the more obscure Latin words, consectetur, from a LoremIpsum passage, and going through the cites of the word in classical literature,discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good andEvil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics,very popular during the Renaissance. The first line of Lorem Ipsum, "Loremipsum dolor sit amet..", comes from a line in section 1.10.32.The standard chunk of Lorem Ipsum used since the 1500s is reproduced belowfor those interested. Sections 1.10.32 and 1.10.33 from "de Finibus Bonorum etMalorum" by Cicero are also reproduced in their exact original form,accompanied by English versions from the 1914 translation by H. Rackham.There are many variations of passages of Lorem Ipsum available, but themajority have suffered alteration in some form, by injected anythingembarrassing hidden in the middle of text. All the Lorem Ipsum generators onthe Internet tend to repeat predefined chunks as necessary, making this the firsttrue generator on the Internet. It uses a dictionary of over 200 Latin words,combined with a handful of model sentence structures, to generate LoremIpsum which looks reasonable. The generated Lorem Ipsum is therefore alwaysfree from repetition, injected humour, or non-characteristic words etc.
  23. 23. Lorem Ipsum is simply dummy text of the printing and typesetting industry.A Lorem Ipsum has been the industrys standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum. It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using Content here, content here, making it look like readable English. Many desktop publishing packages and web page editors now use Lorem Ipsum as their default model text, and a search for lorem ipsum will uncover many web sites still in their infancy. Various versions have evolved over the years, sometimes by accident, sometimes on purpose (injected humour and the like). Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32 and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good and Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics, very popular during the Renaissance. The first line of Lorem Ipsum, "Lorem ipsum dolor sit amet..", comes from a line in section 1.10.32. The standard chunk of Lorem Ipsum used since the 1500s is reproduced below for those interested. Sections 1.10.32 and 1.10.33 from "de Finibus Bonorum et Malorum" by Cicero are also reproduced in their exact original form, accompanied by English versions from the 1914 translation by H. Rackham. There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected anything embarrassing hidden in the middle of text. All the Lorem Ipsum generators on the Internet tend to repeat predefined chunks as necessary, making this the first true generator on the Internet. It uses a dictionary of over 200 Latin words, combined with a handful of model sentence structures, to generate Lorem Ipsum which looks reasonable. The generated Lorem Ipsum is therefore always free from repetition, injected humour, or non-characteristic words etc.
  24. 24. Lorem Ipsum is simply dummy text of the printing and typesetting industry.A Lorem Ipsum has been the industrys standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum. B It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using Content here, content here, making it look like readable English. Many desktop publishing packages and web page editors now use Lorem Ipsum as their default model text, and a search for lorem ipsum will uncover many web sites still in their infancy. Various versions have evolved over the years, sometimes by accident, sometimes on purpose (injected humour and the like). Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old. Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32 and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good and Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics, very popular during the Renaissance. The first line of Lorem Ipsum, "Lorem ipsum dolor sit amet..", comes from a line in section 1.10.32. The standard chunk of Lorem Ipsum used since the 1500s is reproduced below for those interested. Sections 1.10.32 and 1.10.33 from "de Finibus Bonorum et Malorum" by Cicero are also reproduced in their exact original form, accompanied by English versions from the 1914 translation by H. Rackham. There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected anything embarrassing hidden in the middle of text. All the Lorem Ipsum generators on the Internet tend to repeat predefined chunks as necessary, making this the first true generator on the Internet. It uses a dictionary of over 200 Latin words, combined with a handful of model sentence structures, to generate Lorem Ipsum which looks reasonable. The generated Lorem Ipsum is therefore always free from repetition, injected humour, or non-characteristic words etc.
  25. 25. Lorem Ipsum is simply dummy text of the printing and typesetting industry.A Lorem Ipsum has been the industrys standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum. B It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout. The point of using Lorem Ipsum is that it has a more-or-less normal distribution of letters, as opposed to using Content here, content here, making it look like readable English. Many desktop publishing packages and web page editors now use Lorem Ipsum as their default model text, and a search for lorem ipsum will uncover many web sites still in their infancy. Various versions have evolved over the years, sometimes by accident, sometimes on purpose (injected humour and the like). Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a piece of classical Latin literature from 45 BC, making it over 2000 years old.C Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia, looked up one of the more obscure Latin words, consectetur, from a Lorem Ipsum passage, and going through the cites of the word in classical literature, discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32 and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good and Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics, very popular during the Renaissance. The first line of Lorem Ipsum, "Lorem ipsum dolor sit amet..", comes from a line in section 1.10.32. The standard chunk of Lorem Ipsum used since the 1500s is reproduced below for those interested. Sections 1.10.32 and 1.10.33 from "de Finibus Bonorum et Malorum" by Cicero are also reproduced in their exact original form, accompanied by English versions from the 1914 translation by H. Rackham. There are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected anything embarrassing hidden in the middle of text. All the Lorem Ipsum generators on the Internet tend to repeat predefined chunks as necessary, making this the first true generator on the Internet. It uses a dictionary of over 200 Latin words, combined with a handful of model sentence structures, to generate Lorem Ipsum which looks reasonable. The generated Lorem Ipsum is therefore always free from repetition, injected humour, or non-characteristic words etc.
  26. 26. Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem IpsumA has been the industrys standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.
  27. 27. A B C
  28. 28. ALDA B C
  29. 29. Descripción de la plataforma
  30. 30. Entrenamiento
  31. 31. EntrenamientoData Histórica
  32. 32. EntrenamientoData Histórica Limpieza
  33. 33. EntrenamientoData Histórica Limpieza Entrenamiento
  34. 34. EntrenamientoData Histórica Limpieza Entrenamiento Scrapping
  35. 35. EntrenamientoData Histórica Limpieza Entrenamiento Stop Words Frecuencia 1 Frecuencia Transversal
  36. 36. EntrenamientoData Histórica Limpieza Entrenamiento LDA Almacenamiento Filtrado
  37. 37. Categorización
  38. 38. Categorización Texto Plano
  39. 39. Categorización Texto Plano Limpieza
  40. 40. Categorización Texto Plano Limpieza Clasificación
  41. 41. Categorización Texto Plano Limpieza Clasificación Stop Words
  42. 42. Categorización Texto Plano Limpieza Clasificación Stop Words Modelo LDA Naïve Bayes Multinomial Otros métodos
  43. 43. Python
  44. 44. PythonDjango
  45. 45. Python DjangoModelo Vista Controlador
  46. 46. Python DjangoModelo Vista Controlador MySql
  47. 47. Python Django Modelo Vista Controlador MySqlWeb Service
  48. 48. Resultados
  49. 49. Entrenamiento4.000 Post
  50. 50. 1000 FayerWayer 1000 WayerLess 1000 Belelu 1000 Ferplei
  51. 51. 5 Temáticas detectadas
  52. 52. chile colo equipo copa partido barcelona futbol universidad jugador partidos seleccion ex jugadores alexis goles clubtecnico fecha sanchez chileno real torneo madrid final america nacional catolica gran primera argentino estadio jugarmundial san ahora luego volante delantero clausura campeon agosto nuevo golargentina primer sostuvo carlos mejor liga
  53. 53. 93% Ferplei
  54. 54. Validación350 Post
  55. 55. FayerWayer y Wayerless Suma Frecuencia TFIDF NB NBMPrecission 88% 85% 83% 91% 92%Recall 96% 97% 91% 94% 93%F-Measure 92% 91% 87% 93% 93%Belelú Suma Frecuencia TFIDF NB NBMPrecission 91% 93% 79% 86% 84%Recall 74% 67% 62% 85% 86%F-Measure 82% 78% 70% 85% 85%
  56. 56. Ferplei Suma Frecuencia TFIDF NB NBMPrecission 96% 94% 91% 100% 100%Recall 96% 96% 96% 89% 94%F-Measure 96% 95% 94% 94% 97%
  57. 57. Entrenamiento14.400 Post
  58. 58. Validación1.600 Post
  59. 59. Video Juegos Futbol Musica, Fiestas y Panoramas Telefonia Movil Automoviles Pareja y Vida Social Medio Ambiente Global Ciencia y TecnologiaMedio Ambiente Pequeña Escala Mujer y Sexualidad Familia y Sociedad Investigación Espacial Tech Gadgets Servicios y Tecnologia Automoviles: Top Gear
  60. 60. Complejidad
  61. 61. Trabajo Futuro
  62. 62. Mejorar limpieza del texto Sistema multimodelo Predicción de BlogMejoras del modelo en el tiempo Incorporar Stemming
  63. 63. Muchas Gracias
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×