The document discusses the design and implementation of an automatic post categorization prototype for Betazeta Networks S.A. using text mining. The objectives are to support the manual processing of large volumes of blog posts by developing a solution for automatic categorization. This will involve selecting historical data and models, establishing evaluation methods, designing the categorization process, and implementing a prototype for users to input data and the company to process, filter, and publish posts according to business criteria.

1. Diseño y construcción de una plataforma de
clasificación y calificación de post para una red
de blogs basada en textmining para Betazeta
Networks S.A.
Camilo López A.

2. Objetivo General
El objetivo general del trabajo es el apoyo al procesamiento manual de
grandes volúmenes de publicaciones en la red de blogs de Betazeta mediante el
diseño e implementación de un prototipo para la categorización automática de
estos datos utilizando text mining.

3. Objetivos Específicos
1. Entender a fondo la problemática y el contexto de la empresa junto
con los conocimientos necesarios respecto a text mining, modelos y
metodologías necesarias.
2. Selección de los datos históricos, las consultas sobre éstos y los
modelos que permitan realizar predicciones exitosas de
categorización.
3. Establecer métodos y métricas para la evaluación de la solución
propuesta.
4. Utilizando el conocimiento adquirido en los objetivos anteriores,
diseñar el proceso de categorización automático de posts.

4. Objetivos Específicos
5. Diseñar e implementar un prototipo que permita al usuario ingresar
información en forma adecuada para su análisis y a la empresa
procesarla, filtrarla y publicarla en base a criterios del negocio.
6. Implementación de la metodología de evaluación.

5. El Problema

6. betazeta

7. 7,5 millones
Visitas Mensuales

9. User Generated Content

10. Volumen

11. Google Categories

12. Spam

13. Filtro de contenido

14. Filtro de contenido
Categorizar

18. Background
Teórico

19. Data Mining

20. Limpieza del texto

21. Stemming

22. Stop Words

23. Diccionario

24. Latent Dirichlet Allocation

25. LDA
Latent Dirichlet Allocation

26. Lorem Ipsum is simply dummy text of the printing and typesetting industry.
Lorem Ipsum has been the industry's standard dummy text ever since the 1500s,
when an unknown printer took a galley of type and scrambled it to make a type
specimen book. It has survived not only five centuries, but also the leap into
electronic typesetting, remaining essentially unchanged. It was popularised in
the 1960s with the release of Letraset sheets containing Lorem Ipsum passages,
and more recently with desktop publishing software like Aldus PageMaker
including versions of Lorem Ipsum.
It is a long established fact that a reader will be distracted by the readable
content of a page when looking at its layout. The point of using Lorem Ipsum is
that it has a more-or-less normal distribution of letters, as opposed to using
'Content here, content here', making it look like readable English. Many desktop
publishing packages and web page editors now use Lorem Ipsum as their default
model text, and a search for 'lorem ipsum' will uncover many web sites still in
their infancy. Various versions have evolved over the years, sometimes by
accident, sometimes on purpose (injected humour and the like).
Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots
in a piece of classical Latin literature from 45 BC, making it over 2000 years old.
Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia,
looked up one of the more obscure Latin words, consectetur, from a Lorem
Ipsum passage, and going through the cites of the word in classical literature,
discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32
and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good and
Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics,
very popular during the Renaissance. The first line of Lorem Ipsum, "Lorem
ipsum dolor sit amet..", comes from a line in section 1.10.32.
The standard chunk of Lorem Ipsum used since the 1500s is reproduced below
for those interested. Sections 1.10.32 and 1.10.33 from "de Finibus Bonorum et
Malorum" by Cicero are also reproduced in their exact original form,
accompanied by English versions from the 1914 translation by H. Rackham.
There are many variations of passages of Lorem Ipsum available, but the
majority have suffered alteration in some form, by injected anything
embarrassing hidden in the middle of text. All the Lorem Ipsum generators on
the Internet tend to repeat predefined chunks as necessary, making this the first
true generator on the Internet. It uses a dictionary of over 200 Latin words,
combined with a handful of model sentence structures, to generate Lorem
Ipsum which looks reasonable. The generated Lorem Ipsum is therefore always
free from repetition, injected humour, or non-characteristic words etc.

27. Lorem Ipsum is simply dummy text of the printing and typesetting industry.
A
Lorem Ipsum has been the industry's standard dummy text ever since the 1500s,
when an unknown printer took a galley of type and scrambled it to make a type
specimen book. It has survived not only five centuries, but also the leap into
electronic typesetting, remaining essentially unchanged. It was popularised in
the 1960s with the release of Letraset sheets containing Lorem Ipsum passages,
and more recently with desktop publishing software like Aldus PageMaker
including versions of Lorem Ipsum.
It is a long established fact that a reader will be distracted by the readable
content of a page when looking at its layout. The point of using Lorem Ipsum is
that it has a more-or-less normal distribution of letters, as opposed to using
'Content here, content here', making it look like readable English. Many desktop
publishing packages and web page editors now use Lorem Ipsum as their default
model text, and a search for 'lorem ipsum' will uncover many web sites still in
their infancy. Various versions have evolved over the years, sometimes by
accident, sometimes on purpose (injected humour and the like).
Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots
in a piece of classical Latin literature from 45 BC, making it over 2000 years old.
Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia,
looked up one of the more obscure Latin words, consectetur, from a Lorem
Ipsum passage, and going through the cites of the word in classical literature,
discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32
and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good and
Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics,
very popular during the Renaissance. The first line of Lorem Ipsum, "Lorem
ipsum dolor sit amet..", comes from a line in section 1.10.32.
The standard chunk of Lorem Ipsum used since the 1500s is reproduced below
for those interested. Sections 1.10.32 and 1.10.33 from "de Finibus Bonorum et
Malorum" by Cicero are also reproduced in their exact original form,
accompanied by English versions from the 1914 translation by H. Rackham.
There are many variations of passages of Lorem Ipsum available, but the
majority have suffered alteration in some form, by injected anything
embarrassing hidden in the middle of text. All the Lorem Ipsum generators on
the Internet tend to repeat predefined chunks as necessary, making this the first
true generator on the Internet. It uses a dictionary of over 200 Latin words,
combined with a handful of model sentence structures, to generate Lorem
Ipsum which looks reasonable. The generated Lorem Ipsum is therefore always
free from repetition, injected humour, or non-characteristic words etc.

28. Lorem Ipsum is simply dummy text of the printing and typesetting industry.
A
Lorem Ipsum has been the industry's standard dummy text ever since the 1500s,
when an unknown printer took a galley of type and scrambled it to make a type
specimen book. It has survived not only five centuries, but also the leap into
electronic typesetting, remaining essentially unchanged. It was popularised in
the 1960s with the release of Letraset sheets containing Lorem Ipsum passages,
and more recently with desktop publishing software like Aldus PageMaker
including versions of Lorem Ipsum.
B
It is a long established fact that a reader will be distracted by the readable
content of a page when looking at its layout. The point of using Lorem Ipsum is
that it has a more-or-less normal distribution of letters, as opposed to using
'Content here, content here', making it look like readable English. Many desktop
publishing packages and web page editors now use Lorem Ipsum as their default
model text, and a search for 'lorem ipsum' will uncover many web sites still in
their infancy. Various versions have evolved over the years, sometimes by
accident, sometimes on purpose (injected humour and the like).
Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots
in a piece of classical Latin literature from 45 BC, making it over 2000 years old.
Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia,
looked up one of the more obscure Latin words, consectetur, from a Lorem
Ipsum passage, and going through the cites of the word in classical literature,
discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32
and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good and
Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics,
very popular during the Renaissance. The first line of Lorem Ipsum, "Lorem
ipsum dolor sit amet..", comes from a line in section 1.10.32.
The standard chunk of Lorem Ipsum used since the 1500s is reproduced below
for those interested. Sections 1.10.32 and 1.10.33 from "de Finibus Bonorum et
Malorum" by Cicero are also reproduced in their exact original form,
accompanied by English versions from the 1914 translation by H. Rackham.
There are many variations of passages of Lorem Ipsum available, but the
majority have suffered alteration in some form, by injected anything
embarrassing hidden in the middle of text. All the Lorem Ipsum generators on
the Internet tend to repeat predefined chunks as necessary, making this the first
true generator on the Internet. It uses a dictionary of over 200 Latin words,
combined with a handful of model sentence structures, to generate Lorem
Ipsum which looks reasonable. The generated Lorem Ipsum is therefore always
free from repetition, injected humour, or non-characteristic words etc.

29. Lorem Ipsum is simply dummy text of the printing and typesetting industry.
A
Lorem Ipsum has been the industry's standard dummy text ever since the 1500s,
when an unknown printer took a galley of type and scrambled it to make a type
specimen book. It has survived not only five centuries, but also the leap into
electronic typesetting, remaining essentially unchanged. It was popularised in
the 1960s with the release of Letraset sheets containing Lorem Ipsum passages,
and more recently with desktop publishing software like Aldus PageMaker
including versions of Lorem Ipsum.
B
It is a long established fact that a reader will be distracted by the readable
content of a page when looking at its layout. The point of using Lorem Ipsum is
that it has a more-or-less normal distribution of letters, as opposed to using
'Content here, content here', making it look like readable English. Many desktop
publishing packages and web page editors now use Lorem Ipsum as their default
model text, and a search for 'lorem ipsum' will uncover many web sites still in
their infancy. Various versions have evolved over the years, sometimes by
accident, sometimes on purpose (injected humour and the like).
Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots
in a piece of classical Latin literature from 45 BC, making it over 2000 years old.
C
Richard McClintock, a Latin professor at Hampden-Sydney College in Virginia,
looked up one of the more obscure Latin words, consectetur, from a Lorem
Ipsum passage, and going through the cites of the word in classical literature,
discovered the undoubtable source. Lorem Ipsum comes from sections 1.10.32
and 1.10.33 of "de Finibus Bonorum et Malorum" (The Extremes of Good and
Evil) by Cicero, written in 45 BC. This book is a treatise on the theory of ethics,
very popular during the Renaissance. The first line of Lorem Ipsum, "Lorem
ipsum dolor sit amet..", comes from a line in section 1.10.32.
The standard chunk of Lorem Ipsum used since the 1500s is reproduced below
for those interested. Sections 1.10.32 and 1.10.33 from "de Finibus Bonorum et
Malorum" by Cicero are also reproduced in their exact original form,
accompanied by English versions from the 1914 translation by H. Rackham.
There are many variations of passages of Lorem Ipsum available, but the
majority have suffered alteration in some form, by injected anything
embarrassing hidden in the middle of text. All the Lorem Ipsum generators on
the Internet tend to repeat predefined chunks as necessary, making this the first
true generator on the Internet. It uses a dictionary of over 200 Latin words,
combined with a handful of model sentence structures, to generate Lorem
Ipsum which looks reasonable. The generated Lorem Ipsum is therefore always
free from repetition, injected humour, or non-characteristic words etc.

30. Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum
A
has been the industry's standard dummy text ever since the 1500s, when an unknown
printer took a galley of type and scrambled it to make a type specimen book. It has
survived not only five centuries, but also the leap into electronic typesetting, remaining
essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets
containing Lorem Ipsum passages, and more recently with desktop publishing software
like Aldus PageMaker including versions of Lorem Ipsum.

31. A B C

33. A
LDA
B
C

39. Descripción de la
plataforma

40. Entrenamiento

41. Entrenamiento
Data Histórica

42. Entrenamiento
Data Histórica Limpieza

43. Entrenamiento
Data Histórica Limpieza Entrenamiento

44. Entrenamiento
Data Histórica Limpieza Entrenamiento
Scrapping

45. Entrenamiento
Data Histórica Limpieza Entrenamiento
Stop Words
Frecuencia 1
Frecuencia
Transversal

46. Entrenamiento
Data Histórica Limpieza Entrenamiento
LDA
Almacenamiento
Filtrado

47. Categorización

48. Categorización
Texto Plano

49. Categorización
Texto Plano Limpieza

50. Categorización
Texto Plano Limpieza Clasificación

51. Categorización
Texto Plano Limpieza Clasificación
Stop Words

52. Categorización
Texto Plano Limpieza Clasificación
Stop Words Modelo LDA
Naïve Bayes Multinomial
Otros métodos

53. Python

54. Python
Django

55. Python
Django
Modelo Vista Controlador

56. Python
Django
Modelo Vista Controlador
MySql

57. Python
Django
Modelo Vista Controlador
MySql
Web Service

58. Resultados

59. Entrenamiento
4.000 Post

60. 1000 FayerWayer
1000 WayerLess
1000 Belelu
1000 Ferplei

61. 5 Temáticas detectadas

62. chile colo equipo copa partido barcelona
futbol universidad jugador partidos
seleccion ex jugadores alexis goles club
tecnico fecha sanchez chileno real torneo
madrid final america nacional catolica
gran primera argentino estadio jugar
mundial san ahora luego volante delantero
clausura campeon agosto nuevo gol
argentina primer sostuvo carlos mejor liga

63. 93% Ferplei

64. Validación
350 Post

65. FayerWayer y Wayerless
Suma Frecuencia TFIDF NB NBM
Precission 88% 85% 83% 91% 92%
Recall 96% 97% 91% 94% 93%
F-Measure 92% 91% 87% 93% 93%
Belelú
Suma Frecuencia TFIDF NB NBM
Precission 91% 93% 79% 86% 84%
Recall 74% 67% 62% 85% 86%
F-Measure 82% 78% 70% 85% 85%

66. Ferplei
Suma Frecuencia TFIDF NB NBM
Precission 96% 94% 91% 100% 100%
Recall 96% 96% 96% 89% 94%
F-Measure 96% 95% 94% 94% 97%

67. Entrenamiento
14.400 Post

68. Validación
1.600 Post

69. Video Juegos
Futbol
Musica, Fiestas y Panoramas
Telefonia Movil
Automoviles
Pareja y Vida Social
Medio Ambiente Global
Ciencia y Tecnologia
Medio Ambiente Pequeña Escala
Mujer y Sexualidad
Familia y Sociedad
Investigación Espacial
Tech Gadgets
Servicios y Tecnologia
Automoviles: Top Gear

70. Complejidad

71. Trabajo Futuro

72. Mejorar limpieza del texto
Sistema multimodelo
Predicción de Blog
Mejoras del modelo en el tiempo
Incorporar Stemming

73. Muchas Gracias