The document discusses data engineering techniques, particularly focusing on altering table structures in databases, such as changing data types and managing large textual fields. It evaluates different database engines like MyISAM and InnoDB regarding speed, reliability, and search efficiency, and highlights the inefficiencies of querying text fields. It concludes that while there is no one-size-fits-all solution, certain implementations can enhance performance and application architecture.

1. strip your
`TEXT` fields
Gabi
@gabidavila
http://gabriela.io

2. about
• Data Engineer
• @Crowdcube

3. uses
• serialised Arrays
• JSON
• large strings
• images*

5. how does it
work?

6. alter table
•creates a temporary table with the new structure
•copies the data from the old table to the new one
•consolidates the new table

7. example

8. alter PK: INT ->
BIGINT
- 2.147.483.648
TO
2.147.483.647
INT
- 922.337.203.854.775.808
TO
922.337.203.854.775.807
BIGINT
Illustrationoutofscale

9. case
• > 750 GB
• > 380 million lines
• 3 TEXT fields
• Auto increment: 898.191.090

10. how long did it
take?

11. 2 days

12. why?

13. speed
fast slow

14. storage
• 1 file in the disk for each row for each TEXT
field
• stored in different location than the table
data itself
• each field up to 4 MB

15. engines

16. MyISAM
• fastest read speed
• supports FULLTEXT indexes
• non transactional
• less data reliability

17. InnoDB
• transactional
• better data integrity
• until MySQL 5.6 it didn’t support FULLTEXT
indexes in TEXT fields

18. querying into
TEXT fields
• inefficient search with LIKE statement
• slow DDL operations (like alter table)
• unnecessary increase of the table size

19. possible
alternatives

20. fastest search

21. search servers
• indexes large bodies
• api bindings
• decoupled

22. smart retrieval

23. content delivery
network
• high availability
• cheap

24. TEXT field within
the RDBMS

26. conclusion
• there is no silver bullet solution
• some implementations may add an
additional layer of complexity to the
application
• some implementation works better with
decoupled applications

27. thanks!
come say hi :)