The document discusses optimizations for Hive and Hadoop at Facebook including RCFile columnar storage, which saves up to 30% storage space and reduces I/O and CPU costs. It also discusses harnessing the sort and bucket properties of data to optimize the group by operator, reducing CPU costs by half. Future work mentioned includes join skew optimization, indexing, and data sharing across multiple machines in parallel.

2. Hive and Hadoop in Data-InfraYongqiang He12/09/09

3. Agenda

4. RCFile (columnar storage on Hive)Open source effortEffectSaves up to 30% storage spaces. on average >20%Reduces IO, CPU costs, memoryWhat others we can save?An outside fb usage reports it is 8 times faster in readingOn deployment to Facebook Hadoop Hive cluster now

5. Harness Sort/bucket propertyData is grouped, and sometimes sortedBut this property is not used right now

6. Why it is useful?

7. Okay, CPU and MemoryEffectAn Optimization drop CPU costs to half

8. Group by operator in Hive (group by is used everywhere in fb)