Cost Based Optimizer - Part 1 of 2


Published on

This paper describes how the optimizer uses statistics and determines plans for executing SQL statement. It explains how the 10053 trace file can be used to understand Oracle's decisions on execution plans.

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Note that without properly collected statistics, the CBO will do one of two things: if no statistics exist for any object used in the SQL statement, the CBO may use rule-based optimization (prior to v10) or use dynamic sampling if statistics exist for any single object but not others in the SQL statement, the CBO may use a set of default statistics for the object without statistics or use dynamic sampling. CBO default statistics for objects without collected stats (prior to v10…in v10 dynamic sampling is typically used instead of defaults): TABLE SETTING DEFAULT STATISTICS cardinality (number of blocks * (block size – cache layer) / average row length average row length 100 bytes number of blocks 100 or actual value based on the extent map remote cardinality (distrib) 2000 rows remote average row length 100 bytes INDEX SETTING DEFAULT STATISTICS levels 1 leaf blocks 25 leaf blocks/key 1 data blocks/key 1 distinct keys 100 clustering factor 800
  • Plot A illustrates a situation in which the execution plan does not change, but the query response time varies significantly as the number of rows in the table changes. This kind of thing occurs when an application chooses a TABLE ACCESS (FULL) execution plan for a growing table. It’s what causes RBO-based applications to appear fast in a small development environment, but then behave poorly in the production environment. Plot B illustrates the marginal improvement that’s achievable, for example, by distributing an inefficient application’s workload more uniformly across the disks in a disk array. Notice that the execution plan (or “shape of the performance curve”) isn’t necessarily changed by such an operation (although, if the output of dbms_stats.gather_system_statistics changes as a result of the configuration change, then the plan might change). The performance for a given number of rows might change, however, as the plot here indicates. Plot C illustrates what is commonly the most profound type of performance change: an execution plan change. This situation can be caused by a change to any of CBO inputs. For example, an accidental deletion of a segment’s statistics can change a plan from a nice fast plan (depicted by the green curve, which is O(log n)) to a horrifically slow plan (depicted by the red curve, which is O(n 2 )). The phenomenon illustrated in plot C is what has happened when a query that was fast last week now runs for 14 hours without completing before you finally give up and kill the session.
  • Cost Based Optimizer - Part 1 of 2

    1. 1. Cost Based Optimizer – 1 of 2 Hotsos Enterprises, Ltd. Grapevine, Texas Oracle. Performance. Now. [email_address]
    2. 2. Agenda <ul><li>Cost Based Optimizer and its impact in performance </li></ul><ul><li>Data Points Collected by the Statistics Gathering Process </li></ul><ul><li>The 10053 Trace File </li></ul><ul><li>Case Studies </li></ul>
    3. 3. Cost Based Optimizer
    4. 4. Cost Based Optimizer (CBO) <ul><li>The CBO in reality is a complex decision making software </li></ul><ul><ul><li>Use several Database Initialization Parameters </li></ul></ul><ul><ul><ul><li>These are listed in the 10053 trace file </li></ul></ul></ul><ul><ul><li>Uses several session level initialization parameter </li></ul></ul><ul><ul><ul><li>These are parameters at the session level that override the database initialization parameters </li></ul></ul></ul><ul><ul><li>Uses statistics about the objects </li></ul></ul><ul><ul><ul><li>These statistics will be discussed later </li></ul></ul></ul><ul><ul><li>Hints to the optimizer </li></ul></ul><ul><ul><li>Uses Statistics about the system (CPU, Disk etc) </li></ul></ul><ul><ul><li>Use this information and makes decisions on the “best way” to generate an execution plan </li></ul></ul>
    5. 5. CBO will be part of your life if you keep working with Oracle. <ul><li>The cost-based query optimizer (CBO)… </li></ul><ul><ul><li>Uses data from a variety of sources </li></ul></ul><ul><ul><li>Estimates the costs of several execution plans </li></ul></ul><ul><ul><li>Chooses the plan it estimates to be the least expensive </li></ul></ul><ul><li>Characteristics </li></ul><ul><ul><li>Adapts to changing circumstances </li></ul></ul><ul><ul><li>Frustrating if you don’t know what it considers as input </li></ul></ul><ul><ul><ul><li>Works great if you know how to use it </li></ul></ul></ul><ul><ul><ul><li>But produces very poor results if you lie to it </li></ul></ul></ul><ul><ul><li>The only query optimizer supported by Oracle Corporation from release 10 onward </li></ul></ul>
    6. 6. The cost-based query optimizer chooses the plan that it computes as having the lowest estimated cost. <ul><li>Don’t assume the following are identical </li></ul><ul><ul><li>CBO’s estimated cost of an execution plan </li></ul></ul><ul><ul><li>The actual cost of an execution plan </li></ul></ul><ul><li>CBO’s cost estimate can be imperfect </li></ul><ul><ul><li>Are your CBO inputs perfect? </li></ul></ul><ul><ul><li>CBO isn’t perfect, but by 9.2 it’s almost always good enough </li></ul></ul><ul><li>Without properly collected statistics, the CBO will </li></ul><ul><ul><li>use RBO if no statistics exist on any object in the statement </li></ul></ul><ul><ul><li>use default statistics if statistics exist for a single object in the statement but not others </li></ul></ul><ul><ul><li>use dynamic sampling to generate statistics (based on parameter setting and Oracle version) </li></ul></ul>
    7. 7. Cost Based Optimizer
    8. 8. Execution plan changes can result in profoundly different application performance. <ul><li>Table size change </li></ul><ul><li>Device latency change </li></ul><ul><li>Execution plan change </li></ul><ul><li>Type C performance changes are the most profound </li></ul>size change performance change performance change performance change
    9. 9. Recap <ul><li>The CBO is a complex piece of software </li></ul><ul><li>It uses several data points to calculate the cost of the execution plan and will choose the plan with the lowest cost </li></ul><ul><li>It is dynamic and will adapt to changing data better than the Cost Based Optimizer </li></ul><ul><li>A good understanding of the Cost Based Optimizer is imperative in understanding the rationale behind some of the choices </li></ul>
    10. 10. Data Points Collected by the Cost Based Optimizer
    11. 11. Data Points Collected by the Cost Based Optimizer <ul><li>Table Statistics </li></ul><ul><li>Column Statistics </li></ul><ul><li>Index Statistics </li></ul><ul><li>System Statistics </li></ul>
    12. 12. Table Statistics <ul><li>Statistics collected for tables appear in the data dictionary views *_TABLES </li></ul><ul><ul><li>Number of rows ( NUM_ROWS ) </li></ul></ul><ul><ul><li>Number of data blocks below the high water mark ( BLOCKS ) </li></ul></ul><ul><ul><li>Number of data blocks allocated to the table that have never been used ( EMPTY_BLOCKS ) </li></ul></ul><ul><ul><li>Average available free space in each data block in bytes ( AVG_SPACE ) </li></ul></ul><ul><ul><li>Number of chained rows ( CHAIN_CNT ) </li></ul></ul><ul><ul><li>Average row length, including the row's overhead, in bytes ( AVG_ROW_LEN ) </li></ul></ul>
    13. 13. Column Statistics <ul><li>Statistics collected for columns appear in the data dictionary views *_TAB_COLUMNS and *_TAB_COL_STATISTICS. </li></ul><ul><ul><li>Number of distinct column values ( NUM_DISTINCT ) </li></ul></ul><ul><ul><li>Lowest value in a column ( LOW_VALUE ) </li></ul></ul><ul><ul><li>Highest value in a column ( HIGH_VALUE ) </li></ul></ul><ul><ul><li>Selectivity estimate for column ( DENSITY ) </li></ul></ul><ul><ul><li>Number of null values in a column ( NUM_NULLS ) </li></ul></ul><ul><ul><li>Number of histogram buckets ( NUM_BUCKETS ) </li></ul></ul><ul><ul><li>Average column length ( AVG_COL_LEN ) </li></ul></ul><ul><ul><li>Existence/type of histogram ( HISTOGRAM ) </li></ul></ul>
    14. 14. Index Statistics <ul><li>Statistics collected for indexes appear in the data dictionary views *_INDEXES . </li></ul><ul><ul><li>Depth of the index from its root block to its leaf blocks ( BLEVEL ) </li></ul></ul><ul><ul><li>Number of leaf blocks ( LEAF_BLOCKS ) </li></ul></ul><ul><ul><li>Number of distinct indexed values ( DISTINCT_KEYS ) </li></ul></ul><ul><ul><li>Average number of leaf blocks in which each distinct value in the index appears ( AVG_LEAF_BLOCKS_PER_KEY ) </li></ul></ul><ul><ul><li>Average number of data blocks in the table that are pointed to by a distinct value in the index ( AVG_DATA_BLOCKS_PER_KEY ) </li></ul></ul><ul><ul><li>Clustering factor (how well ordered the rows are with respect to the indexed values) ( CLUSTERING_FACTOR ) </li></ul></ul>
    15. 15. System Statistics Statistic Description sreadtim average time (ms) to complete a single block read mreadtim average time (ms) to complete a multi-block read cpuspeed average CPU cycles per second (in millions) cpuspeednw estimate of CPU cycles (v10) ioseektim estimate of average read seek time (v10) iotfrspeed estimate of transfer speed of I/O system (v10) mbrc average multiblock read count (in blocks) maxthr maximum I/O system throughput (in bytes/second) slavethr average slave I/O throughput (in bytes/second)
    16. 16. DBMS_STATS <ul><li>DBMS_STATS are used to collect the above described data points </li></ul><ul><ul><li>There are several debates on what the % of data collected should be </li></ul></ul><ul><ul><li>What the options to the DBMS_STATS should be on </li></ul></ul><ul><li>Our perspective </li></ul><ul><ul><li>It should be dependent on the best plans that can be generated for all the SQL being executed in the system </li></ul></ul><ul><ul><li>The frequency and the % of stats being gathered itself should not become a malignant load on the system </li></ul></ul>
    17. 17. Recap <ul><li>There are several data points that are used by the optimizer </li></ul><ul><li>They are collected using DBMS_STATS </li></ul><ul><li>All these data points help the optimizer in deciding to choose a execution plan </li></ul><ul><li>The collection of statistics should be a function of the quality of the plans that can be generated and the special cases that will drive higher frequency and % estimate of statistics collection </li></ul>
    18. 18. Example of Data Points Gathered by the Optimizer
    19. 19. 10053 Trace File
    20. 20. 10053 Trace File <ul><li>The 10053 Trace File is the dump of the analysis done by the Cost Based Optimizer in figuring out the execution plan </li></ul><ul><li>It is unlike the 10046 trace file which has a specific syntax and is usually the same across version barring a few exceptions (Millsap, Optimizing Oracle Performance) </li></ul><ul><li>It should be used as the last resort for trouble shooting </li></ul><ul><li>Usually the difference in the rows estimated by the Cost Based Optimizer and the Actual Rows from the execution plan is good enough to see where the problem is </li></ul>
    21. 21. See Webinar 1 – SQL Tuning with Trace Data and DBMS_XPLAN
    22. 22. 10053 Trace File Structure <ul><li>Peeked Values of the Bind variables in the SQL Statement </li></ul><ul><li>Parameters used by the Optimizer </li></ul><ul><li>Base Table Statistics </li></ul><ul><ul><li>Table Stats </li></ul></ul><ul><ul><li>Column Stats </li></ul></ul><ul><ul><li>Index Stats </li></ul></ul><ul><li>Single Table Access Path </li></ul><ul><li>Join Order Determination (Tries Several Combinations and determines the type of the join and the order of the join) </li></ul><ul><li>Query Itself </li></ul>
    23. 23. Script to Generate 10053 Trace File <ul><li>alter session set max_dump_file_size = unlimited; </li></ul><ul><li>alter session set tracefile_identifier = ‘Query1_10053’; </li></ul><ul><li>alter session set events '10053 trace name context forever, level 1'; </li></ul><ul><li>Explain plan for </li></ul><ul><li>select * from dba_objects where owner=‘SCOTT’; </li></ul><ul><li>alter session set events ’10053 trace name context off’; </li></ul><ul><li>exit; </li></ul>
    24. 24. Recap <ul><li>The 10053 Trace File is a good tool to understand the reasoning and the data behind the optimizer </li></ul><ul><li>Should usually be used a last resort </li></ul><ul><li>Produce large amount of data due to the number of combinations and data points involved in making a decision </li></ul><ul><li>Usually the row difference in the execution plan and the explain plan should be more than enough to understand why the optimizer is making a certain choice of row source operation </li></ul>
    25. 25. Case Studies
    26. 26. Index Scan versus Full Table Scan’s
    27. 27. Setup <ul><li>Create a table which is a copy of dba_objects </li></ul><ul><li>create table myobj as select * from dba_objects; </li></ul><ul><li>We will create indexes on owner and object_type </li></ul><ul><li>We will now gather default statistics and run some queries </li></ul><ul><li>We will create indexes on owner and object_id </li></ul><ul><li>We will then study some simple queries on this table </li></ul>
    28. 28. Should I do a Full Table Scan or Use an Index? <ul><li>Here is a simple query </li></ul><ul><li>Full Table Scan </li></ul><ul><ul><li>select object_type, count(1) from myobj </li></ul></ul><ul><ul><li>where object_id between 10 and 11558 </li></ul></ul><ul><ul><li>group by object_type; </li></ul></ul><ul><li>Index Range Scan </li></ul><ul><ul><li>select object_type, count(1) from myobj </li></ul></ul><ul><ul><li>where object_id between 10 and 11557 </li></ul></ul><ul><ul><li>group by object_type; </li></ul></ul>
    29. 29. Review 10053 Trace Files
    30. 30. Data Points <ul><li>Full Table Scan </li></ul><ul><ul><li>Cost </li></ul></ul><ul><ul><ul><li>180 </li></ul></ul></ul><ul><ul><li>Logical I/O’s </li></ul></ul><ul><ul><ul><li>772 </li></ul></ul></ul><ul><li>Index Scan </li></ul><ul><ul><li>Cost </li></ul></ul><ul><ul><ul><li>179 </li></ul></ul></ul><ul><ul><li>Logical I/O’s </li></ul></ul><ul><ul><ul><li>166 </li></ul></ul></ul>
    31. 31. Shouldn’t Oracle minimize the number of Logical I/O’s? <ul><li>Yes </li></ul><ul><ul><li>Yes, it is probably the right thing to do </li></ul></ul><ul><li>No </li></ul><ul><ul><li>Because, it is probably not the right thing </li></ul></ul><ul><li>The answer is it depends on several things the most important being db_file_multiblock_read_count </li></ul><ul><li>In our experiment it was set to 16 </li></ul><ul><li>At 16, the cost or the time taken to execute the query by Oracle will be faster if it did a full table scan using the multi block read count than if it did a sequential read of the index and the table </li></ul><ul><li>You can set different values for db_file_multiblock_read_count and see you can see the cost calculations and the plans change </li></ul>
    32. 32. Now let us set db_file_multiblock_read_count to 8 <ul><li>Full Table Scan </li></ul><ul><ul><li>select object_type, count(1) from myobj </li></ul></ul><ul><ul><li>where object_id between 10 and 14301 </li></ul></ul><ul><ul><li>group by object_type; </li></ul></ul><ul><li>Index Range Scan </li></ul><ul><ul><li>select object_type, count(1) from myobj </li></ul></ul><ul><ul><li>where object_id between 10 and 14300 </li></ul></ul><ul><ul><li>group by object_type; </li></ul></ul>
    33. 33. Let us set it to 8 and see what happens <ul><li>Full Table Scan </li></ul><ul><ul><li>Cost </li></ul></ul><ul><ul><ul><li>221 </li></ul></ul></ul><ul><ul><li>Logical I/O’s </li></ul></ul><ul><ul><ul><li>772 </li></ul></ul></ul><ul><li>Index Scan </li></ul><ul><ul><li>Cost </li></ul></ul><ul><ul><ul><li>220 </li></ul></ul></ul><ul><ul><li>Logical I/O’s </li></ul></ul><ul><ul><ul><li>211 </li></ul></ul></ul>
    34. 34. So there are a lot of moving parts!!! <ul><li>Yes, there are. </li></ul><ul><li>How do you manage that </li></ul><ul><ul><li>Keep your optimizer environment as stable as possible </li></ul></ul><ul><ul><li>The only thing that should be changing are the data points collected by the DBMS_STATS gathering process </li></ul></ul><ul><ul><li>Keep a copy of your old statistics </li></ul></ul><ul><ul><li>Just in case you may have to revert back to old stats to get back to the good ol’ plan </li></ul></ul>
    35. 35. A couple of more concepts
    36. 36. Cardinality/Selectivity <ul><li>Cardinality </li></ul><ul><ul><li>The predicted number of rows generated by a row source operation </li></ul></ul><ul><ul><li>That is what you see in the explain plan </li></ul></ul><ul><li>Selectivity </li></ul><ul><ul><li>Cardinality is calculated by estimating the selectivity which is the expected fraction of rows that will pass the predicate test </li></ul></ul><ul><ul><li>Is very sensitive to number of distinct values </li></ul></ul><ul><ul><li>Determines join order choice based on cardinality and hence selectivity </li></ul></ul>
    37. 37. Simple Selectivity <ul><li>Calculate Range for the Query (Rng) </li></ul><ul><ul><li>Obtained from Query </li></ul></ul><ul><li>Calculate the difference between column high value and column low value (hival-loval) </li></ul><ul><ul><li>Obtained from dba_tab_columns </li></ul></ul><ul><li>Divide Rng and (hival-loval) </li></ul><ul><li>Add N/num_distinct where N is the range bound count and num_distinct is the number of distinct values for this column which is obtained from dba_tab_columns </li></ul>
    38. 38. Sample Selectivity Calculation <ul><li>select object_type, count(1) from myobj where object_id between </li></ul><ul><li>10 and 11558 group by object_type; </li></ul><ul><li>Range is 11558-10 = 11548 </li></ul><ul><li>Column High Value – 69892 </li></ul><ul><li>Column Low Value – 14 </li></ul><ul><li>N = 2 </li></ul><ul><li>Number of Distinct Values – 55618 </li></ul><ul><li>Note the optimizer see that 10 as a lower bound value does not make sense and upgrades it to 14 </li></ul><ul><li>Select (11548-14)/(69892-14)+(2/55618) * 55618 from dual gives cardinality which is 9190 and let us look at the explain plan cardinality </li></ul>
    39. 39. Cost <ul><li>What do we mean by the cost of the query? </li></ul><ul><ul><li>Oracle Performance Guide </li></ul></ul><ul><ul><ul><li>Cost = (Number of Single Block Reads * single read time </li></ul></ul></ul><ul><ul><ul><li>+ number of multi block reads * multi block read time </li></ul></ul></ul><ul><ul><ul><li>+ CPU Cycles / CPU Cycles / second ) </li></ul></ul></ul><ul><ul><ul><li>/Time to read a single block </li></ul></ul></ul><ul><ul><ul><li>Dimensional Analysis Show Cost is just a number </li></ul></ul></ul><ul><ul><ul><li>Although it is the optimizer guess at how long the query will take to execute </li></ul></ul></ul>
    40. 40. Selectivity and Cardinality Impact cost and join strategies <ul><li>Selectivity and Cardinality impact cost and join strategies as we saw in the example </li></ul><ul><li>You can predict the cost and behavior of your queries as the data volume increases </li></ul><ul><li>The number of distinct values is very important and determines the selectivity and hence cardinality and hence access strategies and hence costs </li></ul><ul><li>The number of distinct values from non-unique columns on which predicates are written and joined can severely be impacted the quality of the statistics being collected </li></ul>
    41. 41. So how can you know if a plan is good? <ul><li>Look at the cardinality of the explain plan </li></ul><ul><li>Look at the rows actually obtained from the execution plan (from the trace file or the row source execution statistics) </li></ul><ul><li>See how closely they match </li></ul><ul><li>If there is a wide variance, study if it is impacting your query performance and determine what additional statistics need to be gathered </li></ul>
    42. 42. How can you predict the nature of plans due to data growth? <ul><li>Use the cardinality hint to study the performance of the query and its behaviour </li></ul>
    43. 43. Scripts
    44. 44. 10053_SCOTT.sql <ul><li>alter session set max_dump_file_size = unlimited; </li></ul><ul><li>alter session set tracefile_identifier = 'MYOBJ_SCOTT'; </li></ul><ul><li>alter session set events '10053 trace name context forever, level 1'; </li></ul><ul><li>explain plan for </li></ul><ul><li>select object_type,count(*) from myobj where owner='SCOTT' </li></ul><ul><li>group by object_type; </li></ul><ul><li>alter session set events '10053 trace name context off'; </li></ul><ul><li>exit; </li></ul>
    45. 45. 10053_sys.sql <ul><li>alter session set max_dump_file_size = unlimited; </li></ul><ul><li>alter session set tracefile_identifier = 'SYS_OBJ'; </li></ul><ul><li>alter session set events '10053 trace name context forever, level 1'; </li></ul><ul><li>explain plan for </li></ul><ul><li>select object_type,count(*) from myobj where owner='SYS' </li></ul><ul><li>group by object_type; </li></ul><ul><li>alter session set events '10053 trace name context off'; </li></ul><ul><li>exit; </li></ul>
    46. 46. Col_stats.sql <ul><li>SELECT table_name &quot;Table Name&quot;, </li></ul><ul><li>column_name &quot;Column Name&quot;, </li></ul><ul><li>num_distinct &quot;Number of Distinct Values&quot;, </li></ul><ul><li>low_value &quot;Low Value&quot;, </li></ul><ul><li>high_value &quot;High Value&quot;, </li></ul><ul><li>density &quot;Density&quot;, </li></ul><ul><li>num_nulls &quot;Number of Nulls&quot;, </li></ul><ul><li>num_buckets &quot;Number of Buckets&quot;, </li></ul><ul><li>avg_col_len &quot;Average Column Length&quot;, </li></ul><ul><li>histogram &quot;Histogram&quot; </li></ul><ul><li>FROM user_tab_columns </li></ul><ul><li>WHERE table_name='MYOBJ'; </li></ul>
    47. 47. Gather_table_stats.sql <ul><li>begin </li></ul><ul><li>dbms_stats.gather_table_stats(USER,'MYOBJ'); </li></ul><ul><li>end; </li></ul>
    48. 48. Index_stats.sql <ul><li>SELECT table_name &quot;Table Name&quot;, </li></ul><ul><li>index_name &quot;Index Name&quot;, </li></ul><ul><li>blevel &quot;Blevel&quot;, </li></ul><ul><li>leaf_blocks &quot;Leaf Blocks&quot;, </li></ul><ul><li>distinct_keys &quot;Distinct Keys&quot;, </li></ul><ul><li>AVG_LEAF_BLOCKS_PER_KEY &quot;Avg Leaf Blocks per Key&quot;, </li></ul><ul><li>AVG_DATA_BLOCKS_PER_KEY &quot;Avg Data Blocks Per Key&quot;, </li></ul><ul><li>CLUSTERING_FACTOR &quot;Clustering Factor&quot; </li></ul><ul><li>FROM user_indexes </li></ul><ul><li>WHERE table_name='MYOBJ'; </li></ul>
    49. 49. Join_strategy_choice.sql <ul><li>select /*+ cardinality (e 6059) cardinality (d 200) */ e.empno,e.ename,d.dname,d.loc from emp e, dept d </li></ul><ul><li>where e.deptno=d.deptno; </li></ul><ul><li>select /*+ cardinality (e 6060) cardinality (d 200) */ e.empno,e.ename,d.dname,d.loc from emp e, dept d </li></ul><ul><li>where e.deptno=d.deptno; </li></ul>
    50. 50. Raw_to_num.sql <ul><li>select </li></ul><ul><li>column_name, </li></ul><ul><li>raw_to_num(low_value) low_val, </li></ul><ul><li>raw_to_num(high_value) high_val </li></ul><ul><li>from </li></ul><ul><li>user_tab_columns </li></ul><ul><li>where </li></ul><ul><li>table_name = 'MYOBJ' </li></ul><ul><li>and column_name = 'OBJECT_ID' </li></ul>
    51. 51. Tab_stats.sql <ul><li>select table_name &quot;Table Name&quot;, </li></ul><ul><li>num_rows &quot;No. of Rows&quot;, </li></ul><ul><li>blocks &quot;Blocks&quot;, </li></ul><ul><li>empty_blocks &quot;Empty Blocks&quot;, </li></ul><ul><li>avg_space &quot;Average Space&quot;, </li></ul><ul><li>chain_cnt &quot;Chain Count&quot;, </li></ul><ul><li>avg_row_len &quot;Average Row Length&quot;, </li></ul><ul><li>sample_size &quot;Sampe Size&quot; from user_tables </li></ul><ul><li>where table_name='MYOBJ'; </li></ul>