6. 1. Filter by multiple taxonomies
2. Combine text, taxonomy terms and numeric values
3. Discover relationships or trends between objects
4. Make huge items volume navigable
5. Simplify "advanced" search UI
What for
7. Tags
• Keyword assigned to an object
• Chosen informally and personally by item's creator or viewer
• Assigned by the viewer + unlimited = Folksonomy
• Assigned by the creator + limited = Taxonomy
8. • Tag-based
• Plain-structure based
Faceted search base
Object Author Category Format Price
Days to
deliver
The Oracle Book: Answers to
Life's Questions
Cerridwen
Greenleaf Fortune telling Paper 18 4
Ask a Question, Find Your Fate:
The Oracle Book Ben Hale Fantasy Kindle 12 2
Object Tags
Book 1 Paper, Fortune telling, Very good book, worth reading
Book 2 Ben Halle, Fantasy, Kindle
19. 1. Limited POC resources: people + time
2. Customer has a license
3. Wide searсh table
4. A lot of rows
5. A lot of equal values:
• Object types
• Facet types
• Tag names
6. Size is fine for InMemory
7. Queries use a lot of aggregate functions
Why to try
35. InMemory size
Options
Volume,
Gb
Load time,
seconds
data in row format 6,5 300
no compress InMememory 7,2 40
memcompress for dml 6 45
memcompress for query low 4 45
memcompress for query high 2,5 49
memcompress for capacity low 3,5 48
memcompress for capacity high 2 50
No significant differences for loading!
46. • Sporadic search degradation – 5-10%
• Happens when a lot of DML operations
Performance spikes
47. 1. Changed records -> Mark as stale
2. Stale records -> Read from row storage
• buffer cache
• disk
3. Stale records -> repopulate IMCU:
• Staleness threshold
• Trickle repopulation process - each 2 minutes
• processes count - INMEMORY_MAX_REPOPULATE_SERVERS
• processes utilization - INMEMORY_TRICKLE_REPOPULATE_SERVERS_PERCENT
Transaction processing
61. InMemory size <> table data size
All data InMemory <> High performance
Decent time to be loaded
8 IM statistics is enough
Big updates => full inmemory repopulation
Small updates => trickle repopulation
Trickle parameters relevant to workload
DBAs findings
62. Advanced IM features <> significant profit our case
Zone maps = numeric and date data type only
Dictionary pruning – not in Oracle 12.2 and 18
Simple data types = High perfomance
High compression <> Slow ingestion
rollback = IMCU repopulation
Developers finding
63. Add extra memory
Implement a POC with IM DWH
Understand all 523 IM statistics + 190 parameters
Use FastStart to speedup database wakeup
Play with Oracle 18c features
Furthers plans
64. Always try and measure
IM works for short queries as well
Understanding of IM internals is a must
Application changes are required
No extra software/hardware introduced
Fast POC followed by production deploy
4x times performance boost
Conclusion
65. Thank you for your time!
Alexander Tokarev
Database expert
DataArt
shtock@mail.ru
https://github.com/shtock
https://www.linkedin.com/in/alexander-tokarev-14bab230
Editor's Notes
Hello, guys. My name is Alex and I work for DataArt as a performance architect. Does someone use Oracle database in your applications? Have you ever used Oracle InMemory option to bring a performance profit?
Perfect.
Today we are going to discuss how we improved faceted search performance with Oracle InMemory option for one of our client.
Важно, что я не буду влезать в технические детали реализации отдельных фишек, а просто расскажу, почему у нас они не взлетели и как мы их заводили, иначе наш доклад будет очень долгим
Let me introduce myself
I work for DataArt as a performance architect and my main focus is data warehousing for big clients. I have huge Oracle experience but I works with Exasol and Tarantool databases either.
We have rather straightforward agenda for today discussion
We’ll discuss what is faceted search,
client application overview and its performance issues
I try to explain how Oracle InMemory option is implemented internally but without digging into details.
My main focus is how we made the customer happy and share with you our issues you could face with as well
For sure I’ll answer your questions.
Everybody know what is faceted search.
We have groups named facets,
Values named constraints and
Figures names facet counts
Common facet types are terms
Intervals
And ranges
We need facets if
Users need to filter content using multiple taxonomy terms at the same time.
Users want to combine text searches, taxonomy term filtering, and other search criteria.
trying to discover relationships or trends between contents.
Navigate inside tons of data
And your end users are already frightened by "advanced" search forms in enterprise applications
There are many ways how to implement faceted search internally and one of them is using tags. Everyone knows what is tags but let me reming.
Tag is a keyword assigned to an object
Mostly is is chosen informaly
If it is assigned by the viewer and tags count is unlimited it is folksonomy
When assigned by the creator and a set of tags is limited it is taxonomy
So we could store data for facets in tags which is extremely convenient for terms faceting by folksonomies and taxonomies
Or in a plain table which is very convenient for range, interval facets defined by taxonomies.
I think everything is clear enough but may be you have some questions about theory?
So, let’s move on
We use taxonomy and foksonomy, so the best part of our facets are terms and we choosen tag-based storage.
Let me show some statistics. We extract objects, tags, facets for 2 years
We have about 3 000 000 objects tagged, 42 000 000 terms applied, 100 000 unique terms, maximum tags count is 15 and there are about 150 facets.
Is it whether decent volume or not? I think it is decent because we are bigger than StackOverflow by tags in 3 times.
We have ordinal architecture for enterprise.
Web browser, balancer, a set of application servers which deal with inmemory grid and oracle with the replica on the backend.
Pay attention that there is no full text search server which has been prohibited in accordance with customer policies.
We have microservice architecture and one of the services it tagging service which is responsible for data ingestion and faceted search. It is widely used by many other services and receives about 10000 requests per second. In order to serve search requests there are about 20 fine-tuned sql queries.
I can’t show you real printscreens from the application because of NDA so I implemented the mockup in Paint. As we could see UI is full of features as
Predefined faceted search templates
Full text search by random tags. Here it is done by names. We could search via audit information and we have faceted navigation for sure. We could filter out objects where some tags are present (not tag values) and user paging.
It is rather sophisticated UI.
Without digging into details I could say that the best part of search is made on a wide Denormalized table with tags which stores only current tag values
We could have a look into the table in details. It contains tagged object types, facet information, tags information and values. There is Denormalized object name and object description tags because they are used in full text search mostly.
Let’s have a look into performance metrics. Pay attention the insert performance is very good so I used logarithmic scale to put everything in one diagram. On a first glance figures look good. We do complicated searches less than zero point two seconds but actually it is bad because we need to serve more and more users and data so queries become slow
In accordance with one of my otherspresentations the best option for faceted search is full text search dedicated server but we had a lack of time to implement it as well as reluctant to introduce additional layers
We started looking into InMemory solutions especialy regarding the customer already had Oracle InMemory license. In accordance with Oracle documentation it should be used for analytical queries which isn’t our case completely but we decided to try.
Why we should try
Because we are under limitations preasure
The customer has a license
The table is rather wide, full of data which has a lot of equal values
So it will fit in memory especialy regarding compression features
And last but not least queries are full of aggregate functions
In accordance with the Oracle documentation InMemory feature is sort of silver bullets so we decided not waste time understanding how it works internaly. What we knew that the option stores row and columnar data in dual mode maintaining their consistency automatically.
Let me remind you our table
We set InMemory area size to 10 gigabytes and assigned the table into the area. We decided that performance will be faster without compression so no compression option was used.
What is significant to stress that InMemory brought some overhead without compression
So we started our tests and faced with the difference in performance was extremely unsignificant which is expect for the search by primary keys but not for search by tags
So we started our tests and faced with the difference in performance was extremely unsignificant which is expect for the search by primary keys but not for search by tags
So we started our tests and faced with the difference in performance was extremely unsignificant which is expect for the search by primary keys but not for search by tags
When we started our POC we relied on Lufthansa usecase when they got average 21 performance profit but our profit was 10 times less. It means that we do something wrong and we need to understand how InMemory works internaly.
Do you recall I mentioned InMemory area? It consists from 2 parts actualy.
The first one in IMCU – InMemory Compression Unit which stores data in columnar format
And SMU – Snapshot Metadata Unit which stores transaction related metadata, dictionaries and Zone Maps. Do you know what is zone map?
Let’s have a look into simplified example. We have sales table which is split by 1 Mb IMCU blocks. Each block stores minimal and maximum values for each column so when we need to search sales for California we scans zone maps and check does our value is located in min/max boundaries so we do not scan extra blocks.
I converted SMU zone maps in readable format. As you could see they are stored per-column base as I told before.
You see numeric zone maps and character ones.
Please pay attention that dictionary_entries column contains only zero.
As we understood it happens because no compression were switched on.
Let’s enable the compression
We’ve seen that dictionary extries had been populated which should give dictionary-based pruning in accordance with the documentation
Кстати, не все zone maps одинаково полезны. Как выдумаете какая из них будет абсолютно бесполезна и не будет использоватьс?
We’ve seen that dictionary extries had been populated which should give dictionary-based pruning in accordance with the documentation
Не работает оно чаще всего из-за того, что zone maps дают слишком широкий диапазон. Более того, нет статистик, которые позволяют отличать причину CU pruning – dictionary based или Zone maps
Compression ration is rather high but it depends from you data for sure. You could see that compression doesn’t influence on ingestion so high compression could be used as the default. Frankly speaking loading 6 gigabytes for 1 minute it is to much so we decided to have a look into FastStart feature.
The feature writes InMemory area in a columnar format and once database is rebooted it loads the data in memory with the speed of file reading so we got 3 times loading profit.
Тем не менее вчера я увеличил коэффициент pruning и иногда компрессию использовав как вы думаете что?
Самое обидное, что до этого слайда я дошёл только вчера и проверил его. Само собой я стал гуглить, чтобы убедиться., что я уникален в своём открытии, но оказалось,, что Мария Колган тоже нашла решение с order by, но решение с clustering это моё секретное ноухау.
It is time for rerun. Try to guess which performance profit we got once we used high compression?
Actually no profit at all.
Поэтому придётся разбираться в деталях
В Oracle InMemory очень много умных трансформаций, которые видно в плане и чем более сложные джоины и агрегаты тем более вероятно их увидеть. Одной из них является InMemory aggregation, которая показывается такой умной картинкой. Ключевое для нас в ней то, что создаётся аналог bloom фильтра Key vector и по нему происходит джоин sales с stores и products и при этом считаются суммы в процессе скана sales.
Помните, я показывал в прошлой презентации sql для поиска по нескольким условиям с хитрыми групп бай. Именно для них использовался механизм vector transformation, который задуман для inmemory агрегаций, но как показал наш опыт серьёзно ускоряет джоины на запросах с агрегатами. Так вот всё это на полях типа binary, которые были в джоинах заменялось на блум фильтр, который был медленнее.
Мы попробовали решить эту задачу через так называемые JOING GROUPS. Это аналоги JOIN CLUSTER
Сделали join group.
Кстати, вопрос. Как понять по плану или по статистикам, что join group используются?
А никак! ни в плане ни в статистиках её не видно. Её видно только в Realtime SQL monitor, поэтому пришлось парсить его XML и через него вычислять.
Стали разбираться когда они используются и результат такой. Они используются только тогда, когда в условие джоина есть поля, указанные в join groups, а по нашим запросам это было не всегда возможно.
Changed all types to bigint and removed indexes
We understood that no reasons to store all columns inmemory because we search by them via Oracle Full Text Search indexes
So the final structure is simple data types without b-tree indexes and 2 fields are removed from inmemory storage
We did some performance tests and found sporadic search degradation which happens where records are inserted
Помните, я говорил, что у меня нет презентаций без звоночков!
Let’s understand how Oracle provides read consistency for InMemory
Once a record is changed it is marked as stale. All stale records are read either from buffer cache or disk.
Oracle repopulates stale records by staleness threshold or each 2 minutes by so named trickle repopulation process. It could be done in parallel plus we could state how much resources could be consumed for stale records repopulation
Собственно я думаю эти параметры отвечают за инвалидацию IMCU, но у нас не хватало времени изучить это детально. На самом деле их около 34, но трогать их весьма боязно.
Как же можно понять, что происходит чтение с диска, а не с InMemory?
Очень просто
Эти две статистики должны быть по нулям
За самим процессом пересчёта IMCU можно наблюдать через 2 представления
Тут пойман очень интересный процесс. Начался процесс trickle_repopulation, было обнаружено. Что затронуто очень много данных и произошла полная перезагрузка
Let’s have a look what we have by default
As soon as we set 4 repopulation servers and 8 percent of time to repopulate stale records and performance spikes disappeared
InMemory option is very easily to trace and maintain but let me share a secret information with you.
We needn’t to know all of them at all.
8 parameters covers the best part of InMemory performance tuning.
Let me share one more secret. only 1 parameter is enough. Try to guess which one. It states how efficient zone maps indexes are used.
3 settings covers the best part of dba activities
And 2 views give us enough information for basic InMemory undertanding
Let’s have a look into final figures. As you could see we have about 4-5 times performance profit which isn’t so good but regarding our use case is fine.
Let’s have a look into final figures. As you could see we have about 4-5 times performance profit which isn’t so good but regarding our use case is fine.
Let’s have a look into final figures. As you could see we have about 4-5 times performance profit which isn’t so good but regarding our use case is fine.
Let’s elaborate DBA findings.
InMemory area size should be estimated taking into account compression and your data
If we have all data in memory it doesn’t mean we are extremely fast
InMemory takes time to be loaded after server reboot
8 statistics is enough to help developers with the best part of their issues
Alling tackle processes settings with your workload
Findings for developers are that in case of our usecase extremely advanced inmemory features like join groups, inmemory aggregations hasn’t brought significant performance profit.
Zone maps work perfect with numeric and date data types.
Dictionary based pruning doesn’t work in our oracle version at least.
More simple data types are used – more performance we have
And extremely significant that high compression doesn’t affect ingestion speed
То, что происходит repopulate даже в случае незакомиченной транзакции радикально отличает от result cache.
Faststart
Add some star schema data with star – try what for IM is intended initialy
Tell about cool stuff with multithread scans
Our main conclusion is that you shouldn’t rely on documentation and experts – try and measure your particular use-case
IM works with short queries as well
Without understanding how it works you will not get performance profit
You have to change application to use simple data types
What is good no extra layers, message queues, hardware need to be added
We had managed to put our IM POC in production very fast (1 week actualy)
And business got 5 times performance boost.
Probably that’s it from my side for today.
Thank you for your time and I’m ready answer your questions