Ekanite

10/15/2016 Ekanite
http://127.0.0.1:3999/gosf-ekanite/ekanite.slide#1 1/24
Ekanite
The syslog server with built-in search
Philip O'Toole
GoSF October 19th 2016

10/15/2016 Ekanite
About me
Director of Data Platform Engineering at Percolate.
Previously Director of Engineering and core developer with In uxDB.
Led the backend team that built Loggly's 2nd generation indexing and search platform.
Big fan of Go, databases, and distributed systems.

10/15/2016 Ekanite
About Ekanite
The goal of Ekanite is to do a couple of things, and do them well -- accept log messages over
the network, and make it easy to search the messages. What it lacks in feature, it makes up
for in focus.
You can nd the source code at github.com/ekanite/ekanite(https://github.com/ekanite/ekanite).
The current release is v1.1.1.

10/15/2016 Ekanite
Why?

10/15/2016 Ekanite
Why another log search system?
Built very large scale log search systems in the past.
It can be quite involved - networking, indexing, search, retention, sharding, and
performance.
Could it all be done in a single program?
Single binary Go program would make deployment really easy.
It would act as a detailed demonstration of building such a system.
New use case for bleve.

10/15/2016 Ekanite
What is bleve?

10/15/2016 Ekanite
bleve
bleve is an indexing and full-text search library for Go.
Supports text analysis and faceting.
Straightforward to use.
funcmain(){
//openanewindex...ignoringallerrors.
mapping:=bleve.NewIndexMapping()
index,err:=bleve.New("example.bleve",mapping)
//indexsomedata
err=index.Index(identifier,"hello,world!")
//searchforsometext
query:=bleve.NewMatchQuery("text")
search:=bleve.NewSearchRequest(query)
searchResults,err:=index.Search(search)
}
bleve is to Go what Lucene is to Java.

10/15/2016 Ekanite
The design of a log search system

10/15/2016 Ekanite
Ekanite architecture
The Ekanite engine receives log messages and routes to indexes.
The engine also receives queries, performs searches, and returns results.

10/15/2016 Ekanite
Indexing by time
And index is a logical concept, grouping physical bleve-based shards.
Indexing by time makes search quicker, and retention easier to enforce.
This diagram shows 3 particular hours, but time extends in both directions forever.

10/15/2016 Ekanite
Why do we batch? Why do we shard?

10/15/2016 Ekanite
Demo

10/15/2016 Ekanite
Go patterns in action

10/15/2016 Ekanite
Decoupling input and indexing
Input subsystem accepts a bu ered channel, to which it sends parsed log lines.
//input/collector.go
func(s*TCPCollector)handleConnection(connnet.Conn,cchan<-*Event){
.
.
//Loglineavailable?
ifmatch{
stats.Add("tcpEventsRx",1)
ifs.parser.Parse(bytes.NewBufferString(log).Bytes()){
c<-&Event{
Text: string(s.parser.Raw),
Parsed: s.parser.Result,
ReceptionTime:time.Now().UTC(),
Sequence: atomic.AddInt64(&sequenceNumber,1),
SourceIP: conn.RemoteAddr().String(),
}
}
}
.
.
}

10/15/2016 Ekanite
Parallel indexing
//engine.go
func(e*Engine)Index(events[]*Event)error{
.
.
//De-multiplexthebatchintosub-batches,onesub-batchforeachIndex.
subBatches:=make(map[*Index][]Document,0)
.
.
//Indexeachbatchinparallel.
forindex,subBatch:=rangesubBatches{
wg.Add(1)
gofunc(i*Index,b[]Document){
deferwg.Done()
i.Index(b)
}(index,subBatch)
}
wg.Wait()
.
.
}

10/15/2016 Ekanite
Parallel sharding
//index.go
//Indexindexesthesliceofdocumentsintheindex.Ittakescareofallshardrouting.
func(i*Index)Index(documents[]Document)error{
varwgsync.WaitGroup
shardBatches:=make(map[*Shard][]Document,0)
for_,d:=rangedocuments{
shard:=i.Shard(d.ID())
shardBatches[shard]=append(shardBatches[shard],d)
}
//Indexeachbatchinparallel.
forshard,batch:=rangeshardBatches{
wg.Add(1)
gofunc(s*Shard,b[]Document){
deferwg.Done()
s.Index(b)
}(shard,batch)
}
wg.Wait()
returnnil
}

10/15/2016 Ekanite
Results returned over a channel
//engine.go
//Searchperformsasearch.
func(e*Engine)Search(querystring)(<-chanstring,error){
e.mu.RLock()
defere.mu.RUnlock()
c:=make(chanstring,1)
gofunc(){
//Sequentiallysearcheachindex,startingwiththeearliestintime.
//Thiscouldbedoneinparallelbutmoresortingwouldberequired.
fori:=len(e.indexes)-1;i>=0;i--{
e.Logger.Printf("searchingindex%s",e.indexes[i].Path())
ids,_:=e.indexes[i].Search(query)
for_,id:=rangeids{
b,_:=e.indexes[i].Document(id)
c<-string(b)
}
}
close(c)
}()
returnc,nil
}

10/15/2016 Ekanite
Sorting hits by time
Ekanite performs time-based sorting in the application.
Earlier versions of bleve did not support sorting on custom elds.
Newer versions now do.
Complex sort method on DocIDs.
//Searchperformsasearchoftheindexusingthegivenquery.ReturnsIDsofdocuments
//whichsatisfyallqueries.ReturnsDocIDsinsortedorder,ascending.
func(i*Index)Search(qstring)(DocIDs,error){
query:=bleve.NewQueryStringQuery(q)
searchRequest:=bleve.NewSearchRequest(query)
searchRequest.Size=maxSearchHitSize
searchResults,_:=i.Alias.Search(searchRequest)
docIDs:=make(DocIDs,0,len(searchResults.Hits))
for_,d:=rangesearchResults.Hits{
docIDs=append(docIDs,DocID(d.ID))
}
sort.Sort(docIDs)
returndocIDs,nil
}

10/15/2016 Ekanite
Retention enforcement is straightforward
//engine.go
//runRetentionEnforcementperiodicallyrunsretentionenforcement.
func(e*Engine)runRetentionEnforcement(){
defere.wg.Done()
for{
select{
case<-e.done:
return
case<-time.After(RetentionCheckInterval):
stats.Add("retentionEnforcementRun",1)
e.enforceRetention()
}
}
}
Shards are deleted from disk and references removed from the engine.

10/15/2016 Ekanite
Next steps
Ekanite is software, and software is never nished.
Use storage engine other than BoltDB.
Performance improvements, both CPU and RAM.
Better query language support
Proper dependency management.
A fully-featured CLI.

10/15/2016 Ekanite
What Ekanite can do
With it you've got an easy-to-deploy and maintain log search system.

10/15/2016 Ekanite
References
www.philipotoole.com/designing-a-search-system-for-log-data-part-i(http://www.philipotoole.com/designing-a-
search-system-for-log-data-part-i)
github.com/ekanite/ekanite(https://github.com/ekanite/ekanite)
www.blevesearch.com(http://www.blevesearch.com)
github.com/blevesearch/bleve(https://github.com/blevesearch/bleve)
www.philipotoole.com/increasing-bleve-performance-sharding(http://www.philipotoole.com/increasing-bleve-
performance-sharding)
github.com/otoolep/bleve-bench(https://github.com/otoolep/bleve-bench)
github.com/ekanite/gosf-ekanite(https://github.com/ekanite/gosf-ekanite)

10/15/2016 Ekanite
Thank you
Philip O'Toole
GoSF October 19th 2016
http://www.philipotoole.com/(http://www.philipotoole.com/)
@general_order24(http://twitter.com/general_order24)

10/15/2016 Ekanite

Ekanite

Recommended

Recommended

More Related Content

Similar to Ekanite

Similar to Ekanite (20)

Recently uploaded

Recently uploaded (20)

Ekanite