Codealike plugins in Visual Studio, Eclipse and Chrome, track developers while they code and perform analytic calculations at the millisecond level. For such write heavy workloads and using RavenDB as the main and only database was not without challenge. In this talk, we will reveal how we built and scaled such a solution, how we were able to improve performance with Voron and glance at our own mistakes and architectural choices down the line.

3. RavenDB Conference 2016
Building Codealike
A journey into the developers analytics world
Federico Lois
federico.lois@corvalius.com
github.com/redknightlois
twitter.com/federicolois

4. What do we do?
• We track how developers write code.
• How interruptions impact developer’s productivity
• What part of the software they worked on.
• An instant (available) extended memory.
• How much they can concentrate.
• How they compare against other developers.
• In the world,
• In their team,
• With themselves (a few months/years ago).

5. What do I mean?
Lets so some actual data!!!

6. What data is stored?
Time
ActivityArtifact

7. Activity
System
Outside the
IDE
Debugging
Building
Coding

8. Artifacts

9. Time

10. + 200,000 Hs.
of detailed activity
+130
countries + 100M
de actividades
+ 8M
de artefactos
Duplicating (roughly)
each 6 months

12. Persistent
Non
persistent
DERIVED
INFORMATION
Volume
We ingest more data than our ability of
“instantly” index it.

13. Timezones

14. Aggregation
Months
Weeks
Days
Hours
Minutes
Seconds
Aggregation
Visualization

15. Aggregation
Months
Weeks
Days
Hours
Minutes
Seconds
Aggregation
Visualization

16. Evil Indexes
public class WebAggregateByHostIndex : AbstractIndexCreationTask<WebActivityCollection, WebActivityAggregateByUser>
{
public WebAggregateByHostIndex()
{
this.MaxIndexOutputsPerDocument = 2048;
Map = activities => from activity in activities
let statesMapping = from state in activity.Activities
let host = state.Url != null ? new Uri(state.Url.ToString()).Host : "unknown"
select new
{
Owner = "membership/account/" + activity.Owner,
Status = state.Status,
Host = host,
StartTicks = state.StartTimeTicks,
DurationTicks = state.DurationTicks,
EndTicks = state.StartTimeTicks + state.DurationTicks,
}
from state in statesMapping
group state by new
{
Owner = state.Owner,
Host = state.Host,
} into g
select new WebActivityAggregateByUser
{
Owner = g.Key.Owner,
Host = g.Key.Host,
MinTimeTicks = g.Min( x => x.StartTicks),
MaxTimeTicks = g.Max( x => x.EndTicks),
Navigation = g.Where(x => x.Status == "navigation").Sum(x => x.DurationTicks),
Debugging = g.Where(x => x.Status == "debugging").Sum(x => x.DurationTicks),
Debugger = g.Where(x => x.Status == "debugger").Sum(x => x.DurationTicks),
NavigationCount = g.Where(x => x.Status == "navigation").Count(),
DebuggingCount = g.Where(x => x.Status == "debugging").Count(),
DebuggerCount = g.Where(x => x.Status == "debugger").Count(),
};
Reduce = results => from result in results
group result by new
{
result.Owner,
result.Host
} into g
select new WebActivityAggregateByUser
{
// This is the bag used on the reduce.
Owner = g.Key.Owner,
Host = g.Key.Host,
// Spatial index to prune the search space.
MinTimeTicks = g.Min(x => x.MinTimeTicks),
MaxTimeTicks = g.Max(x => x.MaxTimeTicks),
// This is the actual data
Navigation = g.Sum(x => x.Navigation),
Debugging = g.Sum(x => x.Debugging),
Debugger = g.Sum(x => x.Debugger),
NavigationCount = g.Sum(x => x.NavigationCount),
DebuggingCount = g.Sum(x => x.DebuggingCount),
DebuggerCount = g.Sum(x => x.DebuggerCount),
};
StoreAllFields(FieldStorage.Yes);
}
}
}

17. 6 month token:
ravendbconf2016

18. Questions?

19. Get in
TOUCH
Let’s do something great together.
info@corvalius.com
@Corvalius
/Corvalius
Av. Federico Lacroze 2352
7th Floor, Buenos Aires, Argentina
T. (+54 11) 4772-0650

20. c o r va l i u s . c o m