Change is a routine in software development. Like any system, libraries also evolve over time. As a consequence, clients are compelled to update and, thus, benefit from the available API improvements. However, some of these API changes may break contracts previously established, resulting in compilation errors and behavioral changes. In this paper, we study a set of questions regarding API breaking changes. Our goal is to measure the amount of breaking changes on real-world libraries and its impact on clients at a large-scale level. We assess (i) the frequency of breaking changes, (ii) the behavior of these changes over time, (iii) the impact on clients, and (iv) the characteristics of libraries with high frequency of breaking changes. Our large-scale analysis on 317 real-world Java libraries, 9K releases, and 260K client applications shows that (i) 14.78% of the API changes break compatibility with previous versions, (ii) the frequency of breaking changes increases over time, (iii) 2.54% of their clients are impacted, and (iv) systems with higher frequency of breaking changes are larger, more popular, and more active. Based on these results, we provide a set of lessons to better support library and client developers in their maintenance tasks.

1. Historical and Impact Analysis of API
Breaking Changes: A Large-Scale Study
Laerte Xavier, Aline Brito, André Hora, Marco Tulio Valente
Federal University of Minas Gerais, Brazil
SANER 2017

2. 2
API

3. APIs are constantly
changing!
3

4. 4

5. 5
default Query<R> setHibernateFlushMode(FlushMode flushMode) {
setFlushMode( flushMode );
return this;
}
default org.hibernate.query.Query<R> setHibernateFlushMode(FlushMode
flushMode) {
setFlushMode( flushMode );
return (org.hibernate.query.Query) this;
}
hibernate/hibernate-orm
Version 5.2.0

6. 6Version 5.2.0
default org.hibernate.query.Query<R> setHibernateFlushMode(FlushMode
flushMode) {
setFlushMode( flushMode );
return (org.hibernate.query.Query) this;
}

7. 7
Breaking Changes
Elements removal
Method signature changes
Non-Breaking
Changes
Changes in deprecated elements
Elements addition
Elements deprecationField changes
(Dig and Johnson, ICSM, 2006)

8. 8
Related Work
Hora and Valente, ICSME, 2015
Hora et al., CSMR, 2014
Henkel and Diwan, ICSE, 2005
Raemaekers et al., ICSM, 2012
API Evolution Change Impact
Brito et al., SANER, 2016
McDonnel et al., ICSE, 2013

9. Trabalhos Relacionados
Hora and Valente, 2015;
Hora et al., 2014;
Henkel and Diwan, 2005.
Raemaekers et al., 2012;
Evolução de APIs Impacto de Mudanças
Brito et al., 2016;
McDonnel et al., 2013.
Open Questions
9
To what extent are clients affected by
API breaking changes?

10. Trabalhos Relacionados
Hora and Valente, 2015;
Hora et al., 2014;
Henkel and Diwan, 2005.
Raemaekers et al., 2012;
Evolução de APIs Impacto de Mudanças
Brito et al., 2016;
McDonnel et al., 2013.
To what extent are clients affected by
API breaking changes?
Open Questions
10
Is this a problem only faced by newer
libraries?

11. 11
RQ1: What is the frequency of API breaking changes?
RQ2: How do API breaking changes evolve over time?
RQ3: What is the impact of API breaking changes in
client applications?
RQ4: What are the characteristics of libraries with high
and low frequency of breaking changes?

12. 12
TOP-317
GITHUB JAVA LIBRARIES

13. 13
TOP-317
1,792
STARS
GITHUB JAVA LIBRARIES

14. 14
TOP-317
1,792
STARS
15
RELEASES
GITHUB JAVA LIBRARIES

15. 15
TOP-317
1,792
STARS
15
RELEASES
3.4
YEARS
GITHUB JAVA LIBRARIES

16. 16
RQ1: What is the frequency of API breaking changes?
RQ2: How do API breaking changes evolve over time?
RQ3: What is the impact of API breaking changes in
client applications?
RQ4: What are the characteristics of libraries with high
and low frequency of breaking changes?

17. 17
...
1 2 NN-1
Design
3 N-2
First
Commit
August
2016

18. 18
...
1 2 NN-1
Design
3 N-2
First
Commit
August
2016
DIFF

19. 19
501,645
LIBRARY
CHANGES
28%
BREAKING
CHANGES

20. 20
501,645
LIBRARY
CHANGES
28%
BREAKING
CHANGES

21. 21
501,645
LIBRARY
CHANGES
28%
BREAKING
CHANGES

22. 22
Libraries often break
backward compatibility!

23. 23
RQ1: What is the frequency of API breaking changes?
RQ2: How do API breaking changes evolve over time?
RQ3: What is the impact of API breaking changes in
client applications?
RQ4: What are the characteristics of libraries with high
and low frequency of breaking changes?

24. 24
...
1 2 NN-1
Design
3 N-2
First
Commit
August
2016

25. 25
...
1 2 NN-1
Design
3 N-2
First
Commit
August
2016
DIFF

26. DIFF
26
...
1 2 NN-1
Design
3 N-2
First
Commit
August
2016
DIFF

27. DIFFDIFF
27
...
1 2 NN-1
Design
3 N-2
First
Commit
August
2016
DIFF

28. DIFFDIFFDIFFDIFFDIFFDIFF
28
1
...
2 NN-1
Design
3 N-2
First
Commit
August
2016
DIFF

29. 29
9,329
RELEASES
5
YEARS

30. 30
9,329
RELEASES YEARS
20%
5

31. 31
Breaking changes frequency
increases over time!

32. 32
RQ1: What is the frequency of API breaking changes?
RQ2: How do API breaking changes evolve over time?
RQ3: What is the impact of API breaking changes in
client applications?
RQ4: What are the characteristics of libraries with high
and low frequency of breaking changes?

33. DIFF
33
NN-1
Design
JAVALI
263,425 Projects
16,386,193
Files
http://java.labsoft.dcc.ufmg.br/javali
Boa

34. 34
140,460
BREAKING CHANGES
16,291
CHANGED TYPES
1,290TYPES WITH
POSSIBLE IMPACT

35. 35
140,460
BREAKING CHANGES
16,291
CHANGED TYPES
1,290TYPES WITH
POSSIBLE IMPACT

36. 36
Most breaking changes
do not have a massive
impact on clients!

37. 37
RQ1: What is the frequency of API breaking changes?
RQ2: How do API breaking changes evolve over time?
RQ3: What is the impact of API breaking changes in
client applications?
RQ4: What are the characteristics of libraries with high
and low frequency of breaking changes?

38. Design
38

39. Design
Top (58)
Bottom (58)
0% BC
100%
BC
39

40. Design
Top (58)
Bottom (58)
0% BC
100%
BC
(Mann-Whitney U
Test & Cliff’s Delta)
40

41. Bold: p-value < 0.05 (statistically significant different) and
d > 0.147 (at least small size effect)
41

42. Bold: p-value < 0.05 (statistically significant different) and
d > 0.147 (at least small size effect)
42

43. 43
Development and social
coding measures are
associated with API
breaking changes!

44. Trabalhos Relacionados
Hora and Valente, 2015;
Hora et al., 2014;
Henkel and Diwan, 2005.
Raemaekers et al., 2012;
Evolução de APIs Impacto de Mudanças
Brito et al., 2016;
McDonnel et al., 2013.
To what extent are clients affected by
API breaking changes?
Open Questions
44
Is this a problem only faced by newer
libraries?

45. 45
Libraries often break backward compatibility

46. 46
Libraries often break backward compatibility
Breaking changes frequency increases over time

47. 47
Libraries often break backward compatibility
Breaking changes frequency increases over time
Most breaking changes do not have a massive
impact on clients

48. 48
Libraries often break backward compatibility
Breaking changes frequency increases over time
Most breaking changes do not have a massive
impact on clients
Development and social coding measures
are associated with API breaking changes

49. 49
Would you need tools to
measure the impact of
breaking changes?
As an API developer...

50. 50
Would you use stability
metrics to choose an API?
Would you need tools to
measure the impact of
breaking changes?
As an API developer...
As an API client...

51. Historical and Impact Analysis of API
Breaking Changes: A Large-Scale Study
Laerte Xavier, Aline Brito, André Hora, Marco Tulio Valente
Federal University of Minas Gerais, Brazil
SANER 2017