Internet Conference 2018: Internet Measurement, how to get the relativities right at scale

Internet Measurement,
how to get the relativities right at scale
George Michaelson
ggm@apnic.net

Weighing the world
one click at a time
George Michaelson
ggm@apnic.net

“weighting” the
world one pixel at a
time
George Michaelson
ggm@apnic.net

“weighting” the
time
George Michaelson
ggm@apnic.net
A reprise of a talk I first gave in 2016 to students from the Ecole Polytechnique Paris

“weighting” the
time
George Michaelson
ggm@apnic.net
A reprise of a talk I first gave in 2016 to students from the Ecole Polytechnique Paris
Because its still a significant problem

How Science works
• Sometimes science is doing experiments and seeing what
happens

How Science works
“an experiment on a bird in an air pump” by Joseph Wright of Derby (1768)

How Science works Next week we can
put your little brother
in the vacuum
chamber

How Science works
• We don’t put tiny birds (or brothers) in vacuum chambers
– We put 1x1 pixels out into the browser and see who can fetch them
– Its not as much fun but it’s a lot less messy
– We’re doing around 10,000,000 experiments/day

How Scientists work
• Sometimes Scientists work by thinking.

How Scientists work
“Et plurima mortis imago” by William Hogarth (1736)

How Scientists work
I wish one of
us had a clue
what was
going on

Science is talking
• So its important we talk about how we measure, get some
sense of the caucus around what measurement is being
done, how it works.
• So this meeting? This is Scienting!
– We’re going to talk about how its done, more than what it is saying.

A typical APNIC Labs talk 1950s
Oculus Rift???

IPv6 or DNSSEC? You decide…
20
I hope you like
The 3D effect.
Headache pill
Warning recommended

The usual APNIC Labs Talk
• We’re talking about worldwide IPv6 uptake, or exploring DNSSEC,
putting stories out there, communicating about the results.
• I’m giving one at JPOPM/IW2018 later this week..
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
2014-01-01
2015-01-01
2016-01-01
2017-01-01
2018-01-01
%IPv6Capable
India
Reliance
Bharti Mobile
Hutchinson
ASAS55836 - RELIANCEJIO-IN Reliance Jio Infocomm
Limited
ASAS38266 - HUTCHVAS-AS Vodafone Essar Ltd.,
Telecommunication - Value Added Services
ASAS45271 - ICLNET-AS-AP Idea Cellular Limited
ASAS45609 - BHARTI-MOBILITY-AS-AP Bharti Airtel Ltd. AS
for GPRS Service
ASAS9829 - BSNL-NIB National Internet Backbone
846 other ASN
%EyeballShare
%IPv6 %IPv4

This talk
• .. Is about how we reflected on what other people are doing
measuring IPv6, and other qualities in the Internet, and
what we changed as a result, and why.
• It’s a look ‘under the covers’ at how we’re doing things.
– No doves, no 3D movies alas.
• It’s about a significant problem we all face measuring the
Internet at scale and what we all need to talk about, to fix it.

This APNIC Labs talk
“when did you last see your father” William Frederick Yeames (1878)

MEPlease,
don’t ask…

Why don’t your
IPv6 figures
agree with
Google

Please, don’t
reply…

Worst Case
30 minutes of
my life I wil
never get
back

APNIC 1x1 tests
• Embedded ads primarily shown in mobile apps, websites
– Written in HTML5
– Shown worldwide continuously
• The HTML5 invokes javascript “getURL()” calls
– Tests DNS, Web fetch asynchronously
– DNS response determines if v4only, v6only, dualstack fetch required
• Packet capture at head for DNS, web traffic
– 120Gb/day of logs

Paying dues
• APNIC 1x1 measurement system 2010->present
• We got our basic techniques from a range of sources
– Emile Aben RIPE NCC (javascript, ideas)
– Tore Andersen
– Jason Fesler
– ???
• Visualisation model shamelessly copied from Cisco!
– Eric Vynke
– Google chart api for timeline, growable model, RESTful API

Eric, Emile.... Data sharing
Herge…
Those clever
APNIC Labs
people have
good IPv6 data!
Its
JSON

Can we do better?
If I could get enough
points of
measurement maybe
they’d agree!

Flaws in the basic model
• Hows that worldwide distribution working out for us?
• Well.. We do get signal of different time-of-day outcomes by
economy:

Weighty matters
Why don’t
your figures
agree with
Google
Does nobody
else weight
their data?

Aha! moment
• Google says 25%
• We say 17%
– Because we adjust for a huge over-count and under-count seen in
some economies, using a single consistent weight
– But (we think) google is comparing un-adjusted numbers to get a
simple total(capable)/total(seen) ratio

• Hows that ”inside each economy” model working for us?
• Well.. We do get some huge variances in the signal per
economy.
• And some economies we probably aren’t measuring right at
all.

IPv6 capable by economy, India, USA
0
20
40
60
80
100
2011-10
2012-02
2012-06
2012-10
2013-02
2013-06
2013-10
2014-02
2014-06
2014-10
2015-02
2015-06
2015-10
2016-02
2016-06
2016-10
2017-02
2017-06
2017-10
2018-02
2018-06
%IPv6Capable
IN
US

India on 10 day sample smoothing.
That’s worrying: we had a huge drop
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
2014-01-01 2015-01-01 2016-01-01 2017-01-01 2018-01-01
%IPv6Capable
Date
India
Reliance
Bharti Mobile
Hutchinson

Whats causing the mis-measure?
CGN/NAT
Device
Moar
IPv6!!
!

Sometimes, its Intermediaries!
CGN/NAT
Device
Moar
IPv6!!
!
How big is
A CGN these
Days?
Maybe it needs
A Cisco Nuclear
Reactor to run?

Whats going on here?
• Modelled on a large smoothing interval, like 3 months, we have
reasonably good overall data
– Down at the day-to-day level, we see huge variances in the signal
• Is this normal?
– Yes. This is normal. Data samples are variant.
• Is this explicable?
– Yes. It is totally about the relative sample counts, by ASN we saw in the interval.
– We saw less overall samples
– We saw more variance in the source of the samples
– Which exposes more underlying variances in how people see the Internet

Our basic model
• Assume random selection of users
– Which is true, broadly speaking. The Advertising industry demands this
• Assume low repeat presentation rate
– Which is only true within limits, and can be hard to model with CGN
• Adjust google advert placement model
– Preference views over clicks
– Avoid keyword selection bias (generic terms)
• But we cannot control device placement, or inter-ASN relativity

• Random selection of users?
– Biassed to some apps because bid-rate so low
– Artificial browser/device selection
• Low Repeat rate?
– Emerging evidence of high repeat rates in some circumstances
• Is this NAT/CGN/Proxy?
• No cookie permitted in google ad framework
– Under low sample count, whats the ‘easiest’ form of uniques?
• Different ASN
• Therefore, under low sample count conditions, we mis-count by ASN
• What is the correct inter-ASN relativity anyway?
– Where would “ground truth” for this come from?

• We have no fix for these at this time
– We need an independent placement regime outside google ads, to
understand the qualities of bias google ads brings to the measurement
• Its possible we could bring a model of the distribution of
NAT/CGN to the table if we could understand how many are out
there
– We had a WebRTC and RTMFP measure but it was too intrusive
– We’re beginning to see signs of short NAT binding lifetime, which may be
a useful signal
• We badly need independent sources of samples about AS
relativities to have some certainty here

Those browser, OS distortions
• Google appear to have models of the ‘expected’ ratio of
mobile device, desktop-device, android/iOS/Windows to be
seen worldwide
– But we know this is probably not globally true
• South America have strong tax/currency barriers in place against Apple, and we
should see significantly more Android than we do
• South Asia, China should see significantly less Apple than we do
– So what can we do?
• Nothing! We don’t have a model for expected ratio of devices

Weight inside the economy?
• Assuming Random placement, eyeball share inside a given
economy is looking like a good approximation for
market(ish) share of eyeballs
– Works in US, GB, FR, DE …
– Doesn’t work in KR. Why?

KR: top-10 by sample count 2016
ASN AS Name IPv6
Capable
IPv6
Preferred
#
Samples
AS4766 KIXS-AS-KR Korea Telecom 0.02% 0.00% 1147955
AS9318 HANARO-AS Hanaro Telecom Inc. 0.02% 0.01% 357278
AS17858 KRNIC-ASBLOCK-AP KRNIC 0.01% 0.00% 255188
AS3786 LGDACOM LG DACOM Corporation 0.01% 0.00% 118949
AS9644 SKTELECOM-NET-AS SK Telecom 23.66% 23.47% 114849
AS17853 LGTELECOM-AS-KR LG Telecom 0.00% 0.00% 70558
AS16509 AMAZON-02 - Amazon.com, Inc. 0.00% 0.00% 43708
AS10036 CNM-AS-KR CM Communication Co.,Ltd. 0.10% 0.10% 34424
AS17864 HANVITIAB-AS-KR Hanvit IB 0.00% 0.00% 13124
AS17839 DREAMPLUS-AS-KR DreamcityMedia 0.01% 0.00% 9416

KR: top-10 by sample count 2018
ASN AS Name IPv6
Capable
IPv6
Preferred
#
Samples
AS4766 KIXS-AS-KR Korea Telecom 2.27% 2.19% 4774829
AS9318 SKB-AS SK Broadband Co Ltd 0.01% 0.00% 2106803
AS9644 SKTELECOM-NET-AS SK Telecom 67.94% 41.49% 1496829
AS17858 POWERVIS-AS-KR LG POWERCOMM 0.00% 0.00% 1364276
AS17853 LGTELECOM-AS-KR LGTELECOM 6.25% 5.57% 915738
AS3786 LGDACOM LG DACOM Corporation 0.01% 0.01% 324091
AS10036 CNM-AS-KR DLIVE 0.04% 0.03% 142122
AS17839 DREAMPLUS-AS-KR CJ Hello Co., Ltd. 0.00% 0.00% 46995
AS9845 CJCKN-AS-KR CJ Hello Co., Ltd. 0.00% 0.00% 39747
AS9770 SPEEDONSTV-AS-KR CJ Hello Co., Ltd. 0.00% 0.00% 25728

…Because google is seeking devices?
• Maybe the KR market cannot deliver iOS, Android at levels
expected
– 50%+ of the Internet is on mobile/cellular
– Significantly less than expected is on home cable
– Huge deployment of CPE in home behind old NAT model
• For whatever reason, the KR eyeball share is severely distorted,
on googles advert feed
• Even though SK Telecom is now higher by samplecount, its still
massively under-represented according to industry insiders who
know the relative market share.

..so we need to find an independent model
• IF we had a model of relative weight per ASN
– We could re-weight the data per economy by ASN relative weight, as
we do for Economies to Regions/World
• IF we had a model of relative weight per device/OS per
economy
– We could re-weight the data per economy yadda yadda
• So it looks like what we want, is good weighting models.
Parametric weighting models we can agree on.

If models, what parameters?
• By Government Fiat or Regulation ?
– Painful, but a statutory reporting shedule might help
– Eg G20/OECD quarterly reports on subscribers?
• By indirect methods
– Packetflows which show relationships to subscribers
– Volumes seen at IX sensitive to subscriber volume
• This is potentially tractable but invasive
• Off-exchange traffic would distort severely (private peering)
• Or.. Find other sources of data and try and account for the
distortions.

If models, what parameters?
• By Government Fiat or Regulation ?
– Painful, but a statutory reporting shedule might help
– Eg G20/OECD quarterly reports on subscribers?
• By indirect methods
– Packetflows which show relationships to subscribers
– Volumes seen at IX sensitive to subscriber volume
• This is potentially tractable but invasive
• Off-exchange traffic would distort severely (private peering)
• Or.. Find other sources of data and try and account for the
distortions. Which.. Is kind-of what Emile and I are doing.

Oh look: they publish statistics …

How important is the Internet to
economics?
• If the internet underpins worldwide trade, and consumer
activity…
– Don’t we all need the best possible data about relative usage, customers
to understand what we’re seeing?
– Yes, this is potentially market-moving, non-neutral information
– But.. What if we need it anyway?
• If we don’t agree to do this amongst ourselves, will we face it
anyway?
– OECD Broadband reporting obligations
– UN & other treaty body agreements on reporting
• Where can we get this kind of data?

Maybe we need new questions?
• How many devices per person is ‘normal’ now?
– I’m carrying 5. I left another 2-3 at home.
• They’re switched on
– Japan, Korea, China is now 2.x per person
• Web services mean web measures may be measuring
applications which never see a human

Maybe we need new questions?
• How many NATs and CGNS?
– Unpublished measurements in webRTC() suggest that over 95% of
all end users lie behind some kind of NAT
– Emerging volumes of ICMP which may relate to the ubiquity of CGN
and NAT with short binding lifetime
• Number of devices per IP seen on the Internet
– The {src, dst, port…..} 5-Tuple is now very important

Telescopes on the size of IPv6
If I could get enough
KINDS of
measurement maybe
they’d CONVERGE!

Maybe we need new goals?
• Find independent sources for the qualities of the global
Internet we can agree on in parametric terms:
– Spans economies
– Spans device types
– Can supply volume, random eyes at scale
• CDN backed services?
• Wikipedia? What are the ubiquitous services?
• Industry neutral. Other major industries have reporting lines
– Or.. Are we driving to statutory reporting?

Udon or
Ramen?
If we put the pigeon
inside the CGN do
we get soup?

Where to from here?
• We are exploring a neutral-point, scale-free (relative) measure of
inter-ASN relativity, as seen by somewhere which has a lot of
global views of end users
– A mid-range CDN with global presence
• We are interested in reaching out to the wider ISP community to
find ways to report end user, device, active counts
– Looking to other industry groups, some kind of reporting is normal
• If we can de-bias the samples in measurement, the upside is
huge
– Better, more reliable information for investment planning
– “Build a bigger market”

We’re going to keep on
measuring.

Internet Conference 2018: Internet Measurement, how to get the relativities right at scale

Recommended

Recommended

More Related Content

Similar to Internet Conference 2018: Internet Measurement, how to get the relativities right at scale

Similar to Internet Conference 2018: Internet Measurement, how to get the relativities right at scale (20)

More from APNIC

More from APNIC (20)

Recently uploaded

Recently uploaded (20)

Internet Conference 2018: Internet Measurement, how to get the relativities right at scale