The TV is Smart and Full of Trackers: Measuring Smart TV Advertising and Tracking - Privacy Enhancing ...
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Proceedings on Privacy Enhancing Technologies ; 2020 (2):129–154
Janus Varmarken† *, Hieu Le† , Anastasia Shuba, Athina Markopoulou, and Zubair Shafiq
The TV is Smart and Full of Trackers:
Measuring Smart TV Advertising and Tracking
Abstract: In this paper, we present a large-scale mea-
surement study of the smart TV advertising and track-
1 Introduction
ing ecosystem. First, we illuminate the network behav-
Smart TV adoption has steadily grown over the last few
ior of smart TVs as used in the wild by analyzing net-
years, with more than 37% of US households owning at
work traffic collected from residential gateways. We find
least one smart TV in 2018, a 16% increase over the
that smart TVs connect to well-known and platform-
previous year [1]. This growth is driven by two trends.
specific advertising and tracking services (ATSes). Sec-
First, over-the top (OTT) video streaming services such
ond, we design and implement software tools that sys-
as Hulu and Netflix have become popular, with more
tematically explore and collect traffic from the top-1000
than 28 million and 60 million subscribers in the US, re-
apps on two popular smart TV platforms, Roku and
spectively [2]. Second, smart TV solutions are available
Amazon Fire TV. We discover that a subset of apps
at relatively affordable prices, with many of the external
communicate with a large number of ATSes, and that
smart TV boxes/sticks priced less than $50, while built-
some ATS organizations only appear on certain plat-
in smart TVs now cost only a few hundreds dollars [3].
forms, showing a possible segmentation of the smart
As a result, a diverse set of smart TV platforms have
TV ATS ecosystem across platforms. Third, we evaluate
emerged, and have been integrated into various smart
the (in)effectiveness of DNS-based blocklists in prevent-
TV products. For example, Apple TV integrates tvOS,
ing smart TVs from accessing ATSes. We highlight that
and TCL and Sharp TVs integrate Roku.
even smart TV-specific blocklists suffer from missed ads
Many of these platforms have their own respective
and incur functionality breakage. Finally, we examine
app store, where the vast majority of smart TV apps
our Roku and Fire TV datasets for exposure of person-
are ad-supported [4]. OTT advertising, which includes
ally identifiable information (PII) and find that hun-
smart TV, is expected to increase by 40% to $2 billion in
dreds of apps exfiltrate PII to third parties and plat-
2018 [5]. Roku and Fire TV are two of the leading smart
form domains. We also find evidence that some apps
TV platforms in number of ad requests [6]. Despite their
send the advertising ID alongside static PII values, ef-
increasing popularity, the advertising and tracking ser-
fectively eliminating the user’s ability to opt out of ad
vices (“ATSes”) on smart TVs are currently not well
personalization.
understood by users, researchers, and regulators. In this
Keywords: Smart TV; privacy; tracking; advertising; paper, we present one of the first large-scale measure-
blocklists ment studies of the emerging smart TV advertising and
DOI 10.2478/popets-2020-0021
tracking ecosystem.
Received 2019-08-31; revised 2019-12-15; accepted 2019-12-16. In the Wild Measurements (§3). First, we analyze
the network traffic of smart TV devices in the wild. We
instrument residential gateways of 41 homes and collect
flow-level summary logs of the network traffic gener-
*Corresponding Author: Janus Varmarken† : University ated by 57 smart TVs from seven different platforms.
of California, Irvine, E-mail: jvarmark@uci.edu The comparative analysis of network traffic by differ-
Hieu Le† : University of California, Irvine, E-mail: ent smart TV platforms uncovers similarities and differ-
hieul@uci.edu. († The first two authors made equal contribu-
ences in their characteristics. As expected, we find that
tions and share the first authorship).
Anastasia Shuba: Broadcom Inc. (The author was a student a substantial fraction of the traffic is related to popular
at the University of California, Irvine at the time the work was video streaming services such as Netflix and Hulu. More
conducted), E-mail: ashuba@uci.edu importantly, we find generic as well as platform-specific
Athina Markopoulou: University of California, Irvine, E- ATSes. Although realistic, the in the wild dataset does
mail: athina@uci.edu
not provide app-level visibility, i.e., we cannot deter-
Zubair Shafiq: University of Iowa, E-mail: zubair-
shafiq@uiowa.edu
mine which apps generate traffic to ATSes. To address
this limitation, we undertake the following major effort.The TV is Smart and Full of Trackers 130
Controlled Testbed Measurements (§4). We de- by multiple apps (“app prevalence”) and keyword search
sign and implement two software tools, Rokustic for (based on ATS related words like “ads” and “measure”).
Roku and Firetastic for Amazon Fire TV, which sys- PII Exposures (§6). We further examine the net-
tematically explore apps and collect their network traf- work traces of our testbed datasets and find that hun-
fic. We use Rokustic and Firetastic to exercise the top- dreds of apps exfiltrate personally identifiable informa-
1000 apps of their respective platforms, and refer to the tion (PII) to third parties and platform-specific parties,
collected network traffic as our testbed datasets. We an- mostly for non-functional advertising and tracking pur-
alyze the testbed datasets w.r.t. the top Internet des- poses. Alarmingly, we find that many apps send the ad-
tinations the apps contact, at the granularity of fully vertising ID alongside static PII values such as the de-
qualified domain names (FQDNs), effective second level vice’s serial number. This eliminates the user’s ability
domains (eSLDs), and organizations. We use “domain”, to opt out of personalized advertisements by resetting
“endpoint”, and “destination” interchangeably in place the advertising ID, since the ATS can simply link an old
of FQDN and eSLD when the distinction is clear from advertising ID to its new value by joining on the serial
the context. We further separate destinations as first, number. We evaluate the blocklists’ ability to prevent
third, and platform-specific party, w.r.t. to the app that exposures of PII and find that they generally perform
contacts them. well for Roku, but struggle to prevent exfiltration of the
First, we find that the majority of apps contact few device’s serial number and the device ID to third parties
ATSes, while about 10% of the apps contact a large and the platform-specific party for Fire TV.
number of ATSes. Interestingly, many of these more
Contributions. In this paper, we analyze the network
concerning apps come from a small set of developers.
behavior of smart TVs, both in the wild and in the lab.
Second, we find what appears to be a segmentation of
Our contributions include the following: (1) providing
the smart TV ATS ecosystem across Roku and Fire TV
an in-depth comparative analysis of the ATS ecosystems
as (1) the two datasets have little overlap in terms of
of Roku, Fire TV, and Android; (2) illuminating the
ATS domains; (2) some third party ATSes are among
key players within the Roku and Fire TV ATS ecosys-
the key players on one platform, but completely absent
tems by mapping domains to eSLDs and parent orga-
on the other; and (3) apps that are present on both plat-
nizations; (3) evaluating the effectiveness and adverse
forms have little overlap in terms of the domains they
effects of an extensive set of blocklists, including smart
contact. Third, we compare the top third party ATS
TV specific blocklists; (4) instrumenting long experi-
domains of the testbed datasets to those of Android.
ments per app to uncover approximately twice as many
Evaluation of DNS-Based Blocklists (§5). Users domains as [11]; and (5) making our tools, Rokustic and
typically rely on DNS-based blocking solutions such as Firetastic, and our testbed datasets available [12].
Pi-hole [7] to prevent in-home devices such as smart
Outline. The structure of the rest of the paper is as fol-
TVs from accessing ATSes. Thus, we evaluate the effec-
lows. Section 2 provides background on smart TVs and
tiveness of DNS-based blocklists, selecting those that
reviews related work. Section 3 presents the in the wild
are most relevant to smart TVs. Specifically, we ex-
measurement and analysis of 57 smart TVs from seven
amine and test four popular blocklists: (1) Pi-hole De-
different platforms in 41 homes. Section 4 presents our
fault blocklist (PD) [7], (2) Firebog’s recommended ad-
systematic testing approach and comparative analysis
vertising and tracking lists (TF) [8], (3) Mother of all
of the top-1000 Roku and Fire TV apps. Section 5 eval-
Ad-Blocking (MoaAB) [9], and (4) StopAd’s smart TV
uates four well-known DNS-based blocklists and shows
specific blocklist (SATV) [10]. Our comparative analysis
their limitations through analysis of missed ATSes and
shows that block rates vary, with Firebog having the
app breakage. Section 6 investigates exfiltration of PII.
highest coverage across different platforms and StopAd
Section 7 concludes the paper, discusses limitations, and
blocking the least. We further investigate potential false
outlines directions for future work. The appendix pro-
negatives (FN) and false positives (FP). We discover
vides additional details and results, as well as an evalu-
that blocklists miss different ATSes (FN), some of which
ation and discussion of the limitations of our approach.
are missed by all blocklists, while more aggressive block-
lists can suffer from false positives that result in break-
ing app functionality. We discuss two ways to discover
false negatives, through observing domains contactedThe TV is Smart and Full of Trackers 131
the main ways for smart TV platforms and app devel-
2 Background & Related Work opers to generate revenue [4]. Roku’s advertising rev-
enue exceeded $250 million in 2018 and is expected to
Background on Smart TVs. A smart TV is es-
more than double by 2020 [13]. Both Roku and Fire TV
sentially an Internet-connected TV that has apps for
take a 30% cut of the advertising revenue from apps on
users to stream content, play games, and even browse
their platforms [14]. The smart TV advertising ecosys-
the web. There are two types of smart TV products
tem mirrors many aspects of the web advertising ecosys-
in the market: (1) built-in smart TVs, and (2) exter-
tem. Most importantly, smart TV advertising uses pro-
nal smart TV boxes/sticks, also referred to as over-the-
grammatic mechanisms that allow apps to sell their ad
top (OTT) streaming devices. Some TV manufacturers,
inventory in an automated fashion using behavioral tar-
such as Samsung and Sony, offer TVs with built-in smart
geting [15, 16].
TV functionality. While several others provide external
The rapidly growing smart TV advertising and asso-
box/stick solutions, such as Roku (by Roku), Fire TV
ciated tracking ecosystem has already warranted privacy
(by Amazon), and Apple TV (by Apple), that convert a
and security investigations into different smart TV plat-
regular TV into a smart TV. In addition, hybrid prod-
forms. Consumer Reports examined privacy policies of
ucts exist, as some TV manufacturers now integrate ex-
various smart TV platforms including Roku, LG, Sony,
ternal box/stick solutions directly into their smart TVs.
and Vizio [17]. They found that privacy policies are of-
For example, TCL and Sharp offer smart TVs that inte-
ten challenging to understand and it is difficult for users
grate Roku TV, while Insignia and Toshiba offer smart
to opt out of different types of tracking. For instance,
TVs that integrate Fire TV. We use “smart TV” as an
many smart TVs use Automatic Content Recognition
umbrella term for TVs with built-in smart TV function-
(ACR) to track their users’ viewing data and use it to
ality and OTT streaming devices.
serve targeted ads [18]. Vizio paid $2.2 million to settle
There is a diverse set of smart TV platforms, each
the charges by the Federal Trade Commission (FTC)
with its own set of apps that users can install on their
that they were using ACR to track users’ viewing data
TVs. Many smart TVs use an Android-based operating
without their knowledge or consent [19]. While smart
system (e.g., Sony, AirTV, Philips) or a modified version
TV platforms now allow users to opt out of such track-
of it (e.g., Fire TV). Regular Android TVs have access
ing, it is not straightforward for users to turn it off [20].
to apps from the Google Play Store, while Fire TV has
Further, even with ACR turned off, users still must
its own app store controlled by Amazon. In both cases,
agree to a basic privacy policy that asks for the right to
applications for such TVs are built in a manner simi-
collect data about users’ location, choice of apps, etc.
lar to regular Android applications. Likewise, Apple TV
apps are built using technologies and frameworks that Related Work. While the desktop [21–23] and mo-
are also available for iOS apps, and both types of apps bile [24–26] ATS ecosystems have been thoroughly stud-
can be downloaded from Apple’s App Store. ied, the smart TV ATS ecosystem has not been exam-
Some smart TV platforms are distinct as compared ined at scale until recently.
to traditional Android or iOS. For example, Samsung Three concurrent papers studied the network be-
smart TV apps are built for their own custom platform havior and privacy implications of smart TVs [11, 27,
called Tizen and are downloadable from the Tizen app 28]. Ren et al. [27] studied a large set of IoT de-
store. Likewise, applications for the Roku platform are vices, spanning multiple device categories. Their re-
built using a customized language called BrightScript, sults showed that smart TVs were the category of de-
and are accessible via the Roku Channel Store. Yet an- vices that contacted the largest number of third parties,
other line of smart TVs such as LG smart TV and Hy- which further motivates our in-depth study of the smart
brid broadcast broadband TV (HbbTV) follow a web- TV ATS ecosystem. Huang et al. [28] used crowdsourc-
based ecosystem where applications are developed using ing to collect network traffic for IoT devices in the wild
HTML, CSS, and JavaScript. Finally, some smart TV and showed that smart TVs contact many trackers by
platforms, such as Chromecast, do not have app stores matching the contacted domains against the Disconnect
of their own, but are only meant to “cast” content from blocklist. Finally, Moghaddam et al. [11] also instru-
other devices such as smartphones. mented the Roku and Fire TV platforms to map the
As with mobile apps, smart TV apps can integrate ATS endpoints and the exposure of PII. Our work in-
third-party libraries and services, often for advertising dependently confirms the findings of these works w.r.t.
and tracking purposes. Serving advertisements is one of the smart TV ATS ecosystem, both by analyzing sevenThe TV is Smart and Full of Trackers 132
different smart TV platforms in the wild and by per- example, hulu.com belongs to the Walt Disney Com-
forming systematic tests of two platforms (Roku and pany and youtube.com belongs to Alphabet.
Fire TV) in the lab. In addition, we further contribute App-Level Party Categorization. The app-level vis-
along two fronts. First, we show that even the same app ibility in our testbed experiments (Sec. 4) enables cate-
across different smart TV platforms contact different gorization of an Internet destination as a first party or a
ATSes, which shows the fragmentation of the smart TV third party w.r.t. app generating the traffic. We provide
ATS ecosystem. Second, we evaluate the effectiveness of an overview of the technique here and defer details to
different sets of blocklists, including smart TV specific Appendix C.1.
blocklists, in terms of their ability to prevent ads and We adopt a technique similar to prior work [24], and
their adverse effects on app functionality. We also sug- we augment it to also include a platform-specific party
gest ways to aid blocklist curation through analysis of for traffic to platform-related destinations. We match
domain usage across apps and PII exposures. tokenized eSLDs with tokenized package/app names
Earlier work in this space includes [29] by Ghiglieri and developer names. If the tokens match, we label the
and Tews, who studied how broadcasting stations could domain as first party. Otherwise, if the traffic originated
track viewing behavior of users in the HbbTV plat- from platform activity rather than app activity, we la-
form. In contrast to the rich app-based platforms we bel it as platform-specific party: for Fire TV, AntMoni-
study, the HbbTV platform studied in [29] contained tor [37] labels connections with the responsible process;
only one HbbTV app that uses HTML5-based overlays for Roku, we check if the eSLD contains “roku”. Other-
to provide interactive content. Related to our work, they wise, if the domain is contacted by at least two different
found that the HbbTV app loaded third-party tracking apps from different developers, we label it as third party.
scripts from Google Analytics. Malkin et al. [30] sur- Lastly, we resort to labeling it as other to capture do-
veyed 591 U.S. Internet users about their expectations mains that are only contacted by a single app.
on data collection by smart TVs. They found that users
would rather enjoy new technology than worry about
privacy, and users thus over rely on existing laws and
regulations to protect their data. 3 Smart TV Traffic in the Wild
In this section, we study the network behavior of smart
2.1 Labeling Methodology TV devices when used by real users by analyzing a
dataset collected at residential gateways of tens of
Throughout this paper, we provide insight into the homes (we refer to this dataset as the in the wild
smart TV ATS ecosystems by labeling a domain ac- dataset). We compare the number of flows and traffic
cording to (1) its purpose (ATS or non-ATS); (2) its volumes generated by smart TVs from seven different
parent organization (i.e., the domain owner); and (3) its platforms. We analyze the most frequently used domains
relation to the app that uses it (first, third, or platform- of each platform by identifying ATS domains and map-
specific party). We detail this methodology below. ping each domain to its parent organization.
ATS Domains. We identify ATS domains as follows. Data Collection. To study smart TV traffic character-
For figures that denote top domains, we check if the istics in the wild, we monitor network traffic of 41 homes
FQDN is labeled as ads or tracking by VirusTotal, in a major metropolitan area in the United States. We
McAfee, OpenDNS [31–33], or if it is blocked by any sniff network traffic of smart TV devices at the resi-
of the blocklists considered in Sec. 5. For figures and dential gateways using off-the-shelf OpenWRT-capable
tables that involve entire datasets, we only consider the commodity routers. We collect flow-level summary in-
blocklists due to the impracticality of manually labeling formation for network traffic. For each flow, we collect
thousands of data points. its start time, FQDN of the external endpoint (using
DNS), and the internal device identifier. We identify
Parent Organizations. To understand the presence of
smart TVs using heuristics that rely on DNS, DHCP,
different organizations on smart TV platforms, we map
and SSDP traffic and also manually verify the identi-
each FQDN to its effective second level domain (eSLD)
fied smart TVs by contacting users. Our data collection
using Mozilla’s Public Suffix List [34, 35], and use
covers a total of 57 smart TVs across 41 homes over
Crunchbase [36]’s acquisition and sub-organization in-
the duration of approximately 3 weeks in 2018. Note
formation to find the parent company of the eSLD. For
that we obtained written consent from users, informingThe TV is Smart and Full of Trackers 133
Number of Flows
Number of Flows vortex.hulu.com
insights-collector.newrelic.com api-global.netflix.com
api-global.netflix.com cws-us-east.conviva.com
vortex.hulu.com api-global.[1].netflix.com
home.hulu.com time-ios.apple.com
aeg-personalization.quickplay.com e673.e9.akamaiedge.net
uwp-aeg-hbs.quickplay.com play.hulu.com
http-v-darwin.hulustream.com home.hulu.com
occ-1-1736-999.1.nflxso.net http-v-darwin.hulustream.com
d2lkq7nlcrdi7q.cloudfront.net occ-1-1736-999.1.nflxso.net
cws-us-east.conviva.com time-ios.g.aaplimg.com
cws-110*57.[2].amazonaws.com 1-courier.push.apple.com
init-p01st.push.apple.com http-e-darwin.hulustream.com
giga.logs.roku.com e1042.b.akamaiedge.net
liberty.logs.roku.com occ-0-1736-999.1.nflxso.net
occ-0-1736-999.1.nflxso.net itunes.apple.com.edgekey.net
ichnaea.netflix.com occ-2-1736-999.1.nflxso.net
dtvn-live-sponsored.akamaized.net pt.hulu.com
midland.logs.roku.com cws-110*57.[2].amazonaws.com
cws-eu-west-1.conviva.com cws-189*47.[2].amazonaws.com
cdn-0.nflximg.com d2hzeyj6b557bu.cloudfront.net
http-e-darwin.hulustream.com http-v-[3].footprint.net
occ-2-1736-999.1.nflxso.net t2.hulu.com
atv-ext.amazon.com
oca-api.geo.netflix.com init-p01st.push.apple.com
b.scorecardresearch.com xp.itunes-apple.com.akadns.net
cws.conviva.com auth.hulu.com
stream.nbcsports.com e1042.e12.akamaiedge.net
tp.akam.nflximg.com mt-ingestion-[4].akadns.net
scribe.logs.roku.com nrdp.nccp.netflix.com
pubads.g.doubleclick.net a1910.b.akamai.net
0 10k 20k 30k 0 10k 20k 30k
(a) Roku (b) Apple
Number of Flows
api-global.netflix.com
mobile-collector.newrelic.com
feed.theplatform.com Number of Flows
hh.prod.sony[6].com log-ingestion.samsungacr.com
android.clients.google.com www.youtube.com
www.fox.com api-global.netflix.com
flingo.tv lcprd1.samsungcloudsolution.net
ichnaea.netflix.com android.clients.google.com
pubads.g.doubleclick.net log-2.samsungacr.com
clients3.google.com i.ytimg.com
www.lookingglass.rocks vortex.hulu.com
i.ytimg.com occ-1-1736-999.1.nflxso.net
clients4.google.com youtube-ui.l.google.com
api.twitter.com
play.googleapis.com clients1.google.com
artist.api.lv3.cdn.hbo.com ypu.samsungelectronics.com
js-agent.newrelic.com i9.ytimg.com
www.googleapis.com occ-0-1736-999.1.nflxso.net
mtalk.google.com nrdp.nccp.netflix.com
comet.api.hbo.com ytimg.l.google.com
profile.localytics.com s.youtube.com
occ-0-586-590.1.nflxso.net clients4.google.com
video-stats.l.google.com
connectivitycheck.gstatic.com occ-2-1736-999.1.nflxso.net
occ-1-586-590.1.nflxso.net osb.samsungqbe.com
assets.fox.com t2.hulu.com
sp.auth.adobe.com ichnaea.netflix.com
occ-2-586-590.1.nflxso.net http-v-darwin.hulustream.com
api.meta.ndmdhs.com upu.samsungelectronics.com
api.fox.com tv.deezer.com
youtubei.googleapis.com dpu.samsungelectronics.com
ocfconnect-[7].samsungiotcloud.com
cdn.meta.ndmdhs.com googleads.g.doubleclick.net
0 2k 4k 0 50k 100k
(c) Sony (d) Samsung
Fig. 1. Top-30 fully qualified domain names in terms of number of flows per device for a subset of the smart TVs in the “in the wild”
dataset. See Appendix C.2 for the other brands. Domains identified as ATS are highlighted with red, dashed bars.
them of our data collection and research objectives, in external smart TV devices. Thus, we do not differentiate
accordance with our institution’s IRB guidelines. between built-in vs. external Roku smart TV devices.
Dataset Statistics. Table 1 lists basic statistics of We expect smart TV devices to generate significant
smart TV devices observed in our dataset. Overall, we traffic because they are typically used for OTT video
note 57 smart TVs from 7 different vendors/platforms streaming [38]. Chromecast devices generate the highest
using a variety of technologies. number of flows (exceeding 200 thousand flows) on aver-
Devices can be built-in smart TVs such as Samsung age, while Samsung, Apple, and Roku devices generate
and Sony, others like Chromecast and Apple TV can nearly 50 thousand flows on average. Roku devices gen-
be external stick/box solutions, while devices like Roku erate the highest volume of flows (exceeding 80 GB) on
can have both forms. For example, 7 out 9 Roku devices average, with one Roku generating as much as 283 GB.
in our dataset were built-in Roku smart TVs, while the Except for LG and Sony devices, all smart TV devices
remaining two were external Roku sticks. Note that a generate at least tens of GBs worth of traffic on aver-
smart TV platform such as Roku supports the same age. Finally, we note that smart TV devices typically
set of apps and a similar interface for both built-in and connect to hundreds of different endpoints on average.The TV is Smart and Full of Trackers 134
Apple
Smart Device Average Average Average NBCUniversal Media
comScore
TV Count Flow Flow eSLD AT&T
Amazon
Platform Count Volume Count AEG
/Device /Device /Device Roku
Conviva
(x 1000) (GB)
Apple TV Unknown/CDN
Apple 16 49.3 46.6 536
Samsung 11 62.6 33.2 369 Walt Disney Company
Chromecast 10 201.9 26.3 354 Roku Twitter
Deezer
Roku 9 48.1 83.0 543 Samsung
Vizio 6 43.4 63.4 278 Samsung
Netflix
LG 4 10.9 0.9 1893
New Relic
Sony 1 33.1 0.1 186
Vizio
NFL
Table 1. Traffic statistics of 57 smart TV devices observed across LG
Vizio
Verizon
41 homes (“in the wild” dataset). LG
Kodi
Sony Bravia Sony
Localytics
Flingo
Comcast
Endpoint Analysis. Fig. 1 plots the top-30 FQDNs in Chromecast Adobe Systems
Time Warner
Fox
terms of flow count for Roku, Apple, Sony, and Samsung
smart TV platforms. The plots for the remaining smart
TV platforms are in Appendix C.2. Alphabet
We note several similarities in the domains accessed
by different smart TV devices. First, video streaming
Fig. 2. Mapping of platforms measured in the wild to the par-
services such as Netflix and Hulu are popular across the
ent organizations of the endpoints they contact (for the top-30
board, as evident from domains such api-global.netflix.
FQDNs of each platform). The width of an edge indicates the
com and vortex.hulu.com. Second, cloud/CDN services number of distinct FQDNs within that organization that was ac-
such as Akamai and AWS (Amazon) also appear for cessed by the platform.
different smart TV platforms. Smart TVs likely connect
zations such as Apple on the other end of the spectrum.
to cloud/CDN services because popular video stream-
Furthermore, it reveals the presence of organizations like
ing services typically rely on third party CDNs [39, 40].
Conviva, comScore, and Localytics, whose main busi-
Third, we note the prevalence of well-known adver-
ness is in the advertising and tracking space. We note
tising and tracking services (ATSes). For example,
that Samsung, Deezer, Roku, LG, and Flingo have the
*.scorecardresearch.com and *.newrelic.com are third
majority of their domains labeled as ATSes while Net-
party tracking services, and pubads.g.doubleclick.net is
flix and Walt Disney Company have less than half of
a third party advertising service.
their domains labeled as ATSes.
We notice several platform-specific differences in
the domains accessed by different smart TV platforms. Takeaway & Limitations. Traffic analysis of differ-
For example, giga.logs.roku.com (Roku), time-ios.apple. ent smart TV platforms in the wild highlights interest-
com (Apple), hh.prod.sonyentertainmentnetwork.com ing similarities and differences. As expected, all devices
(Sony), and log-ingestion.samsungacr.com (Samsung) generate traffic related to popular video streaming ser-
are unique to different types of smart TVs. In addition, vices. In addition, they also access ATSes, both well-
we notice platform-specific ATSes. For example, the fol- known and platform-specific. While our vantage point
lowing advertising-related domains are not in the top-30 at the residential gateway provides a real-world view of
(and therefore not pictured in Fig. 1), but are unique to the behavior of smart TV devices, it lacks granular in-
different smart TV platforms: p.ads.roku.com (Roku), formation beyond flows (e.g., packet-level information)
ads.samsungads.com (Samsung), and us.info.lgsmartad. and does not tie traffic to the app that generate it. An-
com (LG). other limitation of this analysis is that the findings may
be biased by the viewing habits of users in these 41
Organizational Analysis. Figure 2 illustrates the mix
households. It is unclear how to normalize to provide
of different parent organizations contacted by the seven
a fair comparison of endpoints accessed by the differ-
smart TV platforms in our dataset. It shows the preva-
ent smart TV platforms. We address these limitations
lence of Alphabet in smart TV platforms like Chrome-
cast, Sony, and LG, while revealing competing organi-The TV is Smart and Full of Trackers 135
next by systematically analyzing two popular smart TV libraries is still possible. For example, the Ooyala IQ
platforms in a controlled testbed. SDK [46] provides various analytics services that can
be integrated into a Roku app. Thus, such libraries can
help ATSes learn the viewing habits of users by collect-
ing data from multiple apps. In terms of permissions,
4 Systematic Testing of the Roku Roku only protects microphone access with a permis-
and Fire TV Platforms sion and does not require any permission to access the
advertising ID. Users can choose to reset this ID and opt
In this section, we perform an in-depth, systematic out of targeted advertising at any time [45]. However,
study of two smart TV platforms, Roku and Amazon apps and libraries can easily create other IDs or use fin-
Fire TV, which we chose since they are popular [41], gerprinting techniques to continue tracking users even
affordable ($25), and among the leading smart TV plat- after opt-out. We further elaborate on this in Sec 6.
forms in terms of number of ad requests [6]. Sections 4.1 App Selection. The RCS provides a web (and on-
and 4.2 present our measurement approach for system- device) interface for browsing the available Roku apps.
atically testing the top-1000 apps in each platform while To the best of our knowledge, Roku does not pro-
collecting their network traffic. Since app exploration is vide public documentation on how to programmatically
automated, no real users are involved, thus no IRB is query the RCS. We therefore reverse-engineer the REST
needed. Our measurement approach provides visibility API backing the RCS web interface by inspecting the
into the behaviors of individual apps, which was not HTTP(S) requests sent while browsing the RCS, and
possible from the vantage point used for the in the wild use this insight to write a script that crawls the RCS
dataset. In Section 4.3, we analyze the two testbed for the metadata of all (8515 as of April 2019) apps.
datasets, and compare them to each other and to the To test the most relevant apps, we select the
Android ATS ecosystem. top-50 apps in 30 out of the 32 categories. We exclude
“Themes” and “Screensavers” since these apps do not
show up among the regular apps on the Roku and there-
4.1 Roku Data Collection fore cannot be operated using our automation software.
We base our selection on the “star rating count”, which
In this section, we describe the Roku platform and our
we interpret as the review count. Roku apps can be
app selection methodology, and present an overview of
labeled with multiple categories, thus some apps con-
Rokustic—our software tool that automatically explores
tribute to the top-50 of multiple categories. This places
Roku apps. We use Rokustic to explore and collect traf-
the final count of apps in our dataset at 1044.
fic from 1044 Roku apps. The resulting network traces
are analyzed in Sec. 4.3. Automation (Rokustic). To scale testing of apps, we
implement a software tool, Rokustic, that automatically
Roku Platform. We start by describing the Roku plat-
installs and exercises Roku apps. Due to lack of space,
form, which has its own app store—the Roku Chan-
we only provide a brief overview here and defer addi-
nel Store [42] (RCS)—that offers more than 8500 apps,
tional details to Appendix A.1.
called “channels”. For security purposes, Roku sand-
We run Rokustic on a Raspberry Pi that acts as a
boxes each app (apps are not allowed to interact or ac-
router and hosts a local wireless network that the Roku
cess the data of other apps) and provides limited access
is connected to. Rokustic utilizes ECP [47], a REST-like
to system resources [43]. Furthermore, Roku apps can-
API exposed by the Roku device, to control the Roku.
not run in the background. Specifically, app scripts are
Given a set of apps to exercise, Rokustic installs each
only executed when the user selects a particular app,
app by invoking the ECP endpoint that opens up the
and when the user exits the app, the script is halted,
on-device version of the RCS page for the app, and then
and the system resumes control [44].
sends a virtual key press to click the “Add Channel”
To display ads, apps typically rely on the Roku Ad-
button. To exercise apps, Rokustic first invokes the ECP
vertising Framework which is integrated into the Roku
endpoint that returns the set of installed apps. For each
SDK [45]. The framework allows developers to use ad
app, Rokustic then (1) starts tcpdump on the Raspberry
servers of their preference and updates automatically
Pi’s wireless interface; (2) uses ECP to launch the app
without requiring the developer to rebuild the app. Even
and invoke a series of virtual key presses in an attempt
though such a framework eliminates the need for third
to incur content playback; (3) pauses for five minutes to
party ATS libraries, the development and usage of suchThe TV is Smart and Full of Trackers 136
let the content play; (4) exits the app and repeats from Number of Roku Fire TV Both
step 2 an additional two times; (5) terminates tcpdump. Apps exercised 1044 1010 128
Since Roku apps cannot execute in the background Fully qualified domain names (FQDN) 2191 1734 578
FQDNs accessed by multiple apps 669 603 199
(see “Roku Platform”), all captured traffic will belong to
URL paths 13899 240713 74
the exercised app and the Roku system. The total inter-
Table 2. Summary of the Roku and Fire TV testbed datasets.
action time with each app is approximately 16 minutes.
The rightmost column summarizes the intersection between the
We do not attempt to decrypt TLS traffic as we cannot two testbed datasets. For example, there are 128 apps that are
install our own self-signed certificates on the Roku. present both in the Roku dataset and the Fire TV dataset.
4.2 Fire TV Data Collection We rely on AntMonitor [37, 48], an open-source
VPN-based library, to intercept all outgoing network
In this section, we describe the Fire TV platform, our traffic from the Fire TV, and to label each packet with
app selection methodology, and present an overview the package name of the application (or system pro-
of Firetastic—our software tool that automatically ex- cess) that generated it. We enable AntMonitor’s TLS
plores Fire TV apps. By using Firetastic to control six decryption for added visibility into PII exposures. We
Fire TV devices in parallel, we explore and collect traf- analyze the success of TLS decryption in Appendix B.
fic from 1010 Fire TV apps within a one week period. In summary, TLS decryption was generally successful
The resulting network traces are analyzed in Sec. 4.3. with 10% or fewer failures for 55% of all apps, and 20%
Fire TV Platform. Although Fire TV is made by or fewer failures for 80% of all apps.
Amazon, its underlying operating system, Fire OS, is For app exploration, we utilize DroidBot [49], a
a modified version of Android. This allows apps for Python tool that dynamically maps the UI and sim-
Fire TV to be developed in a similar fashion to An- ulates user inputs such as button presses using the An-
droid apps. Therefore, all third-party libraries that are droid Debug Bridge (ADB). To increase the probability
available for Android apps can also be integrated into of content playback, we configure DroidBot to utilize
Fire TV apps. Similarly, application sandboxing and its breadth first search algorithm to explore each app.
permissions in Fire TV are analogous to those of An- The intuition is that the main content is often made
droid, and any permission requested by the app is inher- available from top-level UI elements.
ited by all libraries that the app includes. This allows In summary, for each app, Firetastic: (1) starts
third party libraries to track users across apps using AntMonitor; (2) explores the app for 15 minutes; (3)
a variety of identifiers, such as Advertising ID, Serial stops AntMonitor; and (4) extracts the .pcapng files
Number, and Device ID, etc. We further discuss track- that were generated during testing. We use Firetastic
ing through PII exposure in Section 6. to explore apps on six Fire TV devices in parallel. Our
test setup is resource-efficient and scalable, using only
App Selection. To test the most relevant apps, we pick
one computer to send commands to multiple Fire TVs.
the top-1000 apps from Amazon’s curated list of “Top
Featured” apps. We ignore some apps that use a local
VPN (as they would conflict with AntMonitor), that
could not be installed manually, and utility apps that
4.3 Comparing Roku and Fire TV
can change the device settings (which would affect the
test environment). As a result, we ignore around 200
In this section, we analyze the (ATS) domains accessed
apps while including 1010 testable applications. Ama-
by the apps in the Roku and Fire TV testbed datasets.
zon’s app store offers around 4,000 free apps at the time
We first provide an overview of the datasets, and analyze
of writing, thus our dataset covers approximately 25%.
how many (ATS) domains apps contact. We then look
Automation (Firetastic). We design and implement closer at the eSLDs and third party ATS domains that
a software tool, Firetastic, that integrates the capabil- are contacted by the most apps, including which parent
ities of two open source tools for Android: an SDK for organizations they belong to. Furthermore, we compare
network traffic collection and a tool for input automa- the top third party ATS domains to those of Android.
tion. We provide a brief overview of Firetastic here, and Finally, we compare the domains accessed by apps that
defer additional details to Appendix A.2. are present on both testbed platforms.The TV is Smart and Full of Trackers 137
1.0 1.0
CDF of Tested Apps
0.8
CDF of Tested Apps
0.8
0.6 0.6
0.4 0.4
0.2 0.2
Roku Roku
Fire TV Fire TV
0.0 0.0
0 20 40 60 80 100 0 20 40 60 80
Number of Distinct Domains Number of Distinct ATS Domains
(a) Roku & Fire TV: Distinct domains per app. (b) Roku & Fire TV: Distinct ATS domains per app. A do-
main is considered an ATS if it is labeled as so by any of the
blocklists considered in Sec. 5.
Number of Apps Number of Apps
roku.com amazon.com
doubleclick.net amazonaws.com
googlesyndication.com amazon-adsystem.com
google-analytics.com amazonvideo.com
scorecardresearch.com media-amazon.com
amazonaws.com doubleclick.net
spotxchange.com amazon-dss.com
ifood.tv cloudfront.net
tremorhub.com crashlytics.com
irchan.com googleapis.com
vimeo.com amazonalexa.com
2mdn.net google-analytics.com
1rx.io googlesyndication.com
ravm.tv ssl-images-amazon.com
vimeocdn.com facebook.com
imrworldwide.com google.com
adrta.com flurry.com
akamaized.net scorecardresearch.com
innovid.com ifood.tv
insightexpressai.com gstatic.com
demdex.net youtube.com
gvt1.com uplynk.com
adsrvr.org ytimg.com
akamaihd.net serving-sys.com
monarchads.com unity3d.com
agkn.com Platform Party akamaihd.net Platform Party
cloudfront.net googlevideo.com
springserve.com Third Party moatads.com Third Party
stickyadstv.com First Party adobe.com First Party
ewscloud.com titantv.com
0 250 500 750 1000 0 500 1000
(c) Roku: Top-30 eSLDs. (d) Fire TV: Top-30 eSLDs.
Number of Apps Number of Apps
pubads.g.doubleclick.net e.crashlytics.com
tpc.googlesyndication.com pubads.g.doubleclick.net
www.google-analytics.com ssl.google-analytics.com
search.spotxchange.com pagead2.googlesyndication.com
securepubads.g.doubleclick.net googleads.g.doubleclick.net
ad.doubleclick.net imasdk.googleapis.com
event.spotxchange.com settings.crashlytics.com
b.scorecardresearch.com data.flurry.com
sb.scorecardresearch.com graph.facebook.com
googleads4.g.doubleclick.net b.scorecardresearch.com
ade.googlesyndication.com reports.crashlytics.com
gcdn.2mdn.net csi.gstatic.com
adrta.com www.google-analytics.com
PD[0].ads.tremorhub.com config.uca.cloud.unity3d.com
secure.insightexpressai.com applab-sdk.amazon.com
events.tremorhub.com ad.doubleclick.net
dpm.demdex.net z.moatads.com
cm.g.doubleclick.net ade.googlesyndication.com
insight.adsrvr.org cdp.cloud.unity3d.com
ctv.monarchads.com dpm.demdex.net
0 250 500 0 50 100
(e) Roku: Top-20 third party ATS domains. (f) Fire TV: Top-20 third party ATS domains.
Fig. 3. Analysis of domain usage per app and the top domains across all apps in the Roku and Fire TV testbed datasets.The TV is Smart and Full of Trackers 138
Oracle domains contacted per app in Fig. 3b, the vast majority
Amazon
Unity Tech (80%) of apps from the two platforms behave similarly.
Numitas On the positive side, for both platforms, around
Facebook
Adobe Systems 60% of the apps contact only a small handful ATS
comScore domains. Yet, about 10% of the Roku and Fire TV
Fire TV
The Trade Desk apps contact 20+ and 10+ ATS domains, respectively.
Barons Media
PIxalate
These concerning apps come from a small set of devel-
Bain Capital opers. For instance, for Roku, “Future Today Inc.” [50],
Telaria “8ctave ITV”, and “StuffWeLike” [51] are responsible
RTL Group for 51%, 13%, and 11%, respectively, of these apps.
Roku On the Fire TV side, “HTVMA Solutions, Inc.” [52]
is responsible for 15% of the apps, and “Gray Televi-
sion, Inc.” [53] is responsible for 12% of the apps.
Alphabet Key Players. Figures 3c (Roku) and 3d (Fire TV)
present the top-30 eSLDs in terms of the number of
apps that contact a subdomain of the eSLD. We define
an eSLD’s app penetration as the percentage of apps in
the dataset that contact the eSLD.
Fig. 4. Mapping of platforms measured in our testbed environ- Platform. The top eSLD of each platform has 100%
ment to the parent organizations of the top-20 third-party ATS app penetration and belongs to the platform operator.
domains their apps contact. The width of an edge indicates the While these eSLDs also cover subdomains that provide
number of apps that contact each organization. functionality, we note that both platform operators are
engaged in advertising and tracking, as shown in Fig. 9
Overview. The datasets collected using Rokustic and in Appendix C.3, which separates traffic to the eSLDs
Firetastic are summarized in Table 2. For Roku, we dis- in Figs. 3c and 3d by ATS and non-ATS FQDNs.
cover 2191 distinct FQDNs, 699 of which are contacted Third Parties. As evident from Figs. 3c and 3d, Al-
by multiple apps. For Fire TV, we discover 1734 dis- phabet has a strong presence in the ATS space of both
tinct FQDNs, 603 of which are contacted by multiple platforms, with *.doubleclick.net, an ad delivery end-
apps. We also find 578 FQDNs that appear in both point, achieving 58% and 35% app penetration for Roku
datasets, 199 of which are contacted by multiple apps. and Fire TV, respectively. Its analytic services also rank
Our automation uncovers approximately twice as many high, with google-analytics.com in the top-20 for both
FQDNs as [11], possibly due to longer experiments and platforms and crashlytics.com in the top-10 for Fire TV.
different app exploration goals. We further detail this in To better understand additional key third party ATSes,
Appendix A.3. we strip away the platform-specific endpoints and re-
Leveraging the blocklists from Sec. 5, we identify port the top-20 third party ATS FQDNs for Roku and
314 ATS domains that are unique to the Roku dataset, Fire TV in Figs. 3e and 3f, respectively. We note that
285 that are unique to the Fire TV dataset, and an over- both platforms use distinct third party ATSes. For ex-
lap of 227 between the two datasets. When considering ample, SpotX (*.spotxchange.com), which serves video
eSLDs of the ATS domains, we find 68 eSLDs that are ads, is a significant player in the Roku ATS space with
unique to the Roku dataset, 100 that are unique to the 17% app penetration, but only maintains 1% app pene-
Fire TV dataset, and an overlap of 138 eSLDs. These tration for Fire TV. Even when considering the smaller
numbers suggest that the ATS ecosystems of the two players, we see little overlap between the two platforms,
platforms have substantial differences, which we ana- suggesting these players focus their efforts on a single
lyze in further detail later in this section. platform. For example, Kantar Group’s insightexpres-
sai.com analytics service has 7% app penetration on the
Number of (ATS) Domains Contacted. Figure 3a
Roku platform, but only 0.01% on the Fire TV platform.
presents the empirical CDF of the number of distinct
domains each app in the two testbed datasets contacts. Parent Organization Analysis. We further analyze
Fire TV apps appear more “chatty”—most Fire TV the parent organizations of Roku and Fire TV third
apps contact about twice as many domains as the Roku party ATS endpoints in Fig. 4, using the method de-
apps. However, when we consider the number of ATS scribed earlier in Section 2.1. Interestingly, the set of topThe TV is Smart and Full of Trackers 139
120
Common
FireTV
100
Number of Hostnames
Roku
80
60
40
20
0
Mediterranean Food
CNNgo
VH1
Pluto TV
P. Allen Smith
Free Movies Now
Relax My Dog
Italian Recipes
BET
Philo
Baking by ifood.tv
Hallmark Channel
Bloomberg
Thai recipes
Comedy Central
Cool School
Paramount Network
MTV
Lifetime
Chinese Recipes
A&E
BestCooks
iFood.tv
AMC
Stand-up Comedy
Low Carb 360
CBS Sports Stream
HISTORY
Fun With Roblox by
CraftSmart
Telemundo Deportes
Watch TBS
Watch TNT
The Holy Tales Bible
BBC America
Blippi
Drinks
Fox Business Network
Pilates
Red Bull TV
Om Nom Stories
Haystack TV
TechSmart.tv
MOOVIMEX
NFL Sunday Ticket
PlayStation Vue
ABC News
Smithsonian Channel
Baby By HappyKids.tv
Sling TV
Now You Know
JTV Live
iHeartRadio
Wow, I Never Knew That
WSB-TV Channel 2
Outside TV Features
UP Faith & Family
The BILLIARD Channel
FilmRise Kids
NRA TV
HappyKids
Fig. 5. Top-60 common apps (apps present in both testbed datasets) ordered by the number of domains that each app contacts. Con-
sidering all 128 common apps, there are 597 domains which are exclusive to Roku apps, 496 domains which are exclusive to Fire TV
apps, and 155 domains which are contacted by both the Roku and the Fire TV versions of the same app.
third party organizations is rather diverse, with only a (data.flurry.com) both have a strong presence on both
slight overlap in the shape of Adobe Systems and com- Fire TV and Android. Some of the third party ATS ob-
Score, possibly suggesting that the remaining organiza- served for Fire TV, which were not present for Android,
tions focus their efforts on a single platform. Fire TV include comScore, Adobe (dpm.demdex.net), and Ama-
shows gaming and social media ATSes from Unity Tech zon (applab-sdk.amazon.com).
and Facebook, whereas Roku exhibits more traffic to Common Apps in Roku and Fire TV Next, we
ATSes from companies that focus on video ads such as compare the Roku and Fire TV datasets at the app-
The Trade Desk, Telaria, and RTL Group. Similar to the level by analyzing the FQDNs accessed by the set of
in the wild organization analysis in Fig 2, we again note apps that appear on both platforms, referred to as com-
that Alphabet dominates the set of third party ATSes mon apps. Recall from Table 2 that the datasets col-
on both Roku and Fire TV. lected using Rokustic and Firetastic contain a total of
Comparing to Android ATS Ecosystem. Next, we 128 common apps. We identified common apps by fuzzy
compare the top-20 third party ATS endpoints in our matching app names since they sometimes vary slightly
Roku and Fire TV datasets (Figs. 3e and 3f) with those for each platform (e.g., “TechSmart.tv” on Roku vs.
reported for Android [24]. “TechSmart” on Fire TV). We further cross-referenced
Roku vs. Android. The key third party ATSes with the developer’s name to validate that the apps were
in Roku (Fig. 3e) differ from the Android platform. indeed the same (e.g., both TechSmart apps are created
For example, SpotX (*.spotxchange.com) and comScore by “Future Today”). The 128 common apps contact a
(*.scorecardresearch.com) both have a strong presence total of 1248 distinct FQDNs. Out of these, 597 FQDNs
on Roku, but are not among the key players for An- are exclusively contacted by Roku apps, 496 are exclu-
droid. In contrast, Facebook’s graph.facebook.com is sively contacted by Fire TV apps, and only 155 FQDNs
the second most popular ATS domain on Android, are contacted by both Roku and Fire TV apps.
but insignificant on Roku. The set of top third party Figure 5 reports the amount of overlapping and non-
ATSes in Roku is also more diverse and includes overlapping FQDNs for the top-60 common apps (in
smaller organizations such as Pixalate (adrta.com) terms of the number of distinct FQDNs that each app
and Telaria (*.tremorhub.com). While Alphabet has a contacts). In general, the set of FQDNs contacted by
strong foothold in both ATS ecosystems, it is less sig- both the Roku and the Fire TV versions of the same
nificant for Roku (9 out 20 ATS FQDNs are Alphabet- app is much smaller than the set of platform-specific
owned, vs. 16 out of 20 for Android). FQDNs. From inspecting the common FQDNs for some
Fire TV vs. Android. In contrast to Roku, Fire TV of the apps in Fig. 5, we find that these generally include
is more similar to Android: we see an overlap of 9 endpoints that serve content. For example, for Mediter-
FQDNs, 7 of which are owned by Alphabet. This ranean Food, the only two common FQDNs are subdo-
is expected, given that Fire TV is based off of An- mains of ifood.tv, which belong to the parent organiza-
droid and thus natively supports the ATS services of tion behind the app. This makes intuitive sense as the
Android. Facebook (graph.facebook.com) and Verizon same app presumably offers the same content on bothThe TV is Smart and Full of Trackers 140
platforms and must therefore access the same servers to Block Rate (%)
download said content. On the other hand, the platform- Platform # Domains PD TF MoaAB SATV
specific domains contain obvious ATS endpoints such Dataset obtained “in the wild”
Apple 3179 10% 13% 12% 5%
as ads.yahoo.com and ads.stickyadstv.com for the Roku
Samsumg 1765 14% 19% 15% 8%
version of the app, and aax-us-east.amazon-adsystem.
Chromecast 1576 9% 15% 15% 5%
com and mobileanalytics.us-east-1.amazonaws.com for Roku 2312 15% 19% 18% 7%
the Fire TV version of the app. In conclusion, our anal- Vizio 942 16% 18% 16% 11%
ysis of common apps reveals (to our surprise) little over- LG 627 45% 54% 50% 27%
lap in the ATS endpoints they access, which further sug- Sony 119 16% 24% 16% 7%
Dataset obtained in our testbed
gests that the smart TV ATS ecosystem is segmented
Roku 2191 17% 22% 20% 9%
across platforms.
Fire TV 1734 22% 27% 22% 9%
Takeaway. The ATS ecosystems of the Roku and Table 3. Block rates of the four blocklists when applied to the
Fire TV platforms seem to differ substantially: (1) the domains in our datasets.
full set of ATS domains contacted by apps in the two
datasets have little overlap; (2) some organizations are
key players on one platform, but almost absent on the Setup. We evaluate the following blocklists:
other (e.g., SpotX has a significant presence on Roku, 1. Pi-hole Default (PD): We test blocklists included
but is almost absent on Fire TV, whereas Facebook has in Pi-hole’s default configuration [56] to imitate the
a reasonable foothold Fire TV, but almost zero presence experience of a typical Pi-hole user. This set has
on Roku); and (3) apps present in both datasets have seven hosts files including Disconnect.me ads, Dis-
little overlap in terms of the ATS domains they contact. connect.me tracking, hpHosts, CAMELEON, Mal-
Finally, we find that the key third party ATS players on wareDomains, StevenBlack, and Zeustracker. PD
Android have little overlap with Roku, but substantial contains a total of about 133K entries.
overlap with Fire TV, which intuitively makes sense as 2. The Firebog (TF): We test nine advertising and
Fire TV is built on top of Android. five tracking blocklists recommended by “The Big
Blocklist Collection” [8], to emulate the experience
of an advanced Pi-hole user. This includes: Discon-
5 Blocklists for Smart TVs nect.me ads, hpHosts, a dedicated blocklist target-
ing smart TVs, and hosts versions of EasyList and
In this section, we evaluate four well-known DNS-based EasyPrivacy. TF contains 162K entries total.
blocklists’ ability to prevent smart TVs from accessing 3. Mother of all Ad-Blocking (MoaAB): We test
ATSes and their adverse effects on app functionality. We this curated hosts file [9] that targets a wide-range
further demonstrate how the datasets resulting from au- of unwanted services including advertising, track-
tomated app exploration may aid in curating new can- ing, (cookies, page counters, web bugs), and mal-
didate rules for blocklists. ware (phishing, spyware) to again imitate the expe-
rience of an advanced Pi-hole user. MoaAB contains
a total of about 255K entries.
5.1 Evaluating Popular DNS Blocklists 4. StopAd (SATV): We test a commercial smart TV
focused blocklist by StopAd [10]. This list partic-
DNS-based blocking solutions such as Pi-hole [7] are ularly targets Android based smart TV platforms
used to prevent in-home devices, including smart TVs, such as Fire TV. We extract StopAd’s list by analyz-
from accessing ATS domains [54]. To block advertis- ing its APK using Android Studio’s APK Analyzer
ing and tracking traffic, they essentially “blackhole” [57]. SATV contains a total of about 3K entries.
DNS requests to known ATS domains. Specifically, they
match the domain name in the DNS request against a We applied these blocklists to both our in the wild and
set of blocklists, which are essentially curated hosts files testbed datasets and we report the results next.
that contain rules for well-known ATS domains. If the Block Rates. We start our analysis by comparing how
domain name is found in one of the blocklists, it is typi- many FQDNs are blocked by the different blocklists.
cally mapped to 0.0.0.0 or 127.0.0.1 to prevent outbound We define a blocklist’s block rate as the number of dis-
traffic to that domain [55]. tinct FQDNs that are blocked by the list, over the totalThe TV is Smart and Full of Trackers 141
App Name No List PD TF MoaAB SATV
No No No No No No No No No No
Ads Breakage Ads Breakage Ads Breakage Ads Breakage Ads Breakage
Common
Pluto TV 5 5 5 5 5
iFood.tv 5 5
Tubi — 6
YouTube 5 5 5 5 5
Roku
CBS News Live 5 6 6 6
Top
The Roku Channel 5 6 5
Sony Crackle 5 5 5 5
Common Random
WatchFreeComedyFlix 5 5 5
Live Past 100 Well
SmartWoman 5
Pluto TV 5 5 5 5 5 5 6
iFood.tv 5 — 6 — 6
Tubi
Downloader
Fire TV
The CW for Fire TV 5 — 6 — 6 — 6 5
Top
FoxNow 5 — 6 — 6 5 5
Watch TNT
Random
KCRA3 Sacramento 5 — 6 — 6 5 5
Watch the Weather Channel 5 5
Jackpot Pokers by PokerStars 5
Table 4. Missed ads and functionality breakage for different blocklists when employed during manual interaction with 10 Roku apps
and 10 Fire TV apps. For “No Ads”, a checkmark ( ) indicates that no ads were shown during the experiment, a cross (5) indicates
that some ad(s) appeared during the experiment, and a dash (—) indicates that breakage prevented interaction with the app alto-
gether. For “No Breakage”, a checkmark ( ) indicates that the app functioned correctly, a cross (5) indicates minor breakage, and a
bold cross (6) indicates major breakage.
number of distinct FQDNs in the dataset. Table 3 com- acting with a sample of apps from our testbed datasets
pares the block rates of the aforementioned blocklists on while coding for ads and app breakage. We sample 10
our in the wild and testbed datasets. Overall, we note Roku apps and 10 Fire TV apps, including the top-
that TF, closely followed by MoaAB and PD, blocks the 4 free apps, three apps that are present on both plat-
highest fraction of domains across all of the platforms forms, and an additional three randomly selected apps.
in both the in the wild and testbed datasets. SATV is We test each app five times: once without any blocklist
the distant last in terms of block rate. It is noteworthy and four times where we individually deploy each of the
that TF blocks more domains than MoaAB despite being aforementioned blocklists. During each experiment, we
about one-third shorter. We surmise this is because TF attempt to trigger ads by playing multiple videos and/or
includes a smart TV focused hosts file, and thus catches live TV channels and fast-forwarding through video con-
more relevant smart TV ATSes. This finding shows that tent. We take note of any functionality breakage (due to
the size of a blocklist does not necessarily translate to false positives) and visually observable missed ads (due
its coverage. to false negatives). We differentiate between minor and
Blocklist Mistakes. Motivated by the differences in major functionality breakage as follows: minor breakage
the block rates of the four blocklists, we next compare when the app’s main content remains available but the
them in terms of false negatives (FN) and false posi- application suffers from minor user interface glitches or
tives (FP). False negatives occur when a blocklist does occasional freezes; and major breakage when the app’s
not block requests to ATSes and may result in (visually content becomes completely unavailable or the app fails
observable) ads or (visually unobservable) PII exfiltra- to launch.
tion. False positives occur when a blocklist blocks re- Missed Ads vs. Functionality Breakage. Table 4
quests that enable app functionality and may result in summarizes the results of our manual analysis for missed
(visually observable) app breakage. ads and functionality breakage. Overall, we find that
We first systematically quantify visually observable none of the blocklists are able to block ads from all of
false negatives and false positives of blocklists by inter- the sampled apps while avoiding breakage. In particular,You can also read
