GPC Android

This repo contains code and other resources for dynamically analyzing Android apps, especially, for checking their compliance with Global Privacy Control (GPC). GPC is a privacy preference signal for opting out from ad tracking. Apps are required to respect GPC signals per the California Consumer Privacy Act (CCPA) and other privacy laws.

The code and all other resources in this repo are developed and maintained by Nishant Aggarwal (@n-aggarwal), Zachary Liu (@zatchliu), Samir Cerrato (@samir-cerrato) and Sebastian Zimmeck (@SebastianZimmeck) of the privacy-tech-lab and Konrad Kollnig (@kasnder) of the Law and Tech Lab of Maastricht University. Wesley Tan (@wesley-tan) contributed earlier.

1. Research Publications
2. Repo Overview
3. GPC Android App
4. Scripts
5. Apps CSV
6. Thank You!

1. Research Publications

Sebastian Zimmeck, Nishant Aggarwal, Zachary Liu and Konrad Kollnig, From Ad Identifiers to Global Privacy Control: The Status Quo and Future of Opting Out of Ad Tracking on Android, Under Review, BibTeX.

If you are using code or other resources from this repo, please cite the above paper.

You can find a complete list of our GPC research publications in the GPC OptMeowt repo.

2. Repo Overview

This repo contains the following resources:

• gpc-android-app: GPC Android app written in Java
• scripts: Code for dynamically intercepting and analyzing network traffic
• app_csv: App lists sorted by Google Play Store categories

3. GPC Android App

The gpc-android-app directory contains the code for an app with the following features:

1. Directing people to the AdID setting, where they can disable tracking, which, in our view, is equivalent to turning on GPC
2. Directing people to DuckDuckGo or Brave, two browsers with GPC enabled

You can run the app by cloning this repo and running it in Android Studio.

Also, check out Konrad's GPC Android app!

4. Scripts

The scripts can be used in conjunction with mitmproxy SOCKS5 mode to intercept network traffic and perform dynamic privacy analysis on Android apps. This following is a guide on how to run captures using the gpc-android framework. To begin you should clone the repo.

Dependencies (Prerequisites)

1. Frida Server on Test Device
2. App for SOCKS Proxy on Test Device
3. MITM Proxy in laptop
4. Mullvad VPN Account (and set up in laptop)
5. MITM Proxy installed and setup in laptop
6. ADB Installed and set up on the laptop
7. Rooted Test device set up with the appropriate Magisk Modules
8. USB Debugging is enabled in the test device

Apps to Test

Once the repo is cloned, you will see there is a file called scripts/app-list.txt. In that file you should enter the list of package names that you want to test. Each package name should be on a new line. Make sure to add an empty line after your last package name.

The next step is to upload the actual apk files. You will see a folder called apps. This is where all the apks should be uploaded. It is highly recommended that the apks are deleted (or not staged) prior to any commit.

Setting up the capture infrastructure

Now we can move on to setting up the capture infrastructure. To begin, you will need a usb type-c to type-c cord. Use this to connect the test device to the laptop (via adb). Make sure the device is connected by running the command adb devices. If you see a device in the list, you should be ready. If not, simply try detaching and reconnecting the usb connection.

Running the captures

Before you begin, make sure that your setup has followed all instructions till here and all the pre-requisites have been satisfied. If so, we are now ready to start the captures. The following is a list of steps in order, that explicitly tell you how to run it.

1. Reboot the test device:

adb reboot 

2. Turn on Data Saver and Turn off Private DNS on the Test Device; both the settings can be found in Settings -> Networks
3. Turn on the Frida

adb shell su -c /data/local/tmp/frida-server &

4. Now connect the test device to the laptop using SOCKS Proxy. To do so first find the IP address of the laptop (for Macbooks, this can be found in Settings -> Wifi -> Details. Now set this to be the IP address of the SOCKS Proxy on the test device. Set the port to 8889. Turn on the SOCKS Proxy Connection.
5. Turn on Mullvad VPN on the laptop and make sure, that you are connected to a server in California.
6. Navigate to the scripts directory in the gpc-android repo and run the command

bash multi-app-automation-script.sh

Output files

The output file (.mitm and .har files) are stored in the mitm-captures folder. These files will not be uploaded to the remote. If you do want to upload them, please modify the .gitignore files accordingly.

Troubleshooting

Frida

Frida is the most likely component to break the testing framework. This is because new OTA Android updates can break its functionality. To fix the issue, you can try a couple of things:

• Install the latest version of frida server on android (arm64)
• Make sure the frida client and frida server are the same version
• Check the frida github for relevant information on the issue you may be facing

An error you are likely to encounter when running the Frida server is the following:

• {"type":"error","description":"Error: Unable to perform state transition; please file a bug","stack":"Error: Unable to perform state transition; please file a bug\n at bt (frida/node_modules/frida-java-bridge/lib/android.js:578:1)\n at frida/node_modules/frida-java-bridge/lib/class-model.js:112:1\n at Function.build (frida/node_modules/frida-java-bridge/lib/class-model.js:7:1)\n at k._make (frida/node_modules/frida-java-bridge/lib/class-factory.js:168:1)\n at k.use (frida/node_modules/frida-java-bridge/lib/class-factory.js:62:1)\n at frida/node_modules/frida-java-bridge/index.js:224:1\n at c.perform (frida/node_modules/frida-java-bridge/lib/vm.js:12:1)\n at _performPendingVmOpsWhenReady (frida/node_modules/frida-java-bridge/index.js:223:1)\n at _.perform (frida/node_modules/frida-java-bridge/index.js:204:1)\n at /internal-agent.js:490:6","fileName":"frida/node_modules/frida-java-bridge/lib/android.js","lineNumber":578,"columnNumber":1}

To resolve this issue, you can delete the Android run time library. Steps to uninstall run time:

1. Run "pm uninstall com.google.android.art"

pm uninstall com.google.android.art

2. Reboot your test device

adb reboot 

Network Connection

The path the network traffic takes in this setup is somewhat complicated. As such there could be several different places where the issue could be arising from:

• The SOCKS Proxy: The SOCKS Proxy connects the (Android) phone to the laptop. To make sure the connection is setup properly, first check the IP address of the laptop you are connecting the device to. Now set that to the IP address of to be reached in in SOCKS Proxy. But this alone is not enough. You also need to make sure that the port that you are sending the data to, is listening for incoming data. This would, for us, be MITM Proxy. The port we are using is 8889.
• MITM Proxy: The MITM Proxy is the next stage in the network transfer process. To make sure this is set up correctly make sure that MITM is up to date and running on the same port as defined in the SOCKS Proxy. For us, this should be 8889.
• Mullvad VPN: The last step in our network transfer process is the Mullvad VPN. I have almost never encountered issues with it, but in case you have a connection issue, it may be better to just test the setup without the VPN running to eliminate on potential point of issue.

5. Apps CSV

The apps_csv directory contains a collection of CSV files, each representing a category of apps on the Google Play Store. Each file contains a list of the top 40 free apps for a category.

5.1 Directory Contents

The directory contains the following files:

• Multiple CSV files with the naming convention apps_<CATEGORY>.csv where CATEGORY is the app category name from the Google Play Store, for example apps_ART-AND-DESIGN.csv
• A JavaScript file, trial-play-scraper.js, which can be used to scrape app data from the Google Play Store
• A bash shell script play-store-downloader.sh, which reads a CSV file and downloads the corresponding apps

Each CSV file contains the following columns:

• APP_ID: the unique ID of the app on the Google Play Store
• TITLE: the title of the app
• DEVELOPER: the developer of the app
• SCORE: the score of the app on the Google Play Store

5.2 How to Use

1. Clone this repo to your local machine with:

   git clone https://github.com/privacy-tech-lab/gpc-android.git

   Then, navigate to the app_csv directory.

2. To scrape app metadata from the Google Play Store for a particular category, first make sure you have Node.js installed.

3. Then, run the trial-play-scraper.js script with:

   node trial-play-scraper.js

4. To download APKs from the Google Play Store with the play-store-downloader.sh run:

   chmod +x play-store-downloader.sh
   ./play-store-downloader.sh

   Before running the downloader script replace email@gmail.com and password in the play-store-downloader.sh script with your Google Play Store email and password, respectively. Then, give the script execution permissions and run it. Doing so will download all the apps listed in the apps-ART_AND_DESIGN.csv file. To download apps from a different category, replace "apps-ART_AND_DESIGN.csv" with the desired CSV file name in the script.

5. If downloading apps with the play-store-downloader.sh fails, you can also use Raccoon as follows:

   1. Make sure to have a US-based IP address (e.g., via a VPN)
   2. Set up an account with Google's US Play Store
   3. Get Raccoon and a Raccoon Premium license. Use Raccoon's DummyDroid to extract the configuration from a real Android device
   4. Choose "Import Apps" in Raccoon and paste all apps' links in there (e.g., market://details?id=com.fishbrain.app)
   5. Sit and wait ...

6. If downloading apps with the two previous methods fails, you can also try the google-play method through apkeep.

6. Thank You!

We would like to thank our supporters!

Major financial support provided by the National Science Foundation.

Additional financial support provided by the Alfred P. Sloan Foundation, Wesleyan University, and the Anil Fernando Endowment.

Conclusions reached or positions taken are our own and not necessarily those of our financial supporters, its trustees, officers, or staff.