A performance testing framework for measuring First Frame (Time to Initial Display) values of Fire OS apps on Amazon Fire tablets and Fire TVs.

• 📊 Performance metrics: First Frame (TTID) values for both Cool and Warm Launch Scenario
• 📈 Calculation of Tp50, Tp90, and average values of the execution
• 📝 Log storage for further debugging

Before you begin, ensure you have the following installed:

• Java 8 or above
• Java Development Kit 17 or above
• Android SDK (API Level 34)
• Maven

And remember to disable the screensaver on the Test Device.

1. Clone the repository:

git clone https://github.com/amazonappdev/fireos-perf-testing.git

2. Navigate to the project directory:

cd fireos-perf-testing

3. Build the project and create the JAR:

mvn clean install

4. Locate the ZIP file FOSTTIDPerfTesting-distribution.zip in the 'jar' folder.

1. Copy and unzip FOSTTIDPerfTesting-distribution.zip to your workspace.
2. Place your apk to test in the folder ./Input/APK/ of your workspace where you have extracted FOSTTIDPerfTesting-distribution.zip
3. Rename the apk to test to Input.apk
4. Execute the JAR:

   java -jar PerformanceKpi-jar.jar <KpiType> <DSN(DeviceSerialNumber)>
   

   Where the KpiType can be latency(For both cool and warm launch Execution), cool, warm and <DSN(DeviceSerialNumber)> is the name of your testing device listed executing adb devices
5. Wait for the completion of the tests.

After the tests are complented you can analyze the results opening the logs generated:

• Runtime Logs: Detailed test execution information
• CSV File: Test result values and app details
• HTML Report: UI representation of test results
• Log Folder: Device logs for each test iteration

You can customize the following parameters for the tests to be excuted modifying commonconfig.properties:

• Test iterations (latencyIterations)
• Wait time after app launch (latencyWait)
• Device reboot option (reboot_device)

Tp50 (50th Percentile, Median): Definition: The value below which 50% of the data points fall. Usage: Indicates the median performance, less affected by outliers. Example: If Tp50 of response times is 200 ms, 50% of the responses are 200 ms or faster.

Tp90 (90th Percentile): Definition: The value below which 90% of the data points fall. Usage: Highlights the upper range of typical performance, showing that most responses are this fast or faster. Example: If Tp90 is 500 ms, 90% of the responses are 500 ms or faster.

Average (Mean): Definition: The sum of all values divided by the number of values. Usage: Gives a general sense of overall performance but can be skewed by outliers. Example: If response times are 100 ms, 200 ms, and 1000 ms, the average is 433 ms.

Comparison: Tp50 and Tp90: Less affected by extreme values, providing a clearer picture of typical and upper-range performance. Average: Can be skewed by outliers, giving a less accurate picture of typical performance.

If you found a bug or want to suggest a new [feature/use case/sample], please file an issue.

If you have questions, comments, or need help with code, we're here to help:

• on X at @AmazonAppDev
• on Stack Overflow at the amazon-appstore tag

Get the most up to date Amazon Appstore developer news, product releases, tutorials, and more:

• 📣 Follow @AmazonAppDev and our team on Twitter

• 📺 Subscribe to our Youtube channel

• 📧 Sign up for the Developer Newsletter

• @CharanSingh
• @giolaq