Fix test on emulator workflow failures #734

sunmou99 · 2021-11-04T00:08:39Z

Description

Test on Emualtor sometimes failed when install Android app. https://github.com/firebase/firebase-cpp-sdk/runs/4078767834?check_suite_focus=true

Install testapp: adb install /Users/runner/work/firebase-cpp-sdk/firebase-cpp-sdk/testapps/firestore/iti-debug.apk
cmd: Failure calling service package: Broken pipe (32)
adb: failed to install /Users/runner/work/firebase-cpp-sdk/firebase-cpp-sdk/testapps/firestore/iti-debug.apk: cmd: Failure calling service package: Broken pipe (32)
Performing Streamed Install

This article suggests delete the old emulator and create a new one :
https://medium.com/@aedwin905/solved-failure-calling-service-package-broken-pipe-32-3b860c7e04bb

Also saw some errors when downloading emulator, uninstalling apps.

Thus, I made the following changes:

For critical steps, I added 5 minutes timeout and 3 times retry.
Before retry the step, I also reset and reboot the emulator.
Test on emulator timeout in 90 minutes.
Updated the build-tool version to 29.0.2

Testing

See the testing result in the comment

Type of Change

Place an x the applicable box:

Bug fix. Add the issue # below if applicable.
New feature. A non-breaking change which adds functionality.
Other, such as a build process or documentation change.

github-actions · 2021-11-04T00:09:18Z

❌ Integration test FAILED

Requested by @sunmou99 on commit 9efb188
Last updated: Tue Nov 9 19:05 PST 2021
View integration test log & download artifacts

Failures	Configs
firestore	[TEST] [ERROR] [Android] [ubuntu] [android_target]
messaging	[TEST] [ERROR] [Android] [ubuntu, macos] [emulator_target] [TEST] [ERROR] [Android] [windows] [android_target]

Add flaky tests to go/fpl-cpp-flake-tracker

scripts/gha/test_simulator.py

chkuang-g · 2021-11-09T22:26:27Z

scripts/gha/test_simulator.py

@@ -59,12 +59,12 @@
 Device Information is stored in TEST_DEVICES in print_matrix_configuration.py
 Example:
 sdk id "system-images;android-29;google_apis;x86":
-  --android_sdk "system-images;android-29;google_apis;x86" --build_tools_version "28.0.3"
+  --android_sdk "system-images;android-29;google_apis;x86" --build_tools_version "29.0.2"


Is there a need to update the version here since it is just an example?

Similarly to line 67

No need, but I feel there is not harm too.

chkuang-g · 2021-11-09T22:34:39Z

scripts/gha/test_simulator.py

+  if retry_time > 1:
+    try:
+      subprocess.run(args, shell=shell, check=check, timeout=timeout)
+    except:


We should log the exception here so that we know what is triggering the retry.

subprocess.run itself print out logs and exceptions.
log example: https://github.com/firebase/firebase-cpp-sdk/runs/4145793172?check_suite_focus=true#step:9:78

chkuang-g · 2021-11-09T22:36:57Z

scripts/gha/test_simulator.py

  else:
    time.sleep(45)

+def _run_with_retry(args, shell=False, check=True, timeout=_CMD_TIMEOUT, retry_time=_TEST_RETRY, device=_DEVICE_NONE, type=_RESET_TYPE_REBOOT):


We should log when all the retry attempts failed.

Thanks for the suggestion. I just added log in another PR (due to branch dependency): https://github.com/firebase/firebase-cpp-sdk/pull/733/files#diff-51ade566b474eb1faa5dfc7cfdcc3d943f850740c053a79ba0e4faa9f6f89315R679

chkuang-g · 2021-11-09T22:38:31Z

scripts/gha/test_simulator.py


  args = ["sdkmanager", "--update"]
  logging.info("Update all installed packages: %s", " ".join(args))
-  subprocess.run(args=args, check=True)
+  _run_with_retry(args, check=False)


Why changing check from True to False?

This is a non-critical step.
check=True only when this step block next steps.

chkuang-g · 2021-11-09T22:39:32Z

scripts/gha/test_simulator.py

@@ -129,7 +137,7 @@
    "android_sdk", "system-images;android-29;google_apis;x86",
    "See module docstring for details on how to set and get this id.")
 flags.DEFINE_string(
-    "build_tools_version", "28.0.3",
+    "build_tools_version", "29.0.2",


Isn't this value in build.gradle? Is this the right place to update the config value like this?

We are using tools under $ANDROID_HOME/build-tools/${build_tools_version}

* Fix test on emulator workflow failures (#734) * If simulator install ios app failed, reset simulator and try again (#733) * Trigger workflow move github api cod to github.py (#746) * Fix a data race that could manifest as null pointer dereference in FutureBase::Release() (#747) * Cancel callbacks for messaging (#745) * Cancel callbacks for messaging util::Terminate is referenced counted som when there ar more APIs than messaging active the callbacks will not be canceled until later and still cause a NULL ref due to the FutureData being destroyed now. * Cancel callback earlier * Update readme * Remove "Android" tag from the release notes entry for #747 (#749) * Remove calls to LogInfo, LogError, LogDebug during obj-c +load. (#706) * Remove calls to LogInfo, LogError, LogDebug during obj-c +load. This could be causing an issue in C++ as global class constructors have not yet been run. * Add Objective-C/C++ and Java to code formatter script; format those files. (#755) * Allow format_code to format .m/.mm files; clang-format already knows how. * Run format_code.py on all objective-c/objective-c++ files. * Add Java file extensions to format_code.py * Format all Java source files. * Remove check for objc header, as they are now supported. * Format objective-c .h files. * Don't let lint comment on line length any more; code formatting will report that. * Messaging crash during initialization (#760) * Messaging crash during initialization * Update readme * Don't redeclare inherited state in CredentialsProviderDesktop (#731) * Reduce disk space usage when packaging the built SDK (#763) Remove intermediate build files during desktop packaging step. This should reduce the disk space usage, as those files (*.o and *.obj) are not required when merging libraries. * Workaround for Linux x86 build: downgrade libraries on GitHub runners (#764) When installing 32-bit Linux dependencies on GitHub runners, downgrade libpcre2-8-0 to an earlier version to ensure compatibility with the i386 version of the package. This is something that should be fixed in a subsequent Ubuntu release and so is a temporary workaround. This also adds checks to the various prerequisite commands run by build_desktop.py, which was previously just silently ignoring errors (making this much harder to track down). Now it will error out as soon as a command fails. Co-authored-by: Mou Sun <69009538+sunmou99@users.noreply.github.com> Co-authored-by: Denver Coneybeare <dconeybe@google.com> Co-authored-by: Tobias Barendt <tobias@robotsquid.com> Co-authored-by: Jon Simantov <jsimantov@google.com> Co-authored-by: Sebastian Schmidt <mrschmidt@google.com>

Reset Android when install app failed

06c9c7f

google-cla bot added the cla: yes label Nov 4, 2021

sunmou99 added the tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). label Nov 4, 2021

github-actions bot added tests: in-progress This PR's integration tests are in progress. and removed tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). labels Nov 4, 2021

Wait 210s for emualtor boot

7b666d8

github-actions bot added the tests: failed This PR's integration tests failed. label Nov 4, 2021

firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Nov 4, 2021

build_tools_version 29.0.2

a3c4c1a

sunmou99 added the tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). label Nov 4, 2021

firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Nov 4, 2021

retry 3 time install/uninstall android apps

dd7ceca

sunmou99 added the tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). label Nov 4, 2021

github-actions bot added tests: in-progress This PR's integration tests are in progress. and removed tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). tests: failed This PR's integration tests failed. labels Nov 4, 2021

critical steps 5 minutes timeout. Testing job 90 minutes timeout

823b056

sunmou99 changed the title ~~If emulator install android app failed, reset emulator and try again~~ Nov 4, 2021

sunmou99 requested review from jonsimantov and chkuang-g November 4, 2021 23:21

sunmou99 added the tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). label Nov 4, 2021

github-actions bot added tests: failed This PR's integration tests failed. and removed tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). labels Nov 4, 2021

firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Nov 5, 2021

do not force unistall apps

6092ee9

sunmou99 added the tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). label Nov 5, 2021

github-actions bot added tests: failed This PR's integration tests failed. and removed tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). labels Nov 5, 2021

firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Nov 5, 2021

fix format

1bb22cc

jonsimantov reviewed Nov 5, 2021

View reviewed changes

scripts/gha/test_simulator.py Outdated Show resolved Hide resolved

sunmou99 requested a review from jonsimantov November 5, 2021 23:26

run with retry

0bf525b

sunmou99 added the tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). label Nov 8, 2021

github-actions bot added tests: in-progress This PR's integration tests are in progress. and removed tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). tests: failed This PR's integration tests failed. labels Nov 8, 2021

run with retry support iOS

e9d22bd

sunmou99 added the tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). label Nov 8, 2021

github-actions bot removed the tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). label Nov 8, 2021

fix bug

1411e40

github-actions bot added the tests: failed This PR's integration tests failed. label Nov 8, 2021

firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Nov 9, 2021

chkuang-g reviewed Nov 9, 2021

View reviewed changes

sunmou99 requested a review from chkuang-g November 9, 2021 23:09

chkuang-g approved these changes Nov 9, 2021

View reviewed changes

sunmou99 added skip-release-notes Skip release notes check and removed tests: failed This PR's integration tests failed. labels Nov 10, 2021

sunmou99 merged commit 9efb188 into main Nov 10, 2021

github-actions bot added the tests: in-progress This PR's integration tests are in progress. label Nov 10, 2021

sunmou99 deleted the bugfix/emulator-install-android branch November 10, 2021 00:11

github-actions bot added the tests: failed This PR's integration tests failed. label Nov 10, 2021

firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Nov 10, 2021

firebase locked and limited conversation to collaborators Dec 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix test on emulator workflow failures #734

Fix test on emulator workflow failures #734

sunmou99 commented Nov 4, 2021 •

edited

Loading

github-actions bot commented Nov 4, 2021 •

edited

Loading

chkuang-g Nov 9, 2021

sunmou99 Nov 9, 2021

chkuang-g Nov 9, 2021

sunmou99 Nov 9, 2021

chkuang-g Nov 9, 2021

sunmou99 Nov 9, 2021

chkuang-g Nov 9, 2021

sunmou99 Nov 9, 2021

chkuang-g Nov 9, 2021

sunmou99 Nov 9, 2021

Fix test on emulator workflow failures #734

Fix test on emulator workflow failures #734

Conversation

sunmou99 commented Nov 4, 2021 • edited Loading

Description

Testing

Type of Change

github-actions bot commented Nov 4, 2021 • edited Loading

❌ Integration test FAILED

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunmou99 commented Nov 4, 2021 •

edited

Loading

github-actions bot commented Nov 4, 2021 •

edited

Loading