Fix a data race that could manifest as null pointer dereference in FutureBase::Release() #747

dconeybe · 2021-11-12T06:09:18Z

The FutureBase class has two data members that can be accessed concurrently from multiple threads without any synchronization: api_ and handle_. This data race was detected by the thread sanitizer.

Although there are no known customer reports of crashes resulting from this data race, a recent continuous integration run crashed in Firestore with a null pointer dereference in the following stack trace:

firebase::FutureBase::Release()
firebase::FutureBase::~FutureBase()
firebase::Future<void>::~Future()
firebase::firestore::FirestoreIntegrationTest_FirestoreCanBeDeletedFromTransaction_Test::TestBody()

The test in question is FirestoreCanBeDeletedFromTransaction:

firebase-cpp-sdk/firestore/integration_test_internal/src/firestore_test.cc

Lines 1528 to 1544 in 9efb188

    
           TEST_F(FirestoreIntegrationTest, FirestoreCanBeDeletedFromTransaction) { 
        
             Firestore* db = TestFirestore(); 
        
             DisownFirestore(db); 
        
             auto future = db->RunTransaction( 
        
                 [](Transaction&, std::string&) { return Error::kErrorOk; }); 
        
             std::promise<void> callback_done_promise; 
        
             auto callback_done = callback_done_promise.get_future(); 
        
             future.AddOnCompletion([&](const Future<void>&) mutable { 
        
               delete db; 
        
               callback_done_promise.set_value(); 
        
             }); 
        
             Await(future); 
        
             callback_done.wait(); 
        
           }

This test deletes the Firestore instance from an auxiliary thread, which would cause all of the Future objects it owns to be "released" from that alternate thread. Therefore, in rare conditions, this would race with the deletion of the Future object from the test thread, and yield the null pointer dereference.

This PR fixes the data race by protecting those two members with a mutex.

Googlers can see b/183225305 and b/205966141 for more details.

github-actions · 2021-11-12T06:10:38Z

❌ Integration test FAILED

Requested by @dconeybe on commit a219a97
Last updated: Mon Nov 15 19:40 PST 2021
View integration test log & download artifacts

Failures	Configs
messaging	[TEST] [ERROR] [Android] [ubuntu] [android_target]

Add flaky tests to go/fpl-cpp-flake-tracker

… api_ was always assigned to null

…plicated in the call to the copy/move operators

…e private headers

jonsimantov · 2021-11-15T20:03:43Z

app/src/include/firebase/future.h

@@ -20,6 +20,7 @@
 #include <stddef.h>
 #include <stdint.h>

+#include <mutex>


Can you use firebase::Mutex and firebase::MutexLock instead?

jonsimantov · 2021-11-15T20:06:36Z

release_build_files/readme.md

@@ -569,6 +569,9 @@ code.
 ## Release Notes
 ### Next Release
 -   Changes
+    -   All Products (Android): Fixed a data race that could manifest as null


Can you change this to "General (Android):"?

🍞 Dismissed stale approval on external PR.

github-actions · 2021-11-15T20:20:59Z

app/src/include/firebase/future.h

@@ -20,6 +20,7 @@
 #include <stddef.h>
 #include <stdint.h>

+#include <mutex>


⚠️ Lint warning: <mutex> is an unapproved C++11 header.

github-actions · 2021-11-15T20:20:59Z

app/src/include/firebase/internal/future_impl.h

+    }
+    detail::RegisterForCleanup(api_, this);
+  }
+
  return *this;
 }

 #if defined(FIREBASE_USE_MOVE_OPERATORS)
 inline FutureBase::FutureBase(FutureBase&& rhs) noexcept
    : api_(NULL)  // NOLINT
 {


⚠️ Lint warning: { should almost always be at the end of the previous line

Oh, could you fix the lint warning actually?

Oh wait just noticed it's not in your code - never mind.

jonsimantov · 2021-11-16T00:50:13Z

release_build_files/readme.md

@@ -569,6 +569,9 @@ code.
 ## Release Notes
 ### Next Release
 -   Changes
+    -   General (Android): Fixed a data race that could manifest as null pointer


One last note (it can wait for the followup) - this isn't Android only, right? We can remove the "(Android)" tag.

Ahh right. I've opened #749 to fix the release notes entry.

I had originally thought that the issue was specific to Android, because the bug only surfaced on Android; however, it indeed affects all platforms.

* Fix test on emulator workflow failures (#734) * If simulator install ios app failed, reset simulator and try again (#733) * Trigger workflow move github api cod to github.py (#746) * Fix a data race that could manifest as null pointer dereference in FutureBase::Release() (#747) * Cancel callbacks for messaging (#745) * Cancel callbacks for messaging util::Terminate is referenced counted som when there ar more APIs than messaging active the callbacks will not be canceled until later and still cause a NULL ref due to the FutureData being destroyed now. * Cancel callback earlier * Update readme * Remove "Android" tag from the release notes entry for #747 (#749) * Remove calls to LogInfo, LogError, LogDebug during obj-c +load. (#706) * Remove calls to LogInfo, LogError, LogDebug during obj-c +load. This could be causing an issue in C++ as global class constructors have not yet been run. * Add Objective-C/C++ and Java to code formatter script; format those files. (#755) * Allow format_code to format .m/.mm files; clang-format already knows how. * Run format_code.py on all objective-c/objective-c++ files. * Add Java file extensions to format_code.py * Format all Java source files. * Remove check for objc header, as they are now supported. * Format objective-c .h files. * Don't let lint comment on line length any more; code formatting will report that. * Messaging crash during initialization (#760) * Messaging crash during initialization * Update readme * Don't redeclare inherited state in CredentialsProviderDesktop (#731) * Reduce disk space usage when packaging the built SDK (#763) Remove intermediate build files during desktop packaging step. This should reduce the disk space usage, as those files (*.o and *.obj) are not required when merging libraries. * Workaround for Linux x86 build: downgrade libraries on GitHub runners (#764) When installing 32-bit Linux dependencies on GitHub runners, downgrade libpcre2-8-0 to an earlier version to ensure compatibility with the i386 version of the package. This is something that should be fixed in a subsequent Ubuntu release and so is a temporary workaround. This also adds checks to the various prerequisite commands run by build_desktop.py, which was previously just silently ignoring errors (making this much harder to track down). Now it will error out as soon as a command fails. Co-authored-by: Mou Sun <69009538+sunmou99@users.noreply.github.com> Co-authored-by: Denver Coneybeare <dconeybe@google.com> Co-authored-by: Tobias Barendt <tobias@robotsquid.com> Co-authored-by: Jon Simantov <jsimantov@google.com> Co-authored-by: Sebastian Schmidt <mrschmidt@google.com>

dconeybe added 2 commits November 12, 2021 01:00

Fix race condition in Future/FutureBase

7788946

Add release notes entry skeleton

52c6da1

dconeybe added the api: firestore label Nov 12, 2021

dconeybe self-assigned this Nov 12, 2021

google-cla bot added the cla: yes label Nov 12, 2021

Add a link to the PR in the release notes

18e0cc4

dconeybe added the tests-requested: quick Trigger a quick set of integration tests. label Nov 12, 2021

github-actions bot added tests: in-progress This PR's integration tests are in progress. and removed tests-requested: quick Trigger a quick set of integration tests. labels Nov 12, 2021

github-actions bot added the tests: failed This PR's integration tests failed. label Nov 12, 2021

firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Nov 12, 2021

dconeybe added 2 commits November 15, 2021 11:31

future_impl.h: Fix a bug in FutureBase::operator=(FutureBase&&) where…

17d6beb

… api_ was always assigned to null

future_impl.h: Remove code from the copy/move constructors that is du…

ca4a566

…plicated in the call to the copy/move operators

dconeybe added the tests-requested: quick Trigger a quick set of integration tests. label Nov 15, 2021

github-actions bot added tests: in-progress This PR's integration tests are in progress. tests: failed This PR's integration tests failed. and removed tests-requested: quick Trigger a quick set of integration tests. tests: failed This PR's integration tests failed. labels Nov 15, 2021

Swap out firebase::Mutex with std::mutex since future.h cannot includ…

4be3dd7

…e private headers

dconeybe added the tests-requested: quick Trigger a quick set of integration tests. label Nov 15, 2021

github-actions bot removed tests-requested: quick Trigger a quick set of integration tests. tests: failed This PR's integration tests failed. labels Nov 15, 2021

jonsimantov reviewed Nov 15, 2021

View reviewed changes

jonsimantov previously approved these changes Nov 15, 2021

View reviewed changes

release_build_files/readme.md: tweak release notes entry

4aeb741

github-actions bot reviewed Nov 15, 2021

View reviewed changes

dconeybe enabled auto-merge (squash) November 15, 2021 20:22

github-actions bot added the tests: failed This PR's integration tests failed. label Nov 15, 2021

firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Nov 15, 2021

dconeybe added the tests-requested: quick Trigger a quick set of integration tests. label Nov 15, 2021

github-actions bot added tests: in-progress This PR's integration tests are in progress. tests: succeeded This PR's integration tests succeeded. and removed tests-requested: quick Trigger a quick set of integration tests. tests: failed This PR's integration tests failed. labels Nov 15, 2021

firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Nov 15, 2021

jonsimantov self-requested a review November 16, 2021 00:49

jonsimantov reviewed Nov 16, 2021

View reviewed changes

jonsimantov approved these changes Nov 16, 2021

View reviewed changes

dconeybe merged commit a219a97 into main Nov 16, 2021

github-actions bot added tests: in-progress This PR's integration tests are in progress. and removed tests: succeeded This PR's integration tests succeeded. labels Nov 16, 2021

dconeybe deleted the dconeybe/FixFutureDataRaces branch November 16, 2021 02:04

dconeybe added a commit that referenced this pull request Nov 16, 2021

Remove "Android" tag from the release notes entry for #747

c8dabde

I had originally thought that the issue was specific to Android, because the bug only surfaced on Android; however, it indeed affects all platforms.

github-actions bot added the tests: failed This PR's integration tests failed. label Nov 16, 2021

firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Nov 16, 2021

dconeybe mentioned this pull request Nov 16, 2021

Split mutex.h into .h and .cc files #751

Merged

dconeybe added a commit that referenced this pull request Nov 22, 2021

Remove "Android" tag from the release notes entry for #747 (#749)

29089a7

dconeybe mentioned this pull request Dec 15, 2021

Move mutex.h header to public internal #792

Merged

firebase locked and limited conversation to collaborators Dec 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix a data race that could manifest as null pointer dereference in FutureBase::Release() #747

Fix a data race that could manifest as null pointer dereference in FutureBase::Release() #747

dconeybe commented Nov 12, 2021 •

edited

Loading

github-actions bot commented Nov 12, 2021 •

edited

Loading

jonsimantov Nov 15, 2021

jonsimantov Nov 15, 2021

dconeybe Nov 15, 2021

github-actions bot Nov 15, 2021

github-actions bot Nov 15, 2021

jonsimantov Nov 16, 2021

jonsimantov Nov 16, 2021

jonsimantov Nov 16, 2021

dconeybe Nov 16, 2021

	TEST_F(FirestoreIntegrationTest, FirestoreCanBeDeletedFromTransaction) {
	Firestore* db = TestFirestore();
	DisownFirestore(db);

	auto future = db->RunTransaction(
	[](Transaction&, std::string&) { return Error::kErrorOk; });

	std::promise<void> callback_done_promise;
	auto callback_done = callback_done_promise.get_future();
	future.AddOnCompletion([&](const Future<void>&) mutable {
	delete db;
	callback_done_promise.set_value();
	});

	Await(future);
	callback_done.wait();
	}

Fix a data race that could manifest as null pointer dereference in FutureBase::Release() #747

Fix a data race that could manifest as null pointer dereference in FutureBase::Release() #747

Conversation

dconeybe commented Nov 12, 2021 • edited Loading

github-actions bot commented Nov 12, 2021 • edited Loading

❌ Integration test FAILED

jonsimantov Nov 15, 2021

Choose a reason for hiding this comment

jonsimantov Nov 15, 2021

Choose a reason for hiding this comment

dconeybe Nov 15, 2021

Choose a reason for hiding this comment

github-actions bot Nov 15, 2021

Choose a reason for hiding this comment

github-actions bot Nov 15, 2021

Choose a reason for hiding this comment

jonsimantov Nov 16, 2021

Choose a reason for hiding this comment

jonsimantov Nov 16, 2021

Choose a reason for hiding this comment

jonsimantov Nov 16, 2021

Choose a reason for hiding this comment

dconeybe Nov 16, 2021

Choose a reason for hiding this comment

dconeybe commented Nov 12, 2021 •

edited

Loading

github-actions bot commented Nov 12, 2021 •

edited

Loading