Unity editor crash caused by Firestore lock issue

[vyrin]

Please fill in the following fields:

Unity editor version: 2019.3.7f1
Firebase Unity SDK version: 6.13.0
Source you installed the SDK (.unitypackage or Unity Package Manager): Unity Package Manager
Firebase plugins in use (Auth, Database, etc.): Auth, Firestore, App
Additional SDKs you are using (Facebook, AdMob, etc.): None (replicated in a clean new project)
Platform you are using the Unity editor on (Mac, Windows, or Linux): Mac
Platform you are targeting (iOS, Android, and/or desktop): Editor
Scripting Runtime (Mono, and/or IL2CPP): Mono

Please describe the issue here:

I created a brand new project and installed the Firebase SDK via the Unity Package Manager.

I created a single button that when pressed creates a single listener on a Firestore document.

public void Test() { var store = FirebaseFirestore.DefaultInstance; var doc = store.Document("/versions/1.0/people/foo"); doc.Listen(snapshot => { Debug.Log("Got snapshot"); }); }

I run the project (play button) in the editor, click the Test button, and everything works correctly.

I then stop the project (pause button), and then start the project again (play button).

When I click on the Test button the Unity Editor crashes.

This is the relevant portion of the crash report:

Application Specific Information: dyld: in dlopen() /Applications/Unity/Hub/Editor/2019.3.7f1/Unity.app/Contents/lib/libPackages/com.google.firebase.app/Firebase/Plugins/x86_64/FirebaseCppApp-6_13_0.bundle.so abort() called terminating with uncaught exception of type firebase::firestore::util::FirestoreInternalError: FIRESTORE INTERNAL ASSERTION FAILED: third_party/firebase/ios/Releases/FirebaseFirestore/core/src/firebase/firestore/core/firestore_client.cc(168) void firebase::firestore::core::FirestoreClient::Initialize(const firebase::firestore::auth::User &, const firebase::firestore::api::Settings &): Failed to open DB: Internal: Failed to open LevelDB database at /Users/colin/Library/Application Support/firestore/__FIRAPP_DEFAULT/com-quicksteplabs-wordz/main: LevelDB error: IO error: lock /Users/colin/Library/Application Support/firestore/__FIRAPP_DEFAULT/com-quicksteplabs-wordz/main/LOCK: already held by process (expected created.ok())

So, it appears the Editor is crashing because it attempts to lock the Firestore LOCK file, but it already locked it before (presumably the first time the project ran), and now it is crashing due to an assertion failure.

Please answer the following, if applicable:

This issue was replicated in a brand new project that simply had the Firestore SDK added via the Unity Package Manager.

This happens 100% of the time.

(I'm actually very surprised that I haven't seen anyone else report this issue. It is 100% consistent for me and has halted all of my progress integrating Firestore into my projects.)

[wilhuff]

Hi! Sorry for the trouble.

The issue is essentially as you describe. The prior instance of Firestore hasn't completely shut down and it is still holding the lock on the underlying database. The next instance starts up and then fails to acquire the lock. The trouble comes from the fact that there's some nondeterminism in the way we're handling the shutdown of the AppDomain that represents the test run.

We had recently discovered this issue ourselves, but I'm pretty surprised this is happening every time you pause. We have seen this failure approximately 1 in 20 times.

So the good news is we have a plan for how to fix this (and a start on the implementation), but unfortunately this will take some time to land.

[vyrin]

Thanks for the update. (And thank you so much for all the work on adding Firestore to Unity. I really really appreciate it!)

Do you have an estimate for when this issue will be fixed in an official release? Are we talking days, weeks, or months? (That will help me in my planning.)

And relatedly, is there any workaround that I could try to get it working beforehand?

[wilhuff]

Given the non overlapping release schedules of the components that are required for this, it really couldn’t happen sooner than two weeks from now at the earliest.

[AThilenius]

I also see this every time I reenter play-mode. Any recommendations on a quick (nasty) hack to release the lock until a patch comes out? Would save me lots of restarting-Unity time 😁

[wilhuff]

Unfortunately, the lock is being held by a C++ object that has leaked so there's no way to get back to it. Normally I would suggest that you could disable offline persistence in settings, but we haven't shipped that API yet...

If you want to work around this you need to find some way to get the two instances to use different files for their storage. There are a few flavors of this:

• On UNIX filesystems you also just remove the directory contents. The process will still have the file open, keeping the inode alive, but the filenames will become available for reuse. Once you exit the process the space will be reclaimed. This won't work on Windows.
• An alternative to the above is to move the directory containing the database aside (renaming the directory or similar).
• An alternative that's guaranteed to work but requires a code change is to create a new Firebase App (with a random name) for each run and create your Firestore instance from that App. The app name is part of the path Firestore uses, and a random app name will prompt the SDK to create a new database.

In all of these cases note that the files will still be open and the editor process will eventually run out of files/memory, but it will at least stop you from needing to restart every time.

[Thaina]

Not sure it was related. My editor also eventually keep crashing after switching play mode and edit mode in editor. I could test run my app 2-5 times before the editor will crash again

I don't know where to find unity editor crash log but I have file crash report to unity, the unity team have response back to me as this below

From the crash logs, I can see that this crash happens in FirebaseCppApp-6.0.0.dll

My unity report case is Case 1238725

Is it related or I should open new issue?

[wilhuff]

The Firestore lock issue is very clearly about Firestore failing to open its local database and the crash will be in the Firestore DLL. This sounds like a different issue.

[Thaina]

@wilhuff Thanks, I have file new issue #653

[kirilogan]

Seems the lock occurs when the code is changed while in play mode and the editor does not dispose properly. I've been able to stop it from crashing by making sure that I stop play mode before saving any code changes. Hope this helps for the time being.

[ijaureguialzo]

Following @wilhuff 's advice, I used this as temporary workaround and Unity has stopped crashing on every run:

db = FirebaseFirestore.GetInstance(FirebaseApp.Create());   // was FirebaseApp.DefaultInstance

Hope it helps.

[summitsteven]

Hi, I see PRs from @wilhuff pulled into the ios SDK in this thread 10 days ago. Does that mean the fix has made it in to the Unity SDK (which also released 10 days ago)? Or will that come in the next release? Is there a way we can tell how up to date the Unity SDK is regarding fixes/updates to other repos?

Thanks!

[wilhuff]

Sorry for the delay. The PRs that landed last week will go out in the next iOS release. After that we can make a Unity release that includes the fix.

[maximeLong]

Hey @wilhuff do you have an estimate on when the next Unity release will make it out? I'm getting this error as well and it has halted development on using Firestore for our iOS Unity app. Should we roll a realtime DB solution in the intermit, or will a hotfix be released soon for Unity Firestore SDK? Thanks.

[wilhuff]

As a general policy we can't really make promises about when things will go out. There are a variety of reasons why things may be delayed that are unrelated to these specific changes.

With that said, if all goes well, this fix should be in the next release. It's possible that might not be the case though: for example the next release might be a patch for the prior one for changes unrelated to Firestore.

[maximeLong]

With the addition of the methods TerminateAsync() and ClearPersistenceAsync() in the 6.15 release of Firebase Unity SDK, this issue has been resolved for me.

If I add these methods to the OnApplicationQuit event in my class, this issue seems to be resolved:

FirebaseFirestore db;

void Awake()
{
    Firebase.FirebaseApp.CheckAndFixDependenciesAsync().ContinueWith(task => 
    {
        var dependencyStatus = task.Result;
        if (dependencyStatus == Firebase.DependencyStatus.Available) 
        {
            db = FirebaseFirestore.GetInstance(FirebaseApp.Create());
        }
    });
}

void OnApplicationQuit() {
    db.TerminateAsync();
    db.ClearPersistenceAsync();
}

Note: Unity documentation states iOS applications are usually suspended and do not quit. So for this method to work you might need to tick "Exit on Suspend" in Player settings to cause the game to quit and not suspend. I have not tested this thoroughly yet.

[wilhuff]

Right, that's a workaround you can use for now. The fix that's pending should make it so that an explicit shutdown/clear like this should be unnecessary.

[wilhuff]

The fix for this was released with the Firebase Unity SDK 6.15.1.