Firestore: Attempt to get a record without an in-progress transaction

[Saturnyn]

Hello,

sorry if this is not a proper bug report, but I am out of ideas and thought I'd try and ask for help and advice here.

Our team is maintaining an ionic app that uses firestore with the firebase js sdk.
The persistence is activated on firestore: this is a requirement for the app, it has to be able to save important data while being offline for a long period of time.

For a while now, we've been getting different flavours of the following error:
FIRESTORE (6.0.2) INTERNAL ASSERTION FAILED: AsyncQueue is already failed.

• An internal error was encountered in the Indexed Database server
• The transaction was aborted, so the request cannot be fulfilled
• Transaction timed out due to inactivity

We haven't been able to reproduce those errors consistently on our own devices, and this thread #1670 made us hope that the new iOS version would solve the problem, so we haven't been dealing with it directly and hoped that it would disappear after the iOS update.

However, now we have customers using iOS 13 and iPadOS and they're still getting "AsyncQueue is already failed" errors, expect with new messages:

• Attempt to get a record from database without an in-progress transaction
• Attempt to open a cursor in database without an in-progress transaction

We have run some tests on the device and were able to reproduce those new errors more reliably by just waiting for the iPad to lock itself and then waking it up.
It looks like updating to firebase sdk 6.6 makes this new kind of error disappear (on our testing device at least) but we have no idea if the previous errors are still there or not.

Digging into the bug reports more, we've seen that some of the "AsyncQueue is already failed" errors had actually also been coming from Android devices, so it looks like the problem is more complicated than we thought.

Can anyone tell me if firestore will work reliably on all platforms if we update to 6.6, or if there is anything else we can do to fix the problem ?

We are considering dropping firestore completely, but we want to make sure there is no solution before starting working on an alternative.

[REQUIRED] Describe your environment

• ionic info:

Ionic:
   ionic (Ionic CLI)  : 4.1.0 (/usr/local/lib/node_modules/ionic)
   Ionic Framework    : ionic-angular 3.9.5
   @ionic/app-scripts : 3.2.3
Cordova:
   cordova (Cordova CLI) : 6.5.0
   Cordova Platforms     : android 6.4.0, ios 4.5.5
   Cordova Plugins       : cordova-plugin-ionic-webview 1.2.1, (and 19 other plugins)
System:
   Android SDK Tools : 26.1.1
   ios-deploy        : 1.9.2
   NodeJS            : v8.12.0 (/Users/yann/.nvm/versions/node/v8.12.0/bin/node)
   npm               : 6.9.0
   OS                : macOS
   Xcode             : Xcode 11.0 Build version 11A420a

• Firebase SDK version: 6.0.2
• Firebase Product: firestore (auth and storage also used)

[REQUIRED] Describe the problem

Several INTERNAL ASSERTION FAILED: AsyncQueue is already failed errors happening on different platforms.

Steps to reproduce:

On iPadOS, let the device lock itself, then wake it up, firestore will often stop working after that (looks like it breaks auth to).
Could not find another way to reproduce consistently but it does happen quite often in other contexts.

[schmidt-sebastian]

@Saturnyn Thank you for your detailed report!

First off, I would like you to refer to existing discussions (#1642, #2136, #1670) regarding the issues you encountered on iOS 12. We know that there are still some outstanding problems (especially with MobileSafari), and unfortunately only the "An internal error was encountered in the Indexed Database server" was reported fixed in iOS 13.

Secondly - if I may repurpose your issue - I would like to focus on the new errors you are seeing:

• Attempt to get a record from database without an in-progress transaction
• Attempt to open a cursor in database without an in-progress transaction

Do you have stacktraces for these errors? It would also help us if you could provide debug logs (via https://github.com/firebase/firebase-js-sdk/blob/master/packages/firestore-types/index.d.ts#L129). Thank you!

[Saturnyn]

@schmidt-sebastian Hi, thanks for your message.

First I'd like to amend my initial message:

• it may not be relevant but the project also uses the real time storage database (only to monitor online presence for debugging purposes, the project does not rely on it)
• I am not sure how I messed that up, but the bug doesn't disappear with version 6.6, I actually can't get it to work. I'll need to dig into this a bit more and maybe open another ticket, it looks like for..of loops are compiled as (@firebase/util).forEach which doesn't exist. Not sure if it is a problem with my typescript version or the way our project is set up).

[EDIT] You may want to look at my next post directly as I was actually able to get more useful debug information

I switched back to 6.0.2 and activated the debug logs, you can see the result in the joined text file firestore-error-logs.txt.
Here is what I did:

• open the app, make an operation that edits a simple object
• wait for the iPad to autolock after 2 minutes
• wake it up
• do another edit
  Repeat that a 2 or 3 times until the edit triggers the error.
• plug the iPad and get the logs from safari (can't plug it before because it won't autolock otherwise, so I can't get a breakpoint on the first error)

From what I've seen, something breaks in the code that manages the persistence when I wake the iPad. After that, any operation with firestore seems to fail.

I also joined a screenshot of the debugger after setting a breakpoint on another edit operation, but I am not sure it is relevant, as the root problem seems to happen before I can open the debugger and I can only see what fails after that.

Let me know if I can do something else to provide more information.

[Saturnyn]

Actually, I can reproduce the error by manually just tapping the power button to switch off the srceen while the app is open, then turning it on again.

Here are more precise logs with some annotations of what I was doing at each step, and a screenshot of the debugger at the first point of failure.
firestore-errors-logs2.txt

[Saturnyn]

FYI, here are some stats I have compiled to get a better idea of the scope of the problem.
Our user pool is very small, we have about 10 to 60 single users a day, but we see a few AsyncQueue error everyday.
If the iOS12 error disappear, a good chunk of the errors will disappear, but it looks like they will be replaced by the iOS13 errors at the moment :/

This chart shows the repartition of the errors by type:

Note:

• each bar shows the number of unique users getting one of the errors each day, so users may be counted twice as they can get two different errors
• the light bar shows the total number of users, for scale

[schmidt-sebastian]

Thank you for these updates! From looking through your logs, it makes me wonder if the transactions are started before the device goes to sleep (or locked) and are then picked back up when the app goes back into the foreground. I was not able to confirm this behavior on iOS 13 though.

Can you check if you can reproduce your issue without Cordova? I tested using https://github.com/firebase/friendlyeats-web with iOS 13 on both Simulator and Device and did not see any errors.

[nVitius]

I've spent a bit of time looking through and trying to understand the Webkit patch that was included in iOS 13 for the previous IndexedDB issue (https://bugs.webkit.org/attachment.cgi?id=368148&action=prettypatch)

The patch makes it so that, when Webkit's network process is shutdown, all running transactions will be aborted (https://trac.webkit.org/browser/trunk/Source/WebKit/NetworkProcess/NetworkProcess.cpp#L2104)

Then when the network process is resumed, the transactions are resumed as well (https://trac.webkit.org/browser/trunk/Source/WebKit/NetworkProcess/NetworkProcess.cpp#L2169)

I'm not actually sure how the resuming works. It has to do with how Webkit is using CrossThreadTasks to execute database tasks. From the test that was added with the patch, it seems to be implied that transactions are, in fact, resumed (https://trac.webkit.org/browser/trunk/Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBSuspendImminently.html#L27)

In Firestore, we throw an error whenever a transaction is aborted with an error (https://github.com/firebase/firebase-js-sdk/blob/master/packages/firestore/src/local/simple_db.ts#L376).
When the transaction is aborted, the async-queue is marked as failed (https://github.com/firebase/firebase-js-sdk/blob/master/packages/firestore/src/util/async_queue.ts#L287)
and then it can no longer process any new items (se-js-sdk/blob/master/packages/firestore/src/util/async_queue.ts#L274)

I'm not familiar enough with the way the async-queue works here to know what a good solution would be. The WebKit patch doesn't explicitly label the transactions as aborted because the network is being suspended. So there's no easy way to tell if it will be resumed or not. I don't know what other reasons transactions are suspended for, so I'm not sure if it is safe to just ignore the fact that it was aborted.

[Saturnyn]

@schmidt-sebastian
I installed https://github.com/firebase/friendlyeats-web
I did the test by adding a rating and switching on and off the iPad and could not reproduce the issue.

Note: I don't know if I missed something, but it does not seem to be a good way to test firestore: if you add a rating on a device it won't appear on another (so you can not test the data synchronisation), and you can not add rating while offline (so it is not possible to really test the persistence part of firestore).
I did modify the demo to fix those two points but could not reproduce the AsyncQueue error anyway.

@nVitius Thanks for the info, it looks like the problem is not present in Safari so it could be that cordova doesn't use the same version of webkit ?

For now, I'll work under the assumption that the persistence is not compatible with cordova apps and look for an alternative, maybe try to use a plugin such as https://www.npmjs.com/package/cordova-plugin-firestore to use the native sdks instead.

[nVitius]

I've been able to reproduce the error on both iOS Safari and WkWebView. It is not as straightforward to reproduce as #1670 though. It happens when the app has been in the background or the phone has been blocked for a certain period of time. The network process needs to go to sleep before the error will trigger.

[schmidt-sebastian]

@nVitius The test case you linked to is really interesting (https://trac.webkit.org/browser/trunk/Tools/TestWebKitAPI/Tests/WebKitCocoa/IndexedDBSuspendImminently.html#L27) as it doesn't quite match my understanding of the transaction lifecycle. If we can confirm that a transaction can both be aborted and successful, then we may be able to address this issue in Firestore with only a minor change.

Unfortunately, just changing our SDK to allow for retries requires a substantial rewrite of parts of the SDK. We currently modify our in-memory state while we apply changes to IndexedDb and are not able to rollback the in-memory changes if our IndexedDb commit fails. That being said, if we don't find another way to deal with this on iOS 13 and going forward, this may be an avenue we have to explore.

@Saturnyn You are right - the ratings in FriendlyEats aren't a good example here, since they use Firestore Transactions (via the runTransaction()API). These don't work offline. The rest of the client should work when offline though, and you can search and order restaurants without network connectivity. I will try to come up with a smaller sample that can run in iOS 13 and see if I hit this error without runTransaction().

[nVitius]

@schmidt-sebastian I also thought it was strange that a transaction could be aborted and then resumed. I wasn't sure where to discuss this kind of question with the Webkit developers though. By making a new report on their bug tracker?

As for reproducing the issue, I was able to do it by just patching a document in Firestore every 500ms.

<template>
  <div>
    <pre>{{number}}</pre>
  </div>
</template>

<script>
  export  default {
    data() {
      return {
        count: 0
      }
    },
    computed: {
      number() {
        return this.count
      }
    },
    mounted() {
      setInterval(() => this.count += 1, 500)
    },
    watch: {
      count(value) {
        this.$firebase.firestore().collection('test').doc('ios-count').set({ count: value })
      }
    }
  }
</script>

I used Vue to set up the test project I created.

[schmidt-sebastian]

I have been able to reproduce this issue using a different sample app, but only after running Firestore in the background for a couple minutes and then locking the device. Every time I saw this error, I also saw the following:

I always got the same network error when I first saw the IndexedDb failure. I hope that this is enough for us to narrow down the root cause and find a work around.

[nVitius]

I have also seen that network error and not had the issue come up. It seemed to happen every time the device went to sleep.

[schmidt-sebastian]

I filed a WebKit issue (https://bugs.webkit.org/show_bug.cgi?id=202705), but we are trying to see if we can find a workaround for Firestore.

[tgangso]

Hopefully you can find a workaround, waiting for Apple to fix this can be tedious. Very easy to reproduce this issue, putting the PWA in the background for a few seconds and reading/writing does not work, it when it is opened again.

[schmidt-sebastian]

Just a quick update - we are busy building a workaround for this issue. We will roll this out as soon as possible.

[Pitouli]

Hello, same issue with my app.

It's a PWA, everything works well when loading. But if I close the app and reopen it immediately, then firestore-sync stops working, and when I force a "get", I have the following error :
Error: FIRESTORE (6.3.1) INTERNAL ASSERTION FAILED: AsynsQueue is already failed: An internal error was encountered in the Indexed Database server

This error only appears since iOS13. Before iOS13, the app was working correctly if you close it and reopen it quickly. And if you waited for more than tens of seconds, the app was completely reloaded, so there wasn't this problem (which is not the case any more).

[Saturnyn]

@schmidt-sebastian Hi, it's great that you were able to fix this problem so fast !
Do you know when those changes will be published in the npm registry, or if there is a way to get them otherwise ?

[schmidt-sebastian]

@Saturnyn We don't usually comment on release dates, but this change should be part of this week's JS release, which historically happens on Thursdays.

[hsubox76]

Since this is fairly urgent, we moved up the release, the fix is published now with firebase@7.2.1.

[Saturnyn]

Thanks for the speedy update, I really appreciate it.

Unfortunately, I have to report that I still get the same error in 7.2.1 :/
I am pretty sure the sources are up to date, as the logs show "7.2.1" as expected and I can see the changes from the last merge in the sources.

I used the same process as before:

• make a change (update a field)
• tap the power button on the iPad to turn the screen off
• wait 1 minute (this seems to be required, it works if I turn it back on too early)
• reopen the app (I can see an error in the logs already)
• make a change again => get an exception

firestore logs 7.2.1.txt

[schmidt-sebastian]

I'll take a look at this again. My test app was running with the new build for all of Tuesday on an iPhone 8 with iOS 13. If there is an easy fix, then I can provide a custom build for you so that we can verify this on your device as well.

[MarkChrisLevy]

@Saturnyn what iOS version you have on your device?

[schmidt-sebastian]

@Pitouli The issue you are hitting is #1670, which is supposedly fixed in iOS 13 (and replaced by this one). We haven't been able to work around #1670, and there likely is not a fix since it is an internal IndexedDB error that renders IndexedDB unusable.

[Saturnyn]

@schmidt-sebastian I still get the problem with the custom build (cf attached logs)

I am not sure if that is relevant, but I had to make more retries to get the issue this time.
I did the same thing as I did before update / switch off / wait / update / etc.... Did that 6 times with everything working.
So I killed the app, restarted it, noticed that my database field has the 4th value I entered instead of the 6th. Did another update / switch off / wait / update and got the error (first log file).

Restarted the app again to be sure, and got the crash at the first retry (second log file).

The iPad OS version is 13.1.2 (17A860).

firestore with fix.txt

firestore with fix 2.txt

[Pitouli]

@Pitouli The issue you are hitting is #1670, which is supposedly fixed in iOS 13 (and replaced by this one). We haven't been able to work around #1670, and there likely is not a fix since it is an internal IndexedDB error that renders IndexedDB unusable.

@schmidt-sebastian Since I'm using iOS 13.1.3, it seems it's not fixed in iOS 13 😢

It's even the opposite: before iOS 13, the PWA were not kept in memory more than 10 seconds, so you had a reload almost every time (and when you were killing the app, it was not killed. The lifecycle was completely independent).

Since iOS 13, it behaves almost like a real app: you can leave the app alone for a "long" time, and when you reopen it, it stayed where you were. But with this bug, it becomes an issue rather than an improvement... The only good news is that now you can kill the app soit reload when you reopen it.

Anyway, if it's not related to this thread bug, I'll raise the issue elsewhere! :)

Thanks for your help

[tgangso]

Also still got the same issue, tested 7.2.1 with no luck. iOS 13.1.3, as PWA and in Safari.

[schmidt-sebastian]

The next release will include a more aggressive retry mechanism.

[tgangso]

Looking forward to it, when can we test it?

[schmidt-sebastian]

I uploaded the Web build to https://storage.googleapis.com/firebase-preview-drop/web/firestore/indexeddb-logging/firebase-firestore-rc02.js

[pashton700]

Hi, just wondering if this fix is in the latest version (7.2.2) that's just been released? I just tested it in my application and received the following message while testing:-

Error: FIRESTORE (7.2.2) INTERNAL ASSERTION FAILED: AsyncQueue is already failed: Attempt to delete range from database without an in-progress transaction.

It's not quite the same error as originally reported so I don't know if it's related to the same root cause. My application is a PWA using Ionic/Cordova and I'm testing on iOS 13.1.3

Thanks.

[ngerritsen]

The next release will include a more aggressive retry mechanism.

That would be great. Having the same issue for my app (https://timiks.com). It seems to occur both in PWA and in regular Safari although the PWA one seems to be more problematic. Tried to add my own retries around it but it's hard to reproduce it and test it.

[schmidt-sebastian]

7.2.2 includes the more aggressive fix. I am sorry to hear that the issue has not been resolved. It would be interesting to see if somehow has a small and reliable re-production. 7.2.2 retries any IndexedDB transaction up to 3 times if it fails with an error that is not a FirebaseError (which should catch any error thrown by IndexedDB).

While I have been able to confirm that our recovery logic does kick in on Mobile Safari (at least in my testing), I have not seen this issue in Ionic/Cordova since we published the first fix and am thus unable to confirm whether the recovery works in these environments.

[schmidt-sebastian]

Re-initializing the app and obtaining a new Firestore instance should solve this issue.

[pashton700]

Thanks very much for continuing to work on this. I've built a very small Ionic/Cordova testing app using 7.2.0 to try to consistently reproduce the error and will then re-test that using 7.2.2+. I'll come back when I have some results.

As a side note, I see a very significant improvement with 7.2.2 so thanks very much for your work on that. I have had one or two failures since upgrading about 5 days ago and prior to that, I was seeing errors frequently. I've included a step the reinitialize the app. when it encounters an error and that seems to be working fine.

Thanks.

[schmidt-sebastian]

Thanks for the update! #2307 should be released tomorrow (unless we hit a blocker somewhere).

[ngerritsen]

I definitely see my error rate dropping at 7.2.2. Great work! I still have issues often with opening and closing an iOS homescreen webapp, we’ll see if #2307 improves that.

[schmidt-sebastian]

The new version was just published. We are still working on the release notes, but you can get #2307 via the NPM and CDN builds.

[pashton700]

Great news.. thanks! I've installed 7.2.3 and will run some tests. Thanks again for working on this.

[mikelehen]

I'm optimistically closing this issue in hopes that the improvements @schmidt-sebastian made in the 7.2.3 SDK release will suffice until Apple addresses this in iOS Safari. If folks are still able to reproduce the "Attempt to ... without an in-progress transaction." errors, please comment below and we'll re-open this. If you're seeing any other error, please open a new issue. Thanks!