Firestore: Deleting a couple of documents returned by a limited subcollection query triggers all documents to disappear locally

[aleh]

[REQUIRED] Step 2: Describe your environment

• Xcode version: 9.4 (9F1027a)
• Firebase SDK version: 5.3.0
• Firebase Component: Firestore
• Component version: 5.3.0

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

• Set up a query on any subcollection with a limit smaller than the number of elements in the subcollection already, for example, 5.

• Set up another query on the same subcollection with a limit set to 1. (Possibly unneeded step.)

• Start deleting documents returned by the first query randomly one by one.

• Observe that at some points the documents will begin disappearing from the query snapshot on their own. They will remain gone even if the app restarts but will be still visible via Firestore Console.

Relevant Code:

See example app along with more detailed description and video here.

Also, the following code in Firestore/Source/Remote/FSTRemoteEvent.mm in remoteEventAtSnapshotVersion: method seems to have something to do with this:

      if (targetState.current && [queryData.query isDocumentQuery]) {
        // Document queries for document that don't exist can produce an empty result set. To update
        // our local cache, we synthesize a document delete if we have not previously received the
        // document. This resolves the limbo state of the document, removing it from
        // limboDocumentRefs.
        FSTDocumentKey *key = [FSTDocumentKey keyWithPath:queryData.query.path];
        if (_pendingDocumentUpdates.find(key) == _pendingDocumentUpdates.end() &&
            ![self containsDocument:key inTarget:targetID]) {
          [self removeDocument:[FSTDeletedDocument documentWithKey:key version:snapshotVersion]
                       withKey:key
                    fromTarget:targetID];
        }
      }

[mikelehen]

Thank you for the repro app and video! I'll take a look and get back to you.

[wilhuff]

Thanks for filing this. It turns out to be the cause of this discussion. We'll treat this as the canonical issue for this bug. The problem exists on all Firestore mobile/web SDKs.

For context, when a client is listening to a limit query (and in certain other cases) and a document moves out of the query results the server is allowed to tell the client merely that a document is removed without fully describing why that was. This is an intentional ambiguity that gives us some wiggle room in the server implementation.

When this happens the client considers the document to be in limbo and attempts to figure out what happened to that document by requesting it out of band. Without this limbo resolution process, the next time the client ran the query you would see a flicker: the client would serve stale data from the cache which would disappear when the server gave up-to-date results again. This is something we try to avoid.

So with that background, the bug is in this limbo resolution process. There's a certain order of events from the server that can cause the client to mistakenly determine that a limbo document doesn't exist and this is causing the client to populate negative entries in its cache. The fix for this is in flight here: firebase/firebase-js-sdk#1014.

This alone would not be a problem because the next update from the server would fix the client's cache, but the client is also recording the commit time of the delete as the current time of the limbo resolution read, which is typically later than the last server-side modification. There are corner cases around updates to documents that don't match a query where the server is not obligated to send the absolute latest version of a document so the client will prefer its version of a document if it's later than the server issued one. This compounds the issue and causes it to be persistent. A fix that will allow the client to self-heal its cache is progress here: firebase/firebase-js-sdk#1015.

(Please don't be alarmed: we're fixing the js-sdk first because we have a cross-platform test suite written in typescript. It's easier to fix there and then port.)

[aleh]

@mikelehen @wilhuff Thanks for the detailed update!