Baha Aiman

feat(firestore): refactor pipeline API for uniform stage options (#14322) Design decision [here](https://docs.google.com/document/d/1diPUhXr2kZh69KcAsuOMZMj_rfcykGsE2-phmBIt8vs/edit?usp=sharing) **Overview** This PR standardizes the API surface for the Firestore Pipeline builder in the Go SDK, ensuring idiomatic conformance and aligning closely with cross-platform design paradigms (Java/Node.js). It replaces complex stage-specific configuration structs with a universal functional option pattern and introduces typed slice constants for varied expressions. **Key Features and API Changes:** * **Universal `RawOptions` Escape Hatch:** Introduced `RawOptions` to serve as a universal pass-through for backend options that might not be natively modeled by the Go SDK (e.g., `RawOptions{"limit": 5, "distance_field": "dist"}`). This deprecates intermediate parameter structs like `PipelineFindNearestOptions` across pipeline stages. * **Standardized Variadic Arguments with Slices:** Methods that formerly relied on raw variadic argument unpacking (e.g. `...Ordering`, `...any`) have been updated to accept typed slices. To preserve clean query ergonomics, we introduced domain-specific helper constructors (`Orders()`, `Fields()`, `Accumulators()`, `Selectables()`). * `Pipeline.Sort(Orders(Ascending(FieldOf("f")), Descending(FieldOf("__name__"))))` * `Pipeline.Aggregate(Accumulators(Count(DocumentID).As("total")))` * `Pipeline.Select(Fields("a", "b"))` * **Functional Stage Options Migration:** Implemented standard functional options for specialized stage modifiers: * `WithFindNearestLimit()`, `WithFindNearestDistanceField()` * `WithAggregateGroups()` * `WithUnnestIndexField()` * `WithExplainMode()` **Internal Code Simplification & Testing:** * **Flattened Stage Hierarchy (`baseStage` removal):** Removed the `baseStage` struct embedding across our internal pipeline stages. We replaced it with explicit `options map[string]any` fields within each stage and introduced a shared `stageOptionsToProto(options)` helper. This simplifies the internal struct hierarchy, removes opaque state inherited from struct embedding, and makes configuration serialization significantly more explicit and easier to trace inside each stage's `toProto()` implementation. * **Pipeline Stage Proto Parsing:** Refactored internal `pipelineStage` structures (`newAggregateStage`, `newFindNearestStage`, `newSortStage`, etc.) to reliably map the new `RawOptions` down to standard `*pb.Value` payloads. * **Tests for `AlwaysUseImplicitOrderBy`:** Added dedicated test cases (`TestQuery_AlwaysUseImplicitOrderBy`) to explicitly verify the behavior of the existing `WithAlwaysUseImplicitOrderBy` client toggle, ensuring it appropriately serializes implicit inequality fields and `__name__` into the `OrderBy` proto requests. * **Integration Suite Migration:** Fully migrated the integration suite inside `pipeline_integration_test.go` and `pipeline_test.go` to assert against the newly stabilized `RawOptions` and slice-builder interfaces. --------- Co-authored-by: Alex Hong <9397363+hongalex@users.noreply.github.com>

feat(firestore): Introduce new functions and literals stage (#14114) [go/firestore-query-tracker](http://go/firestore-query-tracker) : Introduce all the functions marked "P" (planned) in Go column O and "C" (Complete) in Backend column H in "Firestore Features (Pipeline)" sheet. The sheet above specified only the function name. The args for all the functions were determined from backend implementation in - google3/java/com/google/cloud/datastore/client/firestorev1/Functions.java - google3/java/com/google/cloud/datastore/client/firestorev1/Stages.java The documentation is only available for the functions marked "C" (Complete) in Docs column I. The same documentation has added to the client in this PR. - google3/googledata/devsite/_common/en/_shared/firestore/pipeline/functions/_shared/

feat(firestore): Add new functions and correct old ones (#14437) This PR significantly expands the Firestore Pipelines API by adding a wide range of new expressions and functions. These additions improve parity with other Firestore SDKs and provide developers with more powerful tools for complex data transformation and analysis within Firestore. Key Changes: * New Aggregations: Added RawAggregate to allow calling any aggregation function supported by the Firestore backend, even if not explicitly defined in the SDK. Java [ref](https://github.com/googleapis/java-firestore/blob/6acac4b0598891f66174e911ccf55375b587d773/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/AggregateFunction.java#L47-L49). * Arithmetic Expressions: Added RoundToPrecision for rounding to specific decimal places. Java [ref](https://github.com/googleapis/java-firestore/blob/6acac4b0598891f66174e911ccf55375b587d773/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java#L4711-L4761) * Array Functions: * Offset: Access array elements by index (supports negative indexing). * ArrayTransform & ArrayTransformWithIndex: Map array elements to new values using expressions. * ArrayFilter: Filter array elements based on a condition (now fully enabled). * Reference & Key Functions: * GetParent: Retrieve the parent reference of a document or field. * ReferenceSlice & ReferenceSliceToEnd: Extract segments from a document reference path. * Logical & General Functions: * Coalesce: Return the first non-null value from a list of expressions. Java [ref](https://github.com/googleapis/java-firestore/blob/6acac4b0598891f66174e911ccf55375b587d773/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java#L318-L348) * Variable: Reference variables within lambda-like expressions (e.g., in ArrayTransform). * StorageSize: Calculate the storage size of a field or expression. * RawBooleanFunction: Support for custom boolean evaluation functions. java [ref](https://github.com/googleapis/java-firestore/blob/6acac4b0598891f66174e911ccf55375b587d773/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/BooleanExpression.java#L87-L90) * Improved Type Safety: Refined SwitchOn and MapSet signatures to enforce required arguments, reducing runtime errors. * Internal Refinements: * Added validation helpers for timestamp units, granularity, and parts. * Unified internal proto conversion logic for binary and numeric operations.

test(bigtable): reduce execution time (#14468) This PR significantly reduces the execution time of the `bigtable` integration test suite by optimizing how test environments and resources are provisioned. It also improves test robustness by refactoring teardown lifecycles. ## Performance Optimizations * **Shared Test Environment:** Refactored `setupIntegration` to utilize `sync.Once`. Instead of creating a new `IntegrationEnv`, `Client`, `AdminClient`, and test table for every single test, a single shared environment is now provisioned and reused across the suite. * **Pre-created Schema:** Common column families (e.g., `follows`, `ts`, `export`, `bulk`, `counter`, `sum`) are now pre-created during the shared initialization. This eliminates the repetitive overhead of schema modifications in individual tests. * **Optimized Teardown:** Updated `deleteTable` to safely ignore `NotFound` errors, bypassing unnecessary retries if a table was already cleaned up. Modified `createColumnFamilyWithConfig` to immediately return on `AlreadyExists` rather than initiating a costly delete/retry loop. ## Lifecycle Robustness * **`t.Cleanup` Migration:** Converted lifecycle-related `defer` statements to `t.Cleanup` across `integration_test.go`. This guarantees that resource teardowns (closing clients, deleting backups/profiles/views) execute safely and correctly even when using `t.Parallel()` or failing subtests. ## Other Changes * Added explicit logic in `export_test.go` to disable DirectPath (`GOOGLE_CLOUD_DISABLE_DIRECT_PATH="true"`) when `AttemptDirectPath` is false.

feat(firestore): refine Pipelines API, error handling, and tests (#14429) This PR introduces several refinements to the Firestore Pipelines experimental feature, aimed at improving API consistency, error reporting, and the reliability of integration tests. Key Changes: - API Refinement: - Renamed ArraySliceWithLength to ArraySliceToEnd to better reflect its behavior similar to [Java](https://github.com/googleapis/java-firestore/blob/9065134fd70da22250f038d7e964f26aebfeb3cf/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java#L2644). - Enabled the ArrayFilter expression (previously commented out). - Updated TimestampTruncateWithTimezone to accept any for the timezone parameter, allowing for dynamic timezone expressions. TimestampTruncateWithTimezone in Java accepts string or expression [src](https://github.com/googleapis/java-firestore/blob/9065134fd70da22250f038d7e964f26aebfeb3cf/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java#L3763-L3908). - Error Handling: - Introduced typed errors ErrPipelineWithoutDatabase and ErrUnionNotSupportRelativeScope to replace generic fmt.Errorf calls https://github.com/googleapis/google-cloud-go/pull/14365#discussion_r3041945116. - Added internal nil checks across various pipeline stages (inputStageDocuments, findNearestStage, removeFieldsStage, etc.) to provide more descriptive error messages during proto conversion. - Integration Test Enhancements: - Replaced skipIfEnterprise with a more flexible skipIfEdition helper to handle feature support across different Firestore editions. - Improved resource cleanup in tests by moving t.Cleanup calls earlier, ensuring documents are deleted even if a test fails prematurely. - Updated deleteCollection to use Select().Documents(ctx) for more efficient document deletion during test setup/teardown. - Bug Fixes & Robustness: - Added nil checks when processing results in TestIntegration_Query_Pipeline. - Standardized test skipping logic across all pipeline-related integration tests.

feat(firestore): Implement Search pipeline stage and related expressions (#14383) **Summary** This PR implements the `search` pipeline stage and its associated expressions in the Go Firestore SDK, providing parity with the Java (https://github.com/googleapis/java-firestore/pull/2346) and Node.js (https://github.com/googleapis/google-cloud-node/pull/7824) SDKs for full-text and geospatial search capabilities within Firestore Pipelines. **Key Changes** * **Search Pipeline Stage**: * Implemented the `Search` stage in `Pipeline` with a functional options pattern. * Introduced `WithSearchQuery`, `WithSearchSort`, `WithSearchAddFields`, and `WithSearchRetrievalDepth` options. * Ensured protobuf encoding places the `query` within the `Options` map (not `Args`) to align with backend requirements and the Java SDK implementation. * **Search Expressions**: * Implemented `DocumentMatches(query string)` and `Matches(field, query string)` for full-text search. * Implemented `GeoDistance(field, location)` for geospatial distance calculations. * Implemented `Score()` to retrieve the search topicality score. * Implemented `Snippet(field, query)` for highlighted search result snippets. * **Fluent API**: * Added `GeoDistance(location)`, `Matches(query)`, and `Snippet(query)` methods to the `Expression` interface to support fluent chaining (e.g., `FieldOf("location").GeoDistance(loc)`). * **Testing**: * Added comprehensive unit tests in `pipeline_test.go` and `pipeline_function_test.go`. * Verified backend request structure consistency via JSON dump comparisons. **Usage Example** ```go pipeline := client.Pipeline(). Collection("restaurants"). Search( WithSearchQuery(DocumentMatches("waffles OR pancakes")), WithSearchSort(Descending(Score())), WithSearchAddFields(Snippet("menu", "waffles").As("highlight")), ) ```

feat(firestore): pipeline subqueries (#14365) **Overview** This PR introduces support for pipeline subqueries, variable definitions, and joins in the Go Firestore SDK, achieving strict 1:1 feature parity with the Java SDK's pipeline APIs https://github.com/googleapis/java-firestore/pull/2323. **What are Subqueries?** In Firestore pipelines, subqueries allow you to embed an entire pipeline execution as a value within a single stage of an outer pipeline. This is incredibly powerful for performing complex "join-like" operations across different collections. For example, while querying a restaurants collection, you can use a subquery to fetch, filter, and aggregate all documents from a nested reviews subcollection, and embed that aggregated result (e.g., average_rating) directly into the restaurant document being returned. Subqueries can be evaluated into either an array of results (ToArrayExpression()) or a single scalar value (ToScalarExpression()). **Key Features & API Additions:** * **Subqueries (Joins):** * Implemented the Subcollection(path string) package-level function to instantiate relative-scope pipelines. * Added ToScalarExpression() and ToArrayExpression() methods on Pipeline to explicitly convert subqueries into expressions, allowing them to be seamlessly embedded inside stages like AddFields and Where. * **Variable Definition & References:** * Introduced the Define pipeline stage and the AliasedExpressions variadic helper to ergonomically bind values to variables. * Added Variable("name") and CurrentDocument() top-level functions to reference bound values in subsequent pipeline stages. * **Field Access & Overloading:** * Implemented GetField (accepting any to support both string and Expression arguments, mirroring Java's method overloading). **Type Safety & Defensive Constraints:** * Strict Aliasing: Expression.As() now explicitly returns *AliasedExpression (similar to Java: https://github.com/googleapis/java-firestore/blob/0c8188520dfbada0d3fef0719e4f95fc231306be/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java#L6608-L6621) rather than a generic Selectable interface, and the Define stage strictly requires []*AliasedExpression. * Pipeline Scope Validation: Added validation to ensure relative-scope pipelines (e.g., those created via Subcollection) cannot be executed directly or passed into a Union() stage. Attempts to do so now return descriptive errors identical to the Java SDK's IllegalStateException and IllegalArgumentException.

test(bigtable): unskip flaky tests and add retry mechanisms to improve test reliability (#14514) ## Description This PR addresses flaky tests, unskips previously broken/flaky test cases, and introduces unary client retry interceptors along with test-level retry helpers to prevent transient test failures. ### Key Changes - **Restored & Re-enabled Flaky/Broken Tests:** - Re-enabled `TestIntegration_AuthorizedView`, `TestIntegration_DataMaterializedView`, and unskipped `TestIntegration_AdminMaterializedView`. - Re-enabled `TestCachingStreamDecrement` in `internal/transport/connpool_test.go`, and corrected the stream descriptors (`ServerStreams: true, ClientStreams: true`). - **Introduced gRPC Unary Retry Interceptor for Admin Client:** - Added `adminUnaryRetryInterceptor()` to both production and emulated test environments. - Retries unary client calls up to 5 times with exponential backoff upon encountering transient error codes (`Unavailable`, `Aborted`, `Internal`, `Unknown`, `ResourceExhausted`). - **Improved Resilience in Integration Tests:** - Uses `newClientWithRetry` for client instantiation to handle transient client setup errors. - Wrapped `CreateAuthorizedView` and `CreateMaterializedView` with retries to handle concurrent operations or propagation delays. - Introduced `updateInstanceAndSyncClustersWithRetry` helper to retry on `ErrPartiallyUnavailable`. - Added `updateTableWithDeletionProtectionWithRetry` with `gax.Backoff` to avoid failures due to resource propagation delays. - Gracefully ignores `NotFound` errors during cleanup in `deleteBackupAndLog`. - **CI and Setup Updates:** - Documented and exported environment variables `GCLOUD_TESTS_BIGTABLE_TAG_KEY` and `GCLOUD_TESTS_BIGTABLE_TAG_VALUE` in `CONTRIBUTING.md` and `internal/kokoro/continuous.sh` to support instance creation and tag tests. Fixes #14498 Fixes #12686 Fixes #12415 Fixes #11927 Fixes #11523 Fixes #11268

Fspq fix stages (#14253) Java stages for reference: https://github.com/googleapis/java-firestore/blob/6e30a6c11efe5d428607bfd78f82ba7b49497bd9/google-cloud-firestore/src/main/java/com/google/cloud/firestore/Pipeline.java Node stages for reference: https://github.com/googleapis/google-cloud-node/blob/c26967ca277fdab196ea94f5561e4f451ac1ec8a/handwritten/firestore/dev/src/pipelines/pipelines.ts 1. Java allows the user to pass in Fields into remove fields. E.g. user can do in Java `.removeFields(field("published"), field("genre"), field("nestedField"))` but in Go user cannot do `RemoveFields(FieldOf("published"), FieldOf("genre"), FieldOf("nestedField"))`. This PR fixes this by allowing FieldOf in RemoveFields args 2. Change unnest signature to a more Go idiomatic one 1. Idiomatic Go Design: Replaced the struct pointer (*UnnestOptions) with a variadic slice of interfaces (...UnnestOption). This standardizes developer ergonomics and aligns with other areas of the Firestore Go SDK. 2. Cleaner Zero-value Defaults: Callers are no longer forced to pass nil if they don't wish to supply specific configuration arguments. 3. Better Hygeine & Robustness: Internal logic initializes default underlying representations implicitly without requiring users to instantiate or check nil pointer values. 3. Refactored the Sample stage The changes to the sample stage from `SampleSpec` to `Sampler`, and `SampleByDocuments` to `ByDocuments` were driven by idiomatic Go design principles regarding readability and removing structural redundancy: 1. Reducing Stuttering in Code In the previous API, calling a sample operation resulted in a redundant "stutter" in the code: ```go // Old Way: Read "Sample" "Sample By Documents" pipeline.Sample(SampleByDocuments(10)) ``` By removing the redundant "Sample" prefix in the builder function name, the code now reads much more cleanly like a sentence ```go // New Way: Read "Sample" "By Documents" pipeline.Sample(ByDocuments(10)) ``` 2. Conciseness and Clarity Renaming `SampleSpec` to `Sampler` represents idiomatic Go naming conventions. Interfaces or types that perform a single conceptual action or "sample" elements are standardly suffixed with `-er` (like `Reader`, `Writer`, `Sampler`) rather than `-Spec`. 3. Exposing `ByPercentage` consistently In addition to rewriting `SampleByDocuments` to `ByDocuments`, the new API now cleanly supports `ByPercentage` directly too: ```go // Old Way: pipeline.Sample(&SampleSpec{Size: 0.6, Mode: SampleModePercent}) // New Way: pipeline.Sample(ByPercentage(0.6)) ``` This is much more expressive than forced instantiation for percentage configurations. 5. Refactored the rawstage The `RawStage` refactoring represents a significant shift from a builder pattern (which is Java-idiomatic) to a direct method invocation with variadic arguments (which is Go-idiomatic). Here is a detailed breakdown of the rationale: 1. Removing Public API Bloat In the previous implementation, `RawStage` was a publicly exported struct (`type RawStage struct`). This forced the SDK to export a constructor `NewRawStage()`, and builder methods like `WithArguments()` and `WithOptions()`. The new API makes `rawStage` internal (lowercase `r`). The user now simply calls a direct method on the pipeline (`pipeline.RawStage(...)`). There are no superfluous constructors or builder methods exposed on the documentation surface. 2. Idiomatic Go (Direct Method Calls) Go prefers passing configuring arguments directly into a method rather than instantiating builder objects. Compare the old chained instantiation against the new design: Old Way (Java Builder style): ```go client.Pipeline().Collection("books"). RawStage( NewRawStage("limit").WithArguments(3), ) ``` New Way (Idiomatic Go): ```go client.Pipeline().Collection("books").RawStage("limit", []any{3}) ``` The new syntax is much shorter, easier to read, and accurately describes the intent without boilerplate. 3. Streamlining Options with Variadics The new method `RawStage(name string, args []any, opts ...RawStageOptions)` safely takes variadic pointers for options. If no options are needed, the caller simply omits them. If they are needed, they can supply any amount of `RawStageOptions` without calling unique chain builders. 4. Architectural Alignment across the SDK This aligns `RawStage` with the rest of the module's recent refactors (shifting `SampleSpec` to `Sampler`, and `*UnnestOptions` to `...UnnestOption`). It unifies the builder's syntax so developers maintain a standard mental model when chained expressions are invoked. 6. Refactored the AddFields, RemoveFields and Select stages to expect at least one arg similar to Java and Node

fix(firestore): bubble up errors in BulkWriter (#14481) This PR addresses an issue where the Firestore `BulkWriter` could silently fail to persist documents without notifying the caller. This behavior was primarily observed when the `BulkWriter`'s context was canceled or when internal limits were reached. #### Root Causes 1. **Dropped Batches on Cancellation:** In the background `send` function, if the context was canceled, the current batch of jobs was dropped immediately without notifying the individual job result channels . 2. **Ignored Queuing Errors:** The `write` function and the retry logic ignored errors returned by the underlying bundler when attempting to queue a write . 3. **Ignored Rate Limit Errors:** Context errors occurring during the rate limiter's wait period were not checked or propagated . #### Changes - Updated the internal `write` method and its public callers (`Create`, `Delete`, `Set`, and `Update`) to return errors from the rate limiter and the bundler. - Modified the `send` function to iterate through and notify all jobs in a batch with a context error if the transmission is aborted due to cancellation. - Added error checking to the retry logic within `send` to ensure failures to re-queue a job are surfaced. These changes ensure that any failure to queue or send a document is explicitly reported through the `BulkWriterJob` results or as an immediate return value, preventing data loss. #11422. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

fix(firestore): fixes for Query to Pipeline Conversion (#14422) This PR updates the Firestore `toPipeline` logic to ensure that queries converted into pipelines follow standard Firestore query semantics. Previously, the pipeline translation was missing several critical behaviors regarding filtering, sorting, and missing-field handling. All the below gaps were identified by running the Java tests in [ITQueryToPipelineTest.java](https://github.com/googleapis/java-firestore/blob/9065134fd70da22250f038d7e964f26aebfeb3cf/google-cloud-firestore/src/test/java/com/google/cloud/firestore/it/ITQueryToPipelineTest.java) and comparing the execute request payload sent by Java with the ones sent by Go equivalent pipelines. Here are the key improvements: ### 1. Correct Stage Ordering (Filters Before Sorts) We now apply the `Where` filters before the `Sort` stage. Applying sorting before filtering is inefficient and diverges from the standard Firestore execution plan, which filters the dataset as early as possible. **Example:** ```go // Before: collection -> sort("name") -> where("age" == 25) // After: collection -> where("age" == 25) -> sort("name") query := client.Collection("users").Where("age", "==", 25).OrderBy("name", firestore.Asc) pipeline := query.toPipeline() ``` ### 2. Enforcing "Missing Field" Semantics In Firestore, a query for `foo == 1` or an ordering on `foo` should implicitly exclude any document where the field `foo` is missing. We now automatically inject `FieldExists` checks into the pipeline to mirror this behavior. **Example:** * **Filter:** `Where("foo", "==", 1)` $\rightarrow$ `And(FieldExists("foo"), Equal("foo", 1))` * **Order:** `OrderBy("bar", Asc)` $\rightarrow$ `Where(FieldExists("bar")) -> Sort("bar", Asc)` ### 3. Support for Composite Filters (OR / Nested AND) The previous implementation could only handle simple top-level field filters. We've added a recursive filter translator that allows the `toPipeline` method to handle complex nested filters, including `OR` queries. **Example:** ```go // This now correctly translates to a nested 'or' function in the pipeline 'where' stage filter := firestore.OrFilter{ firestore.PropertyFilter{Path: "status", Op: "==", Value: "urgent"}, firestore.PropertyFilter{Path: "priority", Op: ">", Value: 10}, } query := client.Collection("tasks").WhereEntity(filter) ``` ### 4. Native `LimitToLast` Implementation Since the pipeline backend doesn't have a specific "limit to last" stage, we now implement this by: 1. Reversing the sorting order. 2. Applying a standard `Limit`. 3. Re-sorting back to the original requested order. **Example:** ```go // Fetch the 5 most recent items // Pipeline: Sort(timestamp DESC) -> Limit(5) -> Sort(timestamp ASC) query := client.Collection("logs").OrderBy("timestamp", firestore.Asc).LimitToLast(5) ``` ### 5. Correct Path Resolution for Subcollections We fixed an issue where subcollection queries were passing the wrong path to the `collection` stage. It now correctly resolves the relative path from the database root. **Example:** ```go // Previously, this might only pass "comments" to the pipeline. // Now it correctly passes "posts/post_123/comments". query := client.Collection("posts").Doc("post_123").Collection("comments") ``` ### 6. Accurate Cursor Boundaries (StartAt / EndAt) We've refined how multi-field cursors are translated into pipeline inequalities. - **Reference Types:** When using a document as a cursor, we now correctly pass it as a `Reference` type instead of a string ID, ensuring the backend can perform accurate comparisons. - **Strict Inequalities:** For multi-field sorts (e.g., `OrderBy("A").OrderBy("B")`), we now use strict inequalities for intermediate fields to ensure the cursor correctly "steps" through the results without including duplicates or skipping valid entries.

feat(firestore): support Transaction ReadTime (#14615) support Transaction ReadTime and prevent invalid WithReadOptions usage --------- Co-authored-by: Alex Hong <9397363+hongalex@users.noreply.github.com>

fix(firestore): rename to match Java and expose functions on expression (#14250) 1. Rename TruncWithPlaces (Java's implementaion [here](https://github.com/googleapis/java-firestore/blob/f9439c92f31c347ed31e42378177630468d5ed29/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java#L3775-L3787)), LtrimWithValues, etc to match Java 2. Change signature of ArrayIndexOf and introduce ArrayLastIndexOf to match Java 3. Earlier the methods such LogicalMaximum, CountDistinct, etc couldn't be chained on expression and were only defined as static methods. E.g. users couldn't do `FieldOf("rating").LogicalMaximum(4.5)` but could do `LogicalMaximum(FieldOf("published"), 1900)`

fix(datastore): fix context leak in Iterator and Transaction spans (#14478) Fixes #9528 This PR fixes a context leak in the `datastore` library where several methods of `Iterator` and `Transaction` were overwriting their internal context field with a new span context on every call (e.g., `t.ctx = trace.StartSpan(t.ctx, ...)`). When these methods are called repeatedly (e.g., in a loop over millions of objects calling `Cursor()` or `Get()`), the context chain grows linearly, eventually leading to a stack overflow in `context.Done()` or `context.Value()` due to deep recursion. Fix is to use a local context variable for the span instead of overwriting.

feat(firestore): update and delete pipeline DML stages (#14331) Corresponding Java PR which was used as reference: https://github.com/googleapis/java-firestore/pull/2317 --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

feat(firestore): add WithAlwaysUseImplicitOrderBy option (#14260) The changes in firestore/query.go were primarily driven by the need to align the Go SDK with the Java and Node.js implementations, ensuring consistent behavior for both standard and pipeline queries. Reasons for the Changes 1. Handling Multiple Inequality Filters: The core change in adjustOrders() and the addition of getInequalityFilterFields() allow the SDK to handle multiple inequality filters (e.g., where("age", ">", 20).where("score", ">", 100)). Previously, Go only added an implicit OrderBy for the first inequality field it encountered. The new implementation collects all inequality fields, deduplicates them, and sorts them lexicographically by FieldPath, matching the long-standing behavior in Java ([Query.createImplicitOrderBy](https://github.com/googleapis/java-firestore/blob/f9439c92f31c347ed31e42378177630468d5ed29/google-cloud-firestore/src/main/java/com/google/cloud/firestore/Query.java#L529)) and Node.js ([Query.getInequalityFilterFields](https://github.com/googleapis/google-cloud-node/blob/1dd586015cbc85409f4737c995c750bc694c14c3/handwritten/firestore/dev/src/reference/query.ts#L895)). 2. alwaysUseImplicitOrderBy Support: The update to toProto() introduces support for the alwaysUseImplicitOrderBy configuration . This mirrors the same setting in Java's [FirestoreOptions](https://github.com/googleapis/java-firestore/blob/f9439c92f31c347ed31e42378177630468d5ed29/google-cloud-firestore/src/main/java/com/google/cloud/firestore/FirestoreOptions.java#L260) and Node.js's [Settings](https://github.com/googleapis/google-cloud-node/blob/1dd586015cbc85409f4737c995c750bc694c14c3/handwritten/firestore/dev/src/index.ts#L859). When enabled, it forces the inclusion of implicit OrderBy clauses (inequality fields + __name__) even for standard queries that don't technically require them for cursor support, ensuring consistent result ordering across different SDKs and platforms. 3. Pipeline Query Parity: The toPipeline() method (used for the experimental Pipeline API and Vector Search) was updated to call q.adjustOrders() instead of just using explicit q.orders. This ensures that pipeline-based queries benefit from the same implicit ordering logic as standard StructuredQuery objects. Java's [pipeline()](https://github.com/googleapis/java-firestore/blob/f9439c92f31c347ed31e42378177630468d5ed29/google-cloud-firestore/src/main/java/com/google/cloud/firestore/Query.java#L2058) and Node.js's [_pipeline()](https://github.com/googleapis/google-cloud-node/blob/1dd586015cbc85409f4737c995c750bc694c14c3/handwritten/firestore/dev/src/reference/query.ts#L742) methods already utilized their respective implicit ordering logic, so this change eliminates a disparity where Go pipelines were missing necessary sort stages. 4. Technical Fixes and API Consistency: Lexicographical Sorting: The sorting logic added to getInequalityFilterFields (segment-by-segment comparison) exactly matches Java's [BasePath.compareTo](https://github.com/googleapis/java-firestore/blob/f9439c92f31c347ed31e42378177630468d5ed29/google-cloud-firestore/src/main/java/com/google/cloud/firestore/BasePath.java#L127) and Node.js's Path.compareTo.

fix(datastore): enable gRPC keepalives and expand retries (#14411) This change improves the library's resilience to abrupt connection losses, such as those caused by Cloud Front End (CFE) host failures or kernel panics. These events can currently lead to extended application hangs (e.g., the 6-minute outage observed in [b/473841984](https://b.corp.google.com/issues/473841984)) because the library lacks proactive health checks and does not retry on certain transient errors

feat(firestore): add raw execute options and move options (#14259) 1. Java and node Allow passing raw options to execute but Go does not. This PR adds the change. 2. removal of WithExecuteOptions: The removal of `WithExecuteOptions` on the `Pipeline` builder and the shifting of its arguments directly into `Execute(ctx, opts...)` is a design pattern choice rooted heavily in idiomatic Go practices and separation of concerns. Here is a detailed breakdown of exactly why this change was made: 1. Separation of Definition vs. Execution A `Pipeline` struct represents an **immutable definition** of a query. It builds a data structure that describes *what* data you want (e.g., `Where(...)`, `Select(...)`, `Aggregate(...)`). Execution options, on the other hand, describe *how* you want the backend to process that query (e.g., "return explain metrics", "use a specific index mode", "pass these raw backend flags"). If `WithExecuteOptions` was left on the `Pipeline`, you would be coupling the query's definition with its execution metadata. By moving `ExecuteOption` to be a variadic argument on the `Execute()` method itself, you completely separate the two concerns. 2. Reusability of the Pipeline Because the definition is now separated from the execution telemetry, a developer can build a pipeline once and execute it in multiple different ways without having to clone or mutate it: ```go // 1. Define the Pipeline once myPipeline := client.Pipeline().Collection("books").Where("published", ">", 1990) // 2. Execute it normally in production res := myPipeline.Execute(ctx) // 3. Execute the exact same pipeline structure in a testing environment to analyze its performance explainRes := myPipeline.Execute(ctx, firestore.WithExplainMode(firestore.ExplainModeAnalyze)) ``` If `WithExecuteOptions` existed, the developer would have to fork the pipeline state to achieve this (`myPipeline.WithExecuteOptions(...)`). 3. The Idiomatic "Functional Options" Pattern In Go, it is standard practice for functions that trigger a network request or a concrete action (`Get`, `Set`, `Execute`) to accept operational modifiers as trailing variadic arguments. If you look at the rest of the Google Cloud SDKs (or even standard library packages like `grpc`), you will see this pattern everywhere: * `client.Collection("c").Doc("d").Get(ctx, firestore.opts...)` * `client.RunTransaction(ctx, f, firestore.opts...)` Moving `opts ...ExecuteOption` directly into `Execute(ctx context.Context, opts ...ExecuteOption)` ensures maximum consistency with the rest of the Go SDK's API surface.

fix(datastore): handle opaque type conversions (#14477) fixes #5225

test(firestore): optimize integration tests and fix resource leaks (#14368) Fixes: #14350 **1. Optimization: Pre-built Indexes** Previously, tests that required composite or vector indexes dynamically generated new collection IDs, built the required indexes programmatically, and waited for them to complete. This added significant overhead (often several minutes) to each test suite run. * **Fix:** Tests requiring indexes (`TestIntegration_QueryDocuments_WhereEntity`, `TestIntegration_QueryUnary`, `TestIntegration_AggregationQueries`, `TestIntegration_AggregationQueries_WithRunOptions`, and `TestIntegration_FindNearest`) now use a static subcollection name (`"indexed_collection"`) under a uniquely generated parent document. This ensures isolation across parallel runs while allowing tests to leverage persistent, pre-built indexes. * **Cleanup:** Removed all programmatic index management logic and API clients used strictly for index creation/deletion during tests. * **Docs:** Updated `CONTRIBUTING.md` with the required `gcloud firestore indexes composite create` commands to build these indexes manually, aligning the Firestore testing workflow with the existing Datastore setup. **2. Fixing Resource Leaks** The test suite was permanently leaking documents due to incomplete cleanup logic and early process termination. * **Fix `TestMain` Execution Order:** Modified `TestMain` so `cleanupIntegrationTest()` runs unconditionally before `os.Exit()`. Previously, if `m.Run()` failed, the process would terminate immediately and bypass the global cleanup. * **Eradicate "Missing" Documents:** Updated the global `deleteCollection` utility to use `DocumentRefs(ctx)` instead of `Documents(ctx)`. `Documents()` only returns existing documents, skipping over empty parent documents and permanently stranding their subcollections. `DocumentRefs()` uses the underlying `ListDocuments` API to ensure all paths—including missing parents—are found and recursively cleaned. * **Enforce Test Hygiene:** Added missing `t.Cleanup` hooks to several tests (e.g., BulkWriter and Aggregation tests) and moved existing cleanup registrations to occur immediately after `NewDoc()`. This guarantees that cleanup handlers are registered even if the subsequent backend creation times out or fails. **3. Aligning with Firestore Native Behavior** * **Fix:** Updated `Aggregations_with_limit` to explicitly use `.OrderBy()` for the `Limit` test to enforce deterministic result sets. **4. Flaky Test Fixes** * **Fix `TestDetectProjectID`:** Explicitly clear the `GOOGLE_CLOUD_PROJECT` environment variable using `t.Setenv("GOOGLE_CLOUD_PROJECT", "")` before attempting to instantiate a client without credentials. This prevents the test from inadvertently detecting a project ID and failing in environments where `GOOGLE_CLOUD_PROJECT` is already set.