github.com-googleapis-google-cloud-go

feat(spanner): complete location-aware routing resilience and observability (#14418) ## Summary This change expands the location-aware routing path in the Spanner Go client with endpoint lifecycle management, stronger endpoint health handling, retry-aware exclusion, latency-aware replica selection, and route-selection tracing. ## What Changed - enabled location-aware routing automatically for experimental-host configurations while preserving env-var override behavior - added endpoint cache support for GetIfPresent, Evict, and DefaultChannel - switched endpoint health evaluation to real gRPC connectivity state, including transient-failure detection - added an endpoint lifecycle manager with: - background probing - idle eviction - endpoint recreation requests - recent transient-failure eviction tracking - integrated lifecycle handling into routing and affinity fallback paths - added request-id keyed one-shot endpoint exclusion for RESOURCE_EXHAUSTED retries - expanded skipped_tablet_uid reporting for transient failures and recent transient-failure evictions, with deduplication - added OpenTelemetry route-selection span attributes and events - updated focused tests and goldens for the new routing behavior ## Behavioral Notes - stopped or transient-failure endpoints are now treated differently from simply non-ready endpoints during tablet selection - retry attempts can avoid previously selected routed endpoints after RESOURCE_EXHAUSTED - leader-preferred requests still favor close leaders, but can fall back to a closer non-leader replica when the leader is too far away - route selection is now observable through tracing attributes and events

feat(workloadmanager): add new clients (#13849) PiperOrigin-RevId: 869327994

feat(storage): merge support for bi-directional multiple range reads (#11377) * feat(storage): merge support for bi-directional multiple range reads * add license header * get object metadata for regular grpc reads * docs change * fix offset and remaining bytes calculation * fix(storage): use mutex for accessing concurrent vars * remove TestOpenAppendableWriterUnsupportedEmulated * Add back retry conformance tests to json * retry tests assume writes are conditionally idempoten --------- Co-authored-by: Daniel B <danielduhh@gmail.com>

feat(auditmanager): add new clients (#13842) PiperOrigin-RevId: 861758611

feat(iam/admin): regenerate client (#11570) This client has some handwritten code in it that causes failures when generating it normally. I generated the client and hand patched around handwritten code as to not create a breaking change. Changes made: Keep old variants of RPCs that did not generate an iterator helper to avoid breaking changes: ListRoles, QueryGrantableRoles, and QueryTestablePermissions Add variants of above RPCs that do return an iterator: ListRolesIter, QueryGrantableRolesIter, and QueryTestablePermissionsIter Preserve API for GetIamPolicy and SetIamPolicy. Fixes: #8219

feat(hypercomputecluster): add new client (#13423) PiperOrigin-RevId: 823382075

chore: onboard a new mapmanagement library (#14467)

chore: onboard new library appoptimize (#14393) Onboard new API client for `google/cloud/appoptimize/v1beta` Internal bug b/500129743

feat(firestore): refactor pipeline API for uniform stage options (#14322) Design decision [here](https://docs.google.com/document/d/1diPUhXr2kZh69KcAsuOMZMj_rfcykGsE2-phmBIt8vs/edit?usp=sharing) **Overview** This PR standardizes the API surface for the Firestore Pipeline builder in the Go SDK, ensuring idiomatic conformance and aligning closely with cross-platform design paradigms (Java/Node.js). It replaces complex stage-specific configuration structs with a universal functional option pattern and introduces typed slice constants for varied expressions. **Key Features and API Changes:** * **Universal `RawOptions` Escape Hatch:** Introduced `RawOptions` to serve as a universal pass-through for backend options that might not be natively modeled by the Go SDK (e.g., `RawOptions{"limit": 5, "distance_field": "dist"}`). This deprecates intermediate parameter structs like `PipelineFindNearestOptions` across pipeline stages. * **Standardized Variadic Arguments with Slices:** Methods that formerly relied on raw variadic argument unpacking (e.g. `...Ordering`, `...any`) have been updated to accept typed slices. To preserve clean query ergonomics, we introduced domain-specific helper constructors (`Orders()`, `Fields()`, `Accumulators()`, `Selectables()`). * `Pipeline.Sort(Orders(Ascending(FieldOf("f")), Descending(FieldOf("__name__"))))` * `Pipeline.Aggregate(Accumulators(Count(DocumentID).As("total")))` * `Pipeline.Select(Fields("a", "b"))` * **Functional Stage Options Migration:** Implemented standard functional options for specialized stage modifiers: * `WithFindNearestLimit()`, `WithFindNearestDistanceField()` * `WithAggregateGroups()` * `WithUnnestIndexField()` * `WithExplainMode()` **Internal Code Simplification & Testing:** * **Flattened Stage Hierarchy (`baseStage` removal):** Removed the `baseStage` struct embedding across our internal pipeline stages. We replaced it with explicit `options map[string]any` fields within each stage and introduced a shared `stageOptionsToProto(options)` helper. This simplifies the internal struct hierarchy, removes opaque state inherited from struct embedding, and makes configuration serialization significantly more explicit and easier to trace inside each stage's `toProto()` implementation. * **Pipeline Stage Proto Parsing:** Refactored internal `pipelineStage` structures (`newAggregateStage`, `newFindNearestStage`, `newSortStage`, etc.) to reliably map the new `RawOptions` down to standard `*pb.Value` payloads. * **Tests for `AlwaysUseImplicitOrderBy`:** Added dedicated test cases (`TestQuery_AlwaysUseImplicitOrderBy`) to explicitly verify the behavior of the existing `WithAlwaysUseImplicitOrderBy` client toggle, ensuring it appropriately serializes implicit inequality fields and `__name__` into the `OrderBy` proto requests. * **Integration Suite Migration:** Fully migrated the integration suite inside `pipeline_integration_test.go` and `pipeline_test.go` to assert against the newly stabilized `RawOptions` and slice-builder interfaces. --------- Co-authored-by: Alex Hong <9397363+hongalex@users.noreply.github.com>

feat: update API sources and regenerate (#14621) This change is from "Update & generate" for Go in the playbook.

feat(spanner): add dynamic channel pool (#14611) Split of https://github.com/googleapis/google-cloud-go/pull/14604 Internal reference: go/go-dcp-design

chore(spanner): integrate location aware routing with RPCs (#13877) - Add `locationAwareSpannerClient` wrapper that intercepts RPCs and routes them to the server endpoint resolved by the existing `channelFinder`, falling back to the default gRPC channel when no endpoint is available - Add `endpointClientCache` that creates and caches per-address gRPC connections - Add transaction affinity tracking so Commit/Rollback route to the same server that handled the transaction, with read-only transactions routed independently per-request based on key ranges - Move request preparation (`prepareReadRequest`, etc.) and response observation (`observePartialResultSet`, etc.) from transaction/batch code into the client wrapper, keeping routing concerns in one place

feat(firestore): Introduce new functions and literals stage (#14114) [go/firestore-query-tracker](http://go/firestore-query-tracker) : Introduce all the functions marked "P" (planned) in Go column O and "C" (Complete) in Backend column H in "Firestore Features (Pipeline)" sheet. The sheet above specified only the function name. The args for all the functions were determined from backend implementation in - google3/java/com/google/cloud/datastore/client/firestorev1/Functions.java - google3/java/com/google/cloud/datastore/client/firestorev1/Stages.java The documentation is only available for the functions marked "C" (Complete) in Docs column I. The same documentation has added to the client in this PR. - google3/googledata/devsite/_common/en/_shared/firestore/pipeline/functions/_shared/

feat(gkerecommender): add new clients (#13238) PiperOrigin-RevId: 805881130

feat(apiregistry): add new clients (#13525) PiperOrigin-RevId: 848064295

chore(internal/librariangen): add librariangen (#12614) * Add internal/librariangen to go.work. * Expect source_roots instead of source_paths in generate-request.json for now. closes: googleapis/librarian#791

feat(firestore): Add new functions and correct old ones (#14437) This PR significantly expands the Firestore Pipelines API by adding a wide range of new expressions and functions. These additions improve parity with other Firestore SDKs and provide developers with more powerful tools for complex data transformation and analysis within Firestore. Key Changes: * New Aggregations: Added RawAggregate to allow calling any aggregation function supported by the Firestore backend, even if not explicitly defined in the SDK. Java [ref](https://github.com/googleapis/java-firestore/blob/6acac4b0598891f66174e911ccf55375b587d773/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/AggregateFunction.java#L47-L49). * Arithmetic Expressions: Added RoundToPrecision for rounding to specific decimal places. Java [ref](https://github.com/googleapis/java-firestore/blob/6acac4b0598891f66174e911ccf55375b587d773/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java#L4711-L4761) * Array Functions: * Offset: Access array elements by index (supports negative indexing). * ArrayTransform & ArrayTransformWithIndex: Map array elements to new values using expressions. * ArrayFilter: Filter array elements based on a condition (now fully enabled). * Reference & Key Functions: * GetParent: Retrieve the parent reference of a document or field. * ReferenceSlice & ReferenceSliceToEnd: Extract segments from a document reference path. * Logical & General Functions: * Coalesce: Return the first non-null value from a list of expressions. Java [ref](https://github.com/googleapis/java-firestore/blob/6acac4b0598891f66174e911ccf55375b587d773/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java#L318-L348) * Variable: Reference variables within lambda-like expressions (e.g., in ArrayTransform). * StorageSize: Calculate the storage size of a field or expression. * RawBooleanFunction: Support for custom boolean evaluation functions. java [ref](https://github.com/googleapis/java-firestore/blob/6acac4b0598891f66174e911ccf55375b587d773/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/BooleanExpression.java#L87-L90) * Improved Type Safety: Refined SwitchOn and MapSet signatures to enforce required arguments, reducing runtime errors. * Internal Refinements: * Added validation helpers for timestamp units, granularity, and parts. * Unified internal proto conversion logic for binary and numeric operations.

feat(hypercomputecluster): add new clients (#13890) PiperOrigin-RevId: 873108574

test(bigtable): reduce execution time (#14468) This PR significantly reduces the execution time of the `bigtable` integration test suite by optimizing how test environments and resources are provisioned. It also improves test robustness by refactoring teardown lifecycles. ## Performance Optimizations * **Shared Test Environment:** Refactored `setupIntegration` to utilize `sync.Once`. Instead of creating a new `IntegrationEnv`, `Client`, `AdminClient`, and test table for every single test, a single shared environment is now provisioned and reused across the suite. * **Pre-created Schema:** Common column families (e.g., `follows`, `ts`, `export`, `bulk`, `counter`, `sum`) are now pre-created during the shared initialization. This eliminates the repetitive overhead of schema modifications in individual tests. * **Optimized Teardown:** Updated `deleteTable` to safely ignore `NotFound` errors, bypassing unnecessary retries if a table was already cleaned up. Modified `createColumnFamilyWithConfig` to immediately return on `AlreadyExists` rather than initiating a costly delete/retry loop. ## Lifecycle Robustness * **`t.Cleanup` Migration:** Converted lifecycle-related `defer` statements to `t.Cleanup` across `integration_test.go`. This guarantees that resource teardowns (closing clients, deleting backups/profiles/views) execute safely and correctly even when using `t.Parallel()` or failing subtests. ## Other Changes * Added explicit logic in `export_test.go` to disable DirectPath (`GOOGLE_CLOUD_DISABLE_DIRECT_PATH="true"`) when `AttemptDirectPath` is false.

feat(bigtable): Add methods to read ResultRow (#11924) * feat(bigtable): Add methods to read ResultRow * revert Scan and Unmarshal * revert GetAllByNames * integration tests for GetByName(string) any * test(bigtable): Add unit tests