Chalmer Lowe

test(gapic-generator): Update gapic generator to resolve pytest-asyncio errors. (#17260) This PR updates the Python GAPIC generator templates to support modern EOL Python support drops natively across the entire SDK. ### Changes * **setup.py.j2**: Bumps the baseline minimum `grpcio` dependency requirement to `>= 1.59.0`. This robust minimum requirement resolves the `pytest-asyncio` event loop regressions universally, removing the need for custom `pytest-asyncio` overrides. * **noxfile.py.j2**: Replaced the hardcoded constraints path with a dynamic constraints file path resolver at runtime, ensuring that monorepos natively resolve to their version-specific constraints files (e.g., `testing/constraints-3.10.txt`). * **pytest-asyncio** Updates the gapic-generator in order to resolve an issue with `pytest-asyncio` failures across Python runtimes 3.10 - 3.14.

feat(scripts): Add dependency version scanner tool (#16867) This adds a utility with the ability to scan for common references to dependencies (Python runtimes and package dependencies) to facilitate updating code when runtimes and dependencies change. * It can be run against an entire repo OR against specific packages within a monorepo * It is customizable with [regex patterns and examples here](https://github.com/googleapis/google-cloud-python/pull/16867/changes#diff-d17423afcf0604a287af7c6a590da37df7105674bbb715248d4f0c53545f3ea9) * The test suite checks each regex against the examples to ensure the efficacy of the patterns * The current patterns account for edge cases such as finding `< 3.8` when searching for references to `3.7` since they are semantically equivalent even if syntactically different. * The scanner produces a CSV report with: ``` path/filename, package name, line number, matching pattern, full line for context, etc. ```

chore(handwritten): centralize mypy configuration and update handwritten package noxfiles (#17409) * Relies on a centralized `mypy.ini` file at the root of the repository to consolidate all shared and package-specific mypy exceptions. * Deletes the individual `mypy.ini` files from handwritten libraries, establishing the root configuration as the source of truth. * Refactored their `noxfile.py` files to define `MYPY_CONFIG_FILE` dynamically and standardized their `session.run("mypy", ...)` invocations. > [!note] > Due to test failures outside the scope of this PR, neither google-cloud-crc32c nor bigframes are included in this PR. This [issue tracks those packages](https://github.com/googleapis/google-cloud-python/issues/17606) to ensure their eventual completion > [!note] > bigquery and spanner were pulled out and placed in separate PRs due to excessive duration of their tests. PR #17640 and PR #17641 > [!note] > Work on generated libraries is outside the scope of this PR and can be found here: https://github.com/googleapis/google-cloud-python/pull/17408 > [!note] > UPDATE (20260706): This PR is no longer blocked by https://github.com/googleapis/google-cloud-python/pull/17607

chore: add missing nox sessions and polish dependencies in sqlalchemy-spanner (#17285) ## Description This PR implements missing nox sessions and refactors dependency management inside `packages/sqlalchemy-spanner/noxfile.py` to align with google-cloud standard practices. --- ## Key Changes ### 1. Constants Architecture All dependencies and version definitions have been concentrated at the top of `noxfile.py` in nestled constants: * `UNIT_TEST_STANDARD_DEPENDENCIES`: Standard testing frameworks (`mock`, `pytest`). * `UNIT_TEST_EXTERNAL_DEPENDENCIES`: Telemetry requirements (fully unpinned: `setuptools`, `opentelemetry-api`, `opentelemetry-sdk`, `opentelemetry-instrumentation` to delegate version resolution strictly to `setup.py` and `constraints`). * `UNIT_TEST_DEPENDENCIES`: SQLAlchemy version specifications under test (`sqlalchemy>=2.0`). ### 2. Expired database management Updated the `create_test_database.py` file to add or tweak code to ensure that expired databases are quickly deleted. During rapid development cycles, the Spanner database count would quickly reach the Spanner limit of 100 and system tests would fail. There were two conditions that contributed being unable to quickly retest. In some cases the system would not delete old databases at all because certain triggers were not created. And in other cases, you needed to wait 4 hours before the databases would age off and you could run the system tests again. ### NOTES: * **`core_deps_from_source` & `prerelease_deps`**: Target packages are cleanly overwritten with local source paths/pre-releases using pip's native `--ignore-installed --no-deps` flags. * **`mypy` & `docs` & `docfx`**: Configured to skip gracefully with standard todos indicating that typehints and docs folders are not present in this package. * **`cover`**: Added standard coverage session which skips gracefully if no `.coverage` database file has been created yet. * **`format`**: Implemented Ruff format and check sessions, and reformatted imports and code to team standards. Fixes #17048 🦖

chore: Adds version scanner CI/CD upgrades (#17425) ### Summary of Changes This PR contains updates to the automated dependency version scanner tool and its associated CI/CD workflow to support decoupled formatting, clean console logs, and advisory (non-signalling) runs during rollout. #### 1. GitHub Actions (GHA) Workflow Modernization * **Triggers & Scheduling:** * Configured the workflow to run on `main` and any branch matching `'**version-scanner**'` * Set the schedule to run hourly to test how the system behaves if we choose to use it nightly * Added a `workflow_dispatch` button in the GHA tab to simplify ad hoc testing and demos during development. #### 2. Scanner Script Refactoring (Decoupled Formatters) * Decoupled formatting code from reporting code. * Introduced specialized formatters: * `format_for_raw_csv`: Generates clean, unformatted raw data for CSV reporting. * `format_for_spreadsheet`: Wraps matches with Google Sheets formulas (such as `HYPERLINK` and string quotes to prevent float truncation) for Google Sheets upload. * `format_for_console`: Prepares a slim, readable console string for stdout/logs (especially GHA logs). #### 3. Output Simplification * Removed some existing outputs that no longer made sense to to declutter GHA runner logs. * Ensure it prints matches in the clean console format and removed some existing duplicate outputs. #### 4. Advisory Runs (`--soft-fail`) * Added a `--soft-fail` CLI flag to the python script to allow it to exit with code `0` even if version matches are found (allowing the scan to run and report findings in the logs without failing the GHA check and blocking merges during development and prototyping phases). * Integrated `--soft-fail` in the GHA workflow for now to support development.

test(spanner): add pytest-xdist parallel execution with state isolation (#17344) This PR adds `pytest-xdist` to parallelize unit tests and the `core_deps_from_source` nox sessions for the `google-cloud-spanner` package. **By running tests in parallel using `-n auto`, the execution time of the Spanner unit tests are reduced from ~8 minutes to 4 minutes.** State isolation and test reliability are achieved by: * **Simplifying Subtests**: We pass simple strings/names to `subTest()` instead of complex objects. This keeps subtests lightweight and prevents serialization errors. * **Cleaning Up Global Singletons**: We reset telemetry singleton states on test teardown (using pytest's idiomatic `monkeypatch` fixture). This ensures metric counters don't leak from one test into another. * **Fixing Concurrent Mock Conflicts**: We return fresh mocked iterators for concurrent calls (using `side_effect` instead of `return_value`). This prevents one thread from exhausting a mock's results before another thread can read them. * **Robust Assertion Checks**: We added a helper method (`_assert_concurrent_transaction_invariants`) that verifies the *behavior* of concurrent threads (ensuring exactly one thread starts the transaction while others wait/reuse it), rather than checking fragile call logs or counting sequential request IDs. This allowed us to safely run four concurrent tests. > [!note] > The long pole in the tent is still the `system` tests, which require about 30 minutes. It is not as simple as just adding xdist because there are other factors that limit velocity including the fact that system tests actually interact with live systems.

fix(bigquery): close GAPIC storage transport and auth sessions to prevent socket leaks (#17508) This PR resolves resource leaks (specifically open sockets left in the `ESTABLISHED` state) that occur during client lifecycle operations and credential refreshing in system/unit tests. ### The Problem 1. **Transport Lifecycle:** When closing the BigQuery Storage client, calling `_transport.grpc_channel.close()` was insufficient for releasing all network resources. The full `_transport.close()` method needs to be invoked to tear down the underlying transport channel correctly. 2. **Dynamic Auth Sessions:** Under certain authentication environments (like Workload Identity/GCE Metadata server inside Kokoro CI), the `google-auth` library dynamically instantiates helper HTTP sessions to fetch access tokens. These sessions are not owned by the BigQuery client and are not closed automatically, leading to leaked sockets. 3. **Flaky Test Assertions:** Socket count assertions in system tests were flaky because Python's garbage collection is non-deterministic, meaning sockets remained open in the operating system even after client close calls until a garbage collection cycle swept them. ### Changes * **Client & Transport Lifecycle:** * Updated [connection.py](file:///usr/local/google/home/chalmerlowe/titan-src/projects/google-cloud-python/main/packages/google-cloud-bigquery/google/cloud/bigquery/dbapi/connection.py), [magics.py](file:///usr/local/google/home/chalmerlowe/titan-src/projects/google-cloud-python/main/packages/google-cloud-bigquery/google/cloud/bigquery/magics/magics.py), and [table.py](file:///usr/local/google/home/chalmerlowe/titan-src/projects/google-cloud-python/main/packages/google-cloud-bigquery/google/cloud/bigquery/table.py) to close the BigQuery Storage transport using `_transport.close()` instead of `_transport.grpc_channel.close()`. * **Testing Improvements:** * Added `patch_tracked_requests` interceptor to system/unit tests to track and explicitly close all dynamically spawned credential-refreshing HTTP sessions when the test context exits. * Added explicit `gc.collect()` calls to socket leak verification tests to force synchronous sweeping of unreferenced socket objects before asserting final socket counts. * **Code Coverage:** * Appended `# pragma: NO COVER` to Python version checks in [`__init__.py`](file:///usr/local/google/home/chalmerlowe/titan-src/projects/google-cloud-python/main/packages/google-cloud-bigquery/google/cloud/bigquery/__init__.py) for code paths that render a deprecation warning code if attempted to run on Python runtimes <3.10 test matrix (these paths do not run in our CI/CD since we never execute code with the older runtimes).

feat: regenerate monitoring using generator v1.32.0 (#17165) Updates post processing to account for dropping support for Python 3.7, 3.8, 3.9 and impacts that has on using 3.10 for lower bounds testing. Changes: * Adds `pandas` and `numpy` to `constraints-3.10.txt` and installs `pandas` extra in `prerelease_deps` and `core_deps_from_source` nox sessions. * Updates `pandas` dependency lower bound to `1.3.4` and updates `constraints-3.10.txt` to use `pandas 1.3.4` and `numpy 1.21.3` to avoid pip build errors on Python 3.10. Partially corrects https://github.com/googleapis/google-cloud-python/pull/17153 🦕

feat(mypy): centralize mypy.ini and update templates (#17523) > [!note] > This is step one of a multi-step process. The work done here is outlined below. Additional steps (to be completed in other PRs) include: > * generate the **generated packages** > * generate and/or post process **hybrid packages** This work: * Adds a centralized `mypy.ini` file at the root of the repository. * Updates GAPIC generator templates to omit local `mypy.ini` and dynamically resolve the root config via a `MYPY_CONFIG_FILE` constant. > [!note] > Work on strictly handwritten libraries is outside the scope of this PR and can be found here: https://github.com/googleapis/google-cloud-python/pull/17409 Partially resolves: #17322 🦕

feat(version-scanner): support target list inputs via --targets (#17478) This pull request adds support for scanning multiple target dependencies and versions concurrently via a YAML file containing targets, while consolidating file handling and error logging. Key changes: - **YAML Targets File Support:** Adds a `--targets-file` CLI argument to read and resolve multiple target dependency-version tuples from a configuration file. - **Consolidated File Handling:** Centralizes file reading and error logging across the codebase under a single helper (`_safe_read_file`) with uniform stderr printing and exit codes. - **Expanded Test Coverage:** Adds parametrized unit tests validating targets file parsing errors (missing files, bad format, null values) and the core file helper error branches, and updates integration tests to utilize soft-fail flags.

chore(ndb): add missing format session and modernize lint and blacken sessions (#17342) This PR adds missing `format` nox session and modernizes `lint` and `blacken` sessions to use Ruff for this package. ### Additional changes: * Also marks `blacken` as deprecated. * Runs Ruff formatter and thus incorporates linting changes. Changes to each nox session are based on the versions found in the **gapic-generator** [`noxfile.py.j2` template](https://github.com/googleapis/google-cloud-python/blob/main/packages/gapic-generator/gapic/templates/noxfile.py.j2). Fixes #17049 --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

chore(versions): update Python version support to 3.10+ in gapic-generator (#16910) > [!note] > This PR has been updated to remove 3.7, 3.8, and 3.9 (with one exception discussed below). ### Overview This Pull Request updates the gapic-generator package and related CI configurations to reflect the deprecation of Python 3.7 through 3.9. It updates documentation, templates, and CI scripts to establish Python 3.10 as the new minimum supported version where applicable. ### Changes #### Documentation: * Updated installation and verification instructions in packages/gapic-generator/docs to reference Python 3.10 #### Templates: * Updated __init__.py.j2 templates in gapic/ads-templates to require Python 3.10+ (updating sys.version_info checks and error messages). * Removed outdated comments regarding Python 3.9 and below features (like module-level __getattr__) to avoid confusion, as the baseline runtime is now higher than the version noted in the comments. * Updated mypy.ini.j2 to target Python 3.10. * Cleaned up dead code (e.g. commented-out Jinja snippets) in the templates. #### CI Configurations: * Removed Python 3.9 and below from the unit test matrix in ci/run_single_test.sh. * Updated comments regarding supported Python versions in ci/run_conditional_tests.sh. ### Reasoning Python 3.9 and below have reached End-of-Life (EOL) and are no longer supported. These changes ensure that our generator and the libraries it produces align with current support policies and reduce maintenance burden by removing dead code paths. > [!note] > because the config that runs this PR checks is not updated yet, it still wants to run 3.9 so we have a temp change to skip that nox session. Once this PR merges future PRs will no longer attempt to run 3.9 and we will follow-up with a small PR to strip out that last 3.9 reference. Fixes internal issue: http://b/482126936 🦕

feat: Add Feature Gating configuration helpers. (#17524) This PR introduces a foundational configuration layer for integrating General Feature gating into Google Cloud Python client libraries. ### Intent & Motivation We are about be embark on adding observability (o11y) to entire suite of Cloud SDK libraries. To enable this functionality on an opt-in basis we are prepping a general feature gating capability. While the initial push is towards enabling o11y, we recognize that this can and should be more general than that. ### Changes Included * **Configuration Resolution Logic**: Added the `resolve_feature_flag` function in the new `_feature_gating_helpers` package to resolve feature gates: Resolves settings in the following order of precedence: 1. Programmatic overrides in configurations (checks for feature_key values) 2. Language-wide Environment Variable: `GOOGLE_CLOUD_PYTHON_<FEATURE>_ENABLED` (natively checks for an `EXPERIMENTAL` token first e.g. `GOOGLE_CLOUD_[EXPERIMENTAL]_PYTHON_<FEATURE>_ENABLED`) 3. Default fallback * **Experimental Fallbacks**: Implemented fallback logic that automatically checks for `EXPERIMENTAL` prefixes within the environment variable definitions to support early-stage adoption and rollout phases safely. * **Test Coverage**: Provided test coverage using `pytest` for precedence rules and environmental scenarios.

feat: regenerate asset w/ dependencies using generator v1.32.0 (#17153) Updates post processing to account for dropping support for Python 3.7, 3.8, 3.9 and impacts that has on using 3.10 for lower bounds testing. Adds `google-cloud-org-policy` to `post-processing` for insertion into `google-cloud-asset` files: * `setup.py` * `constraints-3.10.txt` `google-cloud-asset` was failing to regenerate successfully due to missing dependency on org-policy. Partially corrects #17152 🦕

chore(generator): centralize mypy configuration and regenerate google-cloud-datastore POC (#17408) > [!note] > This is step one of a multi-step process. The work done here is outlined below. Additional steps (to be completed in other PRs) include: > * generate the remaining **generated packages** > * generate and/or post process **hybrid packages** This work: * Adds a centralized `mypy.ini` file at the root of the repository. * Updates GAPIC generator templates to omit local `mypy.ini` and dynamically resolve the root config via a `MYPY_CONFIG_FILE` constant. * Removes `mypy.ini` replacements from `datastore-integration.yaml` post-processing. * Regenerates `google-cloud-datastore` using the updated generator configurations to serve as a proof of concept. > [!note] > Work on strictly handwritten libraries is outside the scope of this PR and can be found here: https://github.com/googleapis/google-cloud-python/pull/17409

feat(spanner): drop Python 3.7-3.9 support and regenerate (#17169) Updates post processing to account for dropping support for Python 3.7, 3.8, 3.9 and the impacts that has on using 3.10 for lower bounds testing. ### Changes * updates the lower bound versions for several libraries to avoid conflicts and install issues in both `setup.py` and `constraints-3.10.txt` * updates post-processing scripts to ensure the above updates persist

fix(proto-plus): make Marshal thread-safe and handle race conditions (#17774) # Thread-safe Marshal Initialization ## Problem A RuntimeError saying `dictionary changed size during iteration` can occur randomly in `BaseMarshal.get_rule`. This happens because one thread is reading the `_instances` dictionary while another thread is adding a new instance to it. This is common when using features like Firestore's on_snapshot in a background thread. ## Solution 1. **Thread-safe instance creation:** Applied a locking mechanism in `Marshal.__new__`. It uses a technique called "double-checked locking" to make sure only one thread creates a new instance at a time without slowing down normal reads. 2. **Copy-on-Write Pattern:** When a new instance is added, we make a copy of the existing instances dictionary, add the new one, and then replace the dictionary atomically. This ensures that threads iterating over the old dictionary are not interrupted. 3. **Defensive Attribute Access:** Added safety in `BaseMarshal.get_rule` to avoid trying to read rules from an instance that has been registered but has not finished initializing yet. ## Notes to Reviewers - The Copy-on-Write pattern allows us to avoid locking during reads (which are very common), keeping performance high while fixing the race condition. - A new test file `test_marshal_thread_safety.py` has been added to cover these concurrency scenarios. - Some portions of files in this package were reformatted automatically by the code linters. Fixes #15100

feat(pandas-gbq): add Ruff format session and implement mypy (#17041) This PR adds a Ruff-based `format` session to `pandas-gbq` and implements the `mypy` session based on the `gapic-generator` template. It also adds `type ignores` and descriptive comments for untyped imports in source files, and applies formatting fixes.

chore(google-auth): drop python 3.7 EOL false positives and refactor metrics telemetry (#17463) This pull request eliminates EOL Python 3.7 false positives from the `google-auth` codebase to ensure a clean signal from version scanner compliance checks. **Why these changes are made:** - **Metrics Telemetry:** Hardcoded Python versions in telemetry headers (like `gl-python/3.7`) have been replaced with abstract version placeholders (`<python-version>`) in comments with formatting examples and in test assertions. This prevents scanner alerts while retaining the ability to verify HTTP client request formatting behavior. - **App Engine Runtime Tests:** GAE standard runtime test cases have been refactored to dynamically construct GAE runtime values from the active Python interpreter at test execution time, avoiding the need for EOL checks and manual updates when Python versions retire. - **Clean Up Transport Properties:** Replaced legacy private `_auto_decompress` internal attribute access in the `aiohttp` transport with the standard public `auto_decompress` property (supported in `aiohttp >= 3.8`), allowing the removal of old TODOs from the source code that referenced 3.7. - **Readme Cleanup**: Removed the manual, out-of-date historical "Unsupported Python Versions" list from the package documentation, relying instead on standard authoritative packaging metadata (`python_requires`) to enforce runtime compatibility. --- Supports resolution of the internal bug: #512225398

fix(gapic): mock os.path.exists in mTLS tests to support newer google auth (#17807) This PR fixes a test failure in the generated goldens related to mTLS (mutual Transport Layer Security) configuration. ## Problem A recent update to the `google-auth` library (specifically how it discovers certificates) introduced stricter checks using `os.path.exists()`. Our generated tests were only mocking `builtins.open()` to simulate configuration files. As a result, the new existence checks failed during tests because the mocked files didn't actually exist on disk, causing assertions to fail. ## Solution 1. **Updated Generator Template**: Modified the generator template (`packages/gapic-generator/gapic/templates/tests/unit/gapic/%name_%version/%sub/test_%service.py.j2`) to mock both `builtins.open()` and `os.path.exists()` in the relevant test blocks. 2. **Regenerated Goldens**: Updated the generated tests across the codebase to include these new mocks. ## Notes to Reviewers * This is a technical fix to adapt our testing infrastructure to newer dependency versions. * The actual business logic of the generated clients is unaffected. * This fixes failures seen in pre-release dependency testing. Fixes # (No specific issue reported, but addresses failures seen in CI/CD)