TrevorBergeron

This commit performs a significant **refactoring** of the **BigFrames UDF (User-Defined Function) subsystem** to simplify its internal architecture and improve maintainability. It introduces a new `ManagedFunctionConfig` dataclass in `udf_def.py` to centralize and consolidate configuration for managed functions, making their provisioning and code generation more robust. Specifically, it refactors methods like `provision_remote_function` and `generate_managed_function_code` across `_function_client.py` and `function_template.py` to leverage this new configuration object, streamlining the process of creating and deploying UDFs. This **maintenance** effort enhances the flexibility and clarity of UDF handling, paving the way for easier future development and extensions within the `bigframes.functions` module.

This commit delivers a **critical fix** for **BigFrames remote user-defined functions (UDFs)**, ensuring that configuration changes (e.g., memory, CPU, environment variables) are correctly applied even if the function's Python logic remains identical. This was achieved through a substantial **refactoring** of the `bigframes.functions` module, introducing new, comprehensive UDF definition objects (`udf_def.py`) that properly encapsulate and track remote function configurations. The update also includes new UDF lowering rewrite steps in the **Ibis and SQLGlot compilers** and refactored remote function provisioning logic to leverage these new structures. This significantly improves the reliability and flexibility of managing and deploying remote functions within BigFrames, preventing silent failures to update critical operational parameters.

This commit significantly **improves the performance** of **BigFrames** data uploads by introducing **asynchronous data loading capabilities** within the session. The **`bigframes.session.loader`** module now includes a new `read_data_async` method and utilizes a thread pool to handle data loading concurrently. Consequently, the **`bigframes.session.bq_caching_executor`** has been refactored, specifically the `_replace_local_sources_with_remote_replacements` function, to leverage these new asynchronous operations for local data source uploads. This internal **performance enhancement** ensures that data uploads no longer block the main execution flow, leading to a more efficient and responsive user experience within the **BigFrames** environment.

This commit performs **test maintenance** by **removing a specific test case** related to the `ingress` subsystem. Specifically, it targets and eliminates the test responsible for verifying the "ingress setting all" functionality, as indicated by the commit message. This action streamlines the **testing suite** for the **ingress module**, potentially by removing an obsolete, redundant, or problematic test, thereby impacting the overall test coverage and execution efficiency for ingress configurations.

This commit **refactors** the **BigFrames caching subsystem** to significantly improve encapsulation and modularity. It extracts the `ExecutionCache` class and its associated logic into a new, dedicated module, `packages/bigframes/bigframes/session/execution_cache.py`. Consequently, the `BigQueryCachingExecutor` in `packages/bigframes/bigframes/session/bq_caching_executor.py` is adapted to utilize this externalized cache, removing its internal cache implementation. Additionally, cache parameter types are updated in `packages/bigframes/bigframes/core/tree_properties.py`, and the executor's initialization is adjusted in `packages/bigframes/bigframes/session/__init__.py`. This **internal architectural improvement** enhances the maintainability and organization of the caching mechanism without altering external behavior.

This commit performs **maintenance** by **fixing ingress settings** within the **VPC tests** for the `bigframes` package. Specifically, it updates the `packages/bigframes/tests/system/large/functions/test_remote_function.py` file to configure ingress for **remote function tests** using `'internal-and-gclb'` instead of the overly broad `'all'`. This **test fix** ensures more precise and secure network configurations during testing, aligning with best practices and preventing potential over-permissioning in test environments. Additionally, the tests are adapted to accommodate a new pandas dataframe index fixture.

This commit introduces comprehensive support for **Common Table Expressions (CTEs)**, fundamentally enhancing the **BigFrames SQL generation capabilities**. It involves defining new AST nodes like `CteNode`, `SqlWithCtesNode`, and `SqlCteRefNode` to explicitly represent CTEs and their references within the internal tree structures. The **SQLGlot compiler** and associated rewrite rules are extensively **refactored** to identify multi-parent nodes as CTEs, integrate them into the compilation process, and generate SQL `WITH` clauses. This significant **architectural refactoring** allows BigFrames to produce more efficient and readable SQL for complex data transformations, improving overall query performance and maintainability.

This commit **introduces `dt` and `str` accessors** to **`bigframes.core.col.Expression` objects**, enabling direct access to string and datetime methods on column expressions. This **new feature** significantly enhances the **BigQuery DataFrames API**, allowing users to perform operations like `bf.col("my_col").str.lower()` or `bf.col("my_date").dt.year` directly on column references. The change involves extensive **refactoring** within the **`bigframes.operations`** module to support these accessors, including the introduction of `DatetimeSimpleMethods` and extending `StringMethods`. Additionally, it includes **fixes and optimizations** for datetime type compatibility and conversion, improving the robustness of **datetime operations**. This makes the API more consistent with pandas and improves expressiveness for **data manipulation** within BigQuery DataFrames.

This commit **introduces a new feature** that allows users to explicitly configure CPU resources for **BigQuery remote functions**. A `cloud_function_cpus` parameter is added to the `remote_function` API within the `bigframes.functions`, `bigframes.pandas`, and `bigframes.session` modules, enabling control over CPU allocation, workers, threads, and concurrency. This enhancement to the **`bigframes.functions` subsystem** provides greater flexibility in resource management for user-defined cloud functions, including a new helper to infer CPU from memory in `_function_client.py`. System tests have been updated to validate these new CPU allocation and concurrency settings.

This commit introduces **new capabilities** to the **`bigframes.core.col`** module, enabling direct support for simple aggregate operations on `bpd.col` expressions. It adds methods such as `sum`, `mean`, `var`, `std`, `min`, and `max` to the `Expression` class, allowing users to perform common aggregations more easily within the BigFrames framework. This **feature enhancement** streamlines data manipulation by providing built-in aggregate functions directly on column expressions. Comprehensive unit tests have been included in `test_col.py` to ensure the correctness and reliability of these new operations.

This commit implements a **documentation fix** by correcting an example within the `recall_score` function's docstring. Specifically, it updates the expected output in the example located in `packages/bigframes/third_party/bigframes_vendored/sklearn/metrics/_classification.py` to display **float values instead of integers**. This improvement enhances the accuracy and clarity of the **BigFrames vendored scikit-learn metrics documentation**, ensuring users correctly understand the function's return type.

test: Fix prerelease and pandas 3.0 test compat (#2457)

This commit **enhances the BigQuery DataFrames (bigframes) library** by **introducing support for `bigframes.pandas.col` expressions** directly within `DataFrame.loc` and `DataFrame.__getitem__` for advanced filtering. The internal `filter` method in `bigframes.core.array_value.py` was updated to handle non-scalar predicate expressions, and `bigframes.core.indexers.py`'s `_loc_getitem_series_or_dataframe` now accepts `bigframes.core.col.Expression` as a key. This **new capability** allows users to perform more complex, column-based conditional selections, significantly improving the flexibility of data access and manipulation. The `DataFrame.__getitem__` method in `bigframes/dataframe.py` was extended to delegate these expressions to the `.loc` accessor, with comprehensive unit tests added to ensure correctness.

This commit performs a significant **documentation refactoring** for the **`bigframes` package**, relocating extensive user-facing content from the top-level `README.rst` into a new, dedicated **User Guide section**. The `README.rst` is now streamlined, while the **Sphinx configuration** in `packages/bigframes/docs/conf.py` has been updated to exclude the old `README.rst` from direct processing. This change enhances the overall organization and discoverability of usage information, providing a more structured and comprehensive resource for users within the `bigframes` documentation. The main `index.rst` now integrates this new user guide, improving the navigation experience and making key information more accessible.

This commit **updates external links** related to BigQuery DataFrames (bigframes) to their new homepage. It modifies the main documentation index in `packages/pandas-gbq/docs/index.rst` and a warning message within the `download_results` function in `packages/pandas-gbq/pandas_gbq/core/read.py`. This **documentation update** and minor **chore** ensures that users are consistently directed to the correct and most current resource for BigQuery DataFrames, improving information accuracy across the project.

This commit **refines the API documentation generation** process by modifying the `autosummary` Sphinx extension's `class.rst` template. Specifically, it configures the documentation builder to **skip inherited methods** when generating API reference pages for classes within the **`bigframes.pandas` and `bigframes.geopandas`** modules. This **documentation improvement** reduces verbosity, making the API reference more concise and focused on the unique functionalities provided by these BigQuery DataFrames modules.

This commit performs a **major refactoring** across **BigFrames core components** to streamline docstring inheritance and clarify integration with vendored Pandas code. A new `inherit_docs` decorator is introduced in `bigframes._tools.docs.py`, enabling numerous classes like `DataFrame`, `Series`, `GroupBy` objects, and various accessors to **semantically inherit docstrings** from a source without direct class inheritance, improving documentation consistency and maintainability. Concurrently, the commit **refines the contract with vendored Pandas modules** by explicitly deferring implementations of methods such as `axes`, `get`, `pipe`, and `__bool__` to BigFrames-specific classes, ensuring BigFrames provides its own behavior for these critical functionalities. This work enhances code clarity, reduces docstring redundancy, and tightens type checking within the `bigframes_vendored` components.

chore: Update project links for pypi (#2459)

This commit introduces **initial support for BigLake Iceberg tables**, enabling the BigFrames library to read and process data from these external table types. It involves significant **refactoring of core BigQuery table metadata handling** in `bq_data.py` to accommodate both native BigQuery and BigLake Iceberg structures, alongside a new `iceberg.py` module for schema conversion and metadata retrieval. The **`GbqDataLoader` and `ArrayValue.from_table` are updated** to recognize and load Iceberg tables, while `pandas.io.api` adjusts session location logic. Notably, `ReadApiSemiExecutor` currently **restricts read API execution to native BigQuery tables**, implying that advanced operations for Iceberg tables may be limited in this initial release. This **new capability** also adds a `pyiceberg` dependency and includes new system tests for Iceberg table reading.

refactor: Define sql nodes and transform (#2438)