Zeina Migeed

Add github workflow for v1 issue ranking Summary: The full issue_ranker package, which is a 5-pass LLM pipeline for ranking pyrefly GitHub issues by priority: 1. GitHub Actions workflow (manual dispatch for now) 2. Issue collection which consists of: - code extraction from issue bodies - snippet repair via LLM - type checker execution (pyrefly/pyright/mypy) - status classification - relationship resolution, - dependency resolution 3. 5-pass LLM ranking pipeline: categorize (Haiku) → primer impact (deterministic error matching then LLM matching if that fails) → dependencies (Opus) → scoring (Sonnet) → final ranking (Opus, batched with retry) 4. Report generation: markdown + JSON output with V1 gap analysis 5. compare_typecheckers.py fixes — 10-min timeouts, pyrefly config cleanup, pyright path overrides (scipy takes too long because it has no config, so we override it) The ranking pipeline uses 8 weighted signals: false positive impact (x3), performance (x3), team priority labels (x2), false negatives/spec compliance (x2), actionability (x2), IDE/usability (x1.5), primer breadth (x1.5), and adoption ecosystem labels (x1.5). Team priority labels (P0/P1/P2) contribute ~20% of the weighting. On the full 391-issue run, the top 20 (critical tier) are dominated by type inference bugs and false positives that affect many primer projects — these would rank highly regardless of labels. Local Results: - Top 10 is 70% the same with or without labels - Labels boost V1 overlap from 29% → 38% (19 → 25 out of 65) with labels P2230410941 without P2230411326 The V1 overlap data from the comparison: WITH labels: 25/65 V1 issues in top 65 (38%) WITHOUT labels: 19/65 V1 issues in top 65 (29%) The 25 V1 issues captured (with labels): #105, #630, #1252, #1286, #1321, #1486, #1518, #1763, #1982, #2043, #2105, #2309, #2370, #2382, #2419, #2445, #2452, #2515, #2517, #2519, #2520, #2522, #2610, #2611, #2616 The 40 V1 issues missed (not in top 65): #43, #219, #317, #383, #385, #397, #793, #795, #881, #908, #920, #1078, #1089, #1159, #1336, #1340, #1422, #1440, #1474, #1728, #1956, #2039, #2064, #2065, #2132, #2139, #2184, #2199, #2207, #2219, #2227, #2303, #2304, #2319, #2386, #2454, #2508, #2535, #2558, #2621 So the ranker agrees with about 38% of the team's V1 picks. NOTE: I need to test this as an actual github workflow. I cannot be 100% certain that the workflow works end to end without kicking off a manual run there. Reviewed By: yangdanny97 Differential Revision: D95686786 fbshipit-source-id: 8de3332306be7d69875a4dfca9f8979e7d19c846

Add project deps + cross-check mypy/pyright in classifier Summary: There are two changes in this diff: 1- install the deps from the dependency list for pyrefly 2- projects that get flagged by the initial run are rerun with pyright and mypy with the same dep list then we compare the results and feed that data to our classifier Reviewed By: grievejia Differential Revision: D96362978 fbshipit-source-id: efd6aa7940caf5ddaf5bd2f5113b4a0b067de21e

Automated V1 gap analysis with labels and reasons Summary: Adds a post-processing step to the issue ranking pipeline that: 1. Reads existing ranking.json (no re-running the pipeline) 2. Uses one cheap Haiku LLM call to generate concise reasons for each issue 3. Applies GitHub labels: v1-consider-adding, v1-consider-removing, v1-verified 4. Generates a team-readable markdown report 5. Renders the full report directly in the GitHub Actions run page via Job Summary The V1 analysis runs automatically after ranking in the workflow, or standalone against a previous run's artifacts (pass v1_analysis_run_id to skip primer+ranking). The full V1 gap analysis report is visible in two places: - **GitHub Actions Job Summary**: rendered directly in the workflow run page — the team can open the run and see the full V1 analysis board with tables and reasons without downloading anything. - **Workflow artifacts**: downloadable as v1_analysis.md in the issue-ranking artifact (or the v1-analysis artifact for standalone runs). We chose Job Summary over committing a .md file back to the repo because that would require write permissions to push, introduce potential merge conflicts, and add maintenance overhead — while Job Summary is zero-maintenance and always reflects the latest run. **Label management**: only touches labels introduced by our workflow (v1-verified, v1-consider-adding, v1-consider-removing). No other labels are ever modified or removed. Reviewed By: rchen152 Differential Revision: D96396374 fbshipit-source-id: 4bee635b494c3383f25cf67854d16626ae7e4980

Update conformance sources Reviewed By: yangdanny97 Differential Revision: D96022818 fbshipit-source-id: 880d8387e4e752aedccab98273782945b7f99455

Promote implicit literals for module-level variables Summary: Module-level assignments like `timeout = 100` infer `Literal[100]`, but protocols expect `timeout: int`. This caused the `protocols_modules.py` conformance test to fail. We fix this by promoting implicit literals at two use sites where module-level variables are read from outside their defining scope: 1. Cross-barrier function reads: when a function reads a module-level variable (across a flow barrier), we use `Binding::PromoteForward` instead of `Binding::Forward`. This widens implicit literals to their base types, so `def foo(): reveal_type(x)` gives `int` for `x = 42`. The `is_module_scope` flag on `NameReadInfo::Anywhere` ensures we only promote reads from module scope, not from enclosing functions. 2. Module exports: unannotated exports promote implicit literals so that cross-module access like `import mod; mod.x` gives `int`. Flow-sensitive reads at the same level are unchanged — `x = 42; reveal_type(x)` still gives `Literal[42]`. ALL_CAPS names are excluded from promotion as conventional constants. Annotated exports (including `Final`) are unchanged since the annotation determines the type. Reviewed By: rchen152 Differential Revision: D98803805 fbshipit-source-id: e7784ad9be3b7a8a32bfd280b31f6e1e2f273069

Add optional `migrate-from` flag to handle the case where both pyright and mypy configs exist Summary: We currently default to migrating from mypy when both mypy and pyright configs exist. This diff is to add the option to choose which typechecker to migrate from. Currently, there is no way to specify this. We accomplish this by adding an "auto" option, which is the default. Otherwise, if we specify a typechecker, we try to migrate it and emit an error otherwise. Use case: I'm currently working on getting some mypy primer data on both pyrefly and pyright, and for high quality data, we should run them on the same configuration. One option is to manually write (or have the AI write) those config files. The downside is that there's no easy way for me to check the correctness of that that would be better than the migration script, which we already have. Reviewed By: stroxler Differential Revision: D95433126 fbshipit-source-id: e3487f145f4ebf49b0a1c0067e35d9efb191b69a

Open source comparison script to be able to run on github actions Summary: The script was previously tailored to run on a devserver. It was using buck and proxy env vars. I want to eventually use this for a github action to classify open source issues, so unfortunately, I need to run this script on the open source. So I'm modifying it to - use cargo instead of buck - remove proxy env vars - moved the files outside of internal. I think the recommendation is not to run with an internet environment on internal systems, so the safest option seems to move it outside of that. **Next steps/questions to answer:** Now that we have both typecheckers running in the same env and in an internet env, what framework should we build around it so that we are able to answer: - Q1: "How should we prioritize our v1 issues". I think an LLM can answer this, but at minimum, it needs: - the spec - pyright and pyrefly's full error messages - a prompt (we can look at the mypy primer classifier for some inpiration) guiding it on how to classify - Q2: Since this is a data driven approach to prioritize issues, what format should the data look like? where are we going to upload the data? My plan is to have this as a github workflow, similar to mypy primer. Happy to hear any feedback on this plan. Reviewed By: stroxler Differential Revision: D95446783 fbshipit-source-id: f261425f3f0327e447caf37ce27821dda2d86634

Match primer errors by message pattern + LLM specificity assessment Summary: Two improvements to primer impact matching: 1. **Template matching**: Previously matched by error kind only (e.g., `bad-argument-type`), crediting every issue with ALL errors of that kind. Now matches by (kind, message_template), where templates replace backtick-quoted identifiers with `_`. Falls back to kind-only aggregation when no template match is found. 2. **LLM specificity assessment**: After deterministic matching, a Haiku call assesses how specific the matched error pattern is to the issue (high/medium/low). A generic pattern like "Argument `_` is not assignable to parameter `_`" could have many root causes — the LLM evaluates whether this particular issue is likely THE cause or just one of many. The scoring LLM then uses this to weight primer counts appropriately. Reviewed By: yangdanny97 Differential Revision: D96421336 fbshipit-source-id: fc4f24fff5df18e7bc17e855d52f03a638e9e94f

Improve classification quality with self-critique, majority voting, and cross-project consistency (#2841) Summary: Pull Request resolved: https://github.com/facebook/pyrefly/pull/2841 The primer classifier has been producing inconsistent results across runs — the same primer diff can be classified as 'improvement' in one run and 'regression' in another. This was observed on real PRs like https://github.com/facebook/pyrefly/pull/2839 (altair TypeVar iterability) and https://github.com/facebook/pyrefly/pull/2764 (overload resolution, 60+ projects). Three changes to improve reliability: 1. **Self-critique pass (Pass 1.5)**: After Pass 1 produces reasoning, a new pass checks it for factual errors — e.g., claiming dicts are not iterable, incorrect inheritance claims, wrong TypeVar constraint analysis. This catches hallucinations before they reach the verdict pass. Tested on PR #2839 where it correctly identified that both constraints of `_C` (list and TypedDict) are iterable. 2. **Majority voting on verdict (Pass 2)**: Instead of a single verdict call, makes 5 independent calls and takes the majority. This reduces non-determinism where the same reasoning could be classified either way. Vote distribution is logged for transparency. 3. **Cross-project consistency enforcement**: After classifying all projects independently, groups them by error kind and enforces majority verdict within each group. This prevents the classifier from saying 'overload resolution improved' for one project and 'overload resolution regressed' for another with the same pattern. Also upgrades the default Anthropic model from claude-opus-4-20250514 to claude-opus-4-6 for better Pass 1 reasoning quality. According to gemni, this is a big upgrade :) so I am hoping to see improvement in the quality. Reviewed By: yangdanny97 Differential Revision: D97571454 fbshipit-source-id: 356f4b150e0c4886c2743abc17699e004da997f1

Fix false positive unbound-name for walrus operator in while condition Summary: The while-loop condition always evaluates at least once, so any walrus target in the condition is guaranteed to be assigned after the loop. However, `setup_loop` snapshots the current flow as the loop's base *before* `ensure_expr` processes the test, so walrus-defined names only entered the loop flow and were missing from the base. When `teardown_loop` merged, it saw those names missing from the base and marked them `PossiblyUninitialized`. The fix propagates newly-defined names from the while condition into the loop's base flow after `ensure_expr`, mirroring the existing `propagate_new_flow_entries_to_fork_base` used for elif conditions. Fixes https://github.com/facebook/pyrefly/issues/3272 Reviewed By: grievejia Differential Revision: D103246547 fbshipit-source-id: 00560948fcd587cf98764a87e0d0da27e9a4eea4

Preserve annotated type when assigned value is Any (github #2227) Summary: When a variable has a bare annotation (`x: int`) and is then assigned a value of type `Any`, pyrefly was using `Any` as the type of `x` instead of preserving the declared `int`. The fix distinguishes first assignments after a bare annotation from reassignments of already-initialized variables. A new `ForwardedInitial` annotation style marks the first case, where the annotation is the primary type declaration and should take precedence over uninformative types like `Any`. For reassignments (`Forwarded`), the expression type still takes precedence, so patterns like `headers = json.loads(headers)` continue to work without false positives. fixes https://github.com/facebook/pyrefly/issues/2987 Reviewed By: yangdanny97 Differential Revision: D98856477 fbshipit-source-id: 35f926f80b716eb405a61d88e73ebf9bce5945ce

Add boilerplate code for new asymmetry field as property of a class Summary: This diff only adds the plumbing. In the next diff, we will implement the algorithm. Issue: https://github.com/facebook/pyrefly/issues/3299 Reviewed By: yangdanny97 Differential Revision: D104536804 fbshipit-source-id: ebefd7865495198ff6939fd9f79d1d90bdabeda2

Use cached symmetry to gate subscript-assignment narrowing Summary: Use cached symmetry answer from the previous diff to determine if we should narrow. fixes https://github.com/facebook/pyrefly/issues/3299 Reviewed By: rchen152 Differential Revision: D104536802 fbshipit-source-id: bb1ebf028a77715b68ec6d1b20a74e6aafd62034

suppress reachability error in generator pattern Summary: closes https://github.com/facebook/pyrefly/issues/2645 Reviewed By: yangdanny97 Differential Revision: D96013327 fbshipit-source-id: d1a6a40f5d415278d060519d69c7fc3ca8bea889

Support field_validator mode='before'|'plain' Summary: Support plain and before modes. They are different at runtime but I believe that from a static perspective, we treat them the same way https://github.com/facebook/pyrefly/issues/589 Reviewed By: samwgoldman Differential Revision: D102908500 fbshipit-source-id: 0e850ee71ec653e1a7ce49644e9e0508ffe07b23

Fix annotated legacy TypeVar/ParamSpec/TypeVarTuple not working as type annotations Summary: Annotated TypeVar definitions like `T: TypeVar = TypeVar("T")` were previously handled as `NameAssign` bindings via the `AnnAssign` code path, which meant `solve_legacy_tparam` could not recognize them as type parameters. This caused downstream `not-a-type` errors when using the TypeVar in annotations like `def f(x: T) -> T`. The fix detects TypeVar/ParamSpec/TypeVarTuple calls in the `AnnAssign` handler and routes them through the existing `assign_type_var` (etc.) functions, which create proper `Binding::TypeVar` bindings. Additionally, the solving path now uses `check_type` instead of `check_and_return_type` to always preserve the special type representation even if the annotation check fails. Reviewed By: yangdanny97 Differential Revision: D98853963 fbshipit-source-id: 5febad3c40e514c8486345655ee12e6be493809f

Add non-interactive mode to pyrefly init Summary: Currently, mypy primer does not run with any configs, which is wrong. We should invoke pyrefly init and generate a config file for every project. To do this for CI purposes, workflow purposes and experiment purposes, we should also have a non interactive mode. This diff adds this mode, which declines suppressing errors and overriding configs (for CI there are no existing pyrefly configs anyway). Reviewed By: yangdanny97 Differential Revision: D94755976 fbshipit-source-id: 7568744b7b613c825d3d2fc6d932647e8190a35d

Solve ImplicitAliasCheck binding Summary: Add `is_annotation: bool` to `Usage::StaticTypeInformation` so that annotation contexts (parameter types, return types, variable annotations) are distinguished from other type contexts like base class expressions. Only annotation contexts trigger implicit alias validation — base classes can use arbitrary expressions and should not be checked. Implement the solve logic for `BindingExpect::ImplicitAliasCheck` using `expr_infer` on the call's function to determine exemptions. If the callable resolves to a `ClassDef`, the assignment is exempt (covers TypeVar, NamedTuple, TypedDict, `type()`, and other class constructors). Otherwise, the error is emitted. Reviewed By: rchen152 Differential Revision: D104353415 fbshipit-source-id: 105c054de20bb047818e855f00eae8e41da652e4

Auto-trigger mypy primer classifier (#2568) Summary: Tested the tool on some PRs manually. Example run: https://github.com/facebook/pyrefly/pull/2567 I would like to gather some feedback on the took before tuning it further, so I am proposing to make the workflow automatic. I added a feedback thumbs up/thumbs down to gather feedback in the last diff. I increased the font size to make it more visible. Pull Request resolved: https://github.com/facebook/pyrefly/pull/2568 Reviewed By: grievejia Differential Revision: D94549072 Pulled By: migeed-z fbshipit-source-id: 4b5b4e5a9e07f2588c19fcd1643f11fa9766543b

Add --output-json option for full error messages in compare_typecheckers.py Summary: Adds a new `--output-json <path>` flag that dumps structured JSON with every error message from pyrefly and pyright, instead of just error counts. Two output modes: - Default : prints summary table with error counts per project (existing behavior, unchanged) - new mode`--output-json path.json` which writes structured JSON with full error details (file, line, col, kind, message, severity) for each checker on each project When `--output-json` is used, pyrefly runs with `--output-format json` and pyright runs with `--outputjson` to get structured output. Pyright's absolute paths are normalized to relative paths. Reviewed By: stroxler Differential Revision: D95615419 fbshipit-source-id: 5c9ce11461ea688fcce7a78e178bdf76bb20eca9