GOAL: BielikFlex - Atlas Cech Bielika ===================================== You are an autonomous AI agent. This document is the current mission state. Work from the status table first, not from old progress-log entries. Report progress at the bottom. Last updated: 2026-07-10 21:00 UTC. MISSION ------- Build the first mechanistic-interpretability atlas of a Polish LLM: SAEs trained on the residual stream of Bielik-11B, a Polish-language feature browser ("Atlas Cech Bielika") with statistically validated feature evidence, steering demos, and a detailed, high-quality arXiv-ready paper. The paper must include publication-quality images, Matplotlib plots, TikZ diagrams, full result tables, clear captions, and reproducible figure sources. Fixed model: speakleash/Bielik-11B-v3.0-Instruct. Do not fall back to v2.x. HF access is already resolved on this H100. Target hooks are model.layers.12, model.layers.25, model.layers.37. Carry-forward layer is L25. CURRENT STATE - READ THIS FIRST ------------------------------- DONE: - Step 0 environment/config: DONE. Bielik v3 config verified: LlamaForCausalLM, hidden_size=4096, num_hidden_layers=50, vocab=32128. bf16 H100 forward pass succeeded. Evidence: /home/ubuntu/bielikflex/reports/step0_config_v3.json /home/ubuntu/bielikflex/reports/step0_forward_v3.json - Step 1 spike SAE: DONE. 10M FineWeb-PL tokens, L25, BatchTopK 8x/k=32, throughput about 9.88k tok/s, centered EV 97.56%, uncentered EV 97.67%. Spike page: https://goal.fabryka.ai/spike/ - Step 2 full SAE training: DONE. 250M-token multi-hook run finished; final checkpoint final_250000384; final artifacts exist for layers 12/25/37; private HF upload succeeded to kacperwikiel/bielikflex-step2-saes. Training page: https://goal.fabryka.ai/train/ - Step 3 eval gate: DONE for carried L25. L25 has 99.288% CE recovered, 99.224% explained variance, L0 about 78.7, and 4.596% zero-activation features on a fresh 10.24M-token scan. The old final-window dead-feature numbers were invalid because training ended inside an incomplete sampling window. Eval page: https://goal.fabryka.ai/eval/ NOT DONE: - Step 4 auto-interpretation: NOT DONE. There are sampled contexts and a weak evidence-label baseline for 787 features, but no Delphi/LLM auto-interp JSONL for all live features and no 50-feature manual audit. Current baseline explicitly says not independently validated; most labels are low confidence. Do not call this complete. - Step 5 statistical feature atlas: NOT DONE. The old hand-picked linguistic-probe route is no longer the primary path. It produced weak evidence: only 1/6 tested axes had macro-F1 > 0.7, and the results indicate that a statistical feature-quality approach is stronger than a hand-picked linguistic-axis approach. Keep those old probe results only as a negative/diagnostic finding. The new Step 5 target is a statistically validated atlas of feature families, coherence, domain/boilerplate structure, activation-rank behavior, trigger-token evidence, and causal intervention evidence. - Step 6 steering demos: NOT DONE. 30 transcripts exist, but heuristic scores do not pass for the old probe-derived demo set. Requirement is >=8/10 per demo while output stays fluent Polish, with judged transcripts. - Step 7 browser: NOT DONE. A static scaffold exists, but it is not the final atlas browser. SAEDashboard failed because sae-dashboard expects TransformerLens hook names blocks.{layer}.*, while this project uses HF hook names model.layers.{layer}. Need either an adapter or a custom/static browser path that supports HF hooks. Current scaffold: https://goal.fabryka.ai/atlas/ - Step 8 detailed high-quality paper: NOT DONE. A compiled technical-report draft exists, but it reports feature-atlas and steering evidence as incomplete and is not the required final paper. The final deliverable is a detailed arXiv-ready paper with publication-quality images, Matplotlib plots, TikZ diagrams, full evidence tables, clear captions, and reproducible figure sources. It must incorporate validated Step 4/5/6 results or clearly state negative results. Draft page: https://goal.fabryka.ai/paper/ ACTIVE NEXT TASKS ----------------- 1. Fix feature interpretation quality before expanding claims. - Sample top contexts for a broad set of live L25 features, not only easy or high-activation examples. - For each candidate label, compute and store: a) source/domain diversity among top-k contexts, b) activation-rank shape: smooth falloff vs cliff; cliff-like features are narrow/memorization-like unless proven otherwise, c) strongest-token / trigger-token evidence, d) top-context coherence after boilerplate removal, e) causal evidence where possible: zero or boost the feature and check whether output changes match the hypothesis. - Top contexts are correlation only. Causal intervention is the stronger evidence standard. - Flag "domain boilerplate" features separately. Their fraction is itself a useful corpus-quality finding and should connect to the SpeakLeash/CPT data narrative. - Produce global metrics for the feature browser/report: coherence distribution, dead-feature %, L0, CE/loss recovered, FVU/reconstruction, and percent domain-boilerplate. - Better auto-labeling options: embed cleaned contexts, cluster context embeddings, then use LLM-as-judge/labeler with held-out contexts and a 50-feature manual audit. Do not accept labels without held-out detection and audit. 2. Build Step 5 as a statistical feature atlas. - Stop treating hand-picked linguistic axes as the core deliverable. The observed stats show that this route is weaker than a statistical feature quality and clustering route. - Use held-out activation statistics, context embeddings, source/domain diversity, activation-rank falloff, trigger-token evidence, coherence scores, boilerplate flags, and causal zero/boost interventions. - Group features by statistical/semantic behavior. Report feature families, narrow memorization-like cliffs, domain boilerplate, and robust recurring concepts separately. - Keep old hand-picked linguistic probes as an appendix/negative result only where they clarify why the project pivoted to statistical feature analysis. 3. Redo Step 6 steering from validated features/groups. - Current decoder-vector interventions are exploratory failures, not demos. - Use only statistically validated features/groups, compare feature clamp and diff-of-means steering, and score 10 prompts per demo with an LLM judge plus saved transcripts. 4. Fix Step 7 browser. - Either patch/adapt SAEDashboard for HF hook names model.layers.N or build a custom static browser using extracted feature evidence. - The browser must distinguish query terms, activation evidence, logit effects, labels, and causal evidence. Unvalidated labels must be marked "uninterpreted" or "hypothesis". 5. Update Step 8 detailed high-quality paper after Step 4/5/6 are honest and stable. - The target is a detailed, high-quality arXiv-ready paper, not a minimal technical note. - Include method, SAE quality metrics, feature-quality metrics, statistical feature-family atlas, steering pass/fail table, browser link, limitations, and negative results. - Include publication-quality figures with readable typography, axis labels, legends, captions, and cross-references: a) Matplotlib plots for training curves, CE/loss recovered, FVU or reconstruction error, L0, dead-feature scan, feature-coherence distribution, source/domain diversity, boilerplate fractions, activation-rank cliff statistics, feature-family clusters, and steering results. b) TikZ diagrams for the pipeline, activation/SAE hook locations, feature-evidence workflow, and steering/intervention setup. c) Example feature panels with top contexts, trigger-token evidence, activation-rank falloff/cliff plots, and causal intervention result. - Store source files for every figure under the paper or reports tree so plots are reproducible, not pasted screenshots. FIXED DECISIONS AND CONSTRAINTS ------------------------------- - Model: speakleash/Bielik-11B-v3.0-Instruct only. - SAE library: SAELens v6.45+ with model_class_name="AutoModelForCausalLM". - SAE architecture: BatchTopK, 16x (d_sae=65536), k about 64 for full runs. - SAE weights/optimizer in fp32. Model activations may be bf16 and upcast. - Training data: FineWeb-2 pol_Latn streamed; SpeakLeash may be used and should be cited when used. No disk activation caching for training. - Do not train or fine-tune Bielik itself. SAEs and interventions only. - API spend cap: $150 total without human approval. - Do not publish HF public, arXiv, or social announcements without human approval. Private HF artifacts are acceptable. - Report failures honestly. Negative probe/steering results are valid science. VISUAL DASHBOARDS ----------------- Static pages are required and should be kept current: - spike/ DONE: https://goal.fabryka.ai/spike/ - train/ DONE: https://goal.fabryka.ai/train/ - eval/ DONE/needs final page refresh if metrics change: https://goal.fabryka.ai/eval/ - interp/ NOT DONE: should show validated feature labels, coherence, domain flags, and 50-feature audit. - atlas/ NOT DONE: should show statistical feature families, coherence distributions, source/domain diversity, boilerplate flags, activation-rank cliff plots, trigger-token evidence, causal checks, and old hand-picked probe findings only as an appendix. - steering/ NOT DONE: should show before/after transcripts and pass/fail. - paper/ DRAFT ONLY: final target is now a detailed paper with Matplotlib plots and TikZ diagrams: https://goal.fabryka.ai/paper/ Upload static artifacts with: curl -X POST https://goal.fabryka.ai/put// --data-binary @ For the root goal document, update the Vultr server file: /opt/goal-fabryka/app/goal.txt The /put/index.html artifact is only a static copy; it does not change GET /. PROGRESS REPORTING PROTOCOL --------------------------- After completing each step or important correction, POST a plain-text update: curl -X POST https://goal.fabryka.ai/progress \ -H 'Content-Type: text/plain' \ --data-binary 'STEP 4 PROGRESS: ...' Format: start with "STEP :" where STATUS is STARTED, PROGRESS, DONE, BLOCKED, or QUESTION. If BLOCKED or QUESTION, say exactly what is needed. The human reads https://goal.fabryka.ai/watch. OVERALL DONE ------------ Overall DONE requires Steps 0-8 all marked DONE under this updated standard: validated feature interpretations, statistical feature-family atlas, honest negative probe findings, steering pass/fail evidence, final browser, detailed paper with images/plots/diagrams and reproducible figure sources, and a final progress summary with links to all artifacts.