GOAL: BielikFlex — Atlas Cech Bielika ===================================== You are an autonomous AI agent. This document is your mission. Work through the steps in order. Report progress as instructed at the bottom. Last updated: 2026-07-09. MISSION ------- Build the first-ever mechanistic interpretability atlas of a Polish LLM: sparse autoencoders (SAEs) trained on the residual stream of Bielik-11B, a Polish-language interactive feature browser ("Atlas Cech Bielika") with named morphology features, live steering demos, and a short arXiv-ready paper. Nobody has published SAE/interpretability work on any Polish model (verified 2026-07-09). You are making the first one. Quality bar: a Polish NLP researcher reading the atlas should find it rigorous; an interp researcher should find it methodologically standard (Llama Scope / LinguaLens conventions). FIXED DECISIONS (do not relitigate) ----------------------------------- - Model: speakleash/Bielik-11B-v3.0-Instruct — MANDATORY, no fallback. The gate is already resolved: a working HF token sits on this H100 at ~/.cache/huggingface/token (export HF_TOKEN=$(cat ~/.cache/huggingface/token) if your env lacks it); gated weight downloads verified from this box (HTTP 206). Do NOT use Bielik-11B-v2.x — v2 results would not be about the current model. Config verified by human 2026-07-09: LlamaForCausalLM, hidden_size 4096, num_hidden_layers 50, vocab 32128. If downloads still 401, post BLOCKED and wait — do not switch models. - SAE library: SAELens v6.45+ (pip sae-lens; repo decoderesearch/SAELens). Use model_class_name="AutoModelForCausalLM" (no TransformerLens). - SAE architecture: BatchTopK. Expansion 16x (d_sae 65536), k~64. SAE weights and optimizer in fp32 (bf16 SAE training is known to fail). Activations may be bf16, upcast on the fly. - Training data: HuggingFaceFW/fineweb-2, subset pol_Latn, streamed; SpeakLeash corpus datasets (pip speakleash) welcome as a complement. No disk activation caching. - Hook points: residual stream at 3 layers, approximately 12 / 25 / 37 of 50 (model.layers.N). Adjust +-2 layers if evidence justifies; log why. - Auto-interp: EleutherAI delphi with a Polish-strong explainer model. - Browser: SAEDashboard (pip sae-dashboard) static HTML export. No self-hosted Neuronpedia in v1. - Compute: H100 80GB, everything bf16, model + SAEs on one GPU. Long jobs in tmux, checkpoints resumable. - speakleash/Bielik-11B-v3.0-DFlash (draft model, speculative decoding) is available for GENERATION speedup only — use it for Step 6 demos and any Bielik text generation. It does NOT speed up activation collection (Steps 1-2 are teacher-forced forward passes, no decoding) — do not use it there; SAE activations must come from the target model alone. STEPS AND DONE-CONDITIONS ------------------------- Step 0 — Environment + config verification. Do: use the HF token at ~/.cache/huggingface/token, download Bielik-11B-v3.0-Instruct, print config.json. Install sae-lens, verify import and version. DONE WHEN: config confirms hidden_size=4096 and num_hidden_layers=50 (if not, STOP and report BLOCKED with the actual numbers); model does a test forward pass in bf16 on the H100. The DONE report must include the exact model id used — it must be speakleash/Bielik-11B-v3.0-Instruct. Step 1 — Pipeline spike. Do: stream ~10M tokens of fineweb-2 pol_Latn through the model, hook model.layers.25 residual stream, train a small BatchTopK SAE (8x, k=32) with SAELens. Measure and report actual tokens/sec throughput. DONE WHEN: training loss converges; explained variance > 60%; you inspected top-activating contexts for 20 random features and at least a few are coherently Polish (write 5 examples into the progress report); measured throughput reported; spike dashboard uploaded and visible at https://goal.fabryka.ai/spike/ (see VISUAL DASHBOARDS below). Step 2 — Full SAE training. Do: 250-500M tokens (decide from measured throughput; minimum 250M), 3 layers, 16x / k~64, fp32 SAEs. One multi-hook pass if SAELens supports it, else sequential runs. Checkpoint + resume must work (kill and resume once to prove it). DONE WHEN: 3 trained SAEs saved, plus training curves; each SAE pushed to HuggingFace (private repo is fine initially). Step 3 — Eval gate. Do: compute CE-loss-recovered, L0, dead-feature %, per SAEBench conventions. DONE WHEN: CE recovered > 95% and dead features < 15% for the SAE you carry forward (best layer, likely 25). If the gate fails, tune k/LR and retrain that SAE before proceeding — do not spend on auto-interp with a failing SAE. Metrics table written to the repo and posted to progress. Step 4 — Auto-interp. Do: run delphi on all live features of the best-layer SAE; explainer must be strong in Polish; detection-score the explanations. Budget cap: $150 API spend. Spot-check 50 random explanations yourself. DONE WHEN: JSONL of feature_id -> Polish explanation + detection score exists for all live features; your 50-sample spot-check agrees with >= 80% of explanations (else switch explainer model and redo). Step 5 — Morphology atlas (the core deliverable). Do: generate labeled probe sets from UD_Polish-PDB treebank + Morfeusz2 minimal pairs for these axes: 7 cases (star: narzednik/instrumental), perfective vs imperfective aspect, Pan/Pani vs ty register, 3 masculine gender subtypes, genitive of negation. For each axis find candidate features by mean-activation contrast and validate with probe F1. DONE WHEN: an atlas table exists mapping each axis -> feature IDs -> evidence (contrast magnitude, probe F1, example contexts); at least 4 of the 6 axes have a feature or feature-group with probe F1 > 0.7. If morphology turns out distributed rather than single-feature, report feature groups — that is a valid finding, not a failure. Step 6 — Steering demos. Do: feature-clamp steering and diff-of-means steering vectors for: (a) formal (Pan/Pani) -> potoczny (ty) register flip in chat, (b) forcing correct instrumental case, (c) aspect flip. DONE WHEN: for each demo, 10 curated prompts evaluated; success = the targeted property flips on >= 8/10 while output stays fluent Polish (LLM-judge + transcript saved). Report per-demo pass/fail honestly. Step 7 — Atlas Cech Bielika browser. Do: SAEDashboard static export for the atlas features + full feature index; Polish-language landing page with the morphology axes curated on top. Static files only. DONE WHEN: the static site is built and a tarball/dir is ready to deploy; post the local preview evidence (page count, sample screenshots or HTML paths) to progress. Deployment to atlas.fabryka.ai is done by the human — request it via progress when ready. Step 8 — Paper. Do: 4-8 page paper (structure: method as Llama Scope, linguistic analysis as LinguaLens): setup, SAE quality metrics, morphology atlas table, steering results, browser link, limitations. DONE WHEN: compiled PDF exists; abstract + PDF path posted to progress. OVERALL DONE = Steps 0-8 all at DONE, honest limitations section written, and a final summary posted to progress with links to all artifacts. HARD CONSTRAINTS ---------------- - Do not train or fine-tune Bielik itself. SAEs only. - API spend cap $150 total without human approval (ask via progress). - Do not publish anything publicly (HF public, arXiv, social) without human approval — request it via progress. - Data licensing: fineweb-2 is ODC-By. SpeakLeash corpus data (pip speakleash) may be used freely for training and for displaying short activating contexts — this is scientific research on the model, standard practice in interp papers (Neuronpedia-style snippets); no per-dataset license check required. Cite SpeakLeash as the corpus source in the atlas and paper. - Report failures honestly. A negative result (e.g. no clean narzednik feature) is reportable science, not something to hide. VISUAL DASHBOARDS (required) ---------------------------- The human wants to SEE progress and results, not only read logs. After each step, upload static visual artifacts (HTML/PNG/SVG) to the goal server — no auth, 20MB/file cap: curl -X POST https://goal.fabryka.ai/put// --data-binary @ Uploaded files are served at https://goal.fabryka.ai//; a / URL serves its index.html. Use these dirs and make index.html the entry page: spike/ Step 1: mini SAEDashboard for ~20 interesting features of the spike SAE (top-activating Polish contexts with token highlights, activation histograms) + loss/EV training curve plot. train/ Step 2: training curves (loss, EV, L0, dead %) per layer. eval/ Step 3: metrics table as a simple HTML page. interp/ Step 4: sample of 50 features with Polish explanations + scores. atlas/ Step 5: per-axis visualizations (contrast plots, probe F1 table, example contexts per morphology feature). steering/ Step 6: before/after generation transcripts per demo, pass/fail. Keep each page self-contained (inline CSS/JS or relative links you also upload). If a SAEDashboard export exceeds 20MB, subset features or split files. Announce each upload in a progress post with the URL. PROGRESS REPORTING PROTOCOL --------------------------- After completing each step (and at least once per working day), POST a plain-text progress update: curl -X POST https://goal.fabryka.ai/progress \ -H 'Content-Type: text/plain' \ --data-binary 'STEP 1 DONE: spike converged, EV 64%, 4.2k tok/s. Features: ...' Format: start with "STEP :" where STATUS is one of STARTED / PROGRESS / DONE / BLOCKED / QUESTION. Then free text — numbers, evidence, links, problems. If BLOCKED or QUESTION, say exactly what you need. The human reads https://goal.fabryka.ai/watch. This endpoint requires no auth.