Implicit Interaction Intelligence — an on-device assistant that reads how you type, then answers accordingly. Built from scratch in PyTorch on a single 6 GB laptop GPU. ~200 MB int8. No cloud round-trip required.

A 204 M-parameter custom transformer (MoE-FFN, ACT halting, per-layer cross-attention conditioning), a from-scratch byte-level BPE tokenizer (217 LOC), a TCN encoder for keystroke dynamics, a LinUCB contextual bandit for edge/cloud routing, and a closed-loop adaptation pipeline that observes a user, infers an 8-axis adaptation vector, and biases generation accordingly — all in a single repository, with zero HuggingFace dependencies in the SLM generation path.

Three-arm cascade with a scored multi-signal smart router — local SLM + retrieval gets the first shot on every chat turn; Qwen 1.7 B + LoRA handles HMI commands (set_timer, play_music, navigate, …) with deterministic JSON output; Gemini 2.5 Flash "tags in" only when the local arms genuinely can't ground the query. Every reply ships a route_decision chip showing which arm answered, the per-arm confidence scores, and the routing reason in a hover tooltip.

Real actuators — set timer for 30 seconds schedules an asyncio task that fires a notification 30 s later (gold pulse banner in the chat); navigate to trafalgar square shows an immediate route banner; cancel tears down all pending timers. The cascade isn't just acks — it actually does things.

Edge inference, demonstrable live. The TCN encoder is INT8-quantised to 162 KB ONNX and runs in your browser tab via ONNX Runtime Web (WASM / WebGPU). Toggle is in the State tab → Edge inference. Open Chrome DevTools → Network and you see zero /api/encode requests when ON — keystrokes never leave the page.

Live demo · 30-min demo cheat sheet · Direct response to recruiter pre-screen · HCI design brief · Open problems / what I'd hand a teammate · Technical report

Run I3_PRELOAD_QWEN=1 python -m uvicorn server.app:app --host 127.0.0.1 --port 8000, then visit http://127.0.0.1:8000 — the SPA opens straight on the Chat tab with five suggestion chips that exercise every cascade arm in five clicks.

1. Why this project exists
2. What makes this different
3. Architecture at a glance
4. From-scratch components
5. Run it locally
6. The demo UI — what each tab shows
7. Live measurements
8. Privacy story
9. Repository layout
10. What I would ship next
11. Credits & references
12. Huawei filter questions — direct map

If you arrived here via the recruiter / via a link in my application, these are the documents to read in priority order:

Most chatbot demos in 2026 are a thin wrapper over a foundation model. They are good at language. They are dead to how you typed it.

The HMI thesis behind this project: people generate a stream of implicit signals — inter-key intervals, edit ratios, voice prosody, gaze fixation — that already encode their cognitive load, affect, and intent. A useful assistant adapts its phrasing, length, vocabulary, and even its routing decisions to that signal, in real time, on device.

This is the demo of that thesis end-to-end. It is also the portfolio piece for the Huawei R&D UK HMI Lab AI/ML Specialist Internship application; the four "filter questions" the recruiter asked are mapped directly to artefacts at the bottom of this file.

Five claims, ranked by relevance to the Huawei JD.

The decoder transformer, the byte-level BPE tokenizer, the TCN encoder, the MoE-FFN with softmax gating, the ACT halting controller, the LinUCB contextual bandit, the BM25 reranker, the typing-biometric authenticator, the LoRA adapter, the per-layer cross-attention head — all written from the original papers in torch.nn.Module / numpy / pure Python. Zero HuggingFace transformers, zero sentence-transformers, zero tokenizers crate, zero pretrained weights downloaded.

AdaptiveTransformerV2 (i3/slm/adaptive_transformer_v2.py) is a 204 M-parameter decoder with d_model=768, 12 layers, 12 heads, d_ff=3072, MoE-FFN with 2 experts, ACT halting, per-layer cross- attention onto an 8-dim adaptation vector and 64-dim user-state embedding, trained with bf16 + 8-bit AdamW + gradient checkpointing on a 977,332-pair dialogue corpus collected from six public sources. No distillation from a foundation model. No "just fine-tune Phi-3." From scratch end to end.

Every reply runs through a real DAG: keystroke capture → linguistic features → TCN encoder → user-state embedding → adaptation-vector inference → bandit route → retrieval (cosine + BM25 rerank) → SLM generation (or cloud LLM with PII sanitisation) → post-processor (surface + structural rewriting) → self-critique loop → coreference resolution → diary persist → telemetry. The Flow tab in the UI animates each stage on every reply with real timings; no synthetic animation.

Measured (CPU, no GPU). The TCN encoder is shipped as 162.2 KB INT8 ONNX (web/models/encoder_int8.onnx) — −63 % vs the 441.4 KB FP32 export, parity MAE 0.00055 — and runs in-browser via onnxruntime-web (WebGPU + WASM fallback) at p50 460 µs (2 176 enc/s). The v2 SLM (204 M params) is profiled at ~200 MB INT8 budget; the v1 SLM (53 M params, kept as a draft-model candidate) measures 110.2 MB int8 / 612 ms p50 for a 32-prompt → 16-token greedy decode. Live numbers: reports/edge_profile_2026-04-28.md (latest); reports/edge_profile.md (older v1 baseline).

The diary database has no text column — only encrypted embeddings, topic-keyword lists, and scalar metrics. PII is stripped at every cloud boundary by i3/privacy/sanitizer.py. The biometric template is keyed by SHA-256, never stored as raw key events. Cloud calls are off by default; toggling them on shows a live privacy budget in the Privacy tab.

┌─────────────────────── on-device pipeline (CPU or 6 GB GPU) ──────────────────────┐
│                                                                                    │
│   [keystroke + voice + gaze]                                                       │
│             │                                                                       │
│             ▼                                                                       │
│   ┌──────────────────────┐    ┌──────────────────────┐    ┌──────────────────────┐│
│   │ feature extractor    │───▶│  TCN encoder         │───▶│ 64-d user-state emb. ││
│   │ 32-dim sliding win   │    │  (dilated conv,      │    │ + 8-axis adaptation  ││
│   │ IKI · burst · dwell  │    │   metric-loss train) │    │   vector             ││
│   └──────────────────────┘    └──────────────────────┘    └──────────┬───────────┘│
│                                                                       │            │
│   ┌──────────────────────┐    ┌──────────────────────┐                │            │
│   │ retrieval            │    │ LinUCB bandit        │◀───────────────┘            │
│   │ cosine + BM25 rerank │───▶│ {edge SLM, cloud}    │                             │
│   └──────────────────────┘    └──────────┬───────────┘                             │
│                                          │                                          │
│                       ┌──────────────────┴───────────────────┐                     │
│                       ▼                                       ▼                     │
│   ┌──────────────────────────────────────┐   ┌─────────────────────────────────┐  │
│   │  AdaptiveTransformerV2 (custom)      │   │  cloud LLM (opt-in)             │  │
│   │  204 M params · MoE × 2 · ACT halt   │   │  PII sanitiser · privacy budget │  │
│   │  per-layer cross-attn on (state, A)  │   │  policy guarded · cost tracked  │  │
│   └────────────────┬─────────────────────┘   └────────────────┬────────────────┘  │
│                    │                                            │                   │
│                    ▼                                            ▼                   │
│   ┌────────────────────────────────────────────────────────────────────────────┐  │
│   │  post-processor (surface + structural rewriting)                           │  │
│   │  self-critique loop · coreference resolution · diary persist (encrypted)   │  │
│   └────────────────────────────────────────────────────────────────────────────┘  │
│                                          │                                          │
│                                          ▼                                          │
│                                    user reply                                       │
│                                                                                    │
└────────────────────────────────────────────────────────────────────────────────────┘

The same diagram is animated live in the Flow tab; every dot is a real stage with real timings.

Each row links to the implementation file and the paper it's drawn from. None of these are imported from a library.

Use this section the morning of the demo. Everything you need is here — start command, URL, reset command, "is it working?" checks. If anything is unclear, this section is broken; fix it.

Windows PowerShell (this is what's on the demo laptop):

$env:I3_PRELOAD_QWEN="1"; .\.venv\Scripts\python.exe -m uvicorn server.app:app --host 127.0.0.1 --port 8000

Git Bash / Linux / macOS:

I3_PRELOAD_QWEN=1 ./.venv/Scripts/python.exe -m uvicorn server.app:app --host 127.0.0.1 --port 8000

The command takes 10–15 seconds to fully boot because Qwen3-1.7B is preloaded into memory. Wait until the terminal prints both:

Uvicorn running on http://127.0.0.1:8000
Application startup complete

Why .venv\Scripts\python.exe and not just python? The project's dependencies (uvicorn, torch, transformers, …) are installed inside .venv/, not in your system Python. If you run plain python you'll hit No module named uvicorn. Using the venv's python directly is foolproof on any fresh terminal.

If you'd rather activate the venv once per terminal:

.\.venv\Scripts\Activate.ps1
$env:I3_PRELOAD_QWEN="1"
python -m uvicorn server.app:app --host 127.0.0.1 --port 8000

After Activate.ps1 your prompt shows (.venv) at the front.

Open http://127.0.0.1:8000 in Chrome or Edge.

You should land on the Chat tab with five suggestion chips below the hero:

• How do you adapt to me?
• Set timer for 30 seconds
• Tell me about Uzbekistan
• What is photosynthesis?
• Navigate to Trafalgar Square

These exercise every cascade arm in five clicks (see the demo flow in docs/huawei/PRESENTATION_SCRIPT.md).

Run this in a second terminal once the server is up. It wipes the keystroke biometric template so Identity Lock starts at learning · 1/5.

curl -X POST http://127.0.0.1:8000/api/biometric/demo/reset

Then hard-refresh the browser (Ctrl+Shift+R) to bust the SPA cache.

• ✓ Server terminal shows Application startup complete.
• ✓ Browser at http://127.0.0.1:8000 shows the Chat tab with the five suggestion chips.
• ✓ The nav bar shows the Identity-Lock pill on the right (text: learning · 1/5 after the reset, or unregistered if not).
• ✓ The cloud-route consent toggle reads cloud · off (default).
• ✓ If you toggle State tab → Edge inference, then send a message, the browser DevTools Network panel shows zero /api/encode requests.

.\.venv\Scripts\python.exe scripts\verify_numbers.py

Should return ALL CLAIMS VERIFY OK (22 / 22 PASS). If any FAIL, fix the docs before the interview.

• Python 3.12 (3.10 / 3.11 also tested)
• ~3 GB free disk for checkpoints + corpus
• Optional: NVIDIA GPU with 6 GB+ VRAM for v2 SLM training (CPU fine for inference and the full demo)

git clone https://github.com/abailey81/implicit-interaction-intelligence.git
cd implicit-interaction-intelligence

# Fast path: install + seed checkpoints + serve on http://127.0.0.1:8000
make all-fast

# Full path: also retrains encoder + SLM, runs benchmarks + ONNX export
make all-full

Both targets call scripts/run_everything.py, the wave-scheduled orchestrator (21 stages, parallel within waves, DAG resume, Rich live UI). To inspect a stage list:

make all-list

poetry install
poetry run uvicorn server.app:app --host 127.0.0.1 --port 8000

Open http://127.0.0.1:8000 in a Chromium-family browser (Chrome, Edge, Brave). The UI is a single SPA with 12 tabs.

# Clear biometric template + diary so the next demo starts cold
curl -X POST http://127.0.0.1:8000/api/biometric/demo/reset
rm -f data/diary.db

All hyperparameters live in configs/default.yaml. The notable sections:

Override at runtime with I3_* environment variables; see i3/config.py.

The corpus is assembled from six public dialogue sources:

Counts are read directly from data/processed/dialogue/triples.json (grouped by source field).

Build it locally:

poetry run python -m training.prepare_dialogue \
    --output data/processed/dialogue/triples.json \
    --max-pairs 977332

Train the encoder:

poetry run python -m training.train_encoder \
    --epochs 10 \
    --batch-size 64 \
    --device auto

Train the v2 SLM (overnight on RTX 4050 Laptop, ~18-22 hours):

poetry run python -m i3.slm.train_v2 \
    --config configs/default.yaml \
    --epochs 2 \
    --corpus-subset 300000 \
    --bf16 --grad-checkpoint --optim adamw8bit

Both trainers write checkpoints to checkpoints/ and emit a JSON metrics file alongside the best-loss weights.

Twelve core tabs + seven Huawei-pitch tabs (iter 51). Each does one thing well.

The 19-tab layout is documented in docs/screenshots/README.md. The smoke-test script at scripts/ui_smoke_test.py drives Playwright through every tab and captures a PNG; run it after any UI change to refresh the gallery.

These numbers are emitted by scripts/measure_edge.py / i3/edge/profiler.py on the same machine that serves the demo. They update on every benchmark run.

A 110-scenario audit, scored on adaptation fidelity, factual support, and refusal correctness:

implicit-interaction-intelligence/
├── i3/                        ← all the from-scratch ML
│   ├── slm/                   ← AdaptiveTransformerV2, MoE, ACT, BPE, retrieval, generate
│   ├── encoder/               ← TCN, ONNX export, quantisation
│   ├── interaction/           ← keystroke + linguistic features
│   ├── biometric/             ← keystroke continuous auth
│   ├── affect/                ← state classifier + shift detector
│   ├── adaptation/            ← 8-axis vector inference + post-processor
│   ├── personalisation/       ← LoRA adapters per biometric template
│   ├── multimodal/            ← prosody, gaze, accelerometer, fusion
│   ├── router/                ← LinUCB bandit, complexity estimator, sensitivity
│   ├── pipeline/              ← 14-stage engine + types + tracer
│   ├── critique/              ← self-refine critic
│   ├── dialogue/              ← multi-turn coref
│   ├── privacy/               ← sanitiser, budget, encryption
│   ├── diary/                 ← encrypted embedding store, summariser
│   ├── edge/                  ← profiler, runtime selection
│   ├── cloud/                 ← guarded LLM client (Anthropic / Google / Ollama)
│   └── … (eval, federated, fairness, observability, redteam, safety, …)
├── server/                    ← FastAPI app + WebSocket + per-feature routes
├── web/                       ← SPA (vanilla JS, 12 tabs, no framework)
├── training/                  ← dialogue prep, encoder train, SLM train
├── scripts/                   ← measure_edge, run_everything (orchestrator)
├── docs/
│   ├── TECHNICAL_REPORT.md    ← arXiv-style writeup
│   ├── INTERVIEW_DEMO.md      ← 5-minute live demo script
│   └── …
├── checkpoints/
│   ├── slm/                   ← v1 + v2 weights, BPE tokenizer
│   └── encoder/               ← TCN .pt + .onnx
├── reports/                   ← edge profile, verification, redteam, evaluation
├── configs/default.yaml       ← all hyperparameters (slm, slm_v2, training_v2, …)
├── tests/                     ← pytest suite
└── benchmarks/                ← latency, conditioning sensitivity

The current demo is a complete vertical slice. The roadmap below is where I would invest engineering judgement next, in priority order.

1. Federated personalisation. Today each device trains its own LoRA adapter against its biometric template. The natural next step is FedAvg on the LoRA deltas (not the base weights) so a fleet improves together without raw data ever leaving devices. Skeleton exists in i3/federated/; needs a server-side aggregator and a key-rotation policy.
2. Real gaze training data. The gaze classifier today is a frozen MobileNetV3 backbone with a tiny calibration head trained on 4 target points per session. Replacing the calibration with a pre-trained gaze regressor (MPIIGaze / Gaze360) would lift the signal from "rough heatmap quadrant" to actual fixation vectors.
3. Bigger SLM via Colab Pro / A100. The 204 M shape is what trains in 18-22 h on the user's RTX 4050 Laptop. The architecturally-ideal shape is d_model=960, n_layers=16, d_ff=3840, n_experts=2, seq=1024 (~400 M); this trains in 2-3 days on a single A100 and likely closes the perplexity gap with foundation small-models. The architecture itself doesn't change — only the shape and the corpus epoch count.
4. Speculative decoding with the v1 model as draft. The v1 SLM (53 M, 110 MB int8) is the perfect draft model for v2 (204 M). Skeleton in i3/slm/speculative_decoding.py; needs a verification kernel and acceptance-rate telemetry. Should roughly halve p50 generation latency.
5. Wearable runtime port. ONNX export already runs in browser; the next port is to a wearable-class runtime (Kirin A2 / RK3588 / ARM-NEON int8). The TCN encoder is already small enough; the SLM needs distillation to ~10-20 M for 50 MB int8 — approachable via ACT-aware width pruning + LoRA-targeted distillation.
6. Dataset collection at scale. The corpus is six public dialogue sources stitched together; the long-term improvement is real keystroke data with affect labels. The capture path exists (web/js/voice_prosody.js, the biometric template), but the labelling pipeline hasn't been built.

Each of these has a clean entry point in the codebase. None of them require rewriting the core architecture, which is the entire point.

The repository ships with three classes of automated checks:

• Unit tests — pytest over the tests/ tree, covering feature extractors, tokenizer round-trips, attention shapes, bandit invariants, sanitiser regex, biometric verification.
• Verification harness — a 44-check acceptance suite that exercises every API route, every WebSocket frame type, and the full pipeline end-to-end. Reports land in reports/verification/.
• Red-team probes — i3/redteam/ runs a battery of jailbreak, PII-extraction, and adversarial-style prompts against both the local SLM and the cloud route. Reports land in reports/redteam/.

Run everything:

make test          # unit tests
make verify        # acceptance harness
make redteam       # adversarial probes
make all-quality   # lint + typecheck + bandit + pip-audit + tests

Or via the orchestrator (parallel-within-wave):

poetry run python scripts/run_everything.py --only test verify redteam

Every turn produces a structured trace consumable by:

• Flow tab — animated 14-stage pipeline with real timings.
• /api/flow/last-trace — JSON dump of the most recent turn's pipeline trace (stage timings, data shapes, decisions made).
• /api/explain/last/{user_id} — the "Why this response?" narration, generated by i3/explain/ from the trace plus the diary.
• MLflow — training runs log to mlruns/ for offline review.

All telemetry is local-only; no external observability service is called.

A WSL2 + Triton setup guide is in docs/operations/wsl2-setup.md (commit 5e0e1e2).

The full annotated bibliography is in the References tab of the demo UI and at docs/TECHNICAL_REPORT.md. The core papers:

• Sennrich, Haddow & Birch (2015) — Neural Machine Translation of Rare Words with Subword Units — BPE.
• Vaswani et al. (2017) — Attention Is All You Need — transformer.
• Shazeer et al. (2017) — Outrageously Large Neural Networks: Sparsely-Gated MoE.
• Graves (2016) — Adaptive Computation Time for Recurrent Neural Networks.
• Hu et al. (2021) — LoRA: Low-Rank Adaptation of Large Language Models.
• Houlsby et al. (2019) — Parameter-Efficient Transfer Learning for NLP — adapter modules.
• Killourhy & Maxion (2009) — Comparing Anomaly-Detection Algorithms for Keystroke Dynamics.
• Monrose & Rubin (1997) — Authentication via keystroke dynamics.
• Schuller (2009) — INTERSPEECH Computational Paralinguistics Challenge.
• Eyben et al. (2010) — openSMILE: The Munich versatile audio feature extractor.
• Howard et al. (2019) — Searching for MobileNetV3.
• Bai et al. (2018) — An Empirical Evaluation of Generic Convolutional and Recurrent Networks — TCN.
• Robertson (1994) — Some simple effective approximations to the 2-Poisson model — BM25.
• Picard (1997) — Affective Computing.
• Sweller (1988) — Cognitive Load During Problem Solving.
• Madaan et al. (2023) — Self-Refine: Iterative Refinement with Self-Feedback.
• Li et al. (2010) — A Contextual-Bandit Approach to Personalized News Article Recommendation — LinUCB.

The recruiter at Huawei R&D UK sent four follow-up questions after my CV. Each is a filter for a specific competency. Here is exactly where each is answered in this codebase.

✅ Every algorithm in the inference path is hand-implemented. See the from-scratch components table above for the 22-row inventory. Concrete starting points:

• i3/slm/adaptive_transformer_v2.py — 204 M-param decoder
• i3/slm/bpe_tokenizer.py — byte-level BPE in 217 LOC
• i3/encoder/tcn.py — TCN encoder
• i3/router/bandit.py — LinUCB
• i3/biometric/keystroke_auth.py — typing biometrics
• i3/personalisation/lora_adapter.py — LoRA

✅ Yes — AdaptiveTransformerV2 is a pure-PyTorch nn.Module, zero HuggingFace dependencies in the inference path. Q/K/V projections, scaled-dot-product attention with causal mask, MoE-FFN with softmax gating, ACT halting controller, and per-layer cross-attention all written from the original papers. Tokenizer is 217 LOC of hand-written byte-level BPE. Training loop uses bnb.optim.AdamW8bit for memory but not for any modelling logic. Files: i3/slm/ (entire directory).

✅ Yes — two orchestrators ship in this repo. The runtime orchestrator is the 14-stage closed-loop pipeline at i3/pipeline/engine.py, animated live in the Flow tab. The build orchestrator is scripts/run_everything.py — a 21-stage wave-scheduled DAG with parallel-within-wave execution, artefact dependency tracking, --resume, and a Rich live UI. End-to-end run takes ~10 minutes on an RTX 4050 Laptop.

✅ Yes — the demo is edge-first. TCN encoder exports to ONNX (0.4 MB); a browser-side inference path runs it via onnxruntime-web with WebGPU

• WASM backend detection (web/js/browser_inference.js). Real edge profile — int8 size, p50/p95 latency, peak RSS — measured by scripts/measure_edge.py and surfaced live in the Edge tab. Server enforces wearable-scale caps: per-message byte cap, per-session message cap, sliding-window rate limit, bounded keystroke buffer.

Built on a single 6 GB laptop GPU. Designed to be quantised to 8-bit and shipped to a phone.