Hennessy Ng

Built a full-stack piano practice app that turns PDF sheet music into an interactive “falling notes” waterfall, so learners can play along without needing to read complex scores first.

Shipped end-to-end

• PDF upload → AI/rule-based music recognition
• Synced sheet music + keyboard UI
• MIDI piano input
• Hand-separate practice
• Wait-for-correct-note mode

Stack

Production-style multi-service deployment: React/TypeScript + WebGL frontend, Python/FastAPI backend, async workers, PostgreSQL/object storage, and Docker across services.

Converts URLs, HTML, PDFs and images into structured Obsidian markdown, with AI-powered wiki generation and semantic embeddings on top.

Why

Clipping tools save pages. They do not turn a pile of clippings into something you can reason across. senji treats ingestion as the first step of a knowledge base, not the last.

How it works

• Ingestion — a single entry point accepting URLs, raw HTML, PDFs and images; scanned input falls through to OCR.
• Normalisation — everything lands as structured markdown with consistent front matter, so the vault stays queryable rather than becoming a dumping ground.
• Wiki generation — an LLM pass links related notes and drafts connective summaries, turning isolated clips into a navigable wiki.
• Semantic search — embeddings over the corpus so retrieval works on meaning, not filename.

Self-hosted by design: the corpus is personal, so it stays on infrastructure I control.

Clients delivered scanned, non-selectable Japanese PDFs and expected editable English DOCX back. The pipeline handles ingestion, OCR, translation and delivery, and feeds its own validated output back into MT training.

Pipeline

1. Scan intake — watched path picks up incoming PDFs and queues a job with language direction and the relevant MT engine plus linguistic resources (TM/TB).
2. Rasterisation — PDF2Image at 300 DPI, JPEG output. Tuned for a balance between OCR accuracy and iteration speed; JPEG keeps intermediate storage manageable across large batches.
3. OCR — GPU-enabled EasyOCR with a Japanese model, exposed behind a REST API.
4. Machine translation — GCP AutoML Translation and Azure Translator, routed per job.
5. Delivery — reassembled as DOCX for human validation and post-editing (MTPE).
6. Feedback loop — validated post-edited output (roughly 40% of volume) trains a dedicated MT engine, improving quality recursively.

Engineering notes

OCR accuracy on Japanese scans. EasyOCR outperformed Tesseract on this corpus. Line breaking and bounding box detection needed tuning for A3-sized documents at 2,560px. Contrast adjustment mattered most on fax-quality scans, where the baseline model degraded badly.

Fine-tuning path. Generating synthetic training data with trdg (image/text pairs) was the route to a domain-tuned Japanese model.

Open issue — worker scaling. Increasing workers beyond five did not improve throughput. Suspected CUDA driver / cuDNN configuration rather than an application-level bottleneck; GPU utilisation did not scale with worker count as expected.

MT quality estimation. Scoring machine output before it reached a human meant post-editors spent time on segments that needed it rather than reading everything.

Every completed project leaves behind parallel corpora. Extracting the terminology out of that data turns a delivery byproduct into a reusable asset — termbases that improve MT training and keep linguists consistent with corporate language.

Approach

1. Segmentation — Jieba handles Chinese word segmentation as pre-processing, since Chinese has no whitespace token boundaries.
2. Keyword extraction — KeyBERT over BERT embeddings, using maximal marginal relevance to keep candidate terms diverse rather than clustered around near-duplicates. This is the core of the project.
3. Cross-lingual alignment — extract terms in the source language (English), translate the candidates with EasyNMT, then match against the target-language corpus by semantic similarity rather than string matching.
4. Similarity scoring — Sentence-BERT for sentence-level similarity; Manhattan LSTM evaluated as an alternative during testing.

Why embeddings rather than statistics

Frequency-based extraction surfaces common words, not terminology. Embedding-based candidate selection with MMR surfaces terms that are distinctive to the document, which is what a termbase actually needs.

MT engines need parallel data. A lot of it exists as PDFs scattered across the public web, in inconsistent formats and mixed languages. This pipeline acquires it, cleans it, and gets it into parallel form fit for training.

Pipeline

Acquisition. Scrapy and requests with regex-driven target discovery. Downloads are deliberately rate-limited — crawling fast enough to matter is also fast enough to get blocked.

Selectability check. fitz from PyMuPDF determines whether a PDF has an extractable text layer. If not, it drops through to OCR.

OCR quality gate. Spell checking with pyenchant measures the misspelled-word rate. High rates flag OCR output that is too noisy to use as training data — cheap signal, catches most garbage before it contaminates the corpus.

Language detection. langdetect and spaCy classify documents. ASCII ratio and PDF-level language flags act as supporting signals where detection is uncertain.

Known limitation

Detection is per-document, so genuinely multilingual PDFs (English and French in one file) are mislabelled. The fix is word-level alignment — awesome-align over mBERT — to segment mixed-language documents before classification.

Language technology grew up as desktop software. Translation memories lived on local disks, terminology in files, MT in installed engines. That model does not survive a distributed team or integrate with anything.

What the migration involved

Rewriting for the API model. Tooling that assumed local file access had to be re-architected around RESTful services with proper request/response boundaries.

Vendor integration. Bulk connections to cloud MT vendors, with routing by language pair and content type.

CMS connectors. Sync with content platforms including Drupal, so localization was a step in the content workflow rather than an export-and-email round trip.

Centralised linguistic data. Translation memories and termbases moved into maintained databases with API access, plus backup and maintenance procedures — the resources are the compounding asset, so losing them is unacceptable.

Multi-cloud evaluation. Surveying language technology available on GCP, Azure and AWS to pick the right service per workload rather than defaulting to a single vendor.