Open-weight local AI workflow

Local AI Coding Assistant

A practical setup for running Swift-Qwen locally, serving it through an OpenAI-compatible API, and using it for safer repo reading, issue triage, coding help, and code review.

Quick start Safety rules Model

What This Project Shows

Local model serving Runs a quantized Hugging Face model through llama.cpp.
OpenAI-compatible API Exposes a local endpoint that coding tools can call.
Repo-aware workflow Connects the model to tools like aider for real code tasks.
Safety-first setup Starts with read-only file tools before shell commands or edits.

Quick Start

Install the core tools, log in to Hugging Face, and start the model server from a demo repo. 

brew update
brew install llama.cpp jq python@3.12 pipx tmux
pipx install huggingface_hub
pipx install --python /opt/homebrew/bin/python3.12 aider-chat
hf auth login

Read-only repo server

cd /path/to/demo-repo

tmux new-session -d -s local-ai-code-assistant \
  'llama-server -hf ukisai/Swift-Qwen3.8-27B-GGUF:Q4_K_M \
    --host 127.0.0.1 \
    --port 8000 \
    --jinja \
    -fa on \
    -ngl 99 \
    -c 32768 \
    --tools read_file,file_glob_search,grep_search,get_info \
    2>&1 | tee /tmp/local-ai-code-assistant.log'

Smoke test

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"ukisai/Swift-Qwen3.8-27B-GGUF:Q4_K_M","messages":[{"role":"user","content":"Reply with exactly LOCAL_MODEL_READY."}],"max_tokens":80}' \
  | jq -r '.choices[0].message.content'

Hardware Notes

Start with Q4_K_M on an Apple Silicon Mac with at least 32 GB of memory. Keep at least 30 GB of free disk space.

Quantization Approx size Best use
Q4_K_M 18 GB First local test and normal repo work
Q5_K_M 20 GB Better quality with more memory use
Q6_K 23 GB Longer coding runs and stricter output

Use It With Aider

cd /path/to/demo-repo
git worktree add -b local-ai-test ../demo-repo-local-ai-test HEAD
cd ../demo-repo-local-ai-test

export OPENAI_API_BASE="http://127.0.0.1:8000/v1"
export OPENAI_API_KEY="none"

aider --model openai/ukisai/Swift-Qwen3.8-27B-GGUF:Q4_K_M --no-auto-commits

A good first prompt is to ask the model to read repo instructions, explain a plan, and wait before editing.

Safety Rules

  1. Use demo repos and fake issue text for public work.
  2. Start with read-only tools.
  3. Use a git worktree for code changes.
  4. Keep private data, secrets, logs, and customer data out of prompts.
  5. Review every diff yourself.
  6. Run tests before committing.
Full agent mode is powerful --agent can enable shell commands, file writes, and edits. Use it only in a trusted local setup.

Portfolio Angle

This project is meant to show useful AI engineering judgment: running models, connecting tools, setting safety limits, and testing the result. A good public writeup should include what worked, what failed, and what you would improve next. 

Built a local AI coding workflow using an open-weight Hugging Face model,
llama.cpp, an OpenAI-compatible API, and a repo-aware coding assistant.
Added read-only file access first, verified the setup with smoke tests,
and documented the safety tradeoffs.

Sources