An AI assistant for your own website: chat and voice, self-hosted

It started with a telephone. For a small business I had built an AI receptionist that answers calls, answers questions and suggests appointments. Speech recognition (Parakeet), the voice (a cloned Qwen3-TTS model) and the brain (Claude through Claude Code) all run on my Mac, not in a cloud speech service. At some point the obvious question came up: why not use the same technique for myself, on this website?

The idea

I did not want an intrusive chat pop-up. I wanted something discreet: a small round icon at the bottom right of every page, in the language of the page (DE, EN, IT). One click opens a window with an animated portrait of me that breathes, blinks and moves its mouth when it speaks. The assistant answers questions about me and the content of the site, and if you like you can call it right in the browser. It says from the start that it is an AI, and it does not make things up: what is not on the site it does not know, and then it points to the contact form.

How it works

Browser (widget)
   │  wss://daniele-michelin.com/assistent/ws
   ▼
nginx on the server  ──►  my Mac at home
                           ├─ speech recognition + voice (local)
                           └─ brain (Claude through Claude Code)
   ▲
   └── settings: the Mac pulls them from the website every minute (token)

The browser never talks to my Mac directly. It talks to my own domain and nginx forwards the connection:

location = /assistent/ws {
    proxy_pass http://MY-MAC:5070/ospite/ws$is_args$args;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

That keeps the page's content security policy strict (connect-src 'self') and no visitor ever sees my home address.

All settings live in a protected admin page on the website: widget, tone, knowledge, voice, limits, appointments, mail. The Mac pulls them every minute with a token. I did not want a channel through which the website can change something on my computer; the Mac asks, never the other way round.

When the Mac is off

A Mac at home is not a data centre. If it is off or busy, the widget switches to plain text chat that the website answers by itself. There is deliberately no AI model and no third party behind it: the answers are written from the public content of the site and chosen by keywords. That is less flexible, but honest and nothing leaves the server. I add my own question-and-answer pairs in the admin.

Appointments and messages

The assistant can arrange a conversation with me. Free times come from my opening hours, minus appointments already booked and the entries in my Nextcloud calendar (via CalDAV; read-only for “busy”, and booked appointments are written there). There is also an ICS feed to subscribe to and confirmation e-mails. Messages for me work exactly like the contact form: same recipient, same mail design, same sending.

This is where I found a bug that kept me thinking. In a test the assistant said “appointment confirmed” although nothing had been booked: the model had not triggered the action and still found the right words. Since then the system decides, not the model. A booking waits in a pending state, the system speaks the summary itself, and only after an explicit “yes” is anything booked. The system also reports the outcome: “done”, “that slot has just been taken” or an error. Never leave it to a language model to say whether an action worked.

Small things that make a difference

If nobody says anything for 30 seconds the assistant hangs up and frees the line. Chat keeps working: the next question reconnects automatically, without greeting again and with the conversation so far as context.
Only one person at a time talks to the language model; others get a friendly “back in a moment” or the text mode.
For the voice, only people who agreed are cloned. The assistant now speaks with my own cloned voice (a 16-second German recording that also works for Italian and English), and I recorded it myself.

Privacy

The content of conversations is not stored; the logs hold at most the time and the length of a message. Audio never leaves my Mac. The recognised text goes to the language model (Claude) today, which is the price for the quality. A visitor's IP address lives only in memory, for a daily limit. Appointment data (name, e-mail, topic, time) sits in a non-public file and is deleted automatically 90 days after the appointment. On the privacy page the section about the assistant only appears when the feature is switched on, and the paragraphs on appointments and messages only when those are active.

What went wrong

Microphone works, no sound. I had created the audio context only after a network request. Safari, and sometimes Chrome, leave a context created that way “suspended”. The microphone worked, the assistant understood me, but I heard nothing. It has to be created synchronously inside the click handler.
Ports 5060 and 5061. Those are SIP ports, and fetch and browsers refuse them with “bad port”. I had picked them only because they were free.
Cache. My main.js is cached immutably for a year. After the change I was the only one who could not see the assistant. Now the widget's version comes from a small config file that is read fresh on every page load.
Licence. The voice system I started from is a fork of a project without a licence. So I may not publish it, and my fork stays private.

The code is open

For that reason the public repository contains only my own code: the widget, the PHP endpoints and the admin page, the nginx configuration, the engine logic (guest limits, the system-owned booking flow) and a small text-only reference engine with a pluggable brain that is a local Ollama by default. The voice pipeline itself (recognition and cloned voice) is not included; the protocol it speaks is documented, so any speech stack can be attached. It is under the MIT licence, with English documentation, screenshots, an architecture diagram and a one-command demo: github.com/Mikweb2025-design/ai-site-assistant.

What I still want to reach: a local brain

Today the recognised text goes to Claude. My goal is that nothing leaves the Mac any more: a language model that runs only locally. Honest status: in my earlier tests small local models were too slow and too weak for a conversation at phone speed. Now I want to try newer small models on Apple Silicon (MLX) and make the brain swappable behind an OpenAI-compatible interface. That is why an Ollama / OpenAI-compatible option is on the repository's roadmap. If I find a model that answers fast enough and sticks to the facts, I will write about it.

I already ran a first round with the reference engine and the real text of this site as knowledge. A 2-billion-parameter model answered in 2 to 6 seconds once warm, got a German question right and correctly refused an off-topic one. The 4B and 12B models I had installed took more than 30 seconds per answer with the long prompt, far too slow. So the direction is clear: smaller prompts (only the relevant chunk of knowledge instead of the whole page), streaming sentence by sentence, and a tiny classifier in front for greetings. The numbers and the script to repeat them on your own machine are in the repository (docs/roadmap.md).