What a self-hosted LLM setup is made of
Most setups have three layers. A model runner loads the model file and answers requests. A chat interface gives people a familiar place to type. Optional apps on top add document search, research or automation. Nearly every runner here offers an OpenAI-style API, which means a program written for OpenAI’s service can talk to it with little or no change, so you can mix and match the layers.
Hardware sets the limits. Models are usually quantised, which means their numbers are stored at lower precision so the file is smaller and needs less memory. A quantised model with 7 or 8 billion parameters will usually run on a recent laptop with 16 GB of memory, though it answers slowly without a graphics card (GPU). Larger models need a GPU with plenty of video memory, or a server with several GPUs.
Model runners that load and serve the model
Ollama is the simplest way to run an LLM locally. One command downloads an open model and starts it on macOS, Linux or Windows, and most apps in this kit can connect to it out of the box. For most people it is the first piece of a self-hosted LLM and the right place to start.
llama.cpp is the C/C++ engine that sits underneath many local AI apps, and it runs quantised models on ordinary processors as well as GPUs. Use it directly when you want the most control over settings or need to run on unusual hardware. Expect to work at the command line and to choose the right build for your machine.
vLLM is an inference server built to answer many requests at once on GPUs. It is more than you need for one person on a laptop, but it is the right choice when a whole team or a busy app shares one GPU server.
SGLang is another serving engine for text and multimodal models on GPUs, under Apache-2.0. Like vLLM, it is built to run open models behind an API for many users, so if you are setting up a shared GPU server for a self-hosted LLM, test both with your own model and traffic before you choose.
LocalAI acts as a local stand-in for the OpenAI API and serves text, image and audio models, with or without a GPU. Choose it when existing software expects OpenAI and you only want to change the address it calls.
llamafile from Mozilla AI packs a model and the llama.cpp engine into one executable file. You download the file and run it, with no installation, which makes it a handy way to share a model with someone who has never set up local AI. On Windows, files over 4 GB will not run, so larger models need their weights kept in a separate file. Its licence is listed as “Other”.
Jan is a desktop app for chatting with models fully offline, with the runner and the chat window in one install. It is the easiest option for people who never want to open a terminal. Its licence is listed as “Other”, so read the terms before you redistribute it.
GPT4All from Nomic is a desktop chat app that runs open models on ordinary laptops without a GPU, on Windows, macOS and Linux. Its LocalDocs feature lets you chat with files on your own computer. It is MIT licensed. Its last code update in our data was in May 2025, so check its recent activity before you rely on it.
Chat interfaces your team can log in to
Open WebUI is an Ollama web UI that also works with any OpenAI-compatible server. It gives you a ChatGPT-style interface with document upload, tools and user accounts, and it runs in Docker. Its licence is listed as “Other”, so check the terms if you plan to rebrand or resell it.
LibreChat is a self-hosted chat interface that puts local models and many cloud providers side by side, with agents, file search and multi-user login. Pick it when your team wants local models for private work and cloud models for harder questions in the same place.
LobeHub is a chat workspace with agents, plugins, a knowledge base and team sharing. It has more features than Open WebUI and takes a little more setup, so it suits teams who want a shared workspace rather than a plain chat window.
Apps that answer questions from your documents
These apps use retrieval, often shortened to RAG (retrieval-augmented generation). They search your files for relevant passages and pass them to the model along with your question.
AnythingLLM runs as a desktop app or in Docker and lets you chat with your documents, with a no-code agent builder on top. It is the quickest way to get document Q&A working on a single machine.
DocsGPT answers questions from your documentation and help content, and you can embed it on a website as a chat widget. It suits support and product teams who want customers to search their docs in plain language.
PrivateGPT is an API layer for building private AI apps on local models, with document Q&A, tools and text-to-SQL, which turns plain questions into database queries. It can connect to any of the runners above that offer an OpenAI-style API. It is Apache-2.0 licensed and suits developers who want to build their own document app rather than use a finished one.
Khoj is a personal assistant that answers from your own documents and the web and can run research on a schedule. It works with local or online models. It uses the AGPL-3.0 licence.
SurfSense is a research assistant that searches your notes, Slack, Notion and the web in one place, in the style of Google’s NotebookLM. It suits people whose knowledge is spread across several tools.
Open Notebook is another NotebookLM-style app that also turns your sources into podcast-style audio summaries. Pick it if you like listening to research as well as reading it.
Research tools and connectors for a complete self-hosted LLM stack
Local Deep Research searches the web and academic sources, then writes a report with citations, and it works with local models. It is a private alternative to the deep-research features in cloud chat apps.
LiteLLM is a gateway that puts many model providers behind one OpenAI-style API, with spend tracking and usage limits. It earns its place when you mix local models with paid cloud ones and want one address and one bill to watch.
The n8n self-hosted AI starter kit is a Docker Compose bundle that starts n8n, a workflow automation tool, together with Ollama, the Qdrant vector database and Postgres. It is the fastest way to try local AI workflows, such as summarising new files as they arrive.
How to choose your self-hosted LLM pieces
Every local LLM GitHub project in this kit is free to download, and you can swap pieces later because most of them speak the same API. These pairings cover the common cases.
- One person on a laptop: install Jan or GPT4All, or run Ollama with Open WebUI in front of it.
- You want to hand someone a model that just runs: send them a llamafile.
- A small team on one server: run Ollama, or vLLM or SGLang if you have a GPU and many users, with Open WebUI or LibreChat as the shared interface.
- You mix local and cloud models: put LiteLLM in front of everything so apps only need one address.
- You want answers from your own files: start with AnythingLLM, and use DocsGPT when the answers need to appear on a website.
- You want research reports: try Local Deep Research or Khoj.
There is no single best open source local LLM to recommend, because the right size depends on your memory and the tasks you give it. Start with a small model in Ollama, test it on real questions, and step up in size only when the answers fall short. Running a self-hosted LLM also makes you responsible for updates and access control, so keep these local AI tools patched and put a login in front of anything reachable from the internet.