How do I access a llama.cpp server instance with the Continue extension for VSCodium?

How to Access a llama.cpp Server Instance with the Continue Extension in VSCodium

If you’re working with large language models locally, llama.cpp is a lightweight, high‑performance implementation that lets you run GGML‑converted local models on your own hardware. Pairing it with the Continue extension in VSCodium gives you an integrated, AI‑powered coding assistant without leaving the editor.

Prerequisites

• VSCodium installed (latest stable version).
• Continue extension installed from the VS Code Marketplace (or its open‑source fork if you prefer).
• A compiled llama.cpp server binary (server or llama-server) built for your platform.
• A GGML model file (e.g., ggml-model-q4_0.bin) ready to be served.
• Python 3.9+ (optional, for script‑based launch).

Step 1: Start the llama.cpp Server

Open a terminal and navigate to the directory containing the compiled server binary. A typical launch command looks like this:

./server -m ./models/ggml-model-q4_0.bin -c 2048 -ngl 32 --host 127.0.0.1 --port 8080

Explanation of the flags:

• -m – Path to the GGML model file.
• -c – Context length (adjust based on RAM/GPU).
• -ngl – Number of GPU layers (set to 0 for CPU‑only).
• --host and --port – Define where the HTTP API will listen.

When the server starts you should see a line similar to:

INFO: Server listening on http://127.0.0.1:8080

Step 2: Configure Continue to Use the Local Server

1. Open VSCodium and go to Extensions → Installed → Continue.
2. Click the gear icon → Extension Settings.
3. Find the continue.customModels section (you may need to enable Advanced Settings).
4. Add a new entry with the following JSON:

{
  "name": "Local LLaMA",
  "type": "llama.cpp",
  "host": "http://127.0.0.1",
  "port": 8080,
  "model": "ggml-model-q4_0.bin",
  "maxTokens": 1024,
  "temperature": 0.7
}

Save the settings. Continue will now list “Local LLaMA” as one of the available models when you invoke the assistant.

Step 3: Test the Connection

Open any code file, press Ctrl+Shift+P, and run Continue: Open Chat. In the chat sidebar, select the newly added “Local LLaMA” model. Type a simple prompt such as:

Explain the difference between let and const in JavaScript.

If everything is set up correctly, the response should appear within a few seconds, confirming that VSCodium is successfully communicating with the llama.cpp server.

Common Troubleshooting Tips

• Server not reachable: Verify the host/port match exactly. Use curl http://127.0.0.1:8080/v1/models to check the API endpoint.
• Memory errors: Reduce -c (context size) or switch to a smaller quantized model (e.g., q5_0).
• GPU not used: Ensure you compiled llama.cpp with CUDA or Metal support and set -ngl to the number of layers you want to offload.
• Continue still shows “No model found”: Reload the VSCodium window (Ctrl+Shift+P → Developer: Reload Window) after saving the settings.

Optional: Automate Server Startup

To avoid manually launching the server each time, add a simple script to your package.json or create a systemd service (Linux) or a launch agent (macOS). Example npm script:

"scripts": {
  "llama:start": "cd ~/llama.cpp && ./server -m ./models/ggml-model-q4_0.bin -c 2048 -ngl 32 --host 127.0.0.1 --port 8080"
}

Now you can run npm run llama:start in a terminal and keep the server alive while you code.

Conclusion

By running a local llama.cpp server and linking it with the Continue extension, you get a fast, privacy‑preserving AI assistant right inside VSCodium. This setup works offline, scales with your hardware, and can be customized (different models, temperature, token limits) without leaving the editor. Happy coding!