How do I access a llama.cpp server instance with the Continue extension for VSCodium?

How to Access a llama.cpp Server Instance with the Continue Extension in VSCodium

Running a local llama.cpp inference server is a great way to get high‑performance LLM responses without relying on external APIs. In this guide we’ll walk through the entire process of:

1. Setting up the llama.cpp server.
2. Installing the Continue extension in VSCodium.
3. Configuring Continue to communicate with the running server.

Prerequisites

• VSCodium installed (https://vscodium.com/)
• Git and a C/C++ compiler (GCC, Clang, or MSVC) – required to build llama.cpp
• Python 3.9+ (optional, for easier script handling)
• A compatible model file (e.g., ggml‑model‑q4_0.bin)

1. Build and Run the llama.cpp Server

Step 1 – Clone the repository

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

Step 2 – Compile the server

# Linux / macOS
make server

# Windows (PowerShell)
.\build.ps1 -server

Step 3 – Start the HTTP server

./server -m models/ggml-model-q4_0.bin -p 8080 --host 127.0.0.1

This launches a REST API listening on http://127.0.0.1:8080/v1/completions. Keep the terminal window open – the server must stay alive while you use Continue.

2. Install the Continue Extension in VSCodium

1. Open VSCodium.
2. Press Ctrl+Shift+X to open the Extensions view.
3. Search for Continue and click Install.
4. Reload VSCodium when prompted.

3. Configure Continue to Use the Local Server

3.1 Open Continue Settings

Press Ctrl+Shift+P, type Continue: Open Settings, and select it. This opens a JSON configuration file.

3.2 Add a Custom Provider

Insert the following JSON object under the "providers" array (or create the array if it doesn’t exist):

{
  "id": "local-llama",
  "name": "Local LLaMA (llama.cpp)",
  "type": "custom",
  "model": "llama.cpp",
  "apiBase": "http://127.0.0.1:8080/v1",
  "apiKey": "none",
  "maxTokens": 512,
  "temperature": 0.7,
  "topP": 0.9,
  "requestOptions": {
    "headers": {
      "Content-Type": "application/json"
    }
  }
}

Explanation of key fields:

• apiBase: Base URL of the llama.cpp server (no trailing slash).
• apiKey: Not required for the local server; set to "none".
• maxTokens, temperature, topP: Adjust to match your model’s capabilities.

3.3 Set the Default Provider (Optional)

If you want Continue to use this provider by default, add:

"defaultProvider": "local-llama"

4. Verify the Connection

1. Open any code file in VSCodium.

2. Press Ctrl+Shift+P and run Continue: Ask Assistant (or use the inline chat panel).

3. Type a prompt such as “Explain the difference between map and filter in JavaScript.”

If everything is configured correctly, the response will come from your local llama.cpp server.

5. Common Troubleshooting Tips

• Server not reachable: Ensure the server is running and that the port (default 8080) isn’t blocked by a firewall.
• Invalid JSON response: Verify you’re using a model that supports the /v1/completions endpoint (most recent llama.cpp builds do).
• High latency: Reduce maxTokens or switch to a lower‑precision model (e.g., q4_0 → q8_0).
• Continue shows “No API key provided”: The extension expects a string; using "none" satisfies the check.

6. Going Further

Once you have the basic setup working, you can explore additional tweaks:

• Enable --host 0.0.0.0 to access the server from other devices on the LAN.
• Use --n-gpu-layers (if you have a compatible GPU) for faster inference.
• Integrate other VSCodium extensions (e.g., ChatGPT Sidebar) by pointing them to the same apiBase.

Conclusion

By following these steps, you’ve connected VSCodium’s Continue extension to a locally running llama.cpp server. This setup gives you private, low‑latency AI assistance directly inside your editor, without any external API calls. Happy coding!