Tabby: Self-hosted AI Coding Assistant

This post shows how to use SkyPilot to host an ai coding assistant with just one CLI command.

Background

Tabby is a self-hosted AI coding assistant, offering an open-source and on-premises alternative to GitHub Copilot. It boasts several key features:

• Self-contained, with no need for a DBMS or cloud service.

• OpenAPI interface, easy to integrate with existing infrastructure (e.g Cloud IDE).

• Supports consumer-grade GPUs.

SkyPilot is an open-source framework from UC Berkeley for seamlessly running machine learning on any cloud. With a simple CLI, users can easily launch many clusters and jobs, while substantially lowering their cloud bills. Currently, AWS, GCP, Azure, Lambda Cloud, IBM, Oracle Cloud Infrastructure (OCI), Cloudflare R2 and Samsung Cloud Platform (SCP) are supported. See docs to learn more.

Steps

All YAML files used below live in the SkyPilot repo.

1. Install SkyPilot and check that cloud credentials exist:

   # pip install skypilot
   pip install "skypilot[aws,gcp,azure,lambda]"  # pick your clouds
   sky check
   

2. Get the Tabby SkyPilot config:

   git clone https://github.com/skypilot-org/skypilot.git
   cd skypilot/llm/tabby
   

3. launch a Tabby cluster: You can select the model to be used by --model in the command field of the docker-compose.cuda.yaml(GPU) or docker-compose.yaml(CPU). TabbyML/SantaCoder-1B is used by default, for more model options, see the tabby docs to learn more.

    sky launch -c tabby tabby.yaml
   

4. After seeing tabby server is ready, enjoy!, you can use the following command to obtain the public IP:

   sky status --ip tabby
   20.92.236.53
   

   You can also use the ssh command directly to map the port to localhost.

   ssh -L 8080:localhost:8080 tabby
   

5. Install the Tabby extension in your IDE. For example, in VS Code, you can install the Tabby extension from the marketplace.

6. Configure your IDE to use $(sky status --ip tabby):8080 as the Tabby endpoint.

7. Open a Python file and start coding! Tabby will start suggesting code snippets as you type.

Cleaning up

When you are done, you can stop or tear down the cluster:

• To stop the cluster, run

  sky stop tabby  # or pass your custom name if you used "-c <other name>"
  

  You can restart a stopped cluster and relaunch the tabby (the run section in YAML) with

  sky launch tabby.yaml -c tabby --no-setup
  

  Note the --no-setup flag: a stopped cluster preserves its disk contents so we can skip redoing the setup.

• To tear down the cluster (non-restartable), run

  sky down tabby  # or pass your custom name if you used "-c <other name>"
  

To see your clusters, run sky status, which is a single pane of glass for all your clusters across regions/clouds.

To learn more about various SkyPilot commands, see Quickstart.