Added AI server setup : to be reviewed
This commit is contained in:
@@ -0,0 +1,137 @@
|
||||
# **Port Forwarding Magic: Set Up Bolt.New with Remote Ollama Server and Qwen2.5-Coder:32B**
|
||||
|
||||
This guide demonstrates how to use **port forwarding** to connect your local **Bolt.New** setup to a **remote Ollama server**, solving issues with apps that don’t allow full customization. We’ll use the open-source [Bolt.New repository](https://github.com/coleam00/bolt.new-any-llm) as our example, and we’ll even show you how to extend the context length for the popular **Qwen2.5-Coder:32B model**.
|
||||
|
||||
If you encounter installation issues, submit an [issue](https://github.com/coleam00/bolt.new-any-llm/issues) or contribute by forking and improving this guide.
|
||||
|
||||
---
|
||||
|
||||
## **What You'll Learn**
|
||||
- Clone and configure **Bolt.New** for your local development.
|
||||
- Use **SSH tunneling** to seamlessly forward traffic to a remote server.
|
||||
- Extend the context length of AI models for enhanced capabilities.
|
||||
- Run **Bolt.New** locally.
|
||||
|
||||
---
|
||||
|
||||
## **Prerequisites**
|
||||
|
||||
Download and install Node.js from [https://nodejs.org/en/download/](https://nodejs.org/en/download/).
|
||||
|
||||
---
|
||||
|
||||
## **Step 1: Clone the Repository**
|
||||
|
||||
1. Open Terminal.
|
||||
2. Clone the repository:
|
||||
```bash
|
||||
git clone https://github.com/coleam00/bolt.new-any-llm.git
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **Step 2: Stop Local Ollama Service**
|
||||
|
||||
If Ollama is already running on your machine, stop it to avoid conflicts with the remote server.
|
||||
|
||||
- **Stop the service**:
|
||||
```bash
|
||||
sudo systemctl stop ollama.service
|
||||
```
|
||||
- **OPTIONAL: Disable it from restarting**:
|
||||
```bash
|
||||
sudo systemctl disable ollama.service
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **Step 3: Forward Local Traffic to the Remote Ollama Server**
|
||||
|
||||
To forward all traffic from `localhost:11434` to your remote Ollama server (`ai.mtcl.lan:11434`), set up SSH tunneling:
|
||||
|
||||
1. Open a terminal and run:
|
||||
```bash
|
||||
ssh -L 11434:ai.mtcl.lan:11434 mukul@ai.mtcl.lan
|
||||
```
|
||||
- Replace `mukul` with your remote username.
|
||||
- Replace `ai.mtcl.lan` with your server's hostname or IP.
|
||||
|
||||
2. Keep this terminal session running while using Bolt.New. This ensures your app communicates with the remote server as if it’s local.
|
||||
|
||||
---
|
||||
|
||||
## **Step 4: OPTIONAL: Extend Ollama Model Context Length**
|
||||
|
||||
By default, Ollama models have a context length of 2048 tokens. For tasks requiring larger input, extend this limit for **Qwen2.5-Coder:32B**:
|
||||
|
||||
1. SSH into your remote server:
|
||||
```bash
|
||||
ssh mukul@ai.mtcl.lan
|
||||
```
|
||||
2. Access the Docker container running Ollama:
|
||||
```bash
|
||||
docker exec -it ollama /bin/bash
|
||||
```
|
||||
3. Create a `Modelfile`:
|
||||
|
||||
While inside the Docker container, run the following commands to create the Modelfile:
|
||||
|
||||
```bash
|
||||
echo "FROM qwen2.5-coder:32b" > /tmp/Modelfile
|
||||
echo "PARAMETER num_ctx 32768" >> /tmp/Modelfile
|
||||
```
|
||||
If you prefer, you can use cat to directly create the file:
|
||||
```bash
|
||||
cat > /tmp/Modelfile << EOF
|
||||
FROM qwen2.5-coder:32b
|
||||
PARAMETER num_ctx 32768
|
||||
EOF
|
||||
```
|
||||
|
||||
|
||||
4. Create the new model:
|
||||
```bash
|
||||
ollama create -f /tmp/Modelfile qwen2.5-coder-extra-ctx:32b
|
||||
```
|
||||
5. Verify the new model:
|
||||
```bash
|
||||
ollama list
|
||||
```
|
||||
You should see `qwen2.5-coder-extra-ctx:32b` listed.
|
||||
|
||||
6. Exit the Docker container:
|
||||
```bash
|
||||
exit
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **Step 5: Run Bolt.New Without Docker**
|
||||
|
||||
1. **Install Dependencies**
|
||||
Navigate to the cloned repository:
|
||||
```bash
|
||||
cd bolt.new-any-llm
|
||||
pnpm install
|
||||
```
|
||||
|
||||
2. **Start the Development Server**
|
||||
Run:
|
||||
```bash
|
||||
pnpm run dev
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **Summary**
|
||||
|
||||
This guide walks you through setting up **Bolt.New** with a **remote Ollama server**, ensuring seamless communication through SSH tunneling. We’ve also shown you how to extend the context length for **Qwen2.5-Coder:32B**, making it ideal for advanced development tasks.
|
||||
|
||||
With this setup:
|
||||
- You’ll offload heavy computation to your remote server.
|
||||
- Your local machine remains light and responsive.
|
||||
- Buggy `localhost` configurations? No problem—SSH tunneling has you covered.
|
||||
|
||||
Credits: [Bolt.New repository](https://github.com/coleam00/bolt.new-any-llm).
|
||||
|
||||
Let’s build something amazing! 🚀
|
||||
@@ -0,0 +1,107 @@
|
||||
### **Guide to Set Up Bridge Networking on Ubuntu for Virtual Machines**
|
||||
|
||||
This guide explains how to configure bridge networking on Ubuntu to allow virtual machines (VMs) to directly access the network, obtaining their own IP addresses from the DHCP server.
|
||||
|
||||
By following this guide, you can successfully set up bridge networking, enabling your virtual machines to directly access the network as if they were standalone devices.
|
||||
|
||||
---
|
||||
|
||||
#### **Step 1: Identify Your Primary Network Interface**
|
||||
The primary network interface is the one currently used by the server for network access. Identify it with the following command:
|
||||
|
||||
```bash
|
||||
ip link show
|
||||
```
|
||||
|
||||
Look for the name of the interface (e.g., `enp8s0`) with `state UP`.
|
||||
|
||||
---
|
||||
|
||||
#### **Step 2: Backup Your Current Network Configuration**
|
||||
Before making any changes, back up the existing netplan configuration file:
|
||||
|
||||
```bash
|
||||
sudo cp /etc/netplan/00-installer-config.yaml /etc/netplan/00-installer-config.yaml.bak
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### **Step 3: Configure the Bridge**
|
||||
Edit the netplan configuration file:
|
||||
|
||||
```bash
|
||||
sudo nano /etc/netplan/00-installer-config.yaml
|
||||
```
|
||||
|
||||
Replace its content with the following, adjusted for your environment:
|
||||
|
||||
```yaml
|
||||
network:
|
||||
version: 2
|
||||
ethernets:
|
||||
enp8s0:
|
||||
dhcp4: no
|
||||
bridges:
|
||||
br0:
|
||||
interfaces: [enp8s0]
|
||||
dhcp4: true
|
||||
```
|
||||
|
||||
- `enp8s0`: Your physical network interface.
|
||||
- `br0`: The new bridge interface that will be used by the virtual machines and the host.
|
||||
|
||||
Save and exit the file.
|
||||
|
||||
---
|
||||
|
||||
#### **Step 4: Apply the Configuration**
|
||||
Apply the new network configuration to create the bridge:
|
||||
|
||||
```bash
|
||||
sudo netplan apply
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### **Step 5: Verify the Bridge Configuration**
|
||||
Check that the bridge `br0` is active and has an IP address:
|
||||
|
||||
```bash
|
||||
ip addr show br0
|
||||
```
|
||||
|
||||
You should see an output like this:
|
||||
|
||||
```plaintext
|
||||
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
|
||||
link/ether 46:10:cc:63:f4:37 brd ff:ff:ff:ff:ff:ff
|
||||
inet 192.168.1.10/24 metric 100 brd 192.168.1.255 scope global dynamic br0
|
||||
valid_lft 7102sec preferred_lft 7102sec
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### **Step 6: Configure Virtual Machines to Use the Bridge**
|
||||
For VMs created with tools like `virt-manager` or `virsh`:
|
||||
1. When configuring the VM’s network interface, choose **Bridge** as the network source.
|
||||
2. Set `br0` as the bridge interface.
|
||||
3. The VM will now obtain an IP address dynamically from the same DHCP server as the host.
|
||||
|
||||
For `virt-manager`:
|
||||
- Go to **Add Hardware > Network**.
|
||||
- Choose **Bridge br0** as the source.
|
||||
|
||||
---
|
||||
|
||||
#### **Step 7: Test the Setup**
|
||||
1. Start a VM and ensure it obtains a dynamic IP address from the network.
|
||||
2. Test connectivity by pinging the gateway or external servers from the VM.
|
||||
|
||||
---
|
||||
|
||||
### **Key Considerations**
|
||||
1. **Dynamic IP for Host:** The host server's IP address will now be associated with the bridge (`br0`) instead of the physical interface (`enp8s0`). This is expected behavior.
|
||||
2. **Backup Configuration:** Always maintain a backup of your original network configuration to revert changes if needed.
|
||||
3. **Network Manager vs. Netplan:** Use only one method (`netplan` or `nmcli`) for managing network configurations to avoid conflicts.
|
||||
4. **Alternative Access:** If you are working on a remote server, ensure alternative access (e.g., a second network interface) before applying network changes.
|
||||
|
||||
Reference in New Issue
Block a user