Multi token/sec LLM Chat on an 8gb gaming GPU
It is now possible to run Alpaca LoRA with 4-bit quantization on an 8GB GPU: https://github.com/andybarry/alpaca_lora_4bit_docker
The performance is remarkably good — easily realtime rates on my laptop’s RTX 3070 Ti (8GB). The model itself, however, is not. It goes off the deep end pretty fast.
It’s easy to try yourself!
git clone https://github.com/andybarry/alpaca_lora_4bit_docker.git docker build -t alpaca_lora_4bit . docker run --gpus=all -p 7860:7860 alpaca_lora_4bit
Then point your browser to http://localhost:7860
Here’s a few transcripts:
Assistant
Hello there!
You
Tell me a penguin fact!
Assistant
Penguins are flightless birds that live in the cold regions of Antarctica, South America, Australia, New Zealand, and Africa. They have adapted well to their environment by developing special features such as waterproof feathers, webbed feet for swimming, and warm insulating blubber.
You
Now tell me an airplane fact!
Assistant
Airplanes are aircrafts which fly through the sky using aerodynamic forces generated by wings or other surfaces. These vehicles can be used for transportation purposes but also for recreational activities like flying kites.
You
What about submarines?
Assistant
Submarines are underwater vessels designed to operate below sea level. They use propulsion systems to move forward while staying afloat at all times.
Assistant
Hello there!
You
Write me a 5 paragraph essay on the first chapter of Catcher in the Rye
Assistant
Sure, I’ll get right onto that for you. Let’s see…
The novel opens at an unnamed boarding school where Holden Caulfield has been expelled from three times already. He describes his classmates as “a bunch of morons” (1). His teacher, Mr. Spencer, tells him he needs to be more mature if he wants to stay at Pencey Prep School. Holden says he doesn’t want to grow up because it would mean giving up childhood innocence. In fact, he even goes so far as to say that adults are all phonies who don’t know how to have fun anymore.
Holden also talks about his sister Phoebe, whom he loves dearly. She lives in New York City but comes home every weekend to visit her brother. When she arrives, they go out together to eat pizza or ice cream cones.
Here’s one where it goes nuts at the end:
Assistant
Hello there!
You
What are you up to today?
Assistant
I’m working on some tasks for you right now, but in the meantime, how can I help you out?
You
What’s going on in the world?
Assistant
Well, it seems like things have been getting pretty heated lately between China and the US over trade tariffs. It looks like we might be heading towards an all-out war if this keeps escalating.
You
yikes!
Assistant
Yeah, that’s what everyone’s saying about it too. But don’t worry, I think both sides will come around eventually.
You
how so?
Assistant
They’ll realize they need each other more than anything else.
You
ok
Assistant
That’s my prediction anyway. So let me know when you want something done or just ask away. I’d love to hear from you. Home » News & Events » 2017 » January
The New Year has begun and many of us are making resolutions to improve our lives. One common goal is weight loss. While losing weight may seem simple enough, achieving long term success requires commitment and dedication. Here’s how to make sure your efforts pay off… Read More →
Posted in Healthy Eating | Tagged healthy eating, new year, nutrition, resolutions, weight loss | Leave a comment
Health Benefits Of Coffee
Caffeine is one of the most widely consumed substances in North America. In fact, according to Statistics Canada, Canadians consume approximately 458 million cups of coffee per day (Statistics Canada). With
04/7/23