OpenAI releases a free GPT model that can run on your laptop (www.theverge.com)
from ryujin470@fedia.io to technology@beehaw.org on 06 Aug 12:41
https://fedia.io/m/technology@beehaw.org/t/2524897

OpenAI’s first open source language model since GPT-2

#technology

threaded - newest

nathan@piefed.alphapuggle.dev on 06 Aug 13:30 next collapse

*if you have a laptop with 16gb of vram. Otherwise you'll be watching ollama hit your CPU for 5 minutes with no output

sefra1@lemmy.zip on 06 Aug 14:16 collapse

Isn’t that true for most models until someone destiles and quantises them so they can run on common hardware?

fuckwit_mcbumcrumble@lemmy.dbzer0.com on 06 Aug 14:23 next collapse

This is the internet, we’re only allowed to be snarky here.

Ghoelian@lemmy.dbzer0.com on 06 Aug 16:04 next collapse

I mean yeah, but that doesn’t make the title any more true.

CyberSeeker@discuss.tchncs.de on 07 Aug 00:46 collapse

Yes, but 20 billion parameters is too much for most GPUs, regardless of quantization. You would need at least 14GB, and even that’s unlikely without offloading major parts to the CPU and system RAM (which kills the token rate).

fuckwit_mcbumcrumble@lemmy.dbzer0.com on 08 Aug 14:49 collapse

I tried it out last night and it ran quite well on my heavily thermally limited i9 11950h/rtx 3080 laptop. I had maybe 6 or 7 gigs of main ram used in total, with docker running. It was only using about 12 gigs of vram in my very limited testing.

Bebopalouie@lemmy.ca on 06 Aug 15:06 next collapse

No thanks.

SweetCitrusBuzz@beehaw.org on 06 Aug 16:32 collapse

Agreed.

PoisonedPrisonPanda@discuss.tchncs.de on 06 Aug 16:36 collapse

paywall free version?

edit: huggingface.co/openai/gpt-oss-20b