What an incredibly interesting use of LLMs (generating text to practice typing). It leans in on what LLMs are good at. That said. I would love to see a middle tier pricing which had some features but avoided the AI use.
Why avoid AI use? Genuine question, I see this around and it seems usually based on a mental model of the environmental cost of AI that does not match impact in the real world.
Environmental cost is a concern, though for me not the main one. In this case it's two things.
1. AI interactions cost the service money, which is inevitably passed on to the consumer. The if it's a feature I do not wish to use, I like to have options to avoid paying for that feature. So in this case, avoiding AI use is a purely economic decision.
2. I am concerned about the content LLMs are trained on. Every major AI has (in my opinion) stolen content as training material. I prefer not to support products which I believe are unethically built. In the future, if models can be trained solely on ethically sourced material where the authors have been properly compensated, I would think this position.
I'm active in the /r/localllama community and on the llama.cpp GitHub. For this use-case you absolutely do not need a big LLM. Even an 8B model will suffice, smaller models perform extremely well when the task is very clear and you provide a few shot prompt.
I've experimented in the past with running an LLM like this on a CPU-only VPS, and that actually just works.
If you host it on a server with a single GPU, you'll likely be able to easily fulfil all generation tasks for all customers. What many people don't know about inference is that it's _heavily_ memory bottlenecked, meaning that there is a lot of spare compute left over. What this means in practice is that even on a single GPU, you can serve many parallel chats at once. Think 10 "threads" of inference at 20 Tok/s.
Not only that, but there are also LLMs trained only on commons data.
Yeah, LLMs are indeed really good for this use case.
> That said. I would love to see a middle tier pricing which had some features but avoided the AI use.
Only paid features are AI features. Everything else is free and no ads :)
You can type anything and as much as you want, you have access to all the advanced stats, you can create a custom theme from a photo of your keyboard, etc.
Everything but AI features is free right now. (Might change in future as we’re adding a lot more features so we will definitely consider a mid tier price )
