The BS that ChatGPT generates is especially odious because it requires so much effort to detect if you're not a SME in whatever output it's giving.
Plus, the output, when wrong, is subtly wrong- it's usually not obvious BS, it's credible BS. If you are not competent in the particular area you're asking about, you likely don't have the skills to recognize BS.
It also is a time saver, doing work that most people find unrewarding. So you get a chorus of fans saying "it saved me a bunch of time doing my own research and just spit out what I needed". Maybe it did, or maybe they didn't have enough expertise in the area to recognize the flaws in the output it presented.
McQuillan’s central point here that ChatGPT is a “bullshit generator” and not “artifically intelligent” is really apt. What we’re often working on in the industry is emulating the worst uses of intelligence processes, that humans also use, like optimized BS generation. Seeing ChatGPT turn into the equivalent of a competent essay-spitting undergrad is distressing; the internet is now such a mass of human-generated BS, it's hard to believe people will be arguing and competing with machine optimized and generated BS speech.
The most powerful and effective and immediately available BS generators will probably rely on machine-generated, unhinged speech. I truly fear for the future of the few reputable internet forums left, because even intelligent people tend to engage with well-optimized BS generators, whether driven by human or machine.
You're not too far off from the arguments people made about the internet not too long ago. It's too easy to access information that could easily be incorrect...even maliciously so!
You're better off sticking to published books and journals from respectable organizations that vet their authors and review their publications!
Then again, who's in control of those printing presses? How can you trust the publishers to not push their own politics and agendas? You're probably better off finding a religious organization you can trust to help filter out the bad stuff. Help you see things through the proper perspective.
The problem is people weren't wrong about the internet. In fact, they couldn't grasp the magnitude of the problem it would create, and the complete transformation it would have on media, politics, and culture.
As someone who vaguely remembers the 90s I can tell you that there is no transformation in media, politics or culture. Well, there is.
But the difference people describe, that politics used to be based on sound science and now, with the internet/facebook/fake news/tiktok changed to be based on total bullshit. Not true.
Not because media and politics aren't currently almost exclusively bullshit. But because that wasn't any different in the 90s. Back then media was full of bullshit, and politics reacted 100 times to media bullshit for every time it reacted to actual science. Hell, there's positive evolution too, I think the BBC has actually improved their fact checking since back then, for example. And I actually know what is trustworthy. I didn't know in the 90s.
But... With books you know who is publishing them. You might know who is in charge of a website. At least with Wikipedia sources are cited. With gpt, nothing.
Just had a dinner conversation where ChatGPT was characterized as automated plagiarism, and then I thought wouldn’t it be cool to get like a set of BibTex entries for all the sources whose content were combined to synthesize an output.
Not sure that’s possible, and even if so, that it would be any kind of reasonable or manageable size whatsoever.
I run the cheaper self hostable OpenAI alternative https://text-generator.io I've been working on automating this manual verification of everything, with a few components we already have like a search engine and an edit API we can both detect and correct most of these errors to at least be reflective of what a reliable source says like Wikipedia, still a lot of reasoning, logic and math issues will remain, but there's a big step up coming soon in factual generation
ChatGPT helps a lot but it is necessary to take everything with a grain of salt - so what? It still can save a lot of time. Just Alt-tab ChatGPT/Google. Adjust your mental model from what you think it should be to what it actually is.
I hear this a lot but I don't find it convincing. If I know enough to know if the answer is good or bad, why would I be asking a chat bot? We could say this problem is somewhat analogous to sorting good StackOverflow answers from bad ones. But I feel that there are things like writing and presentation style (not to mention the votes and comments) that could tip me off to a bad answer that are absent when every answer is in the same voice.
> If I know enough to know if the answer is good or bad, why would I be asking a chat bot?
If you are trying to write a blog/essay/etc on a topic, it can help with “writer’s block”-you may know a lot about the topic already, but struggle with putting that understanding into words. And then it isn’t hard to filter ChatGPT’s output based on your pre-existing knowledge.
Sometimes I don’t know the answer to the specific question I am asking it, but still have enough background knowledge of the topic to pick up when it is likely to be “hallucinating”
What counts as “plagiarism” depends on the context. It is (in most cases) unethical to present the work of another human being as your own, but what about an AI? In an educational setting, it would be unethical to turn in an AI’s work as your own. But if a professional programmer uses GitHub Copilot, are they obliged to acknowledge Copilot’s contribution? If I have a personal blog, and ChatGPT helps me write a post, am I obliged to acknowledge ChatGPT’s contribution?
I think the bigger picture to me is something like are you being disingenuous rather than are you technically plagiarizing. Thought experiment: if I tell people I’m great at multiplication and I don’t tell them I have an earpiece / mic with someone using a calculator on the other end. If I convince these people I have a great mind and they respect my math skills, have I not fooled them? As for the copilot argument, I’m not convinced it shouldn’t be required to cite.
People write for many different reasons. Sometimes it is because they believe in an idea (political, ethical, philosophical, religious, etc), and their main objective is to convince others to believe in that idea too-what you may come to think of their abilities as an author is rather besides their point.
Somewhat of an open question really. I think for all of blogging we have concluded that the words in your blog post were in fact written by you. I see two reasons plagiarism is bad: both and injustice to whom wrote the original but also a false indicator to your cognitive and creative ability.
Well, I think it is more challenging than that. Even if the author writes the piece himself, most professional writing is subject to editing, which can substantially change the finished product. The one thing we can be reasonably, if not entirely, certain of is that the piece is one the author is comfortable endorsing as his own.
> If I know enough to know if the answer is good or bad, why would I be asking a chat bot?
I mean, taken to the extreme, I can probably read through the source of VLC and know that it's correct, given enough years to read it and study video compression standards etc. Does that mean I don't get use out of someone else having written VLC for me?
Knowing something is right and producing it are completely different things. You might be thinking of ChatGPT too narrowly, as a simple question-answer thing, but even now you can ask it to write code that will save you time, and scale it up by a few factors and it's doing the equivalent of writing new libraries for you. (Probably many years before it can write VLC for you, if ever.)
This is not the same thing at all. You know whether a tool like VLC works because it has a pretty well-defined scope for "works": it plays the video or audio file you clicked on.
If you're asking ChatGPT to teach you something, you have no such easy verification you can do: you essentially need to learn the same material from another source in order to cross-check it. Obviously this is easy for small factual questions. If I ask ChatGPT the circumference of the Earth, I can quickly figure out whether it's reliable or not on that point. But at the other extreme if I ask it to do a music theory analysis of the Goldberg Variations, it's going to take me about as much work to validate the output as to have just done the analysis myself.
I'd suggest just trying it yourself for a bit to see if you can find any use for the tool. If not, that isn't a problem either - I suspect it is completely useless in some domains.
I remember playing around with it first and doing similar things with the openai playground. It is amusing but the novelty of this type of usage wears off quickly.
I mostly use it for programming type work now. Write a little snip of code (personal projects only) or tell me how to use a library. I also use it to teach me things. I find it incredibly useful for this.
ChatGPT is still free for me but it is glitchy at times. I would definitely pay the $20/ month one it is available in my region.
I know what a clean house looks like, but I'd still love a machine to clean my house. CatGPT's purpose is not to understand complex ideas, it is to do the tedious task of joining disparate such ideas together in an intelligible way. Much faster to have it do 95% of the work and you only need to modify the 5% that's bad, then for you to do 100% yourself.
That’s a bad analogy. It would actually rather be like: I know what a clean house looks like, but I’d still love a machine to describe it to me.
However, I think you are somewhat on the right track as “AI” shows us which types of work can be removed all together. I think, if “AI” can do it, it’s not necessary work to begin with. Eg. instead of “AI” doing formal correspondence for us, we could then just have an interface to exchange raw information directly, as the human element is lost anyway.
Love your "adjust your mental model" sentence. Let's talk about the judge using it in ruling, or your doctor for diag... No problem for you?
And now that it is listed as co-author on research papers (real)? Then using this source to reinforce his model!
I really think it is a lot more than a grain of salt my man.
[edit] AI is not all bad, just fun ;)
"Is chatGPT accurate?"
https://www.perplexity.ai/?s=u&uuid=c97b4388-958f-45cb-a0ae-...
I suppose it isn't obvious to me and I perhaps don't care. I'm not going to get stressed about being made obsolete by AI, etc., if that is what you mean. That could happen but so could a lot of things. Of course, I may have completely missed what you meant by slippery slope.
No one gets the reference? Rich Hickey gave a talk where he speaks about how easy it is to import dependencies you didn't know you had. That means you were getting complications to your systems in an uncontrolled fashion because it's so easy to do.
Now imagine that same ease with ideas. It's so easy to just let ChatGPT write that up, I'm going to do that. I'm not even going to edit, even where what it wrote doesn't exactly reflect my thinking on the matter.
At that point, you're cluttering up your own thinking with output from ChatGPT because it's easy.
> The BS that ChatGPT generates is especially odious because it requires so much effort to detect if you're not a SME in whatever output it's giving.
I've found a good way to demonstrate to oneself how badly ChatGPT can miss not only the nuance of a subject but just plain get basic facts wrong is to ask it about relatively simple things like movie or book plots and see how the results differ from, well, just actually watching or reading the subject, eg.
/In the movie "AI", the protagonist, a highly advanced android boy named David, embarks on a journey to find the Blue Fairy from the story of Pinocchio in order to become a real boy and be reunited with his human mother. After many trials, David finally reaches the submerged city where the Blue Fairy is said to reside, but instead finds a statue of the fairy. He is then discovered by human survivors of a global flood, who have been in suspended animation for thousands of years. The humans react with fear and attempt to dismantle David, but he is rescued by a mermaid, who takes him to the underwater kingdom of the lost city of Rome. There, he finally meets the Blue Fairy, who reveals that she has no power to grant his wish, but assures him that his love for his mother will live on forever. In the end, David is shown as a frozen statue, in a future where the sun has burned out and the Earth is covered in ice, while the human race has long since vanished. The last shot is of the statue of the Blue Fairy, still underwater, suggesting that David's story and love will endure for eternity./
--------
Its close enough to be believable if you haven't seen the movie (or maybe even if you have seen it but it was 20 years ago) but there's a lot of obvious errors packed into a relatively short paragraph there. Got off to a good start in the first two sentences and then just goes right off the rails... but with such confidence.
I'll risk the downvotes, but I'm genuinely curious why people feel that they need to note that they've made trivial edits, like deleting a duplicated word or fixing a typo. Is there a sense that the historical record of these meanderings should be perfectly memorialized?
Just one guess, but in the "good old days" of the Internet, people would sometimes get suspicious and accuse you of trickery if you edited your posts without explanation, so a culture developed of always explaining even the most trivial edits. "Full transparency" taken to its absurd extreme, I suppose.
And not only does the BS often need expertise to detect, but the tech cheerleaders are claiming that not only is this high-volume automated BS machine useful, but that it is somehow nearly an actual near-generalized AI.
Nothing could be further from the truth. The generative models make sometimes-useful BS, and are sometimes surprising in the similarity of the output to human output (sure, human bullshitters often look good too, so what?), but there isn't even the slightest ability to understand any concept. The thing can't even get right puzzles that children laugh at. These things have no concept of truth vs fiction or ethics vs evil.
Yet the "tech elite" try to feed us the BS us that it is nearly the singularity. What we have is a very amusing parlor game toy.
What we need, whether for the world's sanity and/or to get close to AGI, is not an automated high-volume BS generator.
What we need is an automated high volume BS detector.
I think it comes in a situation that isn't great, but not so bad that it warranted drastic decisions: people are organically generating mountains of BS online everyday, including on blogs, official looking news sites etc.
It's a pain, but as most of that is also generated by humans there's an opt out, saying "good guys" should band together to fight it and it would be fine.
Now that we have ChatGPT and the like, manually dealing with all the BS is just out of the window.
And I'm kinda hoping we try to deal with it in a systematic matter, and find a way to flag and bury the BS down to levels lower than when ChatGPT came to the public.
Basically, I'm hoping the thing that happened for spam happens for bullshit as well, and we get optimized tools to fight to manageable levels (with the arms race adn all, probably)
> I think it comes in a situation that isn't great, but not so bad that it warranted drastic decisions: people are organically generating mountains of BS online everyday, including on blogs, official looking news sites etc.
Indeed, ChatGPT produces bullshit because that’s the bullshit it’s been trained on.
> Basically, I'm hoping the thing that happened for spam happens for bullshit as well, and we get optimized tools to fight to manageable levels
Who is going to do that? I’m genuinely asking, because I’m old enough to remember that BigCo was totally fine to “enshitificate” their search for years and still barely anyone got close to them…
I really don’t know, but I also don’t see it coming from any of the behemoths we have now.
The wildest scenario would for it to come from Mozilla or another web player like Cloudflare. More realistically it could come from players outside of the US seeing it as an opportunity to make a dent in the current quasi-monopolies, with some government backing as it would probably cost amounts no investor would pay otherwise.
When I went to google SME, I also got "Small and Medium-sized Enterprise".
When I asked ChatGPT "Someone used the following sentence 'The BS that ChatGPT generates is especially odious because it requires so much effort to detect if you're not a SME in whatever output it's giving.' What does SME stand for?"
It told me: SME stands for "Subject Matter Expert".
It is a question where people think they can get easy karma if they know the answer, and many people know the answer. That said, I also don't think even most people would know the answer. I further think it is a fair question as they provided the most common expansion and that's what you naively find on Google (so it isn't like the people who ask a question they could have already answered asking Google).
The next problem is that the Hacker News user interface doesn't encourage people to not cause duplicate answers within a short window: you load the full thread and spend 10-15 minutes reading it. At the time you loaded the page there aren't all of these other replies, so you may as well leave the first one.
You then click the reply button and start typing a reply--FWIW, for longer replies this step can matter even more than the previous one--and you are staring at the single now and are given just the empty box for your reply. When you are done you submit, and maybe you then notice it is a duplicate? You might not, based on where the page loads and how much you are paying attention. Even if you do, do you bother to delete it? I would, but I hate the duplication and also know how to delete my comments.
I happened to catch the 4 (four! I've seen 2 dupes before but four in less than 2 minutes have never I seen afore), original comments all at once posted 0 and 1 minutes ago, and because I am in some @"mood" tonight, and for all the reasons you've said, found it to be rather funny a HN tragedy. I thought it would be harmless to slip in a 5th, capitalizing only the M to show the careful reader that I am clearly out of my mind. I promise I wasn't karma farming I only got 1 upvote. Alas, HN isn't outfitted with the latest in web push socket hot page reload sex, yet.
Also your comment is "hidden" for a few minutes after you post it, to allow you to edit it, which is why they most probably didn't even see each other.
In this kind of forum, it doesn't bother me when terms (like SME) are not defined by the person using it in a comment (even when it might be relatively new and matches several possible definitions). I didn't immediately recognize it, so I Googled it and found what I was looking for (before noticing the other replies which defined it for me).
What does bother me, is when someone writes a long paper and fails to identify what it means when it is first used. I will read along sometimes for several paragraphs hoping to get a definition (or at least a good clue) before finally having to break off to go Google the term on another tab.
Common terms like CPU or SSD have been around long enough to be used as is, but newer ones like AGI, LLM, or RHLF need to be written out in long form (which the article did beautifully). Sometimes they also need a basic definition to go along with that.
The funny thing is how many HN users have no real expertise in anything beyond software, and yet consider themselves SMEs on aviation, politics, climate, energy, medicine, warfare, and every other common topic here. The Dunning–Kruger effect is everywhere.
Yeah, as if ChatGPT BSes more than your average internet commenter who fools a huge percentage of people he reaches. People really aren't considering the base rate.
Plus, the output, when wrong, is subtly wrong- it's usually not obvious BS, it's credible BS. If you are not competent in the particular area you're asking about, you likely don't have the skills to recognize BS.
It also is a time saver, doing work that most people find unrewarding. So you get a chorus of fans saying "it saved me a bunch of time doing my own research and just spit out what I needed". Maybe it did, or maybe they didn't have enough expertise in the area to recognize the flaws in the output it presented.
[ed] deleted duplicate word