I might misunderstand the point you are making, but this actually stems from a "one and only one" mentality: there is only a single entry for "yield" in the grammar, and it is defined to take a specific kind of expression as its argument.
(edit:) (Additionally, there is the "one and only one" mentality that if yield is valid in a expression for a normal method, it should be valid in an expression for a lambda as well; however, that doesn't cause this syntax issue: statement vs. expression does.)
This level of expression is allowed to contain a comma. As an example, you can do "yield 1, 2", which will yield a 2-tuple containing 1 and 2 as elements.
However, if you wish to embed it as an argument in a function call, you will end up with an ambiguity, as arguments to functions are comma separated. You therefore must embed this expression in parentheses, lest you get a weird grammar conflict.
To be clear, this is identical to the case with tuples themselves, with the one caveat that they decided to make it actually be a syntax error to use it in a way that causes the ambiguity (I will be clear that I do not know if I agree with their decision on that regard).
"return 1, 2" <- valid
"a, b = 1, 2" <- valid
"a((1, 2))" <- valid
"a(1, 2)" <- not what you meant
(edit:) JavaScript has a similar problem, by the way: they have all sorts of special variants of expressions that are not allowed to start with a brace or the keyword "function" due to ambiguities at the top levels of statements.
However, the most similar issue in JavaScript is with relation to the keyword "in": "for (var x = 5 in a" could either mean "loop over x, initialized to 5, as all keys of a" or "initialize x as whether 5 is a key in a".
You can figure out the difference once you get to the next token: ")" (the former) or ";" (the latter), but once you get that far you've already committed to specific parse tree formulations for the code you've already seen.
Ah, ok; that didn't seem related to this specific discussion of yield (yield is valuable to have as an expression even in a multi-statement function), but I can see how it is reminiscent.
It should be noted, though, that when you use the lambda keyword, the result is a function, and has the same properties and behaviors as a function (including that it can contain yield and be a generator).
The lambda syntax is simply a way to declare a function as an expression. You can't have a multi-statement block embedded in an expression due to fundamental limitations of normative whitespace.
There really, though, is only one kind of "callable": "function". There are two syntaxes for declaring it, however: "def", which takes a block of statements, and "lambda", which takes an expression.
This is (to me) drastically different than the behavior difference that I believe is what you are referring to in Ruby, where Proc and lambda have different semantics for embedded usages of the "return" keyword.
(That said, I want to be clear, to both you and any random audience: I personally agree with Ruby's decision to have these two forms of callable, per real-world conversations I've had with wycats.)
(afterword) Someone else might find this interesting: I was going to also make the point that both lambda and def result in the same class (function), but it turns out that Proc.new and lambda do as well (Proc).
(edit:) (Additionally, there is the "one and only one" mentality that if yield is valid in a expression for a normal method, it should be valid in an expression for a lambda as well; however, that doesn't cause this syntax issue: statement vs. expression does.)
This level of expression is allowed to contain a comma. As an example, you can do "yield 1, 2", which will yield a 2-tuple containing 1 and 2 as elements.
However, if you wish to embed it as an argument in a function call, you will end up with an ambiguity, as arguments to functions are comma separated. You therefore must embed this expression in parentheses, lest you get a weird grammar conflict.
To be clear, this is identical to the case with tuples themselves, with the one caveat that they decided to make it actually be a syntax error to use it in a way that causes the ambiguity (I will be clear that I do not know if I agree with their decision on that regard).
"return 1, 2" <- valid
"a, b = 1, 2" <- valid
"a((1, 2))" <- valid
"a(1, 2)" <- not what you meant
(edit:) JavaScript has a similar problem, by the way: they have all sorts of special variants of expressions that are not allowed to start with a brace or the keyword "function" due to ambiguities at the top levels of statements.
However, the most similar issue in JavaScript is with relation to the keyword "in": "for (var x = 5 in a" could either mean "loop over x, initialized to 5, as all keys of a" or "initialize x as whether 5 is a key in a".
You can figure out the difference once you get to the next token: ")" (the former) or ";" (the latter), but once you get that far you've already committed to specific parse tree formulations for the code you've already seen.