That's the way I've been playing it...the break material and any hops that weren't bagged will take up a lot of space in the kettle but under the weight of a few generations of yeast bodies, it compacts into a nice solid layer and gets out of the way of racking clean beer. As regards off flavors, I'm not sure any break material (protiens that are coagulated out of the wort at boiling) will have much of an effect, unless you ended up aging "on the lees" for an extended period. Protein haze or chill haze might be something to think about, but flavors shouldn't be an issue.
Hops transferring to the fermenter could be an issue. I usually use a hop bag except if I'm using Magnum or another high alpha in 1/4 or 1/2 oz at the 60 minute addition - I just don't worry about that.
I just did an IPA and left all the hops loose in the boil and transferred all to the fermenter. I don't think grassy flavors will result from boiled or whirlpooled hops. Dry hops with no high-temp interaction could have a raw, vegetal, grassy flavor, but I think it depends on the hops. To me, nobel hops and varieties described as "earthy" like Saaz or Willamette have a dirt/grass/hay aftertaste in additions less than 10 minutes from FO - those I try to keep out of the fermenter by boiling in a bag.