Model w/ Guardrails Disabled Still Frequently Refuses to Summarize Text

I know someone will ask if I have filed Feedback.

I have in the past and I will file a new one. That said, none of my Foundation Models feedbacks have any activity on them and I've filed about a dozen since June. Feels like a waste of my time.

Hi @Hunter,

If I'm understanding correctly, you're asking Foundation Models to summarize an article, but it is instead responding with the article verbatim. Is this correct? Or is it simply refusing to respond at all? Either way, yes, the proper path forward would be to file Feedback.

When you do, be sure to included a focused code sample, along with an excerpt of the text that was not properly summarized, so that we can recreate and pinpoint the issue. Please also mention the FB number here so I can keep track of it internally.

As for asking the model itself why it did not summarize the articles, this response should not be treated as an accurate error message due to the nature of probabilistic generation.

Best,

-J

Hi,

No, it's not responding verbatim, sorry if unclear. It's responding with a single sentence saying some variation of "I cannot fulfill this request because..." like you see in the above post.

It will give a reason but it sounds like you're saying that's likely hallucinated so perhaps just nonsense. But it's just a refusal, not the original article text returned back to me.

I seem to see this most consistently on political stories but also strangely on tech stories (it will sometimes refuse to summarize any Apple related rumors which I'm sure is not intentional but pretty odd).

I will post my FBs.

Hi @Hunter, thanks for clarifying.

As you likely know, we have some guidelines around acceptable use for Foundation Models, which may be useful for future visitors of this thread.

For acceptable use-cases, there are likely still behaviors we want the model to avoid, and that's what guardrails are for. They provide your app a way to handle the different kinds of errors that may arise during generation, so be sure to catch those and handle them accordingly.

Even still, there may still be instances where the model refuses to answer for one reason or another. You can attempt to catch these generation refusals, too. While the explanation is not as authoritative as an error message, it can provide some clues for why the unexpected behavior is occurring.

This article explains these concepts in some more depth and lays out information about user input boundaries, language model session instructions, blocked terms, and safety testing and monitoring.

Please still provide that FB# and focused sample project. I'd be interested in taking a look at the text causing the behavior you're experiencing as well as the error handling in the app.

Hope this helps,

-J

Actually, I didn't know about the acceptable use guidelines and I thought I had read all of the various documentation bits on Foundation Models so good to have that link for sure.

Been busy the past few days but I will post the sample/FB, yes.

@DTS Engineer Here is an example feedback taken from Playgrounds. If whatever it captures+provides does not include what you are looking for let me know and I can add more to it.

The session instructions are trying to adhere to requirements like the acceptable use doc linked. Trying to!

FB19916776

Hi @Hunter,

Thanks for filing that Feedback. I've been looking into this more closely.

I can clarify one piece of information from earlier, which is that refusals can only be caught when using guided generation.

In terms of your use-case of user-facing text generation (summarization), there will not always be a way to programmatically tell apart an answer from a natural language refusal. This is true of large language models in general, and it is worth keeping in mind as you design your app and set expectations for users.

In another thread I recently provided some links with guidelines on what to keep in mind when designing for generative AI, and you may find that useful as well.

Best,

-J

Thanks for the engagement on this and I appreciate you taking a look.