Should the matter of fact be hyphenated?
If the phrase can’t be considered a phrasal adjective, then no. If it can be considered a phrasal adjective, then yes. Examples:
Not a phrasal adjective, therefore not hyphenated:
It appears that, even though it is logically possible for space to be curved, as a matter of fact, it is flat.
So far as the head teacher is concerned, it is a matter of fact that he did exclude him.
As a matter of fact, it is not possible to justify this attack on the basis of international law.
A phrasal adjective, describing a noun, therefore hyphenated:
You're speaking about this in a very calm, matter-of-fact way.
Beautiful scenery, combined with melancholy music and matter-of-fact narration, made a lovely little story.
It is expressed not in high-flown language but as a down-to-earth, matter-of-fact promise.