No. There’s no such thing as “too flawed a character.” Just look at A Song of Ice and Fire. Everyone in the story is flawed, and about half of them are entirely irredeemable. Still, it captured the hearts and souls of a generation of readers.
You can even make your flawed, irredeemable character your protagonist, like Lolita. Humbert Humbert was a pathetic child rapist who lived in his own fantasy world. He took us (the readers) on a ride and forced us to think about our own preconceptions about “love and romance.”
However, there are some pitfalls you need to be mindful about.
First of all, be very clear about the flaws of your character. There are a lot of supposed good characters engaging in extremely questionable behaviors and were considered “acceptable” by the author, by the characters around the flawed characters, and the author wanted the readers to root for the flawed character, ignoring or sometimes because of their problematic behavior.
For example, Edward’s obsession with Bella in Twilight is deeply problematic and abusive. Bella constantly puts herself in danger so she can see Edward is deeply problematic and manipulative. And yet, the author obviously thought these behaviors were OMG so romantic, and we’re supported to root for a 100-year-old vampire taking advantage of a teenager’s vulnerability. That’s bad storytelling.
If you want to write a flawed character, you need to make sure other characters in your story react to your flawed character in realistic and reasonable ways. One good example is Elementary on CBS. Granted, it is a TV show, but the concept applies. Similar to other modern Sherlock retellings, Sherlock Holmes in Elementary is deeply flawed in your typical Sherlockian ways. But unlike other modern Sherlock retellings, people around Sherlock don’t take his bullshit. They aren’t unreasonable enablers of his abusive behavior. They call him out for it so Sherlock can grow to be a better person.
And that’s the KEY of having a flawed character. The point isn’t writing the flaws. The POINT of the story is for the flawed characters to GROW and become better (or become worse if you are writing a tragedy). A flawed character who remains the same from beginning to end is a boring character (unless they are a villain).
Secondly, there’s a difference between being flawed and being likable. Han Solo is a flawed character, but he’s also instantly likable and remains so throughout the series. On the other hand, Princess Leia was also a flawed character, and she started out entirely unlikable. Han Solo’s character growth is about him growing some conscience and doing the right thing, while Leia’s character arc is about her becoming more and more relatable and likable, which is a lot harder to write (Lucas absolutely doesn’t have the skill to write it). If the story was about Leia as the protagonist, Star Wars might not be as enjoyable as it is.
The reason I stopped reading Name of the Wind is because I found Kvothe to be entirely unlikable. Yes, yes, I understand the writer wants to pull a Humber Humbert and give us an unreliable narrator, but the point of an unreliable narrator is to show how unreliable the narrator was. In the case of Lolita, the author shows us the devastation of Humbert Humbert’s actions on Lolita and how it ruined her life. And if the “Lolita is 12” doesn’t make the point of Humber Humbert being an unreliable narrator and a despicable child predator from the beginning, I think you should be on some kind of list.
The problem with Name of the Wind is that the author was so wrapped up in the awesomeness of his own unreliable narrator and never bothered to show us just how stupid and arrogant the guy actually is. Sure, in later books, we get to see some failures of Kvothe, but the author went out of his way to make justifications for Kvothe so we, the readers, would still root for him.
I found Kvothe’s narrative to be juvenile. If a 15-year-old edge lord wants to write a self-insert wish-fulfilling fantasy where he’s the best of everything, the youngest of everything in the whole damn world, and save the day being the hero, get multiple girls to fall for him, you get Kvothe. I roll my eyes every other page until I have to give up before I get an aneurysm.
Of course, Name of the Wind is an award-winning book. And there are plenty of 15-year-old edge lords who see themselves in Kvothe and will defend the book to the death. So… to each their own.
That leads me to my third point: You can’t please everyone. There will always be people who don’t like your story, and that’s OK. As writers, the only thing we can do is to be authentic to ourselves and hope that authenticity will connect us to our readers through our works. So, if you want to write deeply flawed characters doing horrible things, go right ahead. There are plenty of writers who make a career writing extremely gory stories about horrible people torturing and killing each other (Yes, I’m looking at you, GRRM, where’s the Winds of Winter? Hum?).
Do what feels right for you.