Though it’s been years since I sat at the mahogany table in the dining room or felt the velvet of the long living room sofas, I remember Grandfather’s house as if it was yesterday. Every three months, the doorman would beckon us into the lobby, and we’d make our way into the foreboding building. In the elevator, I would push the button. After 20 seconds that felt like an eternity, the doors would open without a sound to reveal a pink-tiled foyer and two huge marble lions guarding a black door. Enrique, Grandfather’s butler, would open it as soon as he heard our footsteps.

The evening would begin with hors d’oeuvres in the parlor.

“How’s work going?” my mother would ask.

“Fine,” Harriet, my grandfather’s third wife, would respond, not bothering to elaborate.

“How are the kids?” my father would ask.

“Fine,” Harriet would reply.

At dinner, Maria, the housekeeper and cook, would serve salad and soup, meat and potatoes, wine and whiskey for the grownups. Grandfather sat at the head of the table, Harriet to his right, me to his left. Grandfather was always the master of ceremonies, facilitator of all conversation. When he spoke, you listened. His voice was deep and smooth like honey. However, if you really listened, you could notice the thinly veiled sarcasm, his sinister edge. Grandfather loved to talk about politics, culture, and the latest New York society gossip. Sometimes during these conversations my mother would attempt to voice her opinion. “Did I hear anyone ask you, Cathy?” Grandfather would say.

Even as a child, I knew Grandfather was a powerful man. He had money, friends, influence. I felt drawn to his charisma and strength, yet repelled by his pompous disregard for others. Who was this man, this mysterious enigma, and why he did have such influence on me?

Benjamin B. Anthony was born July 24, 1926, in Springfield, Illinois, the youngest of three children. His mother was a housewife; his father ran a used furniture store. Ben wrote for the high school newspaper and played football. During the Depression, the family’s business struggled, causing animosity between his father and mother, a discontented Midwestern woman who dreamed of fortune. The tension was so high that Ben was forced to have two Bar Mitzvahs, one at his father’s orthodox synagogue, and another at his mother’s reform temple. At 18, during World War II, Ben enlisted in the Air Force. He was headed to Europe when news of the war’s end reached his platoon. With help from the G.I. Bill, he enrolled at the University of Chicago, where he studied English and history.

In 1948, Ben moved to New York City to make a name for himself. He married his college girlfriend, Janice, and they moved to the Upper West Side. In 1954, my mother, Cathy, was born; three years later, my uncle, Chris. Grandfather became a journalist, writing articles for Look, Life, and Esquire. By the end of the 1960s, Ben had made a name in journalism. Yet, as a Midwestern Jew from a lower-middle-class family, he still felt like an outsider.

In 1968, Grandfather unexpectedly divorced my grandmother to focus on his career. Janice, devastated, fell into a deep depression. My mom and uncle would go months without seeing their father. That same year, Grandfather was appointed press secretary for the mayor of New York, a position that thrust him into the public eye. He began to travel in more elite social circles. At one of these gatherings, he met Melanie. She was from an impressive family, related to the governor of New York and oil magnates. She was his ticket to the highest ranks of society, and he, a Jewish liberal, was sure to anger her family. In 1973, she would become his second wife.

By leaving his wife of 20 years and two children to marry a powerful socialite, help her raise her children, and make a name for himself in politics, Ben hurt a lot of people. He neglected to be the loving father my mother and uncle desperately needed. Even when he spent time with them, he would challenge their hopes and degrade their character. When my mother expressed her interest in becoming a writer, he replied simply, “You’ll never make it, Cathy. You just don’t have the smarts.”

In the ’70s and ’80s, Grandfather’s career took off. He served as the chief editor of the Village Voice and New York Magazine, the president of the WNYC Communications Group that controls New York’s public television and radio stations, and director of the United Nations Association. As he became more well known and began achieving his goals, Grandfather no longer needed Melanie, and he divorced her.

Two years later, he married Harriet. A wealthy psychiatrist, she was controlling and anxious. She despised anyone related to my grandfather’s past and made it her mission to isolate him from former friends. Grandfather’s interactions with his children became even more sporadic. After retiring in the early ’90s, Ben spent most of his time hosting luncheons at the Century Club and writing his memoirs. And so, four times per year, I would find myself in his luxurious Park Avenue apartment, seated at a table across from his third and most unpleasant wife, utterly mystified by his world.

Yet, I looked up to my grandfather. I heard stories my entire life of his many achievements. People I met would look at me differently when they found out I was Ben Anthony’s grandson. Even as a young boy, I knew that my grandfather somehow added to my worth. I was Jacob Anthony, grandson of an esteemed member of New York society. Major parts of my identity were formed simply by being related to him. For most of my life, I blindly idealized my grandfather. He had done what I so passionately wished to do. He broke out of the middle class and made something of himself. My grandfather has been, in a nuanced, convoluted way, my major inspiration.

In 2008, Grandfather was diagnosed with dementia, and Harriet sequestered him in their Upstate New York country home. There, he lost all contact with the outside world. He spent his days reading The New York Times in his wheelchair and arguing politics with his right-wing Polish housekeeper. Then, in the winter of 2012, Harriet died quietly and mysteriously in the apartment she had once shared with my grandfather. Reading her will, we realized she had craftily siphoned off his money, leaving him penniless. Her children received everything my grandfather had worked for. The only thing Harriet had left for our family was Benjamin Anthony himself.

Grandfather died last summer in a hospital bed, in a cold, federally subsidized home for the elderly. My mother managed to visit him almost every day. By the time of his passing, he had lost cognitive ability, couldn’t speak, was indigent and immobile. Nothing was left except some pictures and books. Everything he had worked so hard for – the money, the connections, the notoriety – had melted away until all that remained, standing next to his hospital bed, was the very thing he had shunned during his life: his family.

My mother called me that morning. “Jake,” she said through tears, “your grandfather passed away last night.”

“Oh,” I responded numbly. We reminisced for a while, and then I sat on the edge of my bed, searching for some sadness. I should be crying, I thought, but I couldn’t muster up a single tear. I simply wasn’t sad that he was gone. At first, I felt guilty about this, but then I realized that what was so amazing about Grandfather – his work, his achievements, the opulence of his life – wasn’t gone. His legacy remained intact. All that had left this earth was the husk of a man with whom I had no meaningful relationship. The parts of him that were important to me were still very much alive.

Some day I hope to live in a beautiful New York City apartment complete with mahogany tables and velvet couches. I hope to have a butler and a maid, to dine with interesting people, and to make an impact. My grandfather’s legacy gives me hope that I can pursue these dreams and become the person I want to be.

I understand now that I want to do the things he did, but I don’t ever want to be the person he was. He disregarded his family and those who loved him, he looked down his nose at those he deemed less successful or talented, and, though I had dinner with him four times a year, I don’t think he ever felt an emotional connection to his only grandson. I can’t recall a single meaningful conversation or one nice thing he said to me.

In the end, Grandfather’s mistakes caught up with him. He died just as he was born, poor. He never understood that what he had – both at the beginning and at the end of his life – that was so special and important about his existence, wasn’t money; it was family.