If there’s one theme that ran through U.S. Supreme Court Justice Antonin Scalia’s interview Monday with Reuters Editor-in-Chief Stephen Adler, it’s that words matter. Time and time again, Scalia and Bryan Garner, the co-author with Scalia of the book Reading Law: The Interpretation of Legal Texts, endorsed originalism and textualism, doctrines that demand judges stick to interpreting the words in front of them rather than attempting to divine legislative intent or (heaven forbid!) imposing their own policy agendas. According to Garner and Scalia, textualism is a sure-footed guide, regardless of where it leads.

“A textualist will frequently end up with — an uncomfortable result. With a result that feels bad,” Garner said, according to a transcript of the interview, which he also participated in. “That’s the funny thing. The judges who are not textualists will essentially always do what they consider to be the better policy. But textualists will frequently decide cases that they think, ‘Wow, it’s a shame I have to do this.’”

If words alone must determine outcome, let’s take a look at what Scalia had to say when Adler asked a question posed by an audience member who wanted to hear the justice’s opinion on term limits for judges. Scalia called term limits “a solution without a problem,” arguing that, in his experience, William Douglas is the only justice who stayed on the Supreme Court too long. The question also led Scalia to muse, however, on how judicial salaries affect the composition of the federal judiciary. “The salaries of federal judges are so low that you’re not getting the best lawyers anyway,” Scalia said. “You’re [not] getting the, the best private lawyers. You may be getting good people, but they’re people who have been an assistant U.S. attorney, then they’re … you know, a minor state judge, then a bankruptcy judge, and then a magistrate judge. And, you know, they finally get appointed to a federal district court. A huge percentage of our federal judges now have never practiced law privately.”

I heard (and read) Scalia’s comments as a slap at longtime public servants who ascend to the federal bench, a group that, in New York’s federal trial courts, includes such well-regarded judges as John Gleeson and Barbara Jones. Scalia creates a dichotomy between them and private practitioners and suggests that while career public servants may be “good,” they’re not “the best lawyers.” (I called Gleeson and Jones, but neither would comment on Scalia’s remark.) That seemed to me to be an unfair generalization — I’m sure we all know plenty of brilliant federal prosecutors and state court judges, and just as many dim-bulb private practitioners — and a distressing vote of no confidence in the federal judiciary by one of its most prominent members.

But when I called around to speak with some current and retired federal judges about Scalia’s comments, three of the four who spoke on the record said that while the justice’s words may not have been artful, he has identified a real issue: diversity on the federal bench. “It is a significant financial sacrifice to take a judicial position,” said Vaughn Walker, a retired former U.S. chief judge who now has an arbitration business in San Francisco. “The judiciary is poorer for having fewer lawyers who come from the private sector.” A big percentage of the cases a federal trial judge oversees, Walker said, involve business disputes. Judges who have represented clients in such disputes bring valuable experience to the bench. “I would not say that lawyers coming out of government practice are not as smart as lawyers in private practice,” said Walker. “But you learn things from having done things.”