The Federal Reserve has a long list of worries about the unexpected consequences of its quantitative easing (QE) program. There’s the risk that all the bond-buying is quietly stoking bubbles in disparate asset classes, and the lingering risk of a future run-up in inflation, and even the political risk of balance sheet losses down the road. But above all, what really keeps Fed policymakers up at night might be their most immediate concern: that investors will overreact when they finally decide to reduce the $85-billion monthly pace of asset purchases.

In perhaps the starkest words yet from someone in Fed Chairman Ben Bernanke’s inner circle, here’s what New York Fed President William Dudley said about this in a speech to the Japan Society in New York on Tuesday: