Bush said any country found harboring Taliban or AQ will be swiftly invaded. Then why Pakistan not invaded?

Bush said in 2002, "if anyone is found harboring Taliban or AQ, they will be swiftly attacked by our military" Now Pakistan is found harboring Bin Laden. So where is the "swift attack an invasion" against Pakistan? How come invasion of Pakistan proving such a tough decision for US?