MaxLinear has terminated an agreement to buy flash memory maker Silicon Motion for $3.8 billion.