What’s the circuit attached? There’s “charge injection” where sampling an input pulls some current for every sample, as the cap internal to the chip is charged for sampling. The more you sample, the lower the value will be if your input is high impedance.
One way to address this is to put a capacitor (eg 100nF) on the ADC input to supply these current pulses, but it depends on what you’re feeding into the device… hence the first question.
Would like to solve the issue though; an opamp should be able to deal with the current flowing into the ADC. Can you try adding 100nF at the pin input?