Quick answer
How a person treats an animal is treated here as a moral indicator. The verse assumes you can tell something real about someone by watching that.
Read both lines
The word behind "life" is nephesh, the same word the book uses for a human life, and applying it to livestock is the verse's quiet argument: the animal has one too, and the righteous man knows it. The Law took the same view, forbidding a farmer to muzzle an ox while it worked. The second line is the sharper half, because it does not say the wicked are cruel. It says their mercies are, which means that even the kindness of a hard person has something wrong in it. Where one man's decency extends past the boundary of what can complain, another man's kindness does not reach the end of itself.
Related: Deuteronomy 25:4 · Proverbs 27:23 · Matthew 10:29
New King James Version®, Copyright © 1982, Thomas Nelson. All rights reserved. Full copyright information. Scripture provided by API.Bible.
Go Deeper
If this verse has been used to put weight on you that it was never carrying, ask about it directly.