Whatever is the closest match and doesn't draw your eye to the area will be best. If both shades are equally close, then usually lighter is better. If the choice is between "significantly lighter" and "slightly darker", it will probably be better to go with slightly darker. It may also depend on the lighting (dimmer lighting is more forgiving).
When using your color chart, it will be best to view the swatches against the current color using the brightest natural light available (open any curtains or blinds, try to do this at midday). If you tape the closest colors to your wall and stand back, you should be able to get a better visual for contrast or lack thereof.
Some customers have also had great results by mixing colors to get a closer match.
