no, it’s because you didn’t remove the server package postgresql-9.4 in the first place. Otherwise the package would be listed as “The following packages will be installed” or something similar.
Yes, you can (and probably should) remove the client, too, but you really didn’t remove the old server package.