The arp table lists for each interface, which MAC is (directly or indirectly) reachable via that interface. That may be a single directly connected device per interface.
If you have several switches, you should ignore the "trunk" lines between switches in your counting: Every MAC reachable indirectly via the connected switch wll be listed there.
If you even have redundant links between your switches, a specific MAC may be listed only with one of several possible trunk links, depending on which link is currently considered best.
Finally, note that a MAC may not be listed if no traffic with that device has passed the switch yet; typically, at least at boot time they will talk something to the net (e.g. DHCP) and thus "tell" their MAC.
In summary: For interfaces with directly connected devices, your guess should be fine. Just make sure to drop info from links to wother switches.
EDIT: While explaining the arp table in general, I missed the point of your final question: For a router the visibility of a MAC may be less prelevant: Since there will typically be a switch between the router and any other device on the connected LANs, a packet from that device will reach the router only if the device taklks to the router specifically (e.g. in order to reach another net via that router) or vice versa. Thus a router may never learn MACs of devices in the LAN that never bother to talk to any non-LAN host.