A hypothetical network...
My closely related questions.
If copy a file from computer A to B (same LAN segment) do all the packets need to travel first through the firewall to get to it? In other words, despite being on same 10Gb/s switch does traffic travel at slower 1 Gb/s port speed of firewall?
If I copy from computer A to X (i.e. different segments and different IP ranges) do all the packets need to travel first through the firewall to get to it? In other words, despite being on same 10Gb/s switch does traffic travel at slower 1 Gb/s port speed of firewall?
TLDR:
Longer version:
Some assumptions:
Usually switches work on L2 level (lets omit L3 level "smart" switches, which are actually routers in common terminology).
When switch gets packet - it inspects it source MAC address and preserves this information in internal table, so it has association MAC <=> physical port.
Next step is to forward received package to destination, switch uses destination MAC for that, if that address already in internal table - packet only forwarded to already known physical port, if not - to all physical ports (note: packet is forwarded as is via specialized hardware chip without modification, thus it is very fast).
As you see - there is no any kind of IP address involved in this process, so, to make things actually work sender's host first converts destination IP address into MAC and then send package via wire to switch.
To perform this conversion sender checks whether destination IP is in the same subnet as its own address, this is where network mask comes into play. If network mask is 255.255.255.0 (as it looks like from your example) - then for addresses in the same subnet (i.e. first 3 numbers in address are the same) sender will obtain real MAC address (via ARP) of destination and send packet directly to that MAC address.
So, answer to first question - source (A) and destination (B) IPs are on the same subnet (192.168.100.0/24), so packet will contain actual sender/receiver MAC addresses, so switch will forward it directly to correct physical port at 10Gbs line speed (assuming that all hosts' ports are 10GBs, switch's ports are 10GBs and switch's internal table is already populated, which usually is).
In second case networks are different, sender cannot get real MAC address of destination, the only thing it can use - default gateway, which is your pfsense firewall, so it sends packet to firewall's MAC address (MAC address of LAN1 port), thus switch will forward it only to that physical port (pfsense LAN1) at 1Gbs speed.
Then, pfsense will perform routing (software will read full packet into memory, inspect its IP, apply internal rules, etc), in simple case - as soon as it knows physical port for subnet 192.168.101.0/24, thus it can obtain real destination's MAC address and firewall will send new packet from LAN2 physical port to switch, this new packet will have source MAC set to pfsense LAN2, destination - real MAC address of X. Switch will then happily forward it to real physical port where X is connected.
So, in second case you have much slower communication, not only because router's 1GBs ports, and that packet has to travel twice via them (switch->LAN1->LAN2->switch), but there is also software involved which is much slower in comparison to hardware switching chip.
If pfsense doesn't know to which physical port some network is connected (all other addresses except your local subnets, basically - internet) - then it will use WAN port to route traffic to in similar fashion as A->X, ie it will send packet to ISP's modem (which is default gateway for pfsense) and then modem will perform routing.