what is the optimize argument in pcap_compile( , , , int optimize, ) in npcap library?

pcap_compile(pcap, &fcode, "tcp", 0, PCAP_NETMASK_UNKNOWN)

Here I have set to 0 and it is working, but I want to know what it does. I am trying to filter tcp packets in a pcap file.

And does pcap_setfilter() reconstructs pcap file into a given fcode?

Well, as stated in the pcap_compile man page,

optimize controls whether optimization on the resulting code is performed.

OK, but of course you might be wondering, "What does that really mean?" To answer that question, I think it's best to provide an example. Consider the following capture filter: icmp or udp port 53 or bootpc. If you run tcpdump, passing it that filter expression, along with the -d and -O options, it will generate a non-optimized BPF program containing 46 instructions that looks like this:

tcpdump -d -O "icmp or udp port 53 or bootpc"

(000) ldh      [12]
(001) jeq      #0x800           jt 2    jf 4
(002) ldb      [23]
(003) jeq      #0x1             jt 44   jf 4
(004) ldh      [12]
(005) jeq      #0x86dd          jt 6    jf 12
(006) ldb      [20]
(007) jeq      #0x11            jt 8    jf 12
(008) ldh      [54]
(009) jeq      #0x35            jt 44   jf 10
(010) ldh      [56]
(011) jeq      #0x35            jt 44   jf 12
(012) ldh      [12]
(013) jeq      #0x800           jt 14   jf 24
(014) ldb      [23]
(015) jeq      #0x11            jt 16   jf 24
(016) ldh      [20]
(017) jset     #0x1fff          jt 24   jf 18
(018) ldxb     4*([14]&0xf)
(019) ldh      [x + 14]
(020) jeq      #0x35            jt 44   jf 21
(021) ldxb     4*([14]&0xf)
(022) ldh      [x + 16]
(023) jeq      #0x35            jt 44   jf 24
(024) ldh      [12]
(025) jeq      #0x86dd          jt 26   jf 32
(026) ldb      [20]
(027) jeq      #0x11            jt 28   jf 32
(028) ldh      [54]
(029) jeq      #0x44            jt 44   jf 30
(030) ldh      [56]
(031) jeq      #0x44            jt 44   jf 32
(032) ldh      [12]
(033) jeq      #0x800           jt 34   jf 45
(034) ldb      [23]
(035) jeq      #0x11            jt 36   jf 45
(036) ldh      [20]
(037) jset     #0x1fff          jt 45   jf 38
(038) ldxb     4*([14]&0xf)
(039) ldh      [x + 14]
(040) jeq      #0x44            jt 44   jf 41
(041) ldxb     4*([14]&0xf)
(042) ldh      [x + 16]
(043) jeq      #0x44            jt 44   jf 45
(044) ret      #262144
(045) ret      #0

And if you run that same tcpdump command, but without the -O option, thus enabling optimization (the default), then the resulting BPF program contains only 24 instructions that looks like this:

tcpdump -d "icmp or udp port 53 or bootpc"
(000) ldh      [12]
(001) jeq      #0x800           jt 2    jf 13
(002) ldb      [23]
(003) jeq      #0x1             jt 22   jf 4
(004) jeq      #0x11            jt 5    jf 23
(005) ldh      [20]
(006) jset     #0x1fff          jt 23   jf 7
(007) ldxb     4*([14]&0xf)
(008) ldh      [x + 14]
(009) jeq      #0x35            jt 22   jf 10
(010) jeq      #0x44            jt 22   jf 11
(011) ldh      [x + 16]
(012) jeq      #0x35            jt 22   jf 21
(013) jeq      #0x86dd          jt 14   jf 23
(014) ldb      [20]
(015) jeq      #0x11            jt 16   jf 23
(016) ldh      [54]
(017) jeq      #0x35            jt 22   jf 18
(018) jeq      #0x44            jt 22   jf 19
(019) ldh      [56]
(020) jeq      #0x35            jt 22   jf 21
(021) jeq      #0x44            jt 22   jf 23
(022) ret      #262144
(023) ret      #0

Both programs are functionally equivalent but the latter is going to be much more efficient, so in general enabling optimization should be preferred.

If you'd like even more information about optimization, then I would recommend visiting the bpfexam man page where you can even enter an arbitrary capture filter and examine the results.

Regarding pcap_setfilter, as the man page indicates, it is used to specify the filter program (such as those you can see above), which ultimately determines which packets are captured and which ones are discarded.