Can someone please explain, in simple language, the meaning of opt-out flag in the NSEC3 RR. I did read RFC 5155 and understand nothing.
NSEC3 records can be created in one of two different ways: for all delegations, or just for secure delegations. The opt-out flag tells you which method is in use. The reason for the flag is to allow very large zones primarily made up of delegations (i.e. TLDs) to not have to generate NSEC3 records and their corresponding RRSIG records for every single delegation. This is important a TLDs like .com with ~100,000,000 delegations; signing every single delegation would make an extremely massive zone file!
There is no difference between secure delegations when the opt-out bit is set or not.
For insecure delegations without the opt-out bit set, you get the delegating NS records plus the NSEC3 and RRSIG for that delegation. Here is an example for house.gov, which is an insecure delegation (no DS record provided) in the .gov zone, which does not use the opt-out bit:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9019 ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 3 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1472 ;; QUESTION SECTION: ;www.house.gov. IN A ;; AUTHORITY SECTION: house.gov. 86400 IN NS chyron.house.gov. house.gov. 86400 IN NS mercury.house.gov. 56j5jp60kq681hl5vtv287p7s16spmcp.gov. 86400 IN NSEC3 1 0 8 4C44934802D3 56K48N2V1IV9H7F4HJ05N62QI9C29JV2 NS 56j5jp60kq681hl5vtv287p7s16spmcp.gov. 86400 IN RRSIG NSEC3 7 2 86400 20120516040019 20120511040019 35464 gov. OlOx3rG7ShAptVt1XTqcXLOtInxEqyfg1b6+vBlWiqSBZ8pkfk/IOFOm 49lbKkrjY2ibw98GaMdjUYCUUDKOX+eTe+HTfxginIfJ3FWOxB+TPFn1 /UEu4QAgEkWdgpT6NbCge9vWnhSxTCYNxTolVuhWq+sp59zodbAMERVi rIdFyoNpX1zijU1tjm9j8a+jFeN7tjf2fgzJQPpk/qNMgmgfp2GerPUX 5kVkhjoXPgRkLmy2W5PwbgWP4zOTRFuLxz0PsRfoqLUHYYEXPMQ0jimW ESl1LDRnHjdQDTD1qYPBCiVNxufaewZMGhTwP901CH3FLr6Gku7ptYkD 5ukEFQ==
If we calculate the NSEC3 hash given a salt of 4C44934802D3 and 8 extra iterations, it is an exact match due to the fact that an NSEC3 records is generated for every delegation:
$ nsec3.py -i 8 -s 4C44934802D3 house.gov 56J5JP60KQ681HL5VTV287P7S16SPMCP house.gov
For insecure delegations with the opt-out bit set, you will not get an NSEC3 record which provides a list of resource record types available for that domain. Instead you get a "closest encloser proof," as described in section 7.2.1 of RFC 5155:
This example is an insecure delegation within the .com zone, which uses the opt-out flag:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48336 ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 9, ADDITIONAL: 6 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 512 ;; QUESTION SECTION: ;www.yahoo.com. IN A ;; AUTHORITY SECTION: yahoo.com. 172800 IN NS ns1.yahoo.com. yahoo.com. 172800 IN NS ns5.yahoo.com. yahoo.com. 172800 IN NS ns2.yahoo.com. yahoo.com. 172800 IN NS ns3.yahoo.com. yahoo.com. 172800 IN NS ns4.yahoo.com. CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0RFQAOES8CTVNVNH4G6Q85NOQAQ8I9 NS SOA RRSIG DNSKEY NSEC3PARAM CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20120515041703 20120508030703 23339 com. RiL6MRN/fPVBjpyANDKurwSgdwgkdsRdB4ADWK7YTJeY2KNnBpjOX+FT +2a/XZR2ylP+G47L8k+DrJCHuAkr1wOYcOj7goiqErDwu+Cm5HZosAL2 EyRqNOHHgTDXlG6PGgyEe2DO0jWgmkyYX7+o0jpYP0m6QNDaRuf166np nkA= GP1945PGQIOH4O61BM3RUL2EVN04SPIA.com. 86400 IN NSEC3 1 1 0 - GPLVOUV0V27L8DPOOBNLQU1VHFRMMPUT NS DS RRSIG GP1945PGQIOH4O61BM3RUL2EVN04SPIA.com. 86400 IN RRSIG NSEC3 8 2 86400 20120518085726 20120511074726 23339 com. VmtH/BYw8H98FJM7YLxLIG0cfReERp5eNh3+bCu7EfWgSuWXn6OXdd4b rIMloxDXe9v/fdyd7RqwDiNLMPMhp8wRJOhKcqT0MczHFEUzy0SnXM3d SABY5d1AJr8YJNL+ZOgbiT445gn7HBET3OL+G5MfZPti+yhBnUvGlPYx UQ8=
The first NSEC3 record is for the com domain itself (it's pretty easy to spot given that it has SOA, DNSKEY, and NSEC3PARAM in the list of RR types. It is the closest provable encloser.
The second NSEC3 record is to "cover" yahoo.com. This is to prove that an NSEC3 record for yahoo.com does not exist, as it is not a secure delegation.
Here are the calculated hashes for comparison:
$ nsec3.py com yahoo.com CK0POJMG874LJREF7EFN8430QVIT8BSM com GPIOV963K81D6QM6IOTOUFUAPRDA6K3V yahoo.com
Unless you are running a TLD zone or other zone with a huge number of delegations, it is not necessary to use the opt-out flag.