It would probably be a good idea to first read up on how claim rules work. I've found the following technet articles quite helpful:
AD FS 2.0 Claims Rule Language Primer
Understanding Claim Rule Language in AD FS 2.0 & Higher
The short version is that you end up stringing together various claim rules that "store" query data and then tweak/filter the data before you actually "issue" the claim with the resulting groups.
Here's an example that we use in our environment. For this particular relying party, we wanted to return all group memberships beginning with "myapp." including nested groups.
- Rule 1: Fetch the user's DN
c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname", Issuer == "AD AUTHORITY"] => add(store = "Active Directory", types = ("http://contoso.com/UserDN"), query = ";distinguishedName;{0}", param = c.Value);
- Rule 2: Fetch all nested group CNs using member attribute
c1:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname", Issuer == "AD AUTHORITY"]
&& c2:[Type == "http://contoso.com/UserDN"]
=> add(store = "Active Directory", types = ("http://schemas.xmlsoap.org/claims/Group"), query = "(member:1.2.840.113556.1.4.1941:={1});cn;{0}", param = c1.Value, param = c2.Value);
- Rule 3: Filter the resulting groups using Regex
c:[Type == "http://schemas.xmlsoap.org/claims/Group", Value =~ "(?i)^myapp\."]
=> issue(claim = c);