Best practices - Using URL properties to whitelist web objects

URL properties, such as URL, URL.Host, URL.Host.BelongsToDomains, and others, can be used in the criteria of rules to whitelist web objects.

When a web object is whitelisted, users are allowed to access it, for example, to view a web page or download a file. Whitelisting rules are inserted into appropriate rule sets within the rule set system of Web Gateway. They usually stop further rule processing with regard to the current request for accessing a web object to prevent other rules from blocking this access.

Different URL properties can be used for different kinds of whitelisting. To allow access to an individual web object, for example, to ensure users can download a particular file, the URL property is best used together with a list that contains the full URL for this file.

The following examples explain which URL properties are best used for different kinds of whitelisting and how to do it.

In addition to this, some tips and examples are given regarding the:

  • Values that different URLs are set to when a sample URL is processed that has been sent to Web Gateway in a request for web access
  • Use of the two operators is in list and matches in list in the criteria of a rule
  • Good and bad entries in the lists that are used with different URL properties

Whitelisting individual web objects – URL

Goal

Allow users to access individual web objects.

For example, download the file Stinger.exe, which can be accessed using the URL http://download.mcafee.com/products/mcafee-avert/Stinger/Stinger.exe.

How to do it Use the URL string property with a list of full URLs in the criteria of a rule.

The rule could, for example, be configured as follows:

URL is in list URLWhiteList –> Stop Rule Set

If you add the URL http://download.mcafee.com/products/mcafee-avert/Stinger/Stinger.exe to the list URLWhiteList, the file Stinger.exe is whitelisted when the rule is processed.

Note:

In a similar way, you can block access to the file using the following rule from the default URL Filtering rule set:

URL matches in list URLBlockList –> Block

If you add the URL in question to the list URLBlockList, the file is blocked when the rule is processed.

If the matches in list operator is used instead of is in list, expressions containing wildcards can be entered into the list that is used by the property. The property can then also be used to whitelist multiple web objects.

However, if all web objects provided by a particular host should be whitelisted, this can be achieved more easily using the URL.Host property.

Whitelisting hosts – URL.Host

Goal

Allow users to access the web objects that are provided on particular hosts.

For example, download the file Stinger.exe or any other file that is provided on the host download.mcafee.com.

How to do it

Use the URL.Host string property with a list for host names in the criteria of a rule.

A rule that the URL.Host property is used in could, for example, be configured as follows:

URL.Host is in list HostWhiteList –> Stop Rule Set

If you add the host download.mcafee.com to the list HostWhiteList, all web objects that are provided by this host are whitelisted when the rule is processed.

If the matches in list operator is used instead of is in list, expressions containing wildcards can be entered into the list that is used by the property. The property can then also be used to whitelist multiple hosts.

However, if all hosts within a particular domain should be whitelisted, this can be achieved more easily using the URL.Host.BelongsToDomains property.

Whitelisting domains – URL.Host.BelongsToDomains

Goal

Allow users to access the web objects that are provided within particular domains.

For example, download the file Stinger.exe and any other file that is provided by the host download.mcafee.com, as well as any other downloadable file provided by any other host within the domain mcafee.com.

How to do it

Use the URL.Host:BelongsToDomains Boolean property with a list of domain names in the criteria of a rule.

The rule could, for example, be configured as follows:

URL.Host.BelongsToDomains("Domain List") equals true –> Stop Rule Set

If you add the domain mcafee.com to the list Domain List, all web objects within this domain are whitelisted when the rule is processed.

The list Domain List is configured as a parameter of the URL.Host:BelongsToDomains property, which is of the Boolean type.

When, for example, the URL http://download.mcafee.com/products/mcafee-avert/Stinger/Stinger.exe is processed, the value of the property (true or false) depends on whether the mcafee.com domain has been entered into the list Domain List or not.

The following example shows which entries in the list Domain List lead to a match when the property is used for whitelisting:

mcafee.com

dell.com

k12.ga.us

twitter.com

xxx

Then the criteria:

URL.Host.BelongsToDomains("Domain List") equals true

matches for the following URLs:

https://contentsecurity.mcafee.com

https://my.mcafee.com

http://my.support.dell.com

http://www.dekalb.k12.ga.us

http://twitter.com

http://www.twitter.com

any.site.xxx

but not for:

https://www.mymcafee.com

http://www.treasury.ga.us

http://malicioustwitter.com

Using the URL.Host.BelongsToDomains property also avoids the effort of creating more complicated solutions to achieve the same, for example:

  • Using two entries in a list of wildcard expressions, such as:

    twitter.com

    *twitter.com

  • Using a single, complex entry in a list of wildcard expressions, such as:

    regex((.*\.|.?)twitter\.com)

Property values for a sample URL

When the sample URL http://www.mcafee.com/us/products/web-gateway.aspx is processed, the URL properties below are set to different values as follows.

Property Value for sample URL
URL http://www.mcafee.com/us/products/web-gateway.aspx
URL.Host www.mcafee.com
URL.Host.BelongsToDomain

true or false

In the list that is configured as a parameter of this property, the following would have to be entered for the domain: mcafee.com.

URL.FileName web-gateway.aspx
URL.Path /us/products/web-gateway.aspx
URL.Protocol http

Use of operators for different types of matches

It makes an important difference whether the is in list or matches in list operator is used in the criteria of a rule.

Operator Description
is in list

Requires an exact string match.

If there are wildcard characters in a list entry, they are interpreted as literal strings.

matches in list

Allows and evaluates wildcards in list entries.

Good and bad entries in lists for URL properties

Entries in the lists that are used by the different URL properties can be good are bad, according to how they fit in with the intended use of a property. The following are examples of good and bad list entries.

URL property Good and bad list entries
URL with is in list operator

Good

http://www.mcafee.com/us/products/web-gateway.aspx

The full URL is entered, as it is required for this property. No wildcards are specified, as these are not evaluated when the is in list operator is used.

Bad

www.mcafee.com/us/products/web-gateway.aspx

The entry does not specify the full URL, as the protocol information, http://, is not included.

URL with matches in list operator

Good

http://www.mcafee.com/*

This entry contains a wildcard for allowing access to any web object provided by the host www.mcafee.com, which is appropriate when the matches in list operator is used.

Note: The entry will not match for http://mcafee.com/.

regex(htt(p|ps)://(.*\.|\.?)mcafee.com(\/.*|\/?))

This entry is more complex, as it uses regular expressions. When matched, it allows access, under the HTTP or HTTPS protocol, to any web object within the domain mcafee.com and its subdomains.

regex(htt(p|ps)://(.*\.|\.?)mcafee.(com|co.us)(\/.*|\/?))

This entry is the same as the previous, but shows how other top-level domains, such as .com or .co.us, can be whitelisted.

Bad

*.mcafee.com*

The entry does not exclude unwanted matches, for example, a match for the URL http://malicious-download-site.cc/malicious-file.exe?url= www.mcafee.com.

URL.Host with is in list operator

Good

www.mcafee.com

A host name is entered, which fits in with the intended use for this property. No wildcards are specified, which is appropriate when the is in list operator is used.

Bad

mcafee.com

The entry specifies a domain name (mcafee.com), whereas the value of the property is a host name (www.mcafee.com if, for example, the URL http://www.mcafee.com/us/products/web-gateway.aspx is processed).

No match will be produced this way.

*.mcafee.com

The entry contains a wildcard, which is not evaluated when the is in list operator is used.

*.mcafee.com/us*

The entry includes path information (/us), which does not fit in with the intended use of the property.

In addition to this, a wildcard is specified, which is not evaluated when the is in list operator is used.

URL.Host with matches in list operator

Good

*.mcafee.com

The entry matches for on any host within the domain mcafee.com, but not for mcafee.com itself.

regex((.*\.|\.?)mcafee.com)

The entry uses regular expressions to whitelist the domain mcafee.com and any of the hosts within it.

Bad

*.mcafee.com*

The entry does not exclude unwanted matches, for example, http://www.mcafee.com .malicious-download-site.cc/.

*.mcafee.com/us*

The entry includes path information (/us), which does not fit in with the intended use of the property.

URL.HostBelongsToDomains

Good

mcafee.com entered in the list Domain List, which is configured as a parameter of the property.

The entry matches for the mcafee.com domain and all hosts within it, for example, www.mcafee.com or secure.mcafee.com.

www.mcafee.com

The entry does not specify a domain, but is valid. It only whitelists the host www.mcafee.com.

Note:

This can also be achieved by adding the entry to a list for the URL.Host property used together with the is in list operator.

Bad

*.mcafee.com

The entry contains a wildcard, which does not fit in with the intended use of the property.

The property was rather developed to avoid the effort of using wildcards in list entries. Instead it requires an exact domain match, for example, a match for mcafee.com.