Customizing IPSec, Part 1

By: Brien M. Posey, MCSE

 Over the past several months, I’ve written a lot about how using the IPSec protocol can increase your network’s security by encrypting data as it flows across the network. However, I’ve always thought that nothing can be considered to be truly secure if it’s configured by using a text book model. After all, the hackers read the books too, and know all of the standard configuration methods. Therefore, if you want to implement good security, you need to use a custom security model. In this article series, I’ll explain how you can go about customizing the IPSec protocol.

 Before we get started, I should point out that there are many different ways to implement an IPSec policy. One of the more common methods is to implement an IPSec policy as a part of a domain’s security policy. This is the security model that I’ll be using in my examples. Therefore, assuming that this is the configuration  that you’ve chosen to use, you may modify your IPSec policy by selecting the Domain Security Policy command from the Administrative Tools menu. When you do, a Microsoft Management Console session will load that displays the current domain’s security policy.

 When the domain security policy loads, navigate through the console tree to Windows Settings | Security Settings | Public Key Policies | IP Security Policies on Active Directory. When you’ve selected this container, the column at the right will display all of the existing IPSec policies. You can tell which policy is the active one by looking at the Policy Assigned column. If the policy has been assigned then it is active.

 To customize an IPSec policy, right click on the policy and select the Properties command from the resulting context menu. When you do, you’ll see the policy’s properties sheet. The properties sheet is divided into two tabs, the Rules tab and the General tab. Let’s begin by looking at the Rules tab.

 The Rules tab controls the basic behavior of the policy. This tab works by making use of an IP filter list. The idea is that the policy can be configured to detect various types of IP traffic and to respond to such traffic by using a predetermined behavior. For example, if you look at the Server (Request Security) policy (this is a built in policy), you’ll see that the first rule in the list tells the policy to request security for all IP traffic, and to use Kerberos for the authentication method.

 If you look at the bottom of the Rules tab, you’ll see that the Use Add Wizard check box is selected by default. This check box tells Windows to use the Add Rules Wizard any time that you want to add a rule, rather than making you go through the process manually.

 To add a rule, click the Add button. When you do, Windows will launch the Add Rules Wizard. Click the Next button to skip the introductory screen. The first real screen that you’ll see deals with tunneling. As you may know, tunneling is the process of creating a virtual private network that passes through a public network such as the Internet. Because of the insecure nature of the Internet, it goes without saying that you’ll need to secure any communications that pass through a tunnel. Therefore, you can use the wizard to specify a tunnel’s end point if you’re working with a virtual private network. If you’re dealing with a local connection instead or a connection over a secure medium, simply tell the wizard that the rule doesn’t apply to a tunnel and click Next to move on.

 The wizard’s next screen asks if the rule will be applied to local network connections, a remote network connection (such as the type used by a dial up connection), or all types of connections. Make your selection and click Next.

 At this point, you’ll see a wizard screen that asks you what authentication method that you want to use. Although Kerberos is the default method, you can choose to use a digital certificate that has been issued by a certificate server, or you can use a preshared key. Select your authentication method and click Next to continue.

 Now, you’ll see a screen that asks you to select your IP filter. You can select All IP Traffic or All ICMP Traffic. If neither of these options meet your needs, you can click the Add button to launch a wizard which allows you to select other types of IP traffic, such as traffic between two specific IP addresses, DNS related traffic, or traffic flowing between two subnets just to name a few. Make your selection and click Next to continue.

 You’ll now be asked to select which filter action that you wish to apply to the rule. By default you can permit the traffic, request security, or require security. However, you may also click the Add button to launch a wizard that allows you to create a custom filter action. Make your selection and click Next to continue. You’ll now see a summary screen. Click Finish to implement the new rule.

 Now that you know the basics of creating a rule, it’s time to build on them. In Part 2, I’ll begin discussing custom filters and filter actions.

If you've found this article helpful then please consider making a donation to help with the cost of keeping this site going. To make a donation, please click on the PayPal link below.