Requirements:

For dfd_keeper, the python/pf implementation, you will need:
OpenBSD or another OS with pf (http://www.openbsd.org/)
python 2.3 or later (http://www.python.org/)
Zope-Interface (http://www.zope.org) (?)
Twisted asynchronous networking framework 3.0.1+ (http://www.twistedmatrix.com/)
dfd_keeper module (http://www.lightconsulting.com/~travis/dfd_keeper/)
your own script which implements the rules you want (see keeper_example.py)

Use:

bash# ./keeper_example.py --test
bash$ nc localhost 8007
Your wish is my command.
dfd_keeper>show
nat on xl2 from xl1:network to any -> (xl2)

# Allow all loopback traffic.
pass quick on lo0

# Default deny.
block all

# Allow LAN to bomb out quickly.
block return on xl1

# Don't allow other networks to impersonate LAN.
antispoof quick for xl1

# Block leakage of LAN stuff to anywhere else.
block out log quick on ! xl1 to xl1:network

# Block hosts we have specified in both directions.
block in log quick on xl2 from [] to any
block out log quick on xl2 from any to []

# Allow firewall to talk to LAN.
pass out quick on xl1 from xl1 to xl1:network keep state

# Allow anything in from LAN that isn't destined to the LAN.
pass in quick on xl1 to ! xl1:network keep state allow-opts

# Allow LAN hosts to SSH into this box.
pass in quick on xl1 proto tcp from any to xl1 port ssh flags S/SA

# Allow connections out WAN, and randomize SEQ #s.
pass out quick on xl2 all modulate state allow-opts

It is done.
dfd_keeper>help
drop_state:
        Drop a particular state table entry.  Takes src and optional dst.
flush:
        Flush the state table.  This is done automatically.
sync:
        Synchronize the rules with pf.  This is done automatically.
show:
        This command shows the active rules to the client.
help:
        Show help to the user.  A command may be provided as an argument.
wan:
        Switches on/off connectivity with the Internet.
        For emergencies only!
block:
        block [add|del] host
        Block an IP from sending in data via WAN interface either direction.
        XXX: Assumes it is on the remote side of that interface.
It is done.
dfd_keeper>block add 1.2.3.4
It is done.
dfd_keeper>show
nat on xl2 from xl1:network to any -> (xl2)

# Allow all loopback traffic.
pass quick on lo0

# Default deny.
block all

# Allow LAN to bomb out quickly.
block return on xl1

# Don't allow other networks to impersonate LAN.
antispoof quick for xl1

# Block leakage of LAN stuff to anywhere else.
block out log quick on ! xl1 to xl1:network

# Block hosts we have specified in both directions.
block in log quick on xl2 from 1.2.3.4 to any
block out log quick on xl2 from any to 1.2.3.4

# Allow firewall to talk to LAN.
pass out quick on xl1 from xl1 to xl1:network keep state

# Allow anything in from LAN that isn't destined to the LAN.
pass in quick on xl1 to ! xl1:network keep state allow-opts

# Allow LAN hosts to SSH into this box.
pass in quick on xl1 proto tcp from any to xl1 port ssh flags S/SA

# Allow connections out WAN, and randomize SEQ #s.
pass out quick on xl2 all modulate state allow-opts

It is done.
dfd_keeper>block add 2.3.4.5
It is done.
dfd_keeper>show
nat on xl2 from xl1:network to any -> (xl2)

# Allow all loopback traffic.
pass quick on lo0

# Default deny.
block all

# Allow LAN to bomb out quickly.
block return on xl1

# Don't allow other networks to impersonate LAN.
antispoof quick for xl1

# Block leakage of LAN stuff to anywhere else.
block out log quick on ! xl1 to xl1:network

# Block hosts we have specified in both directions.
block in log quick on xl2 from { 1.2.3.4 2.3.4.5 } to any
block out log quick on xl2 from any to { 1.2.3.4 2.3.4.5 }

# Allow firewall to talk to LAN.
pass out quick on xl1 from xl1 to xl1:network keep state

# Allow anything in from LAN that isn't destined to the LAN.
pass in quick on xl1 to ! xl1:network keep state allow-opts

# Allow LAN hosts to SSH into this box.
pass in quick on xl1 proto tcp from any to xl1 port ssh flags S/SA

# Allow connections out WAN, and randomize SEQ #s.
pass out quick on xl2 all modulate state allow-opts

It is done.
dfd_keeper>exit
bash$ 

If you don't know what commands are available use the "help" command.

Note that the parsing is crude; you are expected to seperate commands and
arguments are supposed to be seperated by a single space.  Unfortunately,
there is no readline support (and python's readline library appears to be
designed for the interpreter, not other applications, and in any case it
would probably not work well with the Twisted framework).

TODO: Consider making a command-line client by using the python interpreter.
      Could it offer readline and be able to configure a remote daemon?

Overall Design:

The dfd_keeper project is build around two modules.  The dfd_keeper module is
everything you need to set up a static pf ruleset, which doesn't confer many
advantages over writing a regular pf.conf.  The next module is dfd_keeper_rt
which contains everything necessary to make your firewall dynamic (the rt
stands for real-time).  There are also two scripts; static_example, which
shows how to use dfd_keeper, and keeper_example, which shows how to set up
a whole firewall daemon.

The real-time module is built around the Twisted asynchronous framework.
The reason for this is that DFD may have to service multiple clients, and yet
manage one shared data structure (the firewall configuration), so you either
need multiprocessing, multithreading or asynchronous I/O.  I find the latter
to be the safest (fewest gotchas) and most elegant, when it can be used.  For
this case, there are very few blocking operations (perhaps the only one is
when sending the rules to pfctl, which may block on DNS lookups), so asynch
I/O seemed a natural fit.

The end-user is supposed to write a script which uses these two modules.
This script first sets up the rules, and then it sets up a user interface
class (called a command module) which defines methods that a user may call
(which I refer to as commands).  There is a direct relation between the
command name and the method name in the command_module, and I use python's
introspection to avoid having to register these commands in any sort of
lookup table.  Finally, the end-user script tells the reactor to start
running, and that's it.  The Twisted event loop takes over and takes care
of the flow of control.

There are a couple of object-oriented paradigms worth mentioning.  There is
top-level class called pf which encapsulates all of the classes specific to
the OpenBSD packet filter.  First is pf, an instance of which is used to
represent the current pf ruleset.  It is a subclass of enumerated_list, which
is a new container class that can be indexed by name (via square brackets,
such as pf["macros"]), but can be iterated over in a fixed order, which
corresponds to the pf.conf ordering required by OpenBSD.  Each section is
a rule_container object, which is basically an ordered list of rules.  Every
rule is an object.  While I originally allowed creation of free-standing
rules, I ended up creating a factory method make_rule within the
rule_container.

TODO: explain why you can't create free-standing rules any more.
TODO: rename command_module to command_object or command_class.

The end-user sees a command line, similar to a Unix shell.  By default,
the valid commands appear as methods in the command_module object, and
are distinguished from other functions by beginning with a prefix (by
default, this prefix is "cmd_").

TODO: Consider making non-commands begin with an underscore instead.

Originally I had designed the pf instance to completely contain the current
exact state of the pf rules.  In other words, to modify the rules from your
command_module, you had to ask pf to iterate through all the rules, and
provide a callback routine which gets called for each rule, in order.  Since
you can have a hierarchy of rule_containers, this was a depth-first search
and when the search invokes the callback, the ancestors of each rule as
passed as a variable-length list to the callback function.  The callback
then performs manipulations on selected rule objects.  The rules were marked
with "tags", which is usually just a string but could be any python object.
The tags were there so that your callback could figure out which rules it
wanted to modify.  Modification could include changing the text or changing
its active flag.  The idea of inactive rules (rules which do not appear in
the ruleset passed to pf) allows you to keep them as a placeholder for later
operations.

TODO: Allow a user to iterate through all pf rules using generators/iterators.

My first working script merely translated all of my pf.conf as rules,
including the comments which appear in the rules sent to pf.  I found this
to be redundant, so I extracted most of the comments into the python script.
Why send them to pfctl, only to be ignored?

The pf syntax allows you to create variables, called macros or lists, which
allowed you to refer to a string using a symbolic name.  For example, you
could use "$wan_if" instead of the literal interface name ("we1").  This
makes the rules easier to use.  However, it was requiring a lot of verbiage
to create rules for macros.  What I wanted to do was define these variables
in python, and automatically create pf macro rules based on the python
variables, basically importing them from python into your pf configuration.
So I created a method which would take a python variable as input and
produce the equivalent in pf syntax.  So to make the macro mentioned above,
you might type:

wan_if="we1"
pf["macro"].make_rule("wan_if=" + pf.python($wan_if))

As you can see, this is cumbersome.  At first I put all the variables in
a dictionary and looped through each one, running code as above.  However,
it occurred to me that it is not necessary to create symbolic names in pf,
as I could just use python variables in later rules.  For example, I could
write this:

wan_if="we1"
pf["filter"].make_rule("block in on " + pf.python($wan_if) + " proto tcp")

As a first refinement, I decided that instead of converting variables via
pf.python(), I could create python classes which overload their str()
method to do the conversion automatically (string concatenation with an
object implicitly calls str(object)).  So the code now looked like this:

wan_if = pf.macro("we1")
pf["filter"].make_rule("block in on " + $wan_if + " proto tcp")

Or, using python's printf-style operator:

pf["filter"].make_rule("block in on %s proto tcp" % $wan_if)

This is getting better, but I still don't have the economy of typing that is
in the original pf.conf syntax.  What I really wanted was interpolated strings,
like Perl's double-quoted string literals.  Fortunately, python 2.4 has a
string.Template class which does exactly that.  However, there is no port for
python 2.4 for OpenBSD yet, so I hacked up a quick Template class which does
variable substitution.  The Template.substitute() function takes a string
and a dictionary of name-to-value mappings.  However, from where should we
get the dictionary?  I solved this by placing my pf variables in a dictionary
to begin with, and passing that dictionary to the pf object when it is created.

d = { "wan_if" : pf.macro("we1"), "lan_if" : pf.macro("ex0"), ... }
...
packet_filter = pf.pf(..., d) # Pass d into constructor.
...
packet_filter["filter"].make_rule("block in on $wan_if proto tcp")

The interpolation gets done during a process called rendering, which is where
we iterate through all the rules and concatenate their contents.  This is
done during a synchronization, which is what I call it when you invoke pf
with a new set of rules.

Passing a mutable dictionary into the pf constructor has the happy side effect
that you may make changes to the dictionary entries, and any changes you make
will be incorporated next time the rules are rendered.  So now instead of
using callbacks and pf's cooperation in iterating through all the rules, you
can just change the dictionary d from with For example,
if you want to be able to change who can ssh in, you could do it like this:

d["ssh"] = pf.macro("1.2.3.4")
...
pf["filter"].make_rule("pass in proto tcp from $ssh to server port = 22")
...
d["ssh"] = pf.macro("2.3.4.5")

Now let's say that you now don't want anyone to be able to ssh in.  You could
set the ssh macro to contain an empty string, but this would render to the
following rule, which is invalid:

pass in proto tcp from  to server port = 22

Similar things happen with empty lists; pf considers an empty list to be an
error.  So what you really want is for the rule to be skipped if a variable
in it is empty.  So I built into the substitute function a test to see if
the length of the variable is equal to zero.  For strings/macros, this means
the empty string, and for lists this means it contains no elements.  In
either case, an attmempt to use them in a pf rule would probably be an error,
so the substitute function just returns an empty string.

I had compiled up python 2.4 around this time, and I found that the real
string.Template class does not have this special test and it was not clear
that I would be able to easily add it.  So instead of having this test occur
within the substitute method, I decided to put this test in the pf.macro and
pf.list __str__ method, which is called by string.Template.  If the macro or
list are empty, it throws an exception which goes up the stack through
string.Template and gets caught by render(), which knows to not render this
rule.

This presents a new paradigm for making rules which do not appear in the
rendered text passed to pfctl.  Formerly, you had to explicitly mark a
rule as inactive, but now any rule that references an empty variable
will also not appear during rendering.

TODO: Consider rendering all pf macros/lists in one class (implemented as set).
     If "{ we0 }" is valid pf syntax, this will work easily, and if not you
     can special-case __str__ to not use braces for one element.

Where do we go from here?  Can we get rid of the active flag on each rule
using this paradigm?  Perhaps rules could be tagged with a list of
variables, where if any are unset the rule is not rendered.

I am re-thinking the tagging concept.  Instead of tagging a rule, I could just
give it a name in python and access that name instead of searching the entire
pf structure for it.  In other case, the single tag is limiting.  For example,
I may have outbound wan rules which tag certain packets for queuing; I'd like
to be able to activate/deactivate queueing rules, but I'd also like to be able
to activate/deactivate WAN rules, and some of these rules overlap.

TODO: See if make_rule returns the rule object so I can do it this way.

TODO: Talk about rendering, namespaces.

TODO: Consider using str(pf) instead of a render method.

TODO: Quote README with regard to why no generic firewall syntax.


Interaction With Twisted:

Explain the protocol and factory classes.
LineOnlyReceiver, tcp_command_factory.


Helper Classes:

syncing_proxy

helper_functions


Other Capabilities:

Rules may be grouped into containers.

Rules may be inserted into queues.

There are a number of optional built-in commands used for administration
(stop, quit, help).

The port number and all of the prompts may be easily customized.


Types of Commands:

In order to make things easier, I have abstracted out the code necessary to
implement a few kinds of commands.

The simplest form of command is the toggle command.  When invoked, it inverts
the "active" flag on all rules with a specified tag.  This is convenient for
a single human user, but a program would have to maintain the state of the
toggle internally to know whether it should be toggled or not.  This makes it
inconvenient for use by automated systems.

The next form of command is the switch command.  You pass an optional argument
to the command, either "on" (default) or "off".  This makes all rules with the
appropriate tag active or inactive, respectively.

Another form of command is called the list command. hostname or IP as an argument.  The argument is put into
or removed from a pf list by the same name.

I'd like to have a kind of command which switches between one of several
mutually exclusive sets.


Future Directions:

Someone is working with me to incorporate this into a bootable OpenBSD
distribution. [...]

The current system does not do any form of access control apart from defaulting
to listening on the loopback address (127.0.0.1) only.
XXX: This might change.

Let the show command be able to number the lines, so if pfctl gives an error
on a particular line you can easily figure out what's wrong.

For a complete list of planned improvements, grep the source for "TODO".