I recently had to figure out how to add custom routes through a VPN connection under Mac OS X. It took me awhile to work through the problem, but I was thankfully able to find a reliable solution. Hopefully the notes below will help anyone else facing the same problem.

Configuring the Mac OS X VPN Client

I was working on a project that required me to use a VPN to connect to a remote network. I was provided with a pre-configured VPN setup to install on a Windows machine, but I also wanted to be able to connect using my Mac.

I was able to use the VPN client built into Mac OS X to setup the connection. (I also added the search domain “remote.main” for the VPN connection under the Advanced DNS settings.) The client is easy to use and even supports using an RSA SecurID token for authentication, but it does not provide any options for advanced networking routing. Everything would work fine if I configured the machine to send all traffic over the VPN connection, either by setting the service order or in the Advanced Options for the VPN connection.

The problem was that I didn’t want to send all of my traffic over the VPN, just the traffic between my machine and the remote network. The rest of my internet traffic for streaming music, instant messaging, Skype and web browsing would unnecessarily burden the VPN connection. This would not only slow down those activities of mine, but also slow down other people using the VPN.

If I didn’t tell the VPN client to send all traffic over the VPN, then I wasn’t able to connect to all of the machines on the remote network. I needed to explicitly control the network routing.

Analyzing the Problem at the Command-line

The first step was to open up the Terminal and understand what was happening. (I have omitted some of the extraneous text below, only the key lines are shown.) First I looked at how my network interfaces were configured:

%: ifconfig
…
en1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        inet 192.168.1.20 netmask 0xffffff00 broadcast 192.168.1.255
…
ppp0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST>> mtu 1396
        inet 172.19.25.11 --> 172.19.25.1 netmask 0xffff0000

The en1 interface is my wireless connection with a local IP of 192.168.1.20 and the ppp0 interface is the VPN connection with an IP on the remote network of 172.19.25.11.

Next, I wanted to see how my various internet traffic was being routed. Were my connections to the internet using the VPN?

%: traceroute www.google.com
traceroute to www.l.google.com (74.125.230.112), 64 hops max, 52 byte packets
 1  192.168.1.1 (192.168.1.1)  55.163 ms  35.808 ms  14.201 ms
 2  83-169-169-58-isp.superkabel.de (83.169.169.58)  227.858 ms  29.495 ms  43.557 ms
…

That looks good, it goes directly to my local gateway and then out to the internet via my ISP.

What about my connections to the remote network? Is the VPN properly connected? For this I just used my assigned VPN IP address from above.

%: traceroute 172.19.25.11
traceroute to 172.19.25.11 (172.19.25.11), 64 hops max, 52 byte packets
 1  172.19.25.1 (172.19.25.1)  193.050 ms  164.557 ms  196.617 ms
 2  172.19.25.11 (172.19.25.11)  321.015 ms  538.553 ms  371.819 ms

That looks good too, no mention of my local gateway or my ISP. Now to look at a machine on the remote network that I was having trouble connecting to, I’ll call it host.remote.main:

%: traceroute host.remote.main
traceroute to host.remote.main (172.20.50.44), 64 hops max, 52 byte packets
 1  192.168.1.1 (192.168.1.1)  40.987 ms  26.728 ms  50.998 ms
 2  * * *
^C

That confirmed that my machine knew the IP address of the remote machine (172.20.50.44), but that instead of using the VPN connection it was trying to use my local gateway. To understand why I looked at the routing tables:

%: netstat -r
Routing tables

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
default            192.168.1.1        UGSc           68       51     en1
default            172.19.25.1        UGScI           0        0    ppp0
…
172.19             ppp0               USc             2       11    ppp0
172.19.25.1        172.19.25.11       UH              1        9    ppp0
…

I had two default entries, one for my wireless connection and one for my VPN. The wireless connection (en1) was listed first, meaning it took precedence. Unless any of the specific rules below applied, my traffic would be sent over the wireless connection. There were two rules that specified the use of my VPN connection (ppp0). The first of these rules had the Destination 172.19, meaning that it applied to any IP address that started with that value. I had seen that route work when I did a traceroute to 172.19.25.11. This rule did not apply to the machine I was having trouble connecting to, since its IP started with 172.20.

Adding a Route Over the VPN

To fix the problem I added a new route to the table to apply to all IPs starting with 172.20:

%: sudo route -v add -net 172.20 -interface ppp0
u: inet 172.20.0.0; u: link ppp0; RTM_ADD: Add Route: len 136, pid: 0, seq 1, …
locks:  inits:
sockaddrs: <DST ,GATEWAY,NETMASK>
 172.20.0.0 ppp0 (0) 0 ffff
add net 172.20: gateway ppp0

%: traceroute host.remote.main
traceroute to host.remote.main (172.20.50.44), 64 hops max, 52 byte packets
 1  172.19.25.1 (172.19.25.1)  61.289 ms  67.725 ms  126.306 ms
 2  172.19.1.1 (172.19.1.1)  364.456 ms  69.946 ms  106.473 ms
 3  192.168.100.9 (192.168.100.9)  98.471 ms  83.573 ms  120.419 ms
 4  host.remote.main (172.20.50.44)  74.289 ms  282.272 ms  90.462 ms

Now my connection was working! To get to the remote machine, the traffic was now being sent to the ppp0 Gateway of 172.19.25.1 and successfully routed through the remote network. The new route can be viewed by running netstat again and will be deleted when the VPN connection is lost. I didn’t find anyway to automatically recreate this route when the VPN connection is made, so I just keep the command handy and run it every time I reconnect the VPN.

Recently I have read a lot of hyperbolic statements about how the Open Verification Methodology (OVM) is “fully interoperable” and a “single implementation”. Unfortunately these claims overstate the interoperability and portability of OVM.

I have worked on two different OVM-based verification environments where interoperability was a required feature. One was a verification component for the Open Core Protocol (OCP) and the other was a conformance test environment for the FlexRay Communications System. Both environments are compatible with two OVM compliant simulators, but that interoperability was not automatic. A significant amount of time and effort was spent throughout the development process to ensure that the final products would be portable.

The Java Analogy

When Sun Microsystems released the first version of the Java programming language, they used the phrase “write once, run everywhere” to promote the language’s portability. The theory was that a developer could write a Java program (“once”) and users would then be able to run it on any platform (“everywhere”), regardless of the underlying system and/or vendor.

In reality, each platform had its quirks and limitations that could prevent an application from running properly. If you wanted to ensure that your Java program would run on a particular platform, you actually needed to test it on that platform and modify the code until it worked. Eventually you could end up with an application that would run on multiple platforms, but it required repeated testing and tweaking. Java programmers refer to this sarcastically as, “write once, debug everywhere.”

Every time I hear someone promoting the portability of OVM it reminds me of “write once, debug everywhere.”

When “Run Once” is Enough

One way that OVM does live up to the hype is that it provides a powerful SystemVerilog framework for developing complex verification environments. That alone is pretty respectable. Using OVM as the basis of your testbench allows you to spend more time working on actual verification and less time worrying about designing and building your own framework.

How many OVM environments currently in development actually need to be run on more than one simulator? How many will use third party OVM components? I imagine that most OVM testbenches only need to run on a single simulator; in other words they are “write once, run once.” If the environments are using external OVM verification components, they are probably provided by the simulator vendor. So, for a lot of OVM users interoperability doesn’t matter yet.

On the other hand, if you need interoperability or were expecting that it would come automatically you could run into issues…

Varied SystemVerilog Support

The good news is that SystemVerilog is an actively evolving language. The bad news is that none of the simulators support all of the features in the SystemVerilog specification. Just because you write some code and it does what you expect in one simulator doesn’t mean that it will do the same thing or even compile in a different simulator.

The 2.1.1 release of OVM has 51 ifdefs in the code. That’s 51 places where the experts had to resort to the bluntest of instruments possible to overcome the differences between just two simulators. There were probably several other places where the original implementation had to be modified, but in the end did not require the use of ifdefs. If the OVM code needs patches in order to be portable, then your code probably will too.

During the development of the aforementioned OVM components I worked on, there were several cases where we had to make changes to the architecture or implementation in order to get it to work in both of our simulators. We tried to avoid ifdefs as much as possible and in the end only needed a couple, mostly to get around bugs in the simulators.

Multiple Implementations

At first, the claim that OVM is a “single implementation” doesn’t seem that questionable. OVM is open source, so you can go to OVM World and download one codebase that will run on multiple simulators.

The first problem you will have to deal with is that new releases of OVM appear regularly, as would be expected. If you have multiple OVM components in your environment, you have to use the same version of OVM for every one of them. This means that you can only adopt a new release of OVM if it works all of the components you are using. If some of your components have restrictions on what versions of OVM they are compatible with, hopefully there is at least one common version that they all support.

In my experience, the 2.x series of OVM releases were unfortunately not very stable. Sometimes basic features were broken, only to be fixed in the next release, which itself wouldn’t even compile. Sometimes promised features were mistakenly left out.

If you only have to worry about your own environment, it is easy enough to dictate which version of OVM to use or, worst case, to use a locally modified version of OVM. However, if you are distributing your component to someone else, it is harder to control the version of OVM being used. In the end you have to qualify your component with the last several releases of OVM and sometimes provide patches. Hopefully future OVM/UVM releases will be more stable, but that still doesn’t mean there will be a single implementation.

Each vendor bundles a custom, proprietary version of each OVM release with their simulator. These custom versions provide extra OVM debugging and tracing capabilities for that simulator. I have found these additional debugging features very useful and recommend anyone using OVM to become familiar with them. The downside is that the code required to add the enhanced debugging functionality to OVM is only available from the individual vendors and is not redistributable.

Several times I have had a component that runs fine with the public OVM release code, but does not work with a particular vendor-specific release. In most of these cases the issue caused the simulator to crash, making it completely unusable. Since the vendor’s modifications are usually encrypted, debugging the problem and fixing it on your own is usually impossible.

If you run into problems using the vendor-specific OVM release, you can just stick with the public releases in your own environment. Someone else using your component, however, will probably not want to be told they have to give up those handy debugging tools they have gotten used to. They certainly won’t be impressed with the “interoperability” of your component.

To ensure the portability of your OVM component you need to verify it with multiple implementations.

Interoperability Gotchas

Assuming you have tested your component with several simulator releases and OVM releases, can it now be called fully interoperable? Maybe.

The are a couple of things that you could do in your OVM component that could easily be broken unintentionally by another component. Any use of the ovm_default_* objects could potentially be interfered with by another OVM component that tries to use them in a different way. I detailed this issue in a post to OVM World with regards to the ovm_default_packer. The basic problem is that unless you specify a packer object, any calls to any of the ovm_object’s pack*() or unpack*() methods will use the ovm_default_packer. Since any component can change the way the ovm_default_packer does (un)packing (for example the endianness), you are better off using your own instances of the ovm_packer. The same precautions should be taken to avoid using other ovm_default_* objects in your code.

Tips for Achieving Interoperability

So what are the steps you should take to make your OVM component more interoperable?

• Avoid using ovm_default_* objects, particularly the packer.

Create a static instance of ovm_packer for every class that needs one and make sure that the default packer is never used.

• Test your component with multiple simulators during the entire development process.

Expect that you will encounter places where you will need to modify the architecture and implementation of your component. The earlier you figure out what needs to be changed or patched the better.

• Test your component with multiple releases of each simulator.

A new release might require you to modify your code in order for it to work. (Did I mention that earlier is better?) You don’t want to have to dictate to your customers which versions of the simulator they can or can’t use. If the new release has a bug (it happens), the earlier you notify the vendor, the more likely it is to get fixed in the next release.

• Test your component with multiple releases of OVM, including the vendor-specific releases.

Same reasons as above. By now you should have a matrix of configurations you need to qualify.

Making the Situation Better

I want to end with a special request: Please report the issues you find with the simulators and OVM back to the vendors.

It takes a little extra effort to report problems to the vendors, but it helps make the challenge of interoperability less daunting going forward. In general, the situation is getting better with each new tool and framework release. In my experience it was certainly easier to achieve OVM interoperability in 2010 than it was in 2008.

The more people who report issues back to the vendors, the fewer who have to encounter those issues. Each resolved issue makes the reality of OVM interoperability that much closer to the hype.