Multicast IPv6 Anycast RP Design

Like most folks, I heard of multicast but never configured it. That changed back in 2006 when we deployed music on hold over multicast for our Cisco voice infrastructure and also installed VBrick IP television which required multicast. I did the research and created a PIM sparse mode design based on the Cisco Solution Reference Network Design for campus networks.

We used anycast rendezvous points (RP) peered with Multicast Source Discovery Protocol (MSDP). We used a site local multicast scope and properly filtered on the edge. We peered with the multicast RPs in our headquarters site and passed organization local scope groups. Overall, we had a pretty robust, fault tolerant multicast design.

I did more multicast work at my next client through 2011 including dense-mode and simplified multicast forwarding (SMF), custom code for Internet Group Management Protocol (IGMP) joins for multicast on mobile ad-hoc networks (MANET) and lots of other unique designs.

Of course, this was all on IPv4.

Multicast for IPv6 is a different beast and although I haven't seen a requirement to deploy it for a client yet, I wanted to do some testing to get up to speed before the need manifests. I was fortunate to attend the Cisco IPv6 Fundamentals, Design and Deployment class last week. Although much was review for me given my hands-on IPv6 experience over the last 4 years, the multicast module and lab was very informative.

I understood my IPv4 multicast reference design would not map to IPv6. While IPv6 supports anycast RP, there is no MSDP for IPv6. Some (very basic) background:

In a Protocol Independent Multicast Sparse-Mode (PIM-SM) design, multicast sources send their multicast streams to the Rendezvous Point (RP). Multicast listeners are directed to the RP to make the connection and start receiving traffic. To make the design redundant and fault tolerant, we use anycast RP. Anycast is the practice of configuring the same unicast address on multiple devices. Traffic is routed towards the closest instance of the address based on routing protocol metrics. This of course can result in multicast sources sending traffic to one anycast RP instance and multicast listeners being routed to a different anycast RP instance. No connection would be made in this case.

To overcome this issue, we use Multicast Source Discovery Protocol (MSDP) to peer the anycast RPs so they share information about the multicast sources. This way, no matter which instance of the anycast RP a listener connects to, it will be able to establish the stream from the multicast source and start receiving traffic.

With the above explanation, the absence of MSDP in IPv6 is a problem with an anycast RP design. Foregoing anycast RP and just using a single RP does not provide fault tolerance should the RP go down. What to do?

RFC 4610 addresses this issue and Cisco implements it in IOS 15 and other flavors like XE. However, I learned in class that Cisco had another approach they called "Anycast RP with prefix arbitration".

The concept is simple using standard routing rules of longest match prefix. Simply configure the anycast address on multiple devices with different masks. Instead of equal cost multipath routing in redundant environments, all traffic will be routed to the anycast instance with the longest prefix (anycast RP primary). Should that RP go down, all traffic will be routed to the anycast instance with the second longest prefix (anycast RP secondary), and so on (anycast RP tertiary ...). This is very similar to "floating static routes"; static routes with a manually configured admin distance to bring up a BRI interface when the primary frame-relay goes down (remember 1990's).

• Configure primary anycast RP with longest prefix
• Configure secondary anycast RP with second longest prefix
• Configure tertiary anycast RP with third longest prefix
• And so on ...
• Advertise the anycast network from each device via routing protocol

This eliminates the need for MSDP to peer the anycast RPs since all traffic - both sources and listeners - will be routed to the same anycast RP instance. Or will it?

Consider an instance where multicast sources and / or listeners are directly connected to the devices which host the primary and secondary anycast RPs. Connected routes override any longest prefix match since they are connected. So there is a possibility where listeners won't be able to find sources. And that's what happened to me when I finished the documented multicast lab and decided to test this design.

The lab was only two routers with a client connected off each, so it's pretty obvious why it failed. I decided to test a more real-world scenario. Consider the following:

The relevant configurations:

C1#show run interface Loopback0
interface Loopback0
 description Anycast RP (Primary)
 no ip address
 ipv6 address 2001:DB8:AFE:FE00::1/120
 ipv6 enable
 ipv6 eigrp 1
end

C2#show run interface Loopback0
interface Loopback0
 description Anycast RP (Secondary)
 no ip address
 ipv6 address 2001:DB8:AFE:FE00::1/119
 ipv6 enable
 ipv6 eigrp 1
end

With anycast RP primary configured on C1 (/120 mask) and the anycast RP secondary configured on C2 (/119 mask), we expect all traffic to be routed to C1. Indeed it is from both D1 and D2:

D1#show ipv6 route 2001:db8:afe:fe00::1
Routing entry for 2001:DB8:AFE:FE00::/120
  Known via "eigrp 1", distance 90, metric 156160, type internal
  Route count is 1/1, share count 0
  Routing paths:
    FE80::C804:11FF:FE44:1C, FastEthernet1/0
      Last updated 00:00:12 ago

D2#show ipv6 route 2001:DB8:AFE:FE00::1
Routing entry for 2001:DB8:AFE:FE00::/120
  Known via "eigrp 1", distance 90, metric 156160, type internal
  Route count is 1/1, share count 0
  Routing paths:
    FE80::C804:11FF:FE44:1D, FastEthernet1/0
      Last updated 00:03:05 ago

"FastEthernet1/0" is the telltale that D1 and D2 will send their traffic for the anycast RP to C1 (see diagram above). When we shutdown the Loopback0 interface (anycast RP primary) on C1, traffic fails over:

D1#show ipv6 route 2001:db8:afe:fe00::1
Routing entry for 2001:DB8:AFE:FE00::/119
  Known via "eigrp 1", distance 90, metric 156160, type internal
  Route count is 1/1, share count 0
  Routing paths:
    FE80::C805:11FF:FE44:1C, FastEthernet1/1
      Last updated 00:03:25 ago

D2#show ipv6 route 2001:DB8:AFE:FE00::1
Routing entry for 2001:DB8:AFE:FE00::/119
  Known via "eigrp 1", distance 90, metric 156160, type internal
  Route count is 1/1, share count 0
  Routing paths:
    FE80::C805:11FF:FE44:1D, FastEthernet1/1
      Last updated 00:03:27 ago

So the solution works! But what routes do C1 and C2 see? When operating in steady state (each anycast RP interface on C1 and C2 is up), all routing doesn't point to C1:

C1#show ipv6 route 2001:DB8:AFE:FE00::1
Routing entry for 2001:DB8:AFE:FE00::1/128
  Known via "connected", distance 0, metric 0, type receive
  Route count is 1/1, share count 0
  Routing paths:
    receive via Loopback0
      Last updated 00:00:28 ago

C2#show ipv6 route 2001:DB8:AFE:FE00::1
Routing entry for 2001:DB8:AFE:FE00::1/128
  Known via "connected", distance 0, metric 0, type receive
  Route count is 1/1, share count 0
  Routing paths:
    receive via Loopback0
      Last updated 00:19:13 ago

Notice C1 and C2 each see the anycast RP address as their local Loopback0 interface. In the case of C1, that's correct. In the case of C2, it's correct (according to routing rules), but not desired (according to our anycast RP with prefix arbitration design). There isn't a way to "fix" it as it isn't broken, it just highlights a design constraint on the "Anycast RP with prefix arbitration" design:

• Never directly connect multicast sources or listeners to a device acting as an anycast RP

This may not be a problem in the design I tested as most people won't connect end stations to the core layer. But consider collapsed core designs or instances where the RPs may be configured on distribution layer switches that connect servers which are multicast sources.

"Anycast RP with prefix arbitration" is a pretty easy, straight forward design, but like anything new, test first and understand the limitations.

IPv6 TFTP from IPv4

No, the title doesn't imply a great new translation technology for that indispensable file transfer protocol TFTP. Instead, this is to highlight an "oversight" - I won't go so far as to call it a "bug" - in Cisco IOS.

I'm testing with version Advanced IP Services 12.4 (24)T on a 7200 series router - just in case that matters.

For many services on Cisco routers and switches, I've been using the "source interface" command to explicitly tell the device what address to source the updates from. Normally, I point it to a loopback interface. This makes looking at logs pretty easy when DNS resolves the loopback address to the device name.

So for example:

ip flow-export source Loopback0
logging source-interface Loopback0
snmp-server trap-source Loopback0
snmp-server source-interface informs Loopback0

In most cases, we'll even use "update-source LoopbackX" for iBGP neighbors.

This makes looking at a Syslog and SNMP Trap aggregator easy. As long as they resolve addresses to names, I see content like:

Router1  Informational  Local7  Interface FastEthernet0/0 up
SwitchA  Emergency      Local6  Power supply 1 down  

Instead of:

10.254.254.1  Informational  Local7  Interface FastEthernet0/0 up
10.254.254.2  Emergency      Local6  Power supply 1 down  

Now that we've shown why this is good practice, I'll also add that we track nightly TFTP backups of configurations in TFTP logs and the same principle applies. So we use the 'ip tftp source-interface Loopback0' command. Notice however that all previous commands don't start with 'ip', the TFTP 'source-interface' command does. Big deal? With IPv6 it turns out ... YES, it is.

Granted the backup routine tested connected to the devices via IPv4 and requested a TFTP backup via SNMP to the IPv4 address of the TFTP server - so we didn't lose a night's worth of backups and wake up to an error log. The benefits of testing first! However, with IPv6 enabled and an IPv6 address on the Loopback0 interface, IPv6 TFTP should work. And in the test, it didn't.

Here's the relevant configuration:

ip tftp source-interface Loopback0

interface Loopback0
 ip address 10.254.254.1 255.255.255.255
 ipv6 address 2001:DB8:AFE:FE00::1/128
 ipv6 enable

interface FastEthernet2/0
 description To TFTP Server
 ip address 192.168.100.1 255.255.255.0
 ipv6 address 2001:DB8:192:168::1/64
 ipv6 enable

The TFTP server in the test lives at:

192.168.100.254
2001:db8:192:168::254

Again, IPv4 TFTP worked as expected. The Loopback0 address (10.254.254.1) shows in the TFTP logs. But with IPv6, something strange happened:

R1#copy run tftp
Address or name of remote host []? 2001:db8:192:168::254
Destination filename [r1-confg]?
.....
%Error opening tftp://2001:db8:192:168::254/r1-confg (Timed out)
R1#

And the resultant TFTP server log shows:

TFTP# crapps.pl -S tftpd -6
Starting MODE       -> TFTP Server
Listening on        -> [::]:69 (udp)
TFTP Root directory -> .

afe:fe01::  62506  WRQ  OCTET  r1-confg  STARTED
afe:fe01::  62506  WRQ  OCTET  r1-confg  STARTED
afe:fe01::  62506  WRQ  OCTET  r1-confg  File './r1-confg' already exists
afe:fe01::  62506  WRQ  OCTET  r1-confg  Timeout occurred on DATA packet 1
...

Who the heck is "afe:fe01::"? My Loopback0 IPv6 address is "2001:db8:afe:fe00::1". True, but my Loopback0 IPv4 address is "10.254.254.1", or in hex used directly as an IPv6 address is "afe:fe01::". I remember a saying about computers doing exactly what you tell them to. The Cisco router is sourcing the TFTP from the 'ip tftp source-interface Loopback0' - 'ip' as in "IPv4".

So is IPv6 TFTP broken? No, you just need to remove the 'source-interface' command:

R1#config term
Enter configuration commands, one per line.  End with CNTL/Z.
R1(config)#no ip tftp source-interface Loopback0
R1(config)#end
R1#copy run tftp
Address or name of remote host []? 2001:db8:192:168::254
Destination filename [r1-confg]?
!!
8774 bytes copied in 5.540 secs (1584 bytes/sec)
R1#

And confirmed on the TFTP server:

TFTP# crapps.pl -S tftpd -6
Starting MODE       -> TFTP Server
Listening on        -> [::]:69 (udp)
TFTP Root directory -> .

2001:db8:192:168::1  52000  WRQ  OCTET  r1-confg  STARTED
2001:db8:192:168::1  52000  WRQ  OCTET  r1-confg  SUCCESS [8774 bytes]

Much better. Of course now the source is the interface of the router that the TFTP traverses, in this case, FastEthernet2/0. This will also be the same for IPv4 TFTP now.

Nightly TFTP backups are one of those automated tasks we set and forget. Sure there are monitors in place to catch changes and email alerts, but how often does something go wrong? Imagine waking up to an error log an no backups. Not then end of the world, but certainly not something you want to see before your first cup of coffee. Test and test again, especially when incorporating IPv6.

BGP Redistribution Redo

I've always been weary of redistributing an IGP into BGP rather than using explicit network statements. Sure it automates the process, but if I'm going to be redistributing BGP into the IGP (say for MPLS CPE) - hopefully with a 'route-map' - there is a potential for funky redistribution loops and issues. Better to break the cycle and statically configure BGP with 'network' statements to explicitly advertise only what you want.

Of course, I'll break my own rules in the name of testing or in this case, a lab exercise. I'm in a Cisco IPv6 training class and our BGP routing lab instructed us to:

"Configure IBGP between R1 and R2 using the parameters that are listed in the table."

Parameter	R1	R2
Source	Loopback 1	Loopback 1
Redistribute into BGP	IPv6 Connected Set origin IGP	IPv6 Connected Set origin IGP

The original configuration was pretty straight forward. EIGRP was used as the IGP to which we were to add BGP. EIGRP was already advertising the connected routes so the EIGRP admin distance was modified so IBGP would be preferred. I was already cringing, but decided to play along.

The relevant parts of the provided R1 configuration follow. You can assume R2 was identical with the appropriate corresponding addresses for interfaces and peers.

interface Loopback1
 ipv6 address 2001:DB9:121:100::1/64
!
interface Loopback2
 ipv6 address 2001:DB9:121:200::1/64
!
interface Serial0/0/0
 no ip address
 encapsulation frame-relay IETF
 frame-relay lmi-type cisco
!
interface Serial0/0/0.1 point-to-point
 description To R2
 ipv6 address 2001:DB9:123:1::1/64
 ipv6 eigrp 1
 frame-relay interface-dlci 122
!
ipv6 router eigrp 1
 eigrp router-id 10.12.1.1
 no shutdown
 passive-interface Loopback1
 passive-interface Loopback2
 distance 250 255

At this point, the BGP configuration was pretty easy. I added the following:

router bgp 65012
 bgp router-id 10.12.1.1
 no bgp default ipv4-unicast
 bgp log-neighbor-changes
 neighbor 2001:DB9:122:100::1 remote-as 65012
 neighbor 2001:DB9:122:100::1 update-source Loopback1
 !
 address-family ipv6
  neighbor 2001:DB9:122:100::1 activate
  redistribute connected route-map BGPCONN
  no synchronization
 exit-address-family
!
route-map BGPCONN permit 10
 match source-protocol connected
 set origin igp

It's no surprise that BGP came up and all was working. From R1:

W1P2R1#show bgp ipv6 unicast summary
BGP router identifier 10.12.1.1, local AS number 65012

Neighbor    V       AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
2001:DB9:122:100::1
            4   65012    1027    1044     47   0    0 00:02:05 4

And I could see my R1 routes on R2:

W1P2R2#show ipv6 route 2001:db9:121:200::/64
Routing entry for 2001:DB9:121:200::/64
  Known via "bgp 65012", distance 200, metric 0, type internal
  Backup from "eigrp 1 [250]"
  Route count is 1/1, share count 0
  Routing paths:
    2001:DB9:121:100::1
      Last updated 00:00:59 ago

But a subsequent 'show' command revealed an issue:

W1P2R2#show ipv6 route 2001:db9:121:200::/64
Routing entry for 2001:DB9:121:200::/64
  Known via "eigrp 1", distance 250, metric 20640000, type internal
  Route count is 1/1, share count 0
  Routing paths:
    FE80::219:55FF:FE35:1B90, Serial0/0/0.1
      Last updated 00:00:00 ago

The route was no longer learned via BGP, but through EIGRP. Weird. The previous command shows the admin distance was correct; iBGP [200] should be preferred over EIGRP [250 - modified]. What was happening?

I did the "up-arrow, enter" troubleshooting technique; that is, I ran the previous 'show ipv6 route ...' command over and over. Of course it worked. I noticed the "Last updated" timer reset every minute as the route flip-flopped between EIGRP and BGP. I started thinking about the BGP walker process and how it runs every 60 seconds.

Since we were in a lab, I had no issues with running 'debug bgp ipv6 unicast updates' on R2 and verified my instinct was correct.

*Dec 13 15:05:50.176: BGP(1): no valid path for 2001:DB9:121:1::/64
*Dec 13 15:05:50.176: BGP(1): no valid path for 2001:DB9:121:100::/64
*Dec 13 15:05:50.176: BGP(1): no valid path for 2001:DB9:121:200::/64
*Dec 13 15:05:50.176: BGP(1): no valid path for 2001:DB9:121:300::/64
*Dec 13 15:05:50.196: BGP(1): nettable_walker 2001:DB9:121:1::/64 no best path
*Dec 13 15:05:50.196: BGP(1): nettable_walker 2001:DB9:121:100::/64 no best path
*Dec 13 15:05:50.196: BGP(1): nettable_walker 2001:DB9:121:200::/64 no best path
*Dec 13 15:05:50.200: BGP(1): nettable_walker 2001:DB9:121:300::/64 no best path

*Dec 13 15:06:50.220: BGP(1): Revise route installing 2001:DB9:121:1::/64 -> 2001:DB9:121:100::1 (::) to main IPv6 table
*Dec 13 15:06:50.220: BGP(1): Revise route installing 2001:DB9:121:100::/64 -> 2001:DB9:121:100::1 (::) to main IPv6 table
*Dec 13 15:06:50.220: BGP(1): Revise route installing 2001:DB9:121:200::/64 -> 2001:DB9:121:100::1 (::) to main IPv6 table
*Dec 13 15:06:50.220: BGP(1): Revise route installing 2001:DB9:121:300::/64 -> 2001:DB9:121:100::1 (::) to main IPv6 table

BGP on R2 put the routes for the advertised R1 connected interfaces - including the network containing the address for the R1 Loopback1 interface it was peering with - into the routing table. The routes had a next hop of the R1 Loopback1 interface. When these routes entered the routing table, they displaced the EIGRP routes for the same networks. A minute later when BGP walker ran again, there was no IGP (EIGRP) path to the next hop for those routes, so BGP removed them (as seen in the first 8 lines of the 'debug'). The routes were quickly replaced with the EIGRP routes as seen in the above "show ipv6 route ..." commands. Sixty seconds later when BGP walker ran again, the BGP routes were learned and now had a valid next hop as the IGP routes were in the routing table and BGP introduced the same routes - as seen in the last 4 lines of the above 'debug'. Every sixty seconds this repeated and flip-flopped the routes between EIGRP and BGP.

Knowing the problem, the fix was easy. Just use the existing 'route-map' to block the Loopback1 network from being advertised as a connected route. On R1, I used:

route-map BGPCONN deny 5
 match ipv6 address PEER
!
ipv6 access-list PEER
 permit ipv6 2001:DB9:121:100::/64 any

The complementary statements on R2 and a 'clear bgp ipv6 unicast *' had everything working perfectly!

The lesson learned just reinforced my original statement regarding redistribution - proceed with caution.

Testing DHCPv6

I've been doing some research into DHCPv6 for a client and beyond the talk and text, I needed some hands on testing. I can create a DHCPv6 server on a Cisco router in GNS3, but who knew creating a DHCPv6 client would be so troublesome.

I suppose I could have just used my Windows 7 x64 host and connected to the GNS3 simulation on a loopback interface. I've done this before, but I was looking for a way to do this in the simulation. I took a look at the Tiny Core 3.8.2 linux Qemu image available on the GNS3 appliances page and gave it a go. Unfortunately, Tiny Core uses 'udhcpc' as the DHCP client and it only supports IPv4. Partial success.

I had a look at Tiny Core and found that you can load packages and one package was the ISC DHCP package version 4.2.1 including client, server and relay. According to their website, 4.2.1 supports IPv6 so I thought I'd pursue.

The first step was to get the ISC DHCP package installed. Tiny Core comes with a package manager; however, you need to be online to use it and I hadn't played with the Qemu emulator (to run Tiny Core) outside of GNS3. To get network access, Qemu needs a TAP interface, which isn't available on Windows. I had to get one.

OpenVPN offers Windows software that comes with - among other things - a TAP interface driver. So simply download the latest Windows intstaller and run it. Luckily, it asks what components you'd like to install. I unchecked everything except for the Tunnel TAP interface driver. Once the install was completed, I had a new interface under Network and Sharing Center - Adapters, which I renamed to "TUN-TAP".

Simply select my physical NIC - in my case my wireless card - and the new TUN-TAP interface, right-click and select "Bridge Connections".

Armed with my new interface, I was able to launch Qemu and get Tiny Core online to add the proper packages:

C:\> qemu -hda linux-tinycore-3.8.2.img -net nic -net tap,ifname=TUN-TAP

Once loaded, I used the AppBrowser, pressed the "Connect" button and the available packages loaded. I used the search to look for "dhcp" and found "isc-dhcp.tcz". I selected "OnBoot" and pressed "Go". It installed the prerequisites and finished.

Next, I had to change Tiny Core to use 'dhclient' instead of 'udhcpc' and make the change permanent over reboots. First, I edited the "/etc/init.d/dhcp.sh" file. I commented out the line with 'udhcpc' and added two new lines after it:

/usr/local/sbin/dhclient    -sf /usr/local/sbin/dhclient-script $DEVICE >/dev/null 2>&1 &
/usr/local/sbin/dhclient -6 -sf /usr/local/sbin/dhclient-script $DEVICE >/dev/null 2>&1 &

That will load 'dhclient' (ISC DHCP client) instead of 'udhcpc' and it will do it for all interfaces and for both IPv4 and IPv6.

Finally, to make the change permanent, I edited the "/opt/.filetool.lst" file. I removed the line with "/sbin/dhclient" and replaced with:

/etc/init.d/dhcp.sh

and then ran 'filetool' and did a "Backup". Curiously, I got an error saying the backup failed, but examining the "/mnt/hda1/tce/mydata.tgz" file, I found the "/etc/init.d/dhcp.sh" file with my edits. So, I shutdown and restarted and hoped for the best.

It worked! I verified 'dhclient' was running instead of 'udhcpc':

tc@box:~$ ps -ef | grep dhc
 1562 root     /usr/local/sbin/dhclient -sf /usr/local/sbin/dhclient-script eth0
 1699 root     /usr/local/sbin/dhclient -6 -sf /usr/local/sbin/dhclient-script eth0
 1762 tc       grep dhc

The final test was to check DHCPv4 and DHCPv6 functionality in the GNS3 simulation. I had a 7200 series router configured as a DHCPv4 and DHCPv6 server and connected a Qemu host with the newly updated Tiny Core image. The host booted in the simulation and got both an IPv4 and IPv6 address. This was verified on the router by looking at:

R1>show ip dhcp binding
Bindings from all pools not associated with VRF:
IP address      Client-ID/          Lease expiration       Type
                Hardware address/
                User name
10.100.104.17   00ab.298c.3e00      Dec 07 2012 03:48 PM   Automatic
R1>show ipv6 dhcp binding
Client: FE80::2AB:29FF:FE8C:3E00
  DUID: 000100011855208300AB298C3E00
  Username : unassigned
  IA NA: IA ID 0x298C3E00, T1 43200, T2 69120
    Address: 2001:DB8:A64:6800:9D60:EF10:536A:28BF
            preferred lifetime 86400, valid lifetime 172800
            expires at Dec 09 2012 11:47 AM (172618 seconds)

The one thing to remember is that Qemu doesn't save any changes to the actual image file when in a GNS3 simulation. All changes are saved to a FLASH disk in the working directory. If that file or directory is removed, the Qemu host is back to defaults. I learned this the hard way the first time through. The answer is to load any packages and make any edits by running Qemu itself - as shown in the first command of this post - and then using the updated image in the GNS3 simulation.

NOTE: You can do the same for Micro Core linux by using the command line AppBroswer 'ab' and using the 'filetool.sh -b' command to commit the changes.

Smartphone Snacks

Last night I fed my AT&T Samsung Galaxy S3 some jelly beans - that is, the Android 4.1 Jelly Bean update. AT&T / Samsung announced availability earlier this week. However, they also said there would be no over-the-air update. So this was a manual process - first installing Kies from Samsung.

The Kies install on Windows 7 x64 was kludgy. I started the install and it kept looping at "Installing Hotfix". I pressed "Cancel" and it warned me it hadn't completed the install and asked for a confirmation. I canceled that and allowed the install to continue and that seemed to skip out of the loop as the install proceeded and completed successfully.

The next step was to connect my GS3 to the computer with USB and Kies recognized it. I got a pop-up for the new firmware update and allowed it to go forward. A few confirmations and a very slow loading and update process followed. But again, SUCCESS!

I've only been exploring / using Jelly Bean for a few hours but it seems nice and smooth. I see I have an explicit "Driving Mode" (although I don't know what it does) even though I already side-loaded Google Car Home. I already had loaded Chrome as my default browser so that remained. I didn't like that the Accounts under "Settings" are now all listed in the main Settings menu rather than a sub menu. The previous sub menu had an icon regarding sync status for each account. Now you need to click into each account to see the sync status.

I haven't had a chance to try IPv6 yet. Previously, I would get IPv6 addresses via SLAAC on my home network, but I didn't have access to the IPv6 Internet. I'll do some testing in the next few days when I find the time.

IPv6 over IPv4-only MPLS with 6to4

Last year, I researched, created and presented training on 6VPE technology for our internal consultants. Passing IPv6 routes and traffic over an IPv4-only MPLS network is a problem. 6VPE solves that problem from a carrier perspective allowing the MPLS carrier to provide dual-stack IPv4 and IPv6 at the LAN side interface of the customer edge devices.

But what if your MPLS provider has not rolled out IPv6 support?

I knew you could use tunnels to solve this problem from a client perspective. Further, configuring static tunnels in a full mesh over MPLS would be an awful management burden so automatic 6to4 tunnels could address that. However, I never pursued it further.

Recently, a question about this came up again and forced me to reconsider. Some research yielded Dynamic Routing Over 6to4 Automatic Tunnels which describes how to enable dynamic routing over 6to4 tunnels using BGP - specifically external BGP (eBGP). However, the article doesn't consider an MPLS backbone. I did some testing and it turns out it's not that hard to adapt this concept to an IPv4-only MPLS backbone.

Assume the following scenario:

• MPLS provider offers IPv4-only MPLS VPN access for customers
• Customer A wants IPv6 access
• Customer A has many remote sites so solution must be scalable

Since Customer A wants IPv6 access and the provider doesn’t offer it, the first step is for Customer A to configure tunnel endpoints on each of the remote sites. Static tunnels such as 6in4 will require n(n - 1) tunnel endpoints – essentially an n^2 problem or management burden. Each new site almost exponentially increases the management burden.

The solution is to configure 6to4 tunnel endpoints on each remote site. 6to4 tunnels are dynamic so only a single tunnel endpoint is needed per remote site. Adding a new site now only adds 1 new tunnel interface – a linear increase.

To enable dynamic routing, the customer edge routers already use BGP to connect to the MPLS backbone. We can use the existing BGP configuration to enable internal (iBGP) sessions between the 6to4 addresses and advertise the IPv6 networks.

To begin, let's configure the 6to4 endpoints on each remote site:

---

ACE1

interface Tunnel2002
 ipv6 address 2002:a01:102::/128
 tunnel source Serial1/0
 tunnel mode ipv6ip 6to4

ipv6 unicast-routing

ipv6 route 2002::/16 Tunnel2002

---

ACE2

interface Tunnel2002
 ipv6 address 2002:a01:106::/128
 tunnel source Serial1/0
 tunnel mode ipv6ip 6to4

ipv6 unicast-routing

ipv6 route 2002::/16 Tunnel2002

---

ACE3

interface Tunnel2002
 ipv6 address 2002:a01:10a::/128
 tunnel source Serial1/0
 tunnel mode ipv6ip 6to4

ipv6 unicast-routing

ipv6 route 2002::/16 Tunnel2002

---

At this point, the sites have reachability to the newly configured 6to4 addresses but not the IPv6 addresses on the LAN of each remote site. To enable reachability to those networks we need to enable routing.

The easiest way is to add some static routes pointing to the remote 6to4 address for the remote sites' IPv6 networks. For example, on ACE1, the static IPv6 routes to add for the four (4) IPv6 networks on ACE2 and the four (4) IPv6 networks on ACE3 are:

ipv6 route 2001:DB8:A64:6800::/54 2002:A01:106::
ipv6 route 2001:DB8:A64:6C00::/54 2002:A01:10A::

Of course, with static routing we've simply shifted the n^2 problem of tunnel endpoint management to static route management, which could actually be much worse if routes are not hierarchical like the /54 summaries above. Imagine managing several static routes per site on each remote router.

The best solution is dynamic routing and as the previous linked article described, an internal gateway routing protocol will not work as the 6to4 addresses are not on the same subnet. BGP solves this with explicit peering definitions and since the edge routers are already running BGP to the MPLS provider, we can use that configuration.

BGP peers will be configured to the 6to4 addresses so BGP routing traffic will be passed as over the 6to4 tunnels the same way data traffic is passed. We use iBGP since the customer edge routers are all in the same BGP Autonomous System. Finally, we offloaded the configuration burden of an iBGP full mesh by configuring ACE1 as a route reflector ACE2 and ACE3 as route reflector clients.

---

ACE1

router bgp 65100
 neighbor 2002:A01:106:: remote-as 65100
 neighbor 2002:A01:10A:: remote-as 65100
 address-family ipv6
  neighbor 2002:A01:106:: activate
  neighbor 2002:A01:106:: route-reflector-client
  neighbor 2002:A01:10A:: activate
  neighbor 2002:A01:10A:: route-reflector-client
  network 2001:DB8:A64:6400::/64
  network 2001:DB8:A64:6500::/64
  network 2001:DB8:A64:6600::/64
  network 2001:DB8:A64:6700::/64

---

ACE2

router bgp 65100
 neighbor 2002:A01:102:: remote-as 65100
 address-family ipv6
  neighbor 2002:A01:102:: activate
  network 2001:DB8:A64:6800::/64
  network 2001:DB8:A64:6900::/64
  network 2001:DB8:A64:6A00::/64
  network 2001:DB8:A64:6B00::/64

---

ACE3

router bgp 65100
 neighbor 2002:A01:102:: remote-as 65100
 address-family ipv6
  neighbor 2002:A01:102:: activate
  network 2001:DB8:A64:6C00::/64
  network 2001:DB8:A64:6D00::/64
  network 2001:DB8:A64:6E00::/64
  network 2001:DB8:A64:6F00::/64

---

Note that we've made no changes to the PE router - meaning the MPLS provider does not need to support IPv6 at all for this configuration to work. You will need to make sure the provider doesn't do any exotic filtering at the edge or in the MPLS backbone such as blocking IP protocol 41 (IPv6 in IPv4 encapsulation).

At this point, we can test the configuration with some BGP show commands and pings to test connectivity. Without the benefit of a live demo here, rest assured these commands produced the provided output:

ACE1#show bgp ipv6 unicast
BGP table version is 17, local router ID is 10.100.103.1

   Network          Next Hop        Metric LocPrf Weight Path
*> 2001:DB8:A64:6400::/64
                    ::                   0         32768 i
[...]
*>i2001:DB8:A64:6800::/64
                    2002:A01:106::       0    100      0 i
*>i2001:DB8:A64:6900::/64
                    2002:A01:106::       0    100      0 i
[...]
*>i2001:DB8:A64:6C00::/64
                    2002:A01:10A::       0    100      0 i
*>i2001:DB8:A64:6D00::/64
                    2002:A01:10A::       0    100      0 i

ACE1#ping ipv6 2001:db8:a64:6800::1 source 2001:db8:a64:6400::1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:A64:6800::1, timeout is 2 seconds:
Packet sent with a source address of 2001:DB8:A64:6400::1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 148/205/276 ms
ACE1#ping ipv6 2001:db8:a64:6c00::1 source 2001:db8:a64:6400::1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:DB8:A64:6C00::1, timeout is 2 seconds:
Packet sent with a source address of 2001:DB8:A64:6400::1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 128/167/204 ms

And there you have it! Dynamic routing of IPv6 over an IPv4-only MPLS network without the MPLS provider supporting IPv6.

Fourth Annual RI 6 Hour Ultramarathon

On Sunday, November 11, 2012, I ran in the fourth annual RI 6 Hour Ultramarathon. This is quickly becoming a favorite race of mine - like RTB.

My goal - as in past years, was to do at least as well as I did the previous year. Given a good training build up during the past few months and last year's performance, I didn't think it would be too hard.

I started quite easy on a mild November day but the heat started to become an issue by 11 AM. Sixty degrees may not sound too hot to run in, but when you're getting to marathon distance with the intent of going much farther, it does matter. I started to get a little cramping due to dehydration - I was sweating more than I anticipated and drinking too little to compensate. I grabbed my running bottle for a lap and that eased the cramps.

Food intake was good this time. No stomach issues or lack of appetite. I stayed content throughout the race.

As I crossed the start/finish line to end my 13th lap and start my 14th - which was potentially my last and all I needed to equal last year's distance - I saw my wife and mother pull in to the parking lot beeping and waving at me. I yelled a quick "hello" and "1 more lap" and then started on my 14th lap.

On my 14th lap - the one to complete 37.8 miles and equal last year's distance - I kept saying this was it. My wife and mother were waiting to see me finish - a nice ending to the day. Looking at my watch, I could maybe have enough time to make it past the start/finish line to the first split timer at the 50k mark - an additional 1.35 miles. Although I was well past 50k at this point, the timing mat is still there and crossed on each lap and we were told by the race director at the start that partial laps to either the 50k or marathon split timing mats would count. But I kept talking myself out of it saying I would finish 14 laps and be done.

I crossed the start/finish line to end my 14th lap just shy of 5 hours, 42 minutes. I walked to the aid station and the volunteers asked if I was going to do a partial lap to the 50k split mat. That did it. I had 18 minutes. Even with my slow pace, I could do it. Jen was walking to me to congratulate me and instead, I quickly told her, "get in the car and meet me at the entrance of the park." Then I turned and started running.

I think part of me didn't want to run the extra 1.35 miles - literally half the course - and end up having to walk back to the start/finish line. I liked the idea of finishing at the start/finish line and being done. But with an unexpected taxi service in the form of my wife and mom, it seemed perfect.

There is a slight crest at the 50k timing mat and I crossed it at 5:54:08 completing 39.15 miles. My wife had parked behind another truck, so I didn't see her car right away, but then I saw her, waved and walked (painfully) towards my ride. Some congratulations from them and a quick explanation and thank you from me and we were on our way back to the start/finish for the post race activities.

The official results place me 5/43 with an official total mileage of 39.173 miles in 5:55:55.5 (although that time is wrong).

The table below documents my miles and times (by my watch).

RI 6 Hour Ultramarathon: November 11, 2012
Lap	Mileage	Cumm. Mile.	Lap Split	Cumm. Time	Lap Pace	Avg. Pace
1	2.7	2.7	21:53.56	0:21:53.6	08:06.5	08:06.5
2	2.7	5.4	22:09.94	0:44:03.5	08:12.6	08:09.5
3	2.7	8.1	21:55.90	1:05:59.4	08:07.4	08:08.8
4	2.7	10.8	22:30.58	1:28:30.0	08:20.2	08:11.7
5	2.7	13.5	22:51.23	1:51:21.2	08:27.9	08:14.9
6	2.7	16.2	22:14.74	2:13:35.9	08:14.3	08:14.8
7	2.7	18.9	22:48.34	2:36:24.3	08:26.8	08:16.5
8	2.7	21.6	23:30.16	2:59:54.5	08:42.3	08:19.7
9	2.7	24.3	24:19.56	3:24:14.0	09:00.6	08:24.3
Marathon		26.2	--	3:41:58.0	--	08:28.3
10	2.7	27	25:18.30	3:49:32.3	09:22.3	08:30.1
11	2.7	29.7	26:39.65	4:16:12.0	09:52.5	08:37.6
50K		31.25	--	4:30:26.0	--	08:39.2
12	2.7	32.4	27:53.00	4:44:05.0	10:19.6	08:46.1
13	2.7	35.1	28:55.01	5:13:00.0	10:42.6	08:55.0
14	2.7	37.8	28:53.37	5:41:53.3	10:42.0	09:02.7
14+	1.35	39.15	12:15.56	5:54:08.9	09:04.9	09:02.8
Totals:		39.15		5:54:08		09:02.7

Windows Virtual PC IPv6: Part 3

The original title of the first post in the this series was "Windows Virtual PC Wireless IPv6", but dropped the "Wireless" before publishing. At that point, I had Part 1 and Part 2 written and knew the problem was with the wireless card versus the wired card, but didn't yet know the root cause. And since I had exhausted my troubleshooting tools (tcpdump and network based analysis) and wasn't about to dive into wireless driver software reverse engineering, I figured I may never have the exact answer. Instead, I focused the posts on discussing the problem and how it relates to the question, "is 'X' IPv6 ready?"

I did want to close the loop and exhaust all possible angles. So I called my friend and colleague Randy who had experience with wireless bridging / routing and virtual machines from a previous client. I remember he would lament about the issues he faced and custom drivers and wireless cards needed to get things only partially working. Since I was only an observer, I didn't have the intimate details, so now I sought them out over a phone call.

I explained my set up and issue and Randy immediately knew the problem. It wasn't IPv6 specific nor was it related to Windows Virtual PC. In fact, it wasn't even the wireless card or its respective drivers (per se), it was the 802.11 protocol itself. Randy explained the technical issues and gave me the proper Google terms to search to flush out the entire explanation.

And sure enough, after our call I started finding links such as:

• IPv6 and networking pain
• The problem with wireless bridging

that detail the IPv6 neighbor discovery problem I encountered when using wireless bridging. In my case, it wasn't a standalone bridge device, but the virtual network bridge when using a virtual machine. Note that in my testing the issue happened with Windows Virtual PC, but Randy's previous work involved VMware, which also had the issue. Again, this is protocol / driver related, not specific to IPv6 (network) or application layer software.

Further complicating the matter was the "incorrect" 'tcpdump' decodes. While the Ethernet header in the packets I posted showed source and destination MAC addresses, it wasn't the full picture. Wireless 802.11 has a different frame format than wired 802.3 Ethernet. However, the *pcap (winpcap in my case) libraries can't decode the 802.11 header if the wireless driver software doesn't pass it along. With my stock Broadcom wireless on a Dell laptop running Windows 7, I was seeing the "fake" header.

This final link from the OpenWRT project does a good job visualizing what is happening. Again, the link details physical bridges versus the virtual bridge introduced with machine virtualization software, but the same holds true.

So what does all this mean? Depending on how you look at this problem, it could be hardware, protocol or software related. For me, it manifested when trying IPv6 on Windows Virtual PC - so that was my first thought. It turns out the answer was much more involved and complicated than I imagined. However, broken IPv6 connectivity was the symptom and the casual user experiencing that will probably just say, "the network is down." No amount of planning is going to identify these unique use cases until they occur - hopefully during a pilot, more likely during production. You're going to need some smart people when you start rolling out IPv6.

Windows Virtual PC IPv6: Part 2

In the previous post we saw the wireless card was responsible for some IPv6 connectivity issues between a Windows XP Virtual PC and the Internet due to it rewriting the VPC virtual MAC address to the MAC address of the HOST.

Suspecting the wireless card may be the culprit, I physically connected the HOST to the switch on the router and changed the VPC settings to use the wired NIC instead of the wireless one. A quick reset and back to testing - this time in somewhat reverse order.

I first checked the router to see what the MAC address for the VPC was. It was correct!

ROUTER# arp -a
? (192.168.10.102) at C4:17:FE:12:7D:75 [ether]  on br0
? (192.168.10.104) at 00:03:FF:13:7D:75 [ether]  on br0

So I started my 'tcpdump' captures and launched the pings from the VPC.

HOST> tcpdump -i2 -nevvXX icmp or icmp6
00:03:ff:13:7d:75 > 58:6d:8f:78:ad:40, ethertype IPv6 (0x86dd),
length 94: (hlim 128, next-header: ICMPv6 (58), length: 40)
2001:db8:192:168:203:FFFF:FE13:7D75 > 2607:F8B0:400C:C03::68
[icmp6 sum ok] ICMP6, echo request, length 40, seq 553
  0x0000:  586d 8f78 ad40 0003 ff13 7d75 86dd 6000  Xm.x.@....}u..`.
  0x0010:  0000 0028 3a80 2001 0db8 0192 0168 0203  ...(:....p...|..
  0x0020:  ffff fe13 7d75 2607 f8b0 400c 0c03 0000  ....}u&...@.....
  0x0030:  0000 0000 0068 8000 9120 0000 0229 6162  .....h.......)ab
  0x0040:  6364 6566 6768 696a 6b6c 6d6e 6f70 7172  cdefghijklmnopqr
  0x0050:  7374 7576 7761 6263 6465 6667 6869       stuvwabcdefghi
00:03:ff:13:7d:75 > 58:6d:8f:78:ad:40, ethertype IPv4 (0x0800),
length 74: (tos 0x0, ttl 128, id 25380, offset 0, flags [none], 
proto: ICMP (1), length: 60)
192.168.10.104 > 74.125.228.80
ICMP echo request, id 512, seq 9728, length 40
  0x0000:  586d 8f78 ad40 0003 ff13 7d75 0800 4500  Xm.x.@....}u..E.
  0x0010:  003c 6324 0000 8001 ddbe c0a8 0a68 4a7d  .< c$.........hJ}
  0x0020:  e450 0800 255c 0200 2600 6162 6364 6566  .P..%\..&.abcdef
  0x0030:  6768 696a 6b6c 6d6e 6f70 7172 7374 7576  ghijklmnopqrstuv
  0x0040:  7761 6263 6465 6667 6869                 wabcdefghi

The two packets show the IPv6 and IPv4 pings (echo requests) respectively. Notice the first line of each packet has the VPC source MAC address - '00:03:ff:13:7d:75' - not the HOST MAC address as seen in the previous test on the wireless card.

And the satisfying result:

VM> ping 2607:f8b0:400c:c03::68
Reply from 2607:f8b0:400c:c03::68: time=50ms

VM> ping 74.125.228.80
Reply from 74.125.228.80: bytes=32 time=166ms TTL=52

Just to make sure this whole testing regime wasn't an anomaly, I rebooted and reset to my wireless configuration and tested again with same failed results described in the previous post.

Why do the wireless and wired cards behave differently? I haven't answered that yet. Certainly, they are from different manufacturers and have different settings as verified in the "Properties" page, "Configure..." button, "Advanced" tab of each adapter. I did verify the common settings were matched between cards. I have some more research to do to get to the bottom of this one.

So, "does Windows Virtual PC support IPv6?" It's complicated. As we've seen, it depends on the application, the host and virtual machine operating systems, and in this case, the hardware used.

Windows Virtual PC IPv6: Part 1

I have a Windows XP Virtual PC for testing - among other things - all the Perl experimenting I've been doing lately around IPv6. Windows XP supports IPv6 if you install it - it's not on by default. However, I've had some issues with IPv6 connectivity from my VPC to the rest of the Internet.

I run in bridge mode as Shared Networking (NAT) doesn't make much sense with IPv6; there is no NATv6. With bridge mode, the VPC should act as another computer on the network with it's own IP(v6) address(es) and a unique MAC address. I should also note at this time, that I use wireless throughout the house and never connect over the wired LAN.

The setup is pretty simple. Note: I've changed the IPv6 prefix to the documentation prefix for this example but was using global unicast addressing for the actual troubleshooting tests.

+-----------------------------------------+
| (HOST) Windows 7                        |
| c4:17:fe:12:7d:75                       |
| 192.168.10.102                          |
| 2001:db8:192.168:c417:feff:fe12:7d75    |
|                                         |
| +-------------------------------------+ |
| | (VPC) Windows XP                    | |
| | 00:03:ff:13:7d:75                   | |
| | 192.168.10.104                      | |
| | 2001:db8:192.168:203:FFFF:FE13:7D75 | |
| +-------------------------------------+ |
+-----------------------------------------+
                    |
                    |
     +-----------------------------+
     | ROUTER                      |
     | 58:6d:8f:78:ad:40           |
     | 192.168.10.1                |
     | FE80::5A6D:8FFF:FE78:AD40   |
     | ('radvd' running for SLAAC) |
     +-----------------------------+
                    |
                  __|_
               __/    \__
              /          \
              | Internet |
              \__      __/
                 \____/

When I try to ping the IPv6 address returned by 'dig' for "www.google.com" from the VPC, I get 'destination unreachables'. I can ping the same IPv6 address from the HOST. To be sure, I also tried the IPv4 address returned by 'dig' for "www.google.com" from the VPC and was successful.

VPC> ping 2607:f8b0:400c:c03::68
Destination host unreachable.

VPC> ping 74.125.228.80
Reply from 74.125.228.80: bytes=32 time=22ms TTL=52

So the VPC has network connectivity all the way to the Internet for IPv4, but not for IPv6. Why? A packet capture may help.

HOST> tcpdump -i3 -nevvXX icmp or icmp6
c4:17:fe:12:7d:75 > 58:6d:8f:78:ad:40, ethertype IPv4 (0x0800),
length 74: (tos 0x0, ttl 128, id 19700, offset 0, flags [none], 
proto: ICMP (1), length: 60)
192.168.10.104 > 74.125.228.80
ICMP echo request, id 512, seq 19713, length 40
  0x0000:  586d 8f78 ad40 c417 fe12 7d75 0800 4500  Xm.x.@....}u..E.
  0x0010:  003c 4cf4 0000 8001 f3ee c0a8 0a68 4a7d  .< L..........hJ}
  0x0020:  e450 0800 fe5a 0200 4d01 6162 6364 6566  .P...Z..M.abcdef
  0x0030:  6768 696a 6b6c 6d6e 6f70 7172 7374 7576  ghijklmnopqrstuv
  0x0040:  7761 6263 6465 6667 6869                 wabcdefghi
58:6d:8f:78:ad:40 > c4:17:fe:12:7d:75, ethertype IPv4 (0x0800),
length 74: (tos 0x20, ttl  52, id 62188, offset 0, flags [none], 
proto: ICMP (1), length: 60)
74.125.228.80 > 192.168.10.104
ICMP echo reply, id 512, seq 19713, length 40
  0x0000:  c417 fe12 7d75 586d 8f78 ad40 0800 4520  ....}uXm.x.@..E.
  0x0010:  003c f2ec 0000 3401 99d6 4a7d e450 c0a8  .<....4...J}.P..
  0x0020:  0a68 0000 065b 0200 4d01 6162 6364 6566  .h...[..M.abcdef
  0x0030:  6768 696a 6b6c 6d6e 6f70 7172 7374 7576  ghijklmnopqrstuv
  0x0040:  7761 6263 6465 6667 6869                 wabcdefghi

c4:17:fe:12:7d:75 > 58:6d:8f:78:ad:40, ethertype IPv6 (0x86dd),
length 86: (hlim 255, next-header: ICMPv6 (58), length: 32)
FE80::203:FFFF:FE13:7D75 > FE80::5A6D:8FFF:FE78:AD40
[icmp6 sum ok] ICMP6, neighbor solicitation, length 32, 
who has FE80::5A6D:8FFF:FE78:AD40
source link-address option (1), length 8 (1): 00:03:ff:13:7d:75
  0x0000:  586d 8f78 ad40 c417 fe12 7d75 86dd 6000  Xm.x.@....}u..`.
  0x0010:  0000 0020 3aff fe80 0000 0000 0000 0203  ....:...........
  0x0020:  ffff fe13 7d75 fe80 0000 0000 0000 5a6d  ....}u........Zm
  0x0030:  8fff fe78 ad40 8700 55bb 0000 0000 fe80  ...x.@..U.......
  0x0040:  0000 0000 0000 5a6d 8fff fe78 ad40 0101  ......Zm...x.@..
  0x0050:  0003 ff13 7d75                           ....}u
c4:17:fe:12:7d:75 > 58:6d:8f:78:ad:40, ethertype IPv6 (0x86dd),
length 94: (hlim 128, next-header: ICMPv6 (58), length: 40) 
2001:db8:192:168:203:FFFF:FE13:7D75 > 2607:F8B0:400C:C03::68
[icmp6 sum ok] ICMP6, echo request, length 40, seq 498
  0x0000:  586d 8f78 ad40 c417 fe12 7d75 86dd 6000  Xm.x.@....}u..`.
  0x0010:  0000 0028 3a80 2001 0db8 0192 0168 0203  ...(:....p...|..
  0x0020:  ffff fe13 7d75 2607 f8b0 400c 0c03 0000  ....}u&...@.....
  0x0030:  0000 0000 0068 8000 9157 0000 01f2 6162  .....h...W....ab
  0x0040:  6364 6566 6768 696a 6b6c 6d6e 6f70 7172  cdefghijklmnopqr
  0x0050:  7374 7576 7761 6263 6465 6667 6869       stuvwabcdefghi

The first two packets show the IPv4 ping (echo request) and reply (echo reply). The next two packets are more interesting (although, if you were paying attention, you could see it in the first two also).

The second two packets are an IPv6 ICMPv6 neighbor solicitation from the VPC asking for the router's MAC address - the IPv6 version of address resolution protocol (ARP), and an IPv6 ICMPv6 ping request (echo request). Let's look closer at the neighbor solicitation message.

You can see "who has FE80::5A6D:8FFF:FE78:AD40" (the router IPv6 link-local address) and the source of the request in line that follows, "source link-address option (1), length 8 (1): 00:03:ff:13:7d:75" - the MAC address of the VPC. However, look at the first line of that packet, "c4:17:fe:12:7d:75 > 58:6d:8f:78:ad:40, ethertype IPv6 (0x86dd)". The source MAC address is the HOST MAC, not the VPC MAC! WHAT?!?!

Is this normal behavior for bridge mode, to rewrite the source MAC of the VPC with the source MAC of the HOST? Why the virtual MAC address on the VPC in the first place? The HOST operating systems sees the virtual MAC:

HOST> arp -a
Interface: 192.168.10.102 --- 0xb
  Internet Address      Physical Address      Type
  192.168.10.104        00-03-ff-13-7d-75     dynamic

Could this be just a 'tcpdump' anomaly, perhaps capturing at a point in the stack before/after a rewrite? Apparently not, because the router confirms the MAC address for the VPC IPv4 address (192.168.10.104) is in fact the MAC address of the HOST, not the virtual MAC of the VPC:

ROUTER# arp -a
? (192.168.10.102) at C4:17:FE:12:7D:75 [ether]  on br0
? (192.168.10.104) at C4:17:FE:12:7D:75 [ether]  on br0

This touches on a problem that will manifest in the years to come regarding IPv6 support - "does 'X' support IPv6". The answer isn't really binary (yes or no). In this case, the VM can get IPv6 addresses and ping the HOST over IPv6; however, the wireless card (and I say that because in the next post, I'll show how the wired card does not exhibit this behavior) doesn't handle the virtual and host MAC addresses properly and thus breaks IPv6 ICMPv6 neighbor solicitation.

More on IPv6 in Perl Modules

Recently, I went on a tear updating my Perl modules on CPAN to support IPv6. Since my modules (and scripts) rely on other core modules like Socket, IO::Socket and some non core modules like Net::SNMP and Net::Telnet(::Cisco), I had a task to make sure the underlying modules supported IPv6.

I already dealt with IPv6 in the core Socket module for Windows in a previous post. So now to deal with a higher level of code written by other developers. Would they be responsive?

Turns out ... YES! I worked with the author of Net::TFTPd a while back to get an ASCII / BIN mode bug fix submitted and he was again very responsive with my IPv6 ask and patch. So IPv6 capable TFTP server - Perl has one!

I also rely on Net::Telnet::Cisco which uses Net::Telnet to do all the heavy lifting. Net::Telnet looked like it hadn't been updated since 2002 but I submitted the query / patch and contacted the author. In a few days he got back to me and we had a pretty detailed dialogue about adding IPv6 support. There may be a coming version which does this "natively", but imagine my surprise when I found Net::Telnet was already IPv6-capable!

The "workaround" - and as workarounds go, this is the least kludgey I've seen - goes like this:

Open a socket using an IPv6 capable module - like IO::Socket::IP:

use IO::Socket::IP -register;

my $socket = IO::Socket::IP->new(
    PeerHost => '192.168.10.1' # or 2001:db8:192:168::10:1
    PeerPort => 23,
    Family   => AF_INET # or AF_INET6
) or die "not connected\n";

Then, use the $socket handle as an argument to the 'fhopen' parameter in the Net::Telnet new() constructor:

my $session = Net::Telnet->new(
    fhopen => $socket
);

Jay Rogers (Net::Telnet author) is obviously a genius providing IPv6 support in Net::Telnet before Perl core even supported IPv6! Ok, maybe that's a stretch, but certainly the openness of the Net::Telnet API allows for this kind of "plug-and-play" with IPv4/v6 sockets and enables IPv6 right now, while in the background "native" support may be coming.

Thanks Jay!

Reach the Beach Relay - 2012

Massakruliks - Mixed Open

147/425 - 28/117

Back row from left: Bill (leg 3), Randy (leg 7), Gavin (leg 5), Dave (leg 1), Dan (leg 9)

Middle row from left: Patty (leg 12), Jen (leg 6), Jodi (leg 11), Suzi (leg 10), Ashley (leg 8), Jackie (leg 4)

Front row: Vince [aka: Me] (leg 2), Krulik [aka: Mascot] (on my leg)

			Vert
	Distance	Difficulty	Gain	Loss	Net	Time	Pace
Leg 2	8.96	Hard	285	127	158	69:48	7:47
Leg 14	7.57	Moderate	492	502	-10	54:12	7:10
Leg 25 *	7.1 (wild card)	Moderate				56:05	7:54
Leg 30 *	3.15	Easy	101	173	-72	21:52	6:57
	26.78					3:21:57	7:32

* Due to an injury to Leg 1 runner, I doubled-up on our last set running his leg (Leg 25) and the last leg for our van (Leg 30) - Jen took my original Leg 26.

IPv6 in Perl on Windows

The state of IPv6 support in Perl has long been lamented but there are those trying to change this including Paul Evans who's updating many of the networking modules - including the Socket core module - to support IPv6. It's a shame the C compiler shipping with Strawberry Perl still doesn't accommodate IPv6 on Windows.

I've been using Strawberry Perl after switching from ActiveState since version 5.10. I found myself needing more and more XS modules and the ActiveState PPM process was always lagging behind. I had a copy of MinGW gcc - the free Windows port of the 'gcc' compiler - and with 'dmake', I could build my own modules directly from CPAN. However, I had some issues and when I learned about Strawberry Perl including the gcc compiler in the distribution, I switched. I've used Strawberry Perl on Windows XP 32-bit and now Windows 7 64-bit.

I've also recently been doing a lot of IPv6 work and using Perl as a quick development platform to test the IPv6 protocol proved easy with the Net::Frame modules detailed in an early series of blog posts. However, when looking to create sockets with Perl, I found errors, most notably:

Socket::inet_ntop not implemented on this architecture

The lack of this function prevented the display of IPv6 addresses and it's missing complement - inet_pton - prevents the resolution of IPv6. These functions are certainly present in Windows 7.

The issue lies in the header and library files distributed with MinGW gcc compiler in the Strawberry Perl release. They simply don't identify the functions in the headers, nor provide linkage to them in the libraries. Fortunately, tools shipped with the MinGW gcc compiler in Strawberry Perl can be used to remedy the issue and get full IPv6 functionality. Here's how.

First off, we need to update the header file with the function prototypes. I did this by creating a new header file called 'ws2tcpip-win.h':

---

#ifndef WS2TCPIP_WIN_H
#define WS2TCPIP_WIN_H

typedef USHORT ADDRESS_FAMILY;

#if (NTDDI_VERSION >= NTDDI_VISTA)
WINSOCK_API_LINKAGE INT WSAAPI inet_pton(INT, PCSTR, PVOID);
WINSOCK_API_LINKAGE INT WSAAPI InetPtonW(INT, PCWSTR, PVOID);

WINSOCK_API_LINKAGE PCSTR WSAAPI inet_ntop(INT, PVOID, PSTR, size_t);
WINSOCK_API_LINKAGE PCWSTR WSAAPI InetNtopW(INT, PVOID, PWSTR, size_t);

#define InetPtonA       inet_pton
#define InetNtopA       inet_ntop

#ifdef UNICODE
#define InetPton        InetPtonW
#define InetNtop        InetNtopW
#else
#define InetPton        InetPtonA
#define InetNtop        InetNtopA
#endif

#if INCL_WINSOCK_API_TYPEDEFS
typedef INT (WSAAPI * LPFN_INET_PTONA)(INT Family, PCSTR pszAddrString, PVOID pAddrBuf);
typedef INT (WSAAPI * LPFN_INET_PTONW)(INT Family, PCWSTR pszAddrString, PVOID pAddrBuf);

typedef PCSTR (WSAAPI * LPFN_INET_NTOPA)(INT Family, PVOID pAddr, PSTR pStringBuf, size_t StringBufSize);
typedef PCWSTR (WSAAPI * LPFN_INET_NTOPW)(INT Family, PVOID pAddr, PWSTR pStringBuf, size_t StringBufSize);

#ifdef UNICODE
#define LPFN_INET_PTON          LPFN_INET_PTONW
#define LPFN_INET_NTOP          LPFN_INET_NTOPW
#else
#define LPFN_INET_PTON          LPFN_INET_PTONA
#define LPFN_INET_NTOP          LPFN_INET_NTOPA
#endif

#endif  //  TYPEDEFS
#endif  //  (NTDDI_VERSION >= NTDDI_VISTA)

#endif

---

Next, copy that file to the appropriate location:

copy ws2tcpip-win.h c:\strawberry\c\x86_64-w64-mingw32\include

and update the existing 'ws2tcpip.h' file in that location with the following:

#include <ws2tcpip-win.h>

Now we need to create the library. This is done by first exporting the symbols from the appropriate DLL file - in this case, ws2_32.dll - and then creating the link library. The version of the MinGW gcc compiler with Strawberry Perl has 'pexports' included, so simply create the DEF file:

pexports c:\windows\system32\ws2_32.dll > ws2_32.def

Now create the library:

dlltool --as-flags=--64 -m i386:x86-64 -k --output-lib libws2_32-new.a --input-def WS2_32.def

Then put the new library in the appropriate location:

copy libws2_32-new.a c:\strawberry\c\x86_64-w64-mingw32\lib

Now we're ready to go. Get the latest release of Socket from CPAN, unzip and extract the tar file. Change into the directory and start the build process:

perl Makefile.PL

Now, before doing the 'dmake', edit the generated Makefile. Find the "DEFINE =" and "LDLOADLIBS =" lines and update them as per below:

DEFINE = -DHAS_GETADDRINFO -DHAS_GETNAMEINFO -DHAS_SOCKADDR_IN6 -DHAS_IP_MREQ -DHAS_IP_MREQ_SOURCE -DHAS_IPV6_MREQ -DHAS_INETNTOP -DHAS_INETPTON -DAF_INET6

LDLOADLIBS = -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 -lws2_32-new

Finally, 'dmake', 'dmake test' and 'dmake install' and you're good to go with IPv6!

The Adventures of Rich and Vince

Characters courtesy of SP Studio

Makeshift Mobile IPv6

For all its benefits, IPv6 also has increased support for mobility. This is due to the header structure of IPv6 accommodating extension headers - one of which is a mobility header.

However, given the slow adoption and limited deployment of IPv6, use of the mobility header isn't really prevalent and mobility of IPv6 nodes - my computer in this example - leaves a lot to be desired.

In fact, the only reason I have IPv6 working at home isn't from the forethought of my Internet Service Provider - Comcast, but because I'm tech-savvy and configured a Hurricane Electric 6in4 tunnel through their Tunnel Broker service.

---

NOTE: Comcast is very far ahead of many ISPs when it comes to IPv6 support including dual-stack support to end customers - unfortunately, just not in my area.

---

While the HE 6in4 tunnel works fine from my home - terminating on my home router with DD-WRT and providing all equipment in my home network with IPv4/IPv6 support - it doesn't help me when I'm not at home. When I take my computer anywhere - even into the office - I no longer can use IPv6, unless of course the network I connect to has IPv6 deployed (not likely).

My solution albeit not a very elegant one is to use another of the 5 free 6in4 tunnels HE provides me with. The first is for my home network and that stays static. The second one I have configured to a random endpoint which changes based on my location. HE offers a web-based update both manually and via a simple GET URL to change the local endpoint IP address of any of your 6in4 tunnels. The HE server examines the IP address from which it sees your request come from and adjusts the tunnel configuration on the server side. I even wrote a simple batch script to call the HE URL and configure the local tunnel interface on my computer. This works great ...

... but of course there's a catch. HE requires that the local IP address to terminate the tunnel be reachable via Ping. Therefore, if ICMP echo-requests are blocked or echo-replies are not sent, the tunnel won't get built. This is an issue as many sites prevent ICMP for security purposes - in fact, most all home routers disable ICMP from the Internet by default. For my home router, I can change the setting to have the tunnel built, but when I'm travelling, I'm relying on the network to allow ICMP so my mobile IPv6 tunnel can be built. Again, not the most elegant solution.

There are other options. SixXS offers a tunnel broker service and a client AICCU that provides AYIYA (Anything In Anything) support. This may provide a client-based alternative that doesn't rely on a provider network to support IPv6 or ICMP to build a 6in4 tunnel. I may test this next time I'm away.

The Adventures of Rich and Vince

Characters courtesy of SP Studio

Reach the Beach Relay - MA - 2012

Massakruliks - Mixed Open

46/175 - 13/52

Back row from left: Emily (leg 9), Bayen (leg 8), Bill (leg 7), Dan (leg 2), Robert (leg 3), Jen (leg 12)

Middle row from left: Ashley (leg 10, 11), Patty (leg 5), Jodi (leg 6), Suzi (leg 1)

Front row: Vince [aka: Me] (leg 4)

			Vert
	Distance	Difficulty	Gain	Loss	Net	Time	Pace
Leg 4	8.42	Moderate	678	750	-255	62:51	7:28
Leg 16	4.63 (actual)	Easy	164	127	37	32:10	6:57
Leg 28	8.11	Hard	194	248	-54	59:05	7:17
	21.16 (actual)					2:34:06	7:17

The Adventures of Rich and Vince

Characters courtesy of SP Studio

Network Virtualization - Then and Now ... and Future

Recent events in Software Defined Networking (SDN) - not the least of which was major vendors including Cisco and Juniper partner with universities to launch the Open Networking Research Center (ONRC) - have spurred me to take a closer look at the hype.

It's not like I was totally in the dark about this. "Network Virtualization" has been talked about since just after server virtualization hit the scene. It was more of a theory or pie-in-the-sky idea back then - no one really could articulate the real benefits in terms of cost or new functionality, but if you virtualized servers, networks were the next logical step - right?

So today, we hear about OpenFlow, which is just one implementation of network virtualization. How does all this fit in? What is the evolution? This is the way I break it down:

• Network Virtualization
• Software Defined Networking
• OpenFlow
• ...
• Multi-Topology Routing
• ...

So what's this multi-topology routing? Apparently, it's been around since at least 2007 and ready to go on Cisco devices, among other vendors. In fact, when I was working with mobile ad-hoc networks (MANET) from 2009 to 2011, we looked at various new routing protocols and enhancements to existing routing protocols to compensate for the low bandwidth, unreliable links typical in a MANET. One option was multi-topology versions of IS-IS and OSPF which would create logically separate routing tables over the same physical network links to provide routing based on user-defined categories. For example, one routing table would use higher bandwidth links and route only video traffic while another would use low bandwidth but more reliable links for voice. This would happen simultaneously over the same physical network.

The main difference from this network virtualization approach from the SDN approach that OpenFlow is championing is the control plane. In the multi-topology routing example, the routing processes are still distributed between the nodes that create the network. The network nodes need to be "intelligent"; they need to run a routing process, communicate with neighbors, maintain adjacencies and create a localized view of the network. With SDN and OpenFlow, the control plane is removed from the distributed node model and centralized in a controller.

This has some advantages. From experience, configuring multi-topology routing is complex and the more nodes in the network increases the difficulty and the chance for errors. A centralized controller that automatically distributes routing policy to small footprint agents on commodity network nodes eliminates the configuration distribution issue while also reducing costs of the network deployment. This starts to answer the "benefits in terms of cost or new functionality" question earlier in this post.

Additionally, with treating streams of similar packets as "flows", we can realize the benefits of a circuit switched model where routes through a network can be defined not just by the least hop count, but by other factors such as reliability of links, actual load on links and delay sensitive issues for real-time traffic such as voice and video. This all without compromising or changing the packet nature of IP(v6) networks.

Great - so where do I sign up? Not so fast. While research in this area is hot - trade magazines and conferences are highlighting OpenFlow and major vendors like Cisco and Juniper are taking notice, I don't expect a revolution soon. Places you'll see this first are certainly academia and high-tech content hosting companies with large-scale complex data centers. As always, enterprise adoption will be halted until a major vendor has an offering. And don't confuse Cisco's and Juniper's support of the ONRC as an immanent product offering. SDN has some real implications for major vendors. The removal of the routing smarts to a centralized controller means 'dumb' switches will suffice and for companies that make money off intelligent switches, a migration to SDN will hurt their bottom line unless they can control their destiny and entrance into the SDN fray.

And there are other mitigating factors that will push SDN in the enterprise off for about 5 years. The outsourcing of corporate networks to providers means one place SDN makes sense - the WAN - is offloaded. The other place - the data center - is another outsourcing play between all forms of cloud computing. Enterprises that maintain their own data centers are just coming to grips with data center "fabrics". Typical capital expenditures for networking components are closer to 5 years. Besides, those heavy into virtualization may already be using "network virtualization" - the Cisco Nexus 1000V.

Future File Format: .database

File format wars are way over. The clear winner - now - is databases. Not any particular database, just databases in general.

Why? Because "big data" is now king and to make any analytic use of the quantity of data out there, it needs to be mined in a raw format where the user creates the relationships and presentations in a format that makes the most sense. This isn't the end of documents and spreadsheets - yet - but the excess information to format and structure a static file simply to 'host' data is counterproductive to the real task - the analysis and presentation of the data.

I've used this example before, but the very page you're reading isn't a statically formatted HTML page, but rather built on-demand from a template and a database which contains the text you're reading. The layout of the page is determined by the template and can easily be changed independently from the data that the page presents. In software development, this is called model–view–controller (MVC) concept. MVC separates the tasks of data operation, data presentation and user interaction in a way that is modular and logical.

To leverage big data, you need a database to hold the data - maybe you need relational hierarchies like SQL or maybe you need sheer scale for unstructured information slices implying Hadoop. Other options exist also, but the key is storage of the data is not done in a "file" format, but a database "format". Generating "content" from the data is the creation of documents, spreadsheets and presentations and that can (or will) be done dynamically on-demand. This echoes my previous post about enterprise social platforms doing away with static file formats.

The more I read about big data, the more I see a push to software services, the more I see the traditional operating system become fragmented between local and cloud / processing and storage, the more I believe "database" is the file format of the future.

Enterprise Collaboration Education

My previous post talks about enterprise social software and my theory for slow adoption. Basically, as long as enterprises use email as the de facto collaboration standard, we will not see the growth or benefits of truly collaborative platforms.

To expound upon that point, look how humans collaborate outside of the knowledge workers' enterprise. Collectively working on a task does not imply a serial process. In fact, parallel work streams are more efficient and generally more productive when a cooperative group drives an idea forward. Yet when knowledge workers coalesce to produce a deliverable, why do we still create documents in strict, structured, proprietary standards with little flexibility for parallel processing?

My take on an ideal content management, knowledge management, enterprise social collaboration platform is probably pie in the sky. You can incorporate all the profile, following, "Like" button, activity stream, Facebook-like, status update, blogging, wiki, odds and ends since that's what people expect. Of course there is a document dumping ground to provide a web-based front-end for a file share. But what is really needed is online, real-time, editing of content by multiple people simultaneously in a more free-form forum that allows for the manipulation and aggregation of the information in a - or many - customized template for extraction. Sound like a mouthful of impossibility?

This blog you are reading is dynamically generated from the content posts and all the other links and widgets populating a template called on demand as you navigate to this site. This page does not exist in its entirety in the view you see now (save for on caching servers perhaps). That fundamental principle is what we as document creators and consumers need to grasp. Our documentation does not need to live in a neat file format stored away ready for portability should the need arise. Instead, it should be created on-demand and only when there is a demand for the information. Furhtermore, the type of demand (status report, final deliverable) could also determine the amount and/or order of the content presented.

I've done this many years ago eliminating useless network documentation that cataloged switch and router information like serial numbers and part numbers. The information was needed, but was also accessible via SNMP. A script to collect the information and present it in a tabular format was much more useful than maintaining a static document that begged for constant care and feeding when network changes where made. In effect, we used the network itself as a database and the script was a query launched against the database. The information was guaranteed accurate to represent current state as the network itself created the documentation dynamically.

If we've mastered this dynamic documentation skill in some instances, why not where it really matters - on the day to day tasks of creating documents, status updates, statistic and metric correlating reports, presentations and any number of other creative processes that yield static self-contained information in a structured format not readily available for parsing, sharing or collaborating? The short answer is because no solution exists in a nice package. The real answer is because no one is asking for it - content to collaborate content with email

Enterprise Social Collaboration = Email #FAIL

If you don't understand the "social" significance of the title of this post, all hope is lost for you.

What applications do you regularly have open on your work computer? If you're like the majority of those I asked, your answer is a web browser (flavor independent; IE, Firefox, Chrome, Safari) and an email client (Outlook being the most popular since all those I asked were on an Exchange/Outlook solution, but Lotus Notes or others also count in this regard).

The charge that I've witnessed to push social software into the enterprise has been at best small pilots and at worst an utter failure. Why is that? Because when I (and I use "I" as a generality - I don't have a Facebook account) use Facebook, it's about sharing pictures and status updates and in the enterprise, that is utterly useless. We need to share documents and presentations and don't care about "likes". And of course, we already have the most powerful collaboration software for sharing ideas and documents - email!

For enterprise social software to succeed, it needs to deliver a web-based, content-driven interface that incorporates communications and collaboration. That's not my bright new idea - that's the general industry consensus. My take is that's only half the picture. For enterprise social software to succeed, it needs to also take the focus away from the email client, which today is the de facto enterprise collaboration tool.

When I first started my career, our laptops were issued with Eudora as the email client since Microsoft Outlook wasn't released yet (am I dating myself). In the early 2000's, Outlook (and others) did wonders with regards to incorporating the social fabric of the workplace - appointments, meetings, contacts, scheduling of people and resources, task tracking - into an intuitive and integrated work space. If one were looking at the technology evolution from afar without context and information about how and why events unfolded, one may argue that a sine wave of innovation in enterprise IT forced a swing of innovation into consumer IT as the 2000's went on. After all, who was using Google Mail, Contacts and Calendar to share personal schedules (the way I would do with my work schedules in the enterprise) in 2002?

We are now in the next upswing of that sine curve driving new consumer innovations back into enterprise IT. And as social software tries to find a fit within the enterprise, there is no missing gap to fill for many of the ensconced enterprise employees who have developed a sharing and collaboration system based on the emailing of documents.

I have no argument that the existing method is better; in fact, I've railed against it for years begging for people to just send a link to a file share where the common document could live before Sharepoint found a way into enterprises and became the uber content management system. Still, people continue to maintain local versions of documentation and proliferate copies with obscure file names that act as a poor man's version control. No one has taken the time to learn check-in/check-out features of Sharepoint, and why would they - we're not software developers where this type of collaboration has its roots.

The evolution in enterprise social software that I'm looking for - to pick on Microsoft for example - is "Shoutlook", a merging of Sharepoint and Outlook that delivers a web-based interface for projects, work streams and sharing of documents and incorporates my email and calendaring into one interface. Real-time group editing of the documents is supported as well as the integration with a unified communications solution for quick voice/video/sharing ad-hoc conference break outs and logging to the active activity stream. No one does this. Well. Yet.

Enterprise social software is not going to see success by keeping available the crutch of a bloated email client for "old-school collaboration" by those resistant to change. The problem with new enterprise social software adoption is the lack of existing enterprise social software (email client) abandonment.

IPv6 Forum Certified Engineer

It's official yet again:

Forecast: Partly Cloudy

Lately, I've been getting a lot of queries for "cloud strategy" – "what's my cloud strategy", "what should we be doing with cloud". Now cloud is a broad topic and I could talk for hours about what you can do with cloud in your IT organization and even across your enterprise, but that's the wrong approach. The whole engagement is framed incorrectly. The fundamental flaw is the question about "cloud strategy". If you want an answer to that, how about this: "you don't need one."

That's not to say ignore the onslaught of marketing that is driving the cloud hype or even discount the myriad success stories of companies that have employed cloud technology. Instead, look to cloud for what it's for – a suite of tools to leverage for business advantage. The question isn't "what's my cloud strategy"; it's "what's my business strategy". "What is my strategy for growth and expansion?" "What is my strategy for reaching more customers in innovative ways?" "What is the best path towards frictionless commerce in my global enterprise?" And ultimately, "how can cloud technologies help me achieve those goals?"

Developing a strategy to integrate cloud technologies into your enterprise is a bit like a the expression, "if all you have is a hammer, everything looks like a nail". Cloud is simply the hammer, a tool to help you get your work done. You shouldn't look for things to hammer; rather, reach for the hammer when you find a use for it.

Perhaps you're a manufacturing enterprise whose crown jewels are computers controlling the robots on the assembly line. You may have little need for infrastructure burst capacity to support an online ordering application so a public, private or hybrid approach to infrastructure would be a poor fit in this case. Does that mean you can't leverage any cloud technologies? Of course not. Cost reduction is key in a low margin vertical and may be realized by purchasing email and office applications from a Software as a Service (SaaS) pay-as-you-go provider. They key is not to fit your business to the cloud, but apply cloud technologies to your business.

Remember, cloud isn't the end game. Growth or cost efficiencies or business process improvements or better customer reach – whatever your business goals are – that's the end game. So instead of looking at how to use a cloud service, look at your business and determine where cloud services can help you. Cloud isn't an all or nothing proposition so you don't need a strategy for "cloud". Understand your business, keep your goals in focus, develop your plans and execution strategy and keep your feet on the ground while reaching for the sky.

The Adventures of Rich and Vince

Characters courtesy of SP Studio

The Adventures of Rich and Vince

Characters courtesy of SP Studio

Out with Outsourcing

Maybe you moved to outsourcing so long ago, you didn’t have time to learn the lessons of the early 2000's – and you surly couldn’t rely on your managed service provider to “innovate” on your behalf at that time (no one was using the “innovate” word back then). In the early 2000’s we had this thing called “convergence” – now we call it “unified communications” because that’s a sexier marketing term. Those of us who were doing the innovating were moving voice and video on to the data network to replace aging and dollar-sucking analog phone systems and private line ISDN video systems. (Don't look now, but it’s happening again with storage in the data center. We call it “storage convergence”, but it will probably come to be known as “unified fabric”, what with the success of “unified” in “unified communications”; “unified” is now a marketing darling.)

We also went from Nortel wide area routers, a frame relay network, a Lucent PBX and Cisco switches to an all Cisco solution for LAN, WAN and voice communications over an MPLS core. Along the way, we also bought the Microsoft story and acquired more messaging and collaboration products from them and tied it all together with some new protocols and enterprise wide quality of service to handle the real-time traffic.

This also forced an organizational shift from a tower model supporting the silos of legacy voice, LAN and WAN to cross functional teams supporting the converged infrastructure, each person specializing in routing and switching or servers and storage or IP data communications (voice and video included) all with an underlying knowledge and awareness of security and network management. We consolidated the support organization, developed talent and reduced headcount favoring fewer qualified generalists over many siloed technology specialists. We became more efficient, more automated, smaller and more agile, more strategically focused and highly competent in multiple disciplines. Where do we go from here?

Logically, to save more money, you get rid of the IT team in favor of outsourcing. I won’t turn this into a “they took our jobs” rant as I’ve always worked for a provider of consulting services (read that as “outsourcer”, “managed service provider” or however else fits best). No, instead, I’ll take objection with the way in which most organizations choose to outsource. Have you learned nothing? I guess I can’t blame you, you let the people with the all the experience go. “When experience is not retained […] infancy is perpetual.” I’ll break it down for you.

For the past decade, we’ve just gone through the exercise of “unifying” the network and breaking down the tower support model to streamline IT and processes, to integrate multi-discipline teams to support the converged network. Why does your outsourcing agreement look like your 1995 IT office: multiple vendors, each providing one or more (but not all) “towers” of LAN, WAN, IPT, data center, mobility / wireless and security?

I've seen this done in different combinations, but always a combination that involved more than one partner. Remember how hard it was to change culture in your own organization – to get the network folks to talk with the voice folks, to get the server guys to talk with the storage experts? It was a paradigm shift. Some feared (rightly so) that their jobs would be eliminated. Some didn’t believe in the vision of a unified network and melded support organization. What makes you think separate vendors outside your organization are going to cooperate on your behalf better than your own team?

I’m aware of the perils of sole sourcing, but why is it acceptable to sole source my network for LAN, WAN, data center and telephony from Cisco but not the management of that infrastructure to a single partner? In fact, I see the benefits of increased dollar spend with a single vendor providing me better leverage to command greater discounts. I see the benefit of a single support contract for all infrastructure and a smaller bill for that service as on-site spares for a single vendor’s equipment is more cost-effective for large volume items.

We can no longer draw lines between LAN and WAN, internal and external, voice / video and data networks. We didn’t support them that way in-house, why does it make sense to have them supported by multiple managed service providers? As the lines between networks and the services they provide start to blur and we move to an outsourced model, we need to think more strategically about how we partner in this environment. Want a true end-to-end service level agreement? It may be hard with a single vendor - think it will be easier with multiple partners?

When we supported our own WAN in-house, I didn't deploy Cisco WAAS for the very specific task of WAN acceleration. For that specialization, I chose Riverbed, easily the market leader in that specific discipline of wide area network acceleration. That’s a targeted decision to realize a huge benefit on a critical asset. To that end, I don’t need to foster that holistic solution by splitting my network on arbitrary lines and selecting different partners to manage each piece. Rather, I ensure my chosen outsourcing provider maintains the key strategic partners to make their services best of breed. This shifts the management burden of the partner ecosystem to the provider.

I also make sure they go beyond the standard management, lights-on, day-to-day operations teams and support desk to provide a facility for innovation. We establish an internal team and a team of technical experts from the outsourcing partner (that may change based on technology requirements for current initiatives) to incorporate next generation technology as pilot projects that have a clear path to production roll-out and absorption into the existing management contract.

Sure this isn’t a “one size fits all” approach just as trying to find a customized managed service is something of a fool’s errand. But use this as an example. “Those who cannot remember the past are condemned to repeat it.”