Linux Dscc4+HDLC Mini-HOWTO

With X.21/V.35, the difference between DCE and DTE is really about who generates the clock and who listens to it. You can have two PC's connected back-to-back, set up as DTE and DCE.

Both devices transmit and receive their data as a stream of bits, running at a particular clock frequency. The clock signal has a separate twisted pair channel in the electrical interface (connector pinout).

DCE is the party that generates this clock signal, latches RX and TX bits in and out along its edges, and sends the bare clock signal across the dedicated clock line - for the DTE to receive it and use it as a reference for latching the RX/TX data in/out.

In simple router/PC back-to-back cases, the "DCE" device has to generate the clock using its internal clock oscillator.

In complex telco data networks, the PDH/SDH network cloud itself is providing the clock, and both (or all) our L3 devices at the ends are set to DTE operation, receiving clock from their respective CPE modem/DTU/converter box. Somewhere at the heart of the SDH/PDH network, there's a central clock source - a dedicated high-precision oscillator box, providing clock sync to the core SDH/PDH multiplexers. The clock is then distributed throughout the network hierarchy via numerous master/slave relationships down to the CPE devices.

If it's just two base-band modems connected back to back on a dry copper line, then it's the modems (read: the lower layer) providing clock to DTE's at both ends. The modems themselves have the master/slave relationship sorted out among themselves.

Krzysztof Halasa notes that there's more to DTE vs. DCE at the Frame Relay layer: "the DCE is a 'network side' (usually a FR switch but could be a router as well) and DTE is a 'user side' (usually a router but could be, for example, an end computer or something-over-FR bridge). This assumes 'UNI' or User to Network link, as there are 'NNI' connections which are not (yet?) supported by FR code in the Generic HDLC." This FR-layer aspect is not necessarily hard-wired to the physical-layer DTE/DCE - theoretically the two layers are unrelated.

When our PC's sync serial board is set for DCE operation, it has to generate the clock signal internally - using a reasonably precise oscillator.

There's a single quartz oscillator per board (per four channels). It has a specific constant frequency that must be divided/multiplied to provide the desired per-channel sync clock rate. Tell the dscc4.o driver at insmod what its crystal's frequency is and tell `sethdlc' what clock rate you want at the particular line - the divider ratio is then calculated by the software and set into the hardware. Read on for details if interested.

In modern electronics, the general-purpose device of choice for generating precise carriers in radio equipment and clock signals in computing equipment is a "quartz crystal" - a miniature slice of the quartz mineral (or some other ceramic material) with two metallic electrodes deposited on its surfaces. Such a device has two important features:

If hooked up into an electronic feedback loop circuit, the quartz crystal oscillates at its specific frequency.

The quartz alone has one specific frequency that cannot be changed or adjusted at runtime. In fact the crystal's specific frequency is printed on its package - it's so precise. Crystals are mass-manufactured for certain standard tabulated frequencies, just like resistors or capacitors. The crystal is usually enclosed in a vaguely rectangular or oblong shiny metallic package, that is very prominent on any electronics circuit board among all the black plastic semiconductor packages, colorful resistors and capacitors in insulating sleeves.

So the crystal has a specific frequency, given by its manufacturing process, that can't be changed once mounted in an electronic device. Now we need to set our clock frequency in software. So what?

So we use a high-frequency crystal and a programmable frequency divider chip. A classic divider can provide integer parts of the crystal's main frequency, but there's yet another circuit called a PLL (Phase-Locked Loop) that can releave us of the integer divisor restriction. A PLL-based clock source consists of (again) a constant-frequency crystal oscillator, a variable-frequency voltage-tuned oscillator that's feeding a programmable divider, and a phase comparator, precisely comparing the two, adjusting the voltage-controlled oscillator to achieve precise match. Thus, the divider dividing the VCO's signal is effectively turned into a multiplier - integer multiples of some reference frequency are much more comprehensible than integer division products. Such a circuit can provide precise yet programmable clock frequency. Depending on the precise hookup of one or multiple programmable dividers and the two oscillators (constant and variable), the circuit can be tuned below as well as above the crystal's single frequency, getting quite close to the crystal's own accuracy and stability. Remember "digital tuning" radioes or CPU/FSB clock setting on modern motherboards or the adjustable VGA pixel clock.

It's like a manual gearbox in a car. Just imagine that the car has a constant-RPM engine :)

But back to dscc4 and its quartz vs. clock. The PEB20534-based PCI boards are manufactured with various crystals with different nominal frequencies. Some cards are manufactured without a crystal (intended for DTE-only operation) and some true die-hard users add crystals on their own.

There's no easy way of detecting the crystal's frequency in software. That's why, if our PEB20534 is to work as a DCE device, the card's driver (dscc4.o) needs to be told what is the crystal's own frequency. As for the programmable divider (the gearbox), the driver knows well how to control that - all it needs to know is our intended CLOCK RATE on that particular line. It then shifts the gears accordingly to meet that intention.

Our Linux driver software is divided into two parts: the generic HDLC driver and the card's low-level driver. In the current implementation, the generic HDLC layer is able to instruct the particular low-level driver about the desired clock rate and clock source setup (DCE/DTE). That's why the particular line's clock rate is set using the generic sethdlc util.

The generic hdlc layer doesn't care about the particular card's clock generator implementation. The Xtal frequency uncertainty is specific to this particular hardware. Therefore the local Xtal frequency is required by the dscc4.o module at insmod.

From the basic documents from Francois Romieu about dscc4.o and from Krzysztof Halasa about hdlc.o, I knew that I was after the "Generic HDLC Layer" and its dependencies.

In the kernel configuration, there are a number of options that claim pertinence to the sync serial, HDLC and sync PPP topics. Just try walking the menus of `make menuconfig':

->Networking options
 - WAN router                    [wanrouter.o]
 - X.25 packet layer             (apparently compatible with Generic HDLC)
 - LAPB data link driver         (needed for the X.25 support in Generic HDLC)
->Network device support
 - PPP support                   [ppp_generic.o] - for cooperation with pppd
  - PPP support for async serial [ppp_async.o]   - self-contained
  - PPP support for sync tty's   [ppp_synctty.o] - self-contained
 ->Wan interfaces
  - Comtrol Hostess SV-11      (HW specific)
  - COSA/SRP                   (HW specific)
  - Multigate (COMX)           (HW specific)
  - Etinc PCIsync              (HW specific) - uses the Gen.HDLC Layer
  - LanMedia adapters          (HW specific)
  - Aurora Technology adapters (HW specific)
  - SeaLevel Systems 4021      (HW specific)
  - SyncLink HDLC/SYNCPPP      (HW specific, just a supplement)
  - Generic HDLC layer
   - raw HDLC                  (encapsulation)
   - Cisco HDLC                (encapsulation)
   - Frame Relay               (encapsulation)
   - Sync PPP                  (encapsulation) - uses [syncppp.o]
   - X.25 protocol support     (encapsulation) - depends on LAPB
   - SDL RISCom/N2 support     (HW specific)
   - Moxa C101                 (HW specific)
   - FarSync T-Series          (HW specific)
  - Frame relay DLCI support     [dlci.o]
  ->WAN router drivers
   - Sangoma WANPIPE(tm)       (HW specific)
  - Granch SBNI12              (HW specific)
->ISDN subsystem
  - support synchronous PPP    [isdn_ppp.o]

      

By the help entries, all of the components above might be useful. Especially the "wan router" support seems to be necessary - as if without this module you couldn't do any WAN routing, being restricted to Ethernet rug chewing.

After re-reading the menuconfig help and kernel Documentation, some shallow study of the source code and googling around at dejanews, the picture turns out to be quite different.

Before actually discussing the individual items of the list above, let me reiterate that my goal in this mini-HOWTO is to set up a very basic IP-over-sync-serial functionality.

The "wan router" support indeed seems to be quite generic and hardware-independent, with some interesting features. Nevertheless, it was programmed by Sangoma people and is supported by no hardware drivers except Sangoma's own and also Cyclades'.

As the help says, in order to configure and use the "wan router" infrastructure in the kernel, you need to download a package containing the necessary userland utilities from the Sangoma website. This userland package claims dependency on the latest version of Sangoma hardware drivers...

The "WAN router" contains its own HDLC/SyncPPP/netdev/etc. stack, is completely independent of the "Generic HDLC Layer" and incompatible with it.

So much for the hardware independence of wanrouter.o.

The X.25/LAPB entries in "Networking options" are a slightly weird bunch. Their menuconfig help seems to be out of date where it talks about X.25/LAPB being only supported over Ethernet. This is not true anymore - the "Generic HDLC Layer" does support X.25, at least that's what drivers/net/wan/Config.in suggests. The X.25 "encapsulation" entry under the "Generic HDLC Layer" is dependent on the LAPB support from "Networking options". I do not know whether or not the X.25 from "Networking options" is required too, or at least compatible (if enabled), or completely irrelevant. Does it perhaps provide X.25 routing support? I'm not familiar with X.25, please elucidate me if you know for sure. Another possible point of view is that we don't really need X.25/LAPB for basic IP-over-sync-serial => you can safely skip the related menuconfig entries, thus giving up X.25 for the moment.

In the "Wan Interfaces" section, there are a number of hardware-dependent drivers - for particular sync serial board models or product families from various vendors.

There are six or seven menu items for drivers supporting particular vendors (product families) from the lowest hardware layer through raw HDLC, IP encaps, up to the kernel/user interface (userland-manageable network devices). The "WAN router drivers" foldable subsection should really be called "Sangoma hardware drivers" and pretty much belongs into this category.

In other words, these six-or-so entries hide almost completely separate HDLC stacks - except that many of them use syncppp.o for the PPP encap (see the chapter on PPP below). Consequently, the different stacks support different sets of the common IP-over-sync-serial encapsulations, possibly with different bugs and features.

The Sangoma "WAN router" seems to be a particularly advanced package - the HDLC stack is just a central part, the wanrouter module and user-space utilities seem to comprise complex FR PVC management, maybe redundant IP backup routing according to the detected WAN interface status (just like Cisco does it) etc. Didn't have a chance to test it though.

That "SyncLink HDLC/SYNCPPP" menu item is not really a "dangerously dedicated" hardware driver (not this item alone). The Synclink board by Microgate is really a tty-style sync serial board, with its main hardware driver listed under "Character devices". Normally, it uses the "normal" PPP support and is interoperable with pppd. This menu item makes the card use syncppp.o instead of the tty-style PPP subsystems. More on this in the chapter on PPP below. In other words, the "Synclink HDLC/SYNCPPP" doesn't have anything to do with the "Generic HDLC Layer".

The Generic HDLC Layer by Krzysztof Halasa is compatible with hardware drivers for a few particular cards, whose authors decided that Mr. Halasa did a good job of separating the various HDLC-based sync encapsulations and generic network device operations from their low-level hardware stuff - PCI DMA mappings, passing buffers and raw link frames back'n'forth, specific ioctl()'s, hardware configuration and interfacing, hardware intelligence etc. Compared to others, the "Generic HDLC Layer" seems to have a fairly complete set of encapsulations. The PPP encap relies on syncppp.o (see the PPP chapter for more information).

The hardware drivers compliant with the "Generic HDLC Layer" are listed in the "Generic HDLC Layer" foldable subsection. One notable exception is the "Etinc PCIsync" driver (dscc4.o) by Francois Romieu, that really belongs into this category too, but is listed among the "dangerously-dedicated" drivers, apparently for historical reasons.

The "Frame Relay DLCI" menu item seems to be irrelevant, too - it is not interdependent in any way with the FR encap in the "Generic HDLC Layer". An early hint is, that hdlc.o + sethdlc works with FR DLCI's quite well even without this option compiled in.

The menuconfig help doesn't provide much information, the contents of Config.in seem to append this entry to the Generic HDLC Layer (at least optically). Nevertheless, the source code comments in dlci.c strongly suggest that this is an entirely stand-alone implementation of the Frame Relay layer - I'm not quite sure what hardware drivers and HDLC stack are supposed to be compatible with it. Would it perhaps add some FR bridging functionality to the system? Are there any relevant user-space config tools?

For any hardware compatible with the generic HDLC layer, the ISDN stuff is irrelevant, too. The ISDN stack is completely separate - see the PPP section on ISDN sync PPP support.

And, the "PPP support" under "Network device support" is not needed for the Generic HDLC layer, either - see the chapter on PPP.

To sum up, the whole Sync Serial & HDLC scene in Linux is actually quite fragmented, despite how much the on-line help, bundled with most of the seemingly relevant drivers, preaches universal love.

To be able to use your PEB20534 for IP, you need the "Etinc PCISync" menu item (dscc4.o), the "Generic HDLC Layer" (hdlc.o) and its nested menu items referring to the relevant encapsulations - I recommend that you compile in at least PPP, HDLC and Cisco-HDLC (or just the single encap that you plan to be using). If you want to terminate FR PVC's, enabling the FR "encap" should be enough. If you want to terminate X.25 PVC's, enable LAPB under "Networking options" (an experimental feature) and then the X.25 "encap" under "Generic HDLC Layer". No other menu items here or elsewehere seem to be relevant (PPP,FR,X.25 etc.).

There are three separate PPP implementations in a standard Linux kernel:

Now how come that the "sethdlc" util also supports a "ppp" encapsulation? Where does PPP come in?

You probably know async PPP very well. There's the `pppd', a user-space daemon that attaches to a serial TTY (/dev/something) and does the LCP/AUTH/IPCP negotiation. It can talk to a modem and can dial out to call the dial-in party, or can be set for null-modem connections using the "local" keyword. Its auth parameters are set in a file. It uses some black magic interfacing towards the kernel to create ppp<x> network devices at runtime. There's even a note in `man pppd` that it can be set for sync operation with tty-style sync serial cards, namely Microgate Synclink.

There's also the ipppd used for ISDN (synchronous dialup). The kernel options around ISDN do seem to mention sync PPP, and LAPB might also seem relevant, right?

In the USENET archive at DejaNews (now groups.google.com), you can find numerous but vague references and talks about tighter integration of the user-space `pppd' with some generic kernel-space PPP driver stuff, talking to each other via /dev/ppp, thus allowing for multilink PPP operation, better runtime dynamic setup/shutdown of PPP network interfaces and whatnot. I.e., /dev/ppp is definitely not a serial-port-style TTY and the stock `pppd' doesn't even admit knowing about it (didn't check with strace though).

Now dscc4.o doesn't export any /dev/ttyXYZ device. Reasonable as it may sound at a first look, it certainly doesn't work to launch pppd at /dev/ppp to make it process the sync ppp frames coming from dscc4.o.

Hmm. Just where does all of that meet? How do we combine sethdlc with its PPP option (really setting the hdlc.o kernel-space driver) with the user-space pppd?

The answer is: we don't.

Forget about user-space pppd in the first place. The pppd daemon works with TTY's - character-class devices with a special set of ioctl()s, that are directly attached to their respective serial links. There *are* sync serial boards with drivers that make the sync link accessible as a character-class TTY (the Microgate Synclink). This is not the case with dscc4 though. The pppd daemon cooperates with kernel-space "generic PPP" support, visible in the menuconfig menu right here:

       
->Network device support
 - PPP support                   [ppp_generic.o] - for cooperation with pppd
  - PPP support for async serial [ppp_async.o]   - self-contained
  - PPP support for sync tty's   [ppp_synctty.o] - self-contained
   
      

In other words, we DO NOT NEED these menu items for sync PPP on a PEB20534.

There's an entirely separate kernel-mode sync PPP implementation. The respective driver file is drivers/net/wan/syncppp.c, by Serge Vakulenko of Cronyx, ported to Linux by Yenya Kasprzak (the author of COSA sync serial board's hardware drivers). This code module is called by the Generic HDLC Layer to provide PPP functionality when a particular hdlc<x> device is set to PPP encapsulation. This same code snippet (module) is used by many of the "dangerously-dedicated" hardware-specific drivers, namely COSA and Sangoma/Wanrouter (and others). In effect, the reference to syncppp.c is the only meeting point between the many incompatible hardware-specific HDLC stacks.

Don't get confused by the fact that, unlike the traditional PPP support, syncppp.o doesn't have its own menuconfig entry - it's created automatically by `make` if needed, based on the respective dependency (detected upon `make dep`) from any of the host of HDLC stacks.

BTW, the syncppp.c also claims to support Cisco HDLC. To that note, Krzysztof Halasa confirms that "there is a separate implementation of Cisco HDLC in Generic HDLC. This is because I have/there are plans to merge the new PPP code with syncppp (and possibly with ISDN PPP) thus rendering syncppp obsolete."

From a user-space point of view, you just set the hdlc<x> interface to "mode ppp" and that's it - attach a peer box set up for sync PPP and you get an established PPP link. The interface running PPP is still called hdlc<x> - I've already told you to forget about pppd and its ppp<x> interface naming convention.

The ISDN PPP support seems to be an entirely separate stack, too. The user-space ipppd is a mod of the original pppd, now a separate development branch. It's using a different kernel-space support module (isdn_ppp.o) that doesn't seem to link to ppp_generic.o anymore and certainly not to syncppp.o. Optionally, you can launch a legacy pppd on user-space TTY-style devices (/dev/ttyI*). At least, that's what the kernel documentation on ISDN seems to say. In other words, the ISDN stuff has nothing to do with the "Generic HDLC layer" and PEB20534.

The syncppp.c implementation seems fairly rudimentary - good enough for leased lines (permanent circuits) but lacking some features required with dial-up PPP, such as PAP/CHAP authentication. In other words, if you want PPP on your sync serial WAN link, the PPP will have to be left unauthenticated.

Etinc corp. have their own, closed-source drivers, including a complete HDLC stack and a "bandwidth manager" - a software solution for traffic shaping and monitoring that also handles Ethernet interfaces, intended e.g. for WiFi hotspot transmitters or corporate networks. The software ships with a modified dev.c (a part of the Linux Net3 subsystem) with proprietary hooks for the bandwidth manager. The Etinc software seems to have decent link state handling. Unfortunately, due to the closed-source (binary-only) nature of the driver modules, you are restricted to about three selected kernel versions from the 2.4 branch, with module versioning turned off. I don't have hands-on experience - reportedly it works fine.

Please note that there are no universal Windows drivers for the PEB20534 - there's just some example C source code from Infineon, for DOS or some even more obscure platform.

Open-source drivers for Linux have been written by Francois Romieu and Krzysztof Halasa.

As for the border line (interface) between dscc4.o and hdlc.o: the dscc4.o is responsible for interfacing to the hardware, and hdlc.o does the encapsulation of higher-layer protocols, especially IP.

The PEB20534 parses the RX bitstream in hardware and the driver works with data already broken down into frames. The dscc4 driver essentially just passes these frames straight to the generic HDLC layer. I.e., there's no CPU-intensive parsing of the raw bitstream to find the HDLC frames' preambles, calculate the CRC etc. With TX frames, it works the other way around - the driver doesn't have to deal directly with the sync streaming of bits, it just passes HDLC frames to the hardware.

In the Linux 2.2 and earlier Linux 2.4 kernels, the dscc4.o driver did exist and was accompanied by a util called scctool to set the clock rate and other paraphernalia. Apparently it did work for some people under some circumstances. In Linux 2.5, Krzysztof Halasa implemented the Generic HDLC Layer. Later it was back-ported to Linux 2.4 - first in the form of Krzysztof's patches, then the hdlc.o driver made it to the standard vanilla kernel.

The first 2.4 kernel to feature a mutually compatible and positively operational dscc4.o and hdlc.o was 2.4.21-rc1. Mr. Romieu has provided his own mod of Mr. Halasa's sethdlc v1.12. I have installed all this on a system running RedHat 9, thus replacing its stock 2.4.20-something kernel. Then we applied a patch or two, to prevent some misbehavior when issuing "sethdlc hdlc<x> clock ext" in DTE mode (which is really the default setting, hence an unnecessary command). And such is my first proven-to-work testbed.

The modified dscc4.c and sethdlc.c that worked for me in this setup are available here. This package should also be used with 2.4.21 (release). The changes to dscc4.c have already been submitted for approval to the vanilla kernel.

A note on the modified version of sethdlc v1.12: Francois Romieu has "cut some non-compiling functionality". In other words, some parts are missing from this modified sethdlc. According to Krzysztof Halasa, the missing parts address Ethernet emulation/bridging over HDLC. The compile-time problem with sethdlc v.1.12 used to occur with kernels that did not have the Ethernet over HDLC bridging capability - which is the case with the early Generic HDLC backport in 2.4.21. With newer versions of sethdlc, such as 1.15, the compile-time problem is solved in a more appropriate way - the Ethernet-over-HDLC bridging capability in the kernel is automatically detected and excluded from sethdlc if absent in the kernel.

I don't have any information about the earlier releases of kernels, patches and utils (scctool&sethdlc). The first combination I tried was a RedHat8.0 with a stock 2.4.18-14, where it didn't seem to work at all. I'm not so much of a historian and therefore I carried out no further research of the earlier versions.

A quote from Francois Romieu: "Historically it happened that Krzysztof published his first new code for the hdlc layer just when I pushed the first versions of dscc4 and wondered where I would find the time to write a FR stack. Lucky day."

I don't have a problem to install a reasonably current kernel on a dedicated firewall-style box that's installed from scratch at the time of installing the PEB20534 PCI card. If you have some success stories with the older versions of all the software, feel free to send me precise descriptions of how you arrived at your operational setup - I'll include your instructions here.

In this section, I'm not quite sure what Linux is or is not capable of. Please send corrections if you have some.

In WAN networking, line outages are something you have to deal with on automatic basis. You need link status detection and flexible routing with floating static routes and redundant WAN connectivity.

Imagine that you are a medium-size company connected to your ISP via two lines using different geographical routes, thus minimizing the risk of both lines going down at the same time due to cable cut or so.

In Cisco IOS for instance, this is not a problem - you configure two static default routes to the two WAN interfaces and your redundant topology is hereby finished. You even don't need a dynamic routing protocol/daemon.

Cisco distinguishes three different up/down attributes of an IP interface:

In Cisco IOS, only interfaces with all the three attributes "UP" take part in realtime packet switching. When a link goes down on a particular interface, any static routes pointing to that interface remain in the master config table, but are excluded from the preprocessed real-time switching tables.

This scenario is typical for sync serial WAN links such as X.21, V.35 or E1 circuits. The three attributes may behave different with other media types, subinterfaces at al, but the concept is always the same. Note: the one interface that may have problems detecting end-to-end L2 integrity is Ethernet (no keepalives at L2) - but we're not speaking Ethernet here, Ethernet is not a traditional WAN technology.

On Linux, there are only two attributes: UP and RUNNING. These two attributes are either present or absent in the output of ifconfig.

UP means "administratively UP" - this is what you influence by ifconfig up/down. If an interface is "down", it simply vanishes from an ifconfig listing - you have to add the "-a" option to see all the interfaces, including those that are administratively down.

RUNNING means that the interface appears to have a "carrier", "link up" or whatever we call that. This is a real-time property that signals whether the underlying link layer is in perfect order or not, end-to-end. Arguably, with sync serial point-to-point WAN links, we really care about whether or not the IP-over-L2 encapsulation protocol is ultimately up or down.

And this is where Linux seems to have a problem.

All the Ethernet hardware drivers responsibly report link state to the appropriate status flags in the respective kernel-space `struct net_device'. In contrast, almost none of the host of HDLC stacks take care, two notable exceptions being the drivers for 8253x and Farsync. The dscc4.c only sets the RUNNING attribute once at interface startup (at `ifconfig up').

In terms of encapsulations within the "Generic HDLC Layer", the PPP and Cisco HDLC encaps do have a proprietary flag in their per-interface data structs that holds exactly this information, based on state-machine status and based on whether or not keepalives are coming back. Their proprietary status flags are nevertheless not reported up, to the appropriate flag in the generic struct net_device, and there are no IOCTLs to retrieve this information. Thus, no generic kernel-space let alone user-space software can use this information for routing decisions.

The generic HDLC doesn't even have such a proprietary status flag, perhaps because it doesn't have keepalives (?). The FrameRelay base interface does have a "reliable" flag and the FR PVC does have an "active" flag, but I doubt that these would hold anything resembling the link integrity status that we're after.

It would only be desirable if the WAN interfaces could report the RUNNING status to facilitate routing decisions, just like most all Ethernet drivers do.

Another comment from Krzysztof Halasa: "Correct. I'm currently testing a patch adding IFF_RUNNING support for generic HDLC (not only FR) - it will be available soon. Still, it will only change the RUNNING flag and will not alter routing decisions."

As mentioned earlier, Cisco IOS reports physical carrier status separate from protocol-layer status (derived from handshaking and keepalives). Some interfaces, such as structured PDH ports or ATM with OAM-managed PVC's, report even more - apart from the encapsulation protocol keepalives at a particular subinterface and the physical link carrier presence, there's also the low-level frame sync status and a number of possible PDH/SDH network errors (AIS, FERF etc). Cisco IOS is able to aggregate all these status bits into two relevant per-interface flags: link up/down and protocol up/down. Obviously, when the link is down, the protocol goes down too, automatically. This allows for fastest possible re-routing of IP traffic.

Arguably, such error flags aggregation should also take place in Linux - for the purposes of IP routing, at the generic interface level, the single RUNNING flag is IMO good enough to indicate runtime link failures of any kind. This implies that low-level hardware drivers would have to watch physical link status (if the hardware is capable of that) and report it to upper-level encapsulation/multiplexing subsystems, affecting their per-channel interface up/down bits or even resetting their protocol state-machines. Alternatively, the gathering of such information could be implemented via nested function calls, that would aggregate (logically AND) all the status flags at the different network sublayers involved.

So much for link status detection and reporting. For now, let's suppose that the RUNNING flag is updated correctly. Let's take a look at how IP routing can respond to this information.

As mentioned above, Cisco IOS can do it using a semi-dynamic concept called "floating static routes", without requiring a full-fledged routing demon (RIP,OSPF et al). In Linux terms, this would correspond to a basic Linux kernel doing the routing based on a static table as set up by /sbin/route.

The question is: can the ipv4 routing engine in the Linux kernel respond to the RUNNING flag? Alone, without the help of a user-space demon managing its routing table? The answer is: NO, most certainly it can not!

In the simple dual-backup scenario, if one of the two links failed, the natural result in Linux would be a 50% packet loss, as one of the two equal-cost default routes would keep forwarding packets to a defunct interface.

An obvious counter-argument is that perhaps the performace-tuned kernel-space packet scheduling engine is not the right place to preprocess the routing table based on link state. In that case, there should probably be a simple daemon, user-space or kernel-space, doing the simple job of suppressing routes pointing to interfaces that are not "RUNNING".

As for routed, gated et al, another obvious suggestion is: "Run routed or gated with no peers defined, so that it just watches the link states." I don't have any personal experience with these - do they watch the RUNNING flag? Or do they rely strictly on their IGP/EGP handshaking?

Ultimately, running a full IGP or EGP setup would surely do the job even without link state monitoring - except that it might mean a considerable waste of resources on your ISP's PE boxes. Your ISP would have to choose between starting another dedicated session of the routing demon or enlarging his backbone routing domain by your Linux CE box (thus also allowing you to litter his backbone routing with your own misconfigurations etc.). Compared to the basic floating static routes setup, this also entails significant configuration complexity on both the CE and the PE.

As mentioned above, if the link is OK, `ifconfig' shows the RUNNING flag on the respective interface. In addition to the UP flag, meaning that the interface is administratively up.

       
net_device
{
   unsigned long state;
   unsigned short flags;
}

state: __LINK_STATE_NOCARRIER     (= link UP/DOWN)
flags: IFF_UP                     (= administratively UP/DOWN)

       

       
struct hdlc_device
{
   union
   {
      struct cisco
      {
  int up;
      };
   } state;
};

1 = up
0 = down

       

       
struct sppp
{
   int pp_link_state;
};

#define SPPP_LINK_UP    1       = up
#define SPPP_LINK_DOWN  0       = down

       

       
struct wan_device {
   char state;
};

enum wan_states {
 ...(state-machine states)...,
 WAN_DISCONNECTED,
 WAN_CONNECTED
}; // heavily used by the Sangoma's encap-specific drivers/net/wan/sdla*.c