Technical

HVENS: What services does HVENS provide? What’s in Scope and what is the Co-Op responsible for?

User documentation
08/01/2019

What does HVENS provide for the co-op (primary scope):

  1. Co-Op purchases IP space from broker and pays for ARIN membership/dues. HVENS assists with documentation and process to get these
    going. This way Co-Op owns the space and can take it with them if they want to part ways and engineer their own network apart from HVENS.

HVENS does have an option for leasing IP space if a CO-OP customer needs to go this route. We typically lease only dynamic IP (CG-NAT) ips. Not infrastructure or business customers (permanent).

  1. HVENS manages all BGP routing that a customer needs to do (upstream transit providers are managed at Pixel Factory, all peering relationships, internal routing etc). HVENS customers do NOT need to have routing expertise on staff. HVENS will engineer failover as requested by the customer (dual circuits, multi-path, load balancing, etc).
  2. HVENS provides a managed switch/router at the interconnect points (typically a substation or office the Co-Op runs) to terminate the 1G/10G/100G connection back to Pixel.
  3. HVENS provides and manages fast DNS caches for customers, DHCP services for customer addressing and logging.
  4. HVENS provides A10 clustered load balancer for carrier grade NAT. All customer ipv4 traffic that requires NAT is hauled to Pixel
    factory, NAT+LOG is performed, and the traffic egresses to the internet transit/peering. HVENS provides local ipv4/ipv6 routing for
    customers that do NOT need CGNAT services on the local handoff switch/router. For example 2 business customers that are terminated on
    the same switch at a substation can talk locally w/o backhaul to PixelFactory.

A10 is the industry leading CGN provider with support for universal plug and play protocols (Xbox just works).

  1. Most co-ops will want to re-use the same IP transit for internal operations but have the 2 networks secure / separate. Managing the
    full internal corporate / SCADA / telemetry networks is not in scope but HVENS will provide assistance to the NetOps team to make sure the
    IP delivery, VLANs, security, needs/concerns/questions/asks are all taken care of. Typically a customer that has redunant fiber handoffs
    will want a dual path routed network with failover for both customer and internal traffic. The customer will be responsible for their
    internal firewall config; HVENS helps the customer get what they need to make things as redundant as the customer can afford/manage.
  2. IP Addressing design and use management. DHCP pools, NAT pools, static customer pools, etc need to be allocated properly to avoid
    waste and managed. Subnets cant be too small (waste, complexity) or too large (waste, security and performance issues).

HVENS will handle static IP address allocation for business customers (end customers). Customers may not want to use the gateway; typically
they will have a firewall and want to plug direct into the ONT. Some customers will want to use the co-op provided gateway but will want a
non-dynamic ipv4 address. HVENS will manage this using DHCP reservations. Either way the end customer will get what he/she needs
to work properly.

  1. Network monitoring of links, lite levels, snmp traps, Netflow usage reporting. Note: The A10s will do netflow, our lower cost NAT solution
    (ideal for startups) will also do netflow IPFIX. Some of the cisco ToR switches we use for handoffs have LIMITED Netflow capabilities. In
    most cases this does not matter as the CGN netflow is what matters. Customers that want device level netflow will want to use a HVENS
    suggested switch model that will have a moderately higher hardware cost.
  2. Centralized syslog repository (routers/switches/dhcp/NAT logs)
  3. CALEA response, DMCA response – customers will have full access to look at log and DHCP data but often will want us to help prepare a response or assist in digging thru the logs. Attack traffic response (customer who is hacked/infected or violating AUP) is also included. HVENS does not remediate PCs but we will certainly do what we can to help the customer track down the issue and take the offender offline for remediation by a local service tech / PC rebuild.

Note: Lawful intercept for monitoring real-time traffic is an add-on feature we provide via the A-10. This is an extra license cost. Responding to requests for information, logs, etc is included in the HVENS base offering.

  1. Full assistance on troubleshooting fiber and routing issues (lite loss, packet loss, etc), routing, dns. Routing problems may occur
    with VPN connections to providers that have packet loss. HVENS will investigate and if its something we can help with (either to explain or fix) we will.
  2. HVENS will maintain reverse DNS for all IP blocks to assist with proper geo-location. Most customers that buy IP blocks will need help
    with Netflix, Maxmind, and other geolocation databases. HVENS provides this.

HVENS Optional services:

  1. Most customers are operating layer 2 gear at the substations and only doing routing where a connection to Pixel Factory lands. If a customer
    wanted to route at all substations we can engineer this configuration; hardware costs and are higher to cover the additional routing devices.
  2. A10 add-ons:

A10 DDOS protection
A10 Lawful intercept
A10 ThreatStop

  1. A10+Threatstop malware identification and suppression (DNS based). This helps manage customer infections before they get IPs blacklisted,
    cause reputational harm, legal issues, operational overhead expenses and/or waste your bandwidth.
  2. Help with internal/scada/telemetry network design routing / firewalling, etc if needed/desired. VPNs (Ipsec/DMVPN) as well as failover.
  3. Backup storage and replication – PixelFactory provides rack space and colo services. HVENS configures the switching and routing to be
    able to support replication and failover to DR environments if desired at Pixel Factory.
  4. VM hosting – typically customers will have us host the management vms for Calix CMS and related services. We can be primary or a hot standby environment (failover).
  5. IP blocks or CG Nat ranges – if you run out of space we have extra reserve capacity for NAT pools.
  6. Consulting – call us if you have questions or problems !
  7. NEAT rack non-profit program – take advantage of opportunities to train your next generation of engineers. Re-train your current IT support staff to be able to do more.

So what does the Co-Op have to provide?

  1. Fiber ring between the substations – design and build and maintain your own fiber plant. Make your ring as redundant as possible.
  2. Fiber to the home
  3. Calix or similar ONT/Residental Gateway contract and hardware. I.e. manage the in-home service delivery gateway boxes.
  4. SIP/VOIP Phones contract (Momentum is who most of the current players are using)
  5. Support staff to handle physical installs
  6. Billing/accounting systems – manage your subscribers, payments, and plans.
  7. Technical resource(s) to coordinate with HVENS. You do not have to staff advanced routing or switching engineers. Thats
    what HVENS is here to do for you. Basic understanding of system administration practices, vlans, ip addressing, ethernet services, cabling and service provider concepts is all that is needed. A basic understanding of VLANs and service provider IP networking is helpful.

 

HVENS: What kind of redundancy and logging / tracking does HVENS provide?

User documentation
08/01/2019

What is HVENS ? What kind of architecture is in place?

  1. Routed backhaul from customer substation / handoff to Pixel Factory. We do have bridged (layer 2/vlan) that we can do in test mode but this will not scale beyond a 1000 or so customers due to bridge table mac limitations in carrier transport. DHCP is centralized using helper/DHCP relay so that the edges can be routed (no bridging of customer residential gateways back to a central broadband access server).
  2. The design calls for a layer3 switch (redundant of desired) at the customer handoff (typically a substation or Co-OP HQ) with ipv4 and ipv6 routed back to hvens. Hvens handles the ipv6 DHCPv6 Prefix delegation and the IPV4 CG-NAT + DHCP. Customer v4 traffic is routed to our CG NAT cluster and v6 traffic is routed direct to the internet.
  3. (2) BareMetal A10 CGNAT appliances in an H/A configuration. This provides 40G of CG-NAT capacity.

HVENS uses the A10 platform because the A10 NAT engine provides the best support for Universal plug and play nat protocols like what the XBox needs in order for end users to host games.

IMPORTANT: This tends to be the single biggest scaling challenge that startup providers run into (NAT on a cisco ASR or Juniper works – sort-of). When you scale beyond a few hundred test customers the ASR/SRX/Mikrotik typical firewall or router NAT does not scale (CPU or feature wise). Universal plug and play support is critical for double nat (CGN) environments. There is no fix other than dedicated CGN appliances; thats why we picked A10 – A10 does this UPnP support better than anyone else.

We also offer port forwarded ports on shared ipv4 for customer security cameras. This allows customers that have a small allocation (single /24) to more efficiently utilize IP space for customer that want to setup Home Security systems and need a port forwarded for camera/NVR access.

  1. Redundant customer DNS caches – hosted at HVENS. This way the customer does not have to provide, manage, and maintain a DNS infrastructure. HVENS has 2 sets of caches, 1 set has basic content filtering on it for inexpensive reasonably-family-safe browsing that customers can decide to use.
  2. Redundant DHCP (ISC) server, with syslog log collection. Routers and switches will be configured for syslog collection on the DHCP vm. DHCP logs and syslog data is replicated offsite and kept for historical (DMCA/CALEA) purposes.
  3. RVA-IX /Pixel Factory peering:

RVA-IX is a growing exchange that provides a lot of benefits for traffic optimization. PixelFactory is growing fast with both new private peering and new public ip transit connections to provide capacity as our new HVENS customers come onboard.

Akamai Cache cluster
Google Cache cluster
3 GTLD DNS servers
1 TLD DNS server
Virginia Universities and businesses

Netflix 192TB OCA Cache appliance (available off Pixel Factory vlan)
40G/sec capacity.

In addition Pixel Factory has added 10G of PNNI (private peering) direct to Google in Q1 2019.

Over 1000 direct peers for major CDNs, Colleges, and ISPs.

Peering in Ashburn, Atlanta, Manassas, NewYork, Boston, Chicago, Sandston/QTS (Seattle coming in Q4 2019)

Does HVENS provide DDOS mitigation?

  1. Yes, RTBL (real time blackhole – drop traffic) of individual IPs can be executed as part of the built in service (BGP) and with our A10 appliances we can add in finer grained DDOS protection and traffic scrubbing.

How does the DHCP/ISC / logging work?

We use isc dhcp server on a vm. I have a tool we developed that snapshots the leasedb file once per hour for v4 and v6 to a separate folder. That folder in turn is replicated offsite to a DR node as well in case there is an issue with the DHCP vm. The replication tool uses dir purge to keep the number of files under control. Each snap file is less than a MB, so you
can have 60 or 90 days of lease info more as required by the customer.

Example of what the files look like: – the date and then the hour. So even
if a customer was only using an ip for a little while and did something bad, they will be trackable.

ls /data/dhcp_lease_snap/

2019-04-10_19_v4_leasedb.txt  2019-04-11_02_v6_leasedb.txt

2019-04-10_19_v6_leasedb.txt  2019-04-11_03_v4_leasedb.txt

2019-04-10_20_v4_leasedb.txt  2019-04-11_03_v6_leasedb.txt

2019-04-10_20_v6_leasedb.txt  2019-04-11_04_v4_leasedb.txt

2019-04-10_21_v4_leasedb.txt  2019-04-11_04_v6_leasedb.txt

2019-04-10_21_v6_leasedb.txt  2019-04-11_05_v4_leasedb.txt

2019-04-10_22_v4_leasedb.txt  2019-04-11_05_v6_leasedb.txt

2019-04-10_22_v6_leasedb.txt  2019-04-11_06_v4_leasedb.txt

2019-04-10_23_v4_leasedb.txt  2019-04-11_06_v6_leasedb.txt

2019-04-10_23_v6_leasedb.txt  2019-04-11_07_v4_leasedb.txt

2019-04-11_00_v4_leasedb.txt  2019-04-11_07_v6_leasedb.txt

2019-04-11_00_v6_leasedb.txt  2019-04-11_08_v4_leasedb.txt

2019-04-11_01_v4_leasedb.txt  2019-04-11_08_v6_leasedb.txt

2019-04-11_01_v6_leasedb.txt  2019-04-11_09_v4_leasedb.txt

2019-04-11_02_v4_leasedb.txt  2019-04-11_09_v6_leasedb.txt

The actual lease data provides the client mac and agent.circuit-id and agent.remote-id info.

lease 100.64.24.21 {

starts 4 2019/03/14 13:19:40;

ends 5 2019/03/15 13:19:40;

tstp 5 2019/03/15 13:19:40;

cltt 4 2019/03/14 13:19:40;

binding state free;

hardware ethernet 44:65:7f:13:ef:4f;

set vendor-class-identifier = “844E-1.ENT.dslforum.org”;

option agent.circuit-id “APMTVAHUT01 Eth 2/2/4/399/g1:205”;

option agent.remote-id “CXNK005129C5”;

}

Right now we dont have a dedicated DHCP server for empower b/c you are in
testing mode. I just have the DHCP running on the clustered CGN
appliances. Are you still at 3 test sites ? If you are moving into
go-mode, let me know and we can get a dedicated DHCP vm setup for you.

Some of our customers want ssh access into the isc dhcp vm(s) some want us
to manage it all and they dont want access. Its up to you. If you just
want a copy of the lease files so you can do diagnostic (think CALEA
request) searches on your own we can do that as well.

What kind of redundancy options are baked into the design to avoid single points of failure?

  1. Pixel Factory has peering and transit across a wide range of 10G, 20G, 40G connected peering exchanges. Pixel uses standard based BGP protocol for redundant routing. 100G connections adding in Q4 2019.
  2. Pixel Factory has redundant datacenter fabrics that the HVENS Co-Op aggregation rack attaches to at 20G on each path (total of 40G
    current capacity). HVENS routers peer with Pixel Factory via BGP. When customers order/buy their own dedicated IP blocks for the Co-Op HVENS will ensure that
    the IPs are announced from the CUSTOMER ASN (a globally unique routing number that identifies the customer). If a customer does not have an ASN or does not order its
    own IP block, HVENS will use HVENS ip block and ASN. Either way the routing redundancy is not affected. Full BGP redundancy is in place.
  3. HVENS servers are attached on port channels (dual physical links with per flow packet load balancing).
  4. HVENS NAT appliances are clustered using commercial A-10 H/A high speed hardware CG-NAT appliances inside HVENS. This means that a HVENS
    customer does NOT have to provide rack space, power or gear onsite (like expensive servers or routers) at substations to handle NAT. Its
    all handled in the HVENS cloud and connected via ipv4/ipv6 routed pathway back to HVENs.
  5. Transport from Pixel Factory is typically provided by MBC fiber. Some customers contract with an alternate carrier for a 2nd path (if desired).
  6. HVENS has switching infrastructure H/A clustered on the HVENS side.
  7. Customer switching / routing can be clustered (H/A) if the customer choses this extra level of protection.