2016 Network Issues and plans for 2017

Discussion in 'Volunteer Information' started by gnar killin, Mar 9, 2017.

  1. gnar killin

    gnar killin Physics Don't Apply

    173
    63
    39
    May 11, 2015
    I know I'm early to the party, but I thought it might be a good idea to discuss some of the issues on a more technical level, and to see if the volunteering directors have already address and resolved these issues for 2017 or if solutions are still open for debate.

    Last year I was on Crimp Nation for the majority of the time and had a handful of hours on the NOC and Backbone. It is the latter two that concerned me a little bit.

    On the NOC I noticed that they were patching a series of patch panels together, back to back. I'm assuming this is so they could just put ends on the backbone runs and plug them into strait into the patch panel. For someone who is in the business of installing network infrastructure, this would have been a big no no for any of my jobs as it could create 3 additional points of failure. Not sure if this was just to save time or what, but from what I've heard (and I could be wrong) there was a lot of issues with the backbone connections.

    Which brings me to the Backbone. Were all of these runs thoroughly tested? I imagine they would have been. I was not privy to what issues they were having late Wednesday night.

    I have found cat5e to be highly unreliable, especially when it is laid out on the floor with hundreds of potential areas for people to knock, kick, or step on. Its also much more vulnerable to kinks and knots as it lacks the plastic core that cat6 has. When doing some crimping for the backbone, I noticed a lot of these. I realize that the majority of the cable may be donated. I'm curious to know how many linear feat of cat cable is used for the backbone. I may even be willing to donate some boxes myself.

    Anyways, I'm looking forward to Volunteering this year. Last year was my first and now that I have a general idea of whats involved on network setup, I think I would like to spend most of my time on one of those two teams, NOC and backbone.

    Please feel free to correct me if you find me in error, I can only go off what I experienced. I'd be happy to learn/know more about the full extent of the issues those two areas were having. Thanks guys!
     
    Vargess and LionBattery like this.
  2. Xipher

    Xipher Physics Don't Apply

    170
    18
    28
    Jul 14, 2011
    The QuakeCon network is operational for a week and MUST be built in 3 days, very different from your traditional network in a business. Punch downs are great when you do it once and it's left there for years.

    I have to admit some responsibility for presenting this idea, because past years where they punched the backbone down directly was being a blocker for other work since you can only really fit a single person in the space to terminate the panel. Now that the pass through has been built, we don't have to repunch every year and ends can be terminated on the backbone where it's more physically convenient and can be parallelized.


    Always are, you run as much cable as the infrastructure team does in 3 days and you will find problems.

    Yes, fluke certification tester provided by a volunteer.

    About 16 miles.

    None of that is donated, backbone cables are made from spools that are purchased for the event.

    I'm sure the help will be appreciated, but do understand the senior volunteers will pick people they trust which is built over time. I spent a few years volunteering for security and was later asked to join the IT team after they learned of my skill set. I never asked to join the team, they asked me.

    The feedback is appreciated, and hopefully my explanations will also help you learn more about the event. I'm by no means an authoritative voice on the physical infrastructure by the way, my role has been edge connectivity.
     
    Gameking27 likes this.
  3. gnar killin

    gnar killin Physics Don't Apply

    173
    63
    39
    May 11, 2015
    I hope you understand I wasn't trying to come off as insensitive or trying to point fingers. Just genuinely curious about the process that was taken this past year that I helped with. I'm also not saying that your solution was wrong either, just that it can introduce failure points in the network. If tested and punch correctly I'm sure it would be absolutely fine. And it does provide a nice solution to the time suck of having to painstakingly punch down each backbone run inside the rack. (Just wanted to clarify my intentions)

    I do understand it is not a traditional network for a business as I also have 5+ years in event experience with networking and A/V, and fully understand how complex and complicated issues can get. I also understand that I'm the new guy on the block and am by no means trying to insert myself into any team if the help is not requested or desired. I've been apart of the Qcon community for almost 10 years now and I'm just trying to figure out how to best give back to this great group of volunteers and fellow gamers.

    I appreciate the insight you gave me and hope we can continue to talk/collaborate in the events to come.
     
    jex likes this.
  4. Xipher

    Xipher Physics Don't Apply

    170
    18
    28
    Jul 14, 2011
    Sorry if I came off sounding like a dick.

    Like I said, my role isn't in the physical infrastructure. Someone from that team would definately have more detail then a router jockey.

    Hope to see you at the event this year!
     
  5. gnar killin

    gnar killin Physics Don't Apply

    173
    63
    39
    May 11, 2015
    Indeed you shall!
     
    Gameking27 likes this.
  6. Gameking27

    Gameking27 Physics Don't Apply

    168
    16
    27
    Mar 23, 2015
    Midlothian, Texas
    If i recall correctly, me and my buddy sat a few rows behind you (GORGEOUS PC BY THE WAY) and we had network issues from day one of the con right up until people started packing up and leaving the con. From outages in network to having to replace the switch itself. It was a pretty rough start for the crew im guessing.
     
  7. gnar killin

    gnar killin Physics Don't Apply

    173
    63
    39
    May 11, 2015
    Yeah, that is extremely unfortunate. I remember you guys talking about it at the con. Thanks for the compliment on the build!
     
  8. Signus

    Signus Physics Don't Apply

    206
    18
    23
    Aug 5, 2011
    Denver, CO
    If you've got that much experience, I would suggest you try to get yourself a seat on the Services and Connectivity team. CrimpNation is critical to QuakeCon, but if you know your way around laying out networks you could be of benefit to the improvements to the network.

    That being said, the network has greatly improved since I started attending in 2007, and is continuously improving, but we tend to have even more services running on each machine these days. It's a gargantuan task. Even considering the network is laid perfectly, you still have to get everything above L4 working on that network.
     
  9. gnar killin

    gnar killin Physics Don't Apply

    173
    63
    39
    May 11, 2015
    Most of my experience is actually in the installation of the copper and components, making sure connections from point A to point B are solid. Usually my clients have their own network engineers that configure the switches and routers to their own spec. But I don't think that was the issue they were having last year, I'm pretty sure they have the configuration part licked. I have a good deal of experience with the dell enterprise switches, but have avoided cisco. I'm not sure which they are using.

    I am curious though as to what they are using. I think a couple of their table switches failed last year, but that could of been the hardware crapping out. Someone else here would have to correct me on that one.

    I'd also be interested to know if they are using fiber SFP between the switches in the NOC.
     
  10. Signus

    Signus Physics Don't Apply

    206
    18
    23
    Aug 5, 2011
    Denver, CO
    I didn't get to attend '16 but I've seen it vary based on what's available or based on what dies and what is available as a backup. Generally I've seen the core network comprised of Cisco, but my memory could be of disservice to me.

    They have a huge collection of Dell 24/48P switches. They are usually flashed and checked prior to the event, and once they are connected to the network. Often these are switches stored in a box for the year until QuakeCon (hopefully in a AC unit unlike the BYOC CAT5), and some are bound to fail but there are a number of replacements the NOC staff keep around. Much of this in recent years has gotten better, as NOC staff has created standards as well as having a better budget.

    I know SFP was introduced when AT&T started sponsoring QuakeCon and providing Internet access. I don't know to what extent it is used otherwise.
     
  11. gnar killin

    gnar killin Physics Don't Apply

    173
    63
    39
    May 11, 2015
    Awesome, thanks for the reply. Very insightful.
     
  12. Signus

    Signus Physics Don't Apply

    206
    18
    23
    Aug 5, 2011
    Denver, CO
    No problem. I don't spend my time in the NOC too often these days as I'm nearly strictly on the Tournament team.

    Need to figure out a decent way to generate some load on each switch with a computer or two per switch to test issues in resolution, propagation, etc. so that any issues above L4 are found prior to the event. My friends that work at ISPs have some sexy hardware that does this, but I lack a few 500k-1M in pocket change.
     
  13. gnar killin

    gnar killin Physics Don't Apply

    173
    63
    39
    May 11, 2015
    Nice! Hopefully after a few more years of volunteering I could get on the Tourny team :p
     
    Signus likes this.

Share This Page