Simon

A story about how Jane Street unwittingly reinvented datacenters in 2012, involving office server rooms, natural disasters, market maker obligations, recovery sites, colocation facilities, and old limestone telco buildings in lower Manhattan. When I joined Jane Street from college in 2010, all of the firm’s proprietary trading systems ran out of a 8’X12’ server room on the 33rd floor of 1 New York Plaza, on the southern tip of Manhattan. Jane Street’s ETF market-making, options vol pricing, intercountry ADR arbitrage, and pan-equity stat-arb operated from this small deployment. The room itself was an engineering feat: the Dell server arrays were too heavy for the 1960s building’s floors to support, so the racks were affixed to the ceiling with concrete. By 2012 Jane Street was not yet a member of most major US exchanges, even NYSE, and had virtually no presence in colocation facilities. Nevertheless, it was a ~$2B company experiencing rapid growth, a designated liquidity provider in most listed ETFs, a significant percentage of US and international equity volume, and a critical operator in capital markets. In October 2012, Hurricane Sandy devastated lower Manhattan. 1 New York Plaza, Jane Street’s home, flooded to street level and the basement power generators were fully submerged. The resulting gas leaks made the office building hazardous to enter, and power and water were weeks away from being restored. Unprecedentedly, all major US exchanges closed for two business days after the storm but would be back up by day three. Jane Street was in danger of failing its lead-market-maker obligations if it didn’t relocate its business in time. Jane Street had recently set up a partial office in a central New Jersey office park to comply with FINRA/SEC regulatory obligations for business continuity planning. A few engineers and I showed up to the empty site during the hurricane to begin mass-redeploying the firm’s software and infrastructure for the next critical 48 hours. A truckload of Dell PowerEdge R610s arrived the next morning, some salvaged from 1 New York Plaza’s server room, carried down 33 floors in pitch black by Jane Street’s IT team. For the next few months, everyone at the company from Rob Granieri to new-hires rode rented regional travel buses from Manhattan to NJ to keep Jane Street alive and growing. When we all returned to Jane Street’s repaired 1 New York Plaza office in January 2013, the technology team realized something that’s obvious now: hardware doesn’t need to be colocated with people. The NJ disaster site had accidentally become Jane Street’s first firm-wide datacenter. The site afforded significantly more space, cheaper electricity, ground-floor weight support, and abundant cooling. This decision, to leave Jane Street’s servers in post-hurricane locations, would have profound effects on the firm’s infrastructure footprint and growth. In 2013 it became a top priority for the technology and infrastructure teams to move all latency-sensitive trading into colocation facilities in order to cross-connect directly to exchange matching engine switches. And the search began for a new primary datacenter to house the firm’s research cluster and non-latency-sensitive applications. In 2015 Jane Street found its new electronic home: lower Manhattan, limestone and 32-story windows, originally owned by the New York Telephone Company. It had retina-scan security and neon-colored liquid cooling and everything else we know today. They joined other major financial firms and software companies who had operated there for years. In modeling disaster scenarios, experts tend to underrate the technological redundancies that already exist, or can be willed into rapid deployment, to ensure the US financial system remains secure.