A backtest universe that cannot see the future

Every momentum backtest we have seen on Indian equities starts the same way: download today's NIFTY 500 constituents, pull their history, run the strategy from 2012. The results look great. They should. Every one of those 500 companies is, by construction, a company that survived to today. The ones that went to zero are not in the list, so the strategy never had to hold them.

TejHQ now publishes a universe that does not have this problem. It is rebuilt every night from the raw bhavcopy, one membership list per month, using only rows dated on or before that month's rebalance day. This post covers the construction, the API, and the data bug it surfaced in our own pre-2012 rows.

The construction

Four rules, chosen for being boring and hard to game:

• Rank by trailing 63-trading-day mean turnover, in rupees. Turnover is split-invariant, so raw bhavcopy is the right input; no adjustment layer involved.
• Rebalance on the first trading day of each month, hold until the next. Monthly, not daily, so one noisy session does not churn membership.
• Keep the top 500. liquid100 and liquid250 are rank <= N filters over the same rows, so they nest exactly.
• Require a full 63-day window before a name is eligible. A stock that listed three weeks ago has no trailing turnover to rank on.

Delisted names stay in the months they qualified for. The symbol is the symbol in force on the rebalance date, so a 2014 row for what is now ZOMATO says whatever it was called in 2014. That is the whole point.

df = (
    prices.select("date", "symbol", "isin", "name", "turnover")
    # One row per (instrument, date): a symbol in two series counts once, on the busier one.
    .sort(["key", "date", "turnover"], descending=[False, False, True])
    .unique(subset=["key", "date"], keep="first", maintain_order=True)
    .sort(["key", "date"])
    .with_columns(
        pl.col("turnover")
        .rolling_mean(window_size=63, min_samples=63)
        .over("key")
        .alias("avg_turnover"),
    )
    .drop_nulls("avg_turnover")
)

# First trading day of each month, from the exchange calendar as seen in the data.
rebalance_days = (
    trading_days.with_columns(pl.col("date").dt.truncate("1mo").alias("month"))
    .group_by("month").agg(pl.col("date").min().alias("rebalance_date"))
)

ranked = (
    df.join(rebalance_days, left_on="date", right_on="rebalance_date")
    .sort(["rebalance_date", "avg_turnover", "key"], descending=[False, True, False])
    .with_columns(pl.int_range(1, pl.len() + 1).over("rebalance_date").alias("rank"))
    .filter(pl.col("rank") <= 500)
)

Notice the join: membership on a rebalance date uses the trailing mean as of that date. The window only ever looks backwards. There is no step anywhere that can see a later row.

Reading it

One parquet per exchange, universe/<ex>_liquid.parquet, on R2 and HuggingFace like everything else. Each row is one instrument in one month:

exchange           NSE | BSE
rebalance_date     first trading day of the month
valid_to           day before the next rebalance
rank               1 = highest trailing turnover
symbol             symbol on rebalance_date
isin               ISIN on rebalance_date ("" when the source lacks it)
name               instrument name on rebalance_date
avg_turnover_63d   trailing mean turnover, rupees

The API endpoint takes an as_of date and returns the membership that was in force then, meaning the latest rebalance on or before it:

$ curl -s -H "Authorization: Bearer tej_live_..." \
    "https://api.tejhq.dev/v1/universe/liquid500?exchange=nse&as_of=2012-06-15" \
    | jq '.meta, .data[0]'
{
  "as_of": "2012-06-15",
  "count": 500,
  "rebalance_date": "2012-06-01"
}
{
  "rank": 1,
  "symbol": "SBIN",
  "isin": "INE062A01012",
  "name": "STATE BANK OF INDIA",
  "avg_turnover_63d": ...
}

A backtest loop then becomes: for each month, ask for the universe as of the rebalance day, trade only those names, move on. The names that later disappeared are in there for exactly the months they deserved to be.

What it exposed

The first time we built the universe for 2010 and 2011, the ISIN column was empty for every row. We knew that: NSE bhavcopies before January 2012 do not carry ISIN, and we said so in the backfill post. What we had not noticed is what the rest of the pipeline did with those nulls.

Every derived step keys on ISIN. The adjustment step joins corporate actions to prices by ISIN, so with null ISINs it joined nothing: 2010 and 2011 adj_close equalled raw close, quietly, for two years. The metrics step partitions rolling windows by ISIN, and Polars puts every null into one group, so the 2010 52-week high for RELIANCE was computed over a window that mixed in every other stock on the exchange. No error. Just wrong numbers with a confident schema.

The continuity rule

Backfilling ISIN backwards from 2012 is easy to do badly. Tickers get reused: a symbol that delisted in 2010 and was reassigned to a new listing in 2013 must not inherit the new ISIN. The rule we settled on uses only one fact, that the instrument traded on both sides of the cutover:

def isin_anchors(prices: pl.DataFrame) -> pl.DataFrame:
    has = prices.filter(pl.col("isin").is_not_null() & (pl.col("isin") != ""))
    missing = prices.filter(pl.col("isin").is_null() | (pl.col("isin") == ""))
    first_isin_day = has["date"].min()
    last_null_day = missing.filter(pl.col("date") < first_isin_day)["date"].max()

    after = has.filter(pl.col("date") == first_isin_day).unique("symbol").select("symbol", "isin")
    before = missing.filter(pl.col("date") == last_null_day).select("symbol").unique()
    # A symbol inherits its ISIN backwards only if it traded on the last
    # ISIN-less day AND the first ISIN day. Continuity, or nothing.
    return after.join(before, on="symbol", how="inner")

Symbols that fail the test stay null, and the downstream partition key falls back to the symbol so they can never share a window with anything else:

def series_key() -> pl.Expr:
    """Partition key for per-instrument windows: ISIN, else the symbol."""
    return (
        pl.when(pl.col("isin").is_null() | (pl.col("isin") == ""))
        .then(pl.concat_str([pl.lit("sym:"), pl.col("symbol")]))
        .otherwise(pl.col("isin"))
    )

The raw bhavcopy parquet is untouched; it still says what NSE said. The backfill applies to derived outputs only: adjusted prices, metrics, symbol history, universe. After the rebuild, RELIANCE on 4 January 2010 has adj_close 235.85 against a raw close of 1,075.50, which is what sixteen years of bonuses, splits and dividends should do.

Where it sits

The universe endpoint is on the Pro tier, along with the metrics, batch and resolve endpoints we cover in the next post. The parquet is open like every other tree: pull universe/nse_liquid.parquet from HuggingFace and you have the full month-by-month membership since 2010 with no key at all. If you build something with it, we would like to see the equity curve, especially the parts that look worse than the survivorship-biased version.