A/B Testing the Externship USP

The setup.

A/B testing is one of the most overrated and simultaneously underused tools in product marketing. Overrated, because most teams expect it to generate discoveries. Underused, because most teams test the wrong thing: button color instead of the offer, CTA wording instead of the value proposition, presentation instead of substance.

A test doesn't discover anything. It confirms and monetizes what you already learned through qualitative research. The strongest A/B results always follow deep customer interviews, when the hypothesis is sharp enough to produce a noticeable effect on real traffic.

For companies trying to improve conversion, the right question isn't "what should we test" but "what do we already know well enough to bet money on." Without an answer to that question, a test becomes an expensive way to confirm that you don't know your product or your customer well enough.

What had to be true.

Before the test was designed, two metrics were declining simultaneously: completion rate and purchase conversion. The sales cycle had stretched from 10 days to 30. Two signals pointing at the same root cause: the page was either attracting the wrong people, or failing to convince the right ones that this was worth buying.

By the time the A/B test launched, the externship program had already been validated through customer discovery and an MVP. Interviews with 30% of active and potential users had given a clear answer: people were buying employment, not education. The old landing page with its "Choose your career" headline and four tracks on the first screen wasn't selling that. It was selling optionality.

The test had one job: to check whether the new offer "Apply your skills with partner companies. Add experience to your resume." with real business problems, portfolio projects, and live presentations to employer representatives as proof, produced a measurable lift in visitor → qualified call lead conversion. And to do that in a way where the decision came from statistics, not gut feel.

A PR push around the program launch was running in parallel and was factored in as an additional variable during analysis — standard practice when you can't control all external factors in a real business environment.

How it moved.

What I actually did.

The process started before any test was designed, with customer discovery interviews across 30% of active and potential users. Those conversations revealed the core insight: students weren't evaluating curriculum breadth. They were asking one question: will this get me a real job? The externship program was the most concrete answer the product had. The hypothesis was that leading with it, rather than burying it, would change conversion. Those interviews followed an MVP validation with a small group, which had already confirmed the core hypothesis. The A/B test was the next step: validating it at the scale needed to make a confident product and positioning decision.

Before the test launched, three things were locked in: the baseline conversion rate of the control page (3.0–3.5% visitor → qualified lead), the minimum effect size considered meaningful (MDE of +30% relative, targeting ~4.5%), and standard reliability thresholds: 95% confidence and 80% test power.

From those parameters, the required traffic was calculated: roughly 6,000–11,000 people across both variants. At a daily traffic of ~800 people on the page, that meant approximately two weeks. Two full weeks was a hard floor, not a guideline, to capture complete weekly cycles. Weekday and weekend behavior differ, and stopping early inflates false positives.

One thing was tested: the value proposition on the first screen. Control: the old generic "Choose your career." Variant B: "Apply your skills with partner companies. Add experience to your resume." With real business problems, portfolio projects, and live presentations to employer representatives on the first screen. 50/50 split, single URL with client-side rendering, no redirect. Redirects add latency and break attribution. Each user was assigned to a variant via cookies. The same person always sees the same version.

Tools: Google Optimize initially, then VWO and Optimizely after Google Optimize was discontinued. Measurement layer: GA4.

Primary metric: visitor → qualified call lead. Guardrail metrics: what happened to the lead downstream. Did they show up to the call, enroll, complete the program, reach payment. This turned out to be the key insight: the externship variant lifted not just submission volume but lead quality and downstream conversion to payment. A page that "lost" on raw form submits could win on students who actually paid and completed.

One honest caveat: the parallel PR push brought in a different audience and raised brand awareness simultaneously with the test. That's a confounding factor we couldn't isolate. The 5% conversion figure is a before/after with several simultaneous changes, not a clean A/B. Inside the test itself, the control ran at 3.0–3.5% and variant B at 4.5–5.0%, a relative lift of 30–40%.

The most important finding wasn't in the numbers. The team believed the selling point was breadth of choice: four tracks, a strong curriculum, flexibility. That didn't move conversion. What moved it was one specific proof of employment. Customer interviews predicted this. The test confirmed it.

Beyond conversion, the sales team reported shorter conversations: leads arrived already understanding the externship offer and asking fewer foundational questions. The time from first contact to enrollment decision shortened. Nine months later, an additional finding surfaced: students who enrolled through the externship-led page completed the program at higher rates. In EdTech, completion rate is one of the most important product metrics. This was a bonus the test wasn't designed to measure.

Measurable outcomes.

• Sales cycle dropped from 30 days to 10: leads arrived already understanding the offer, enrollment decisions came faster
• Page conversion lifted 30–40% relative: Variant B showed 4.5–5.0% vs. 3.0–3.5% for control
• CustDev hypothesis confirmed at scale in 2 weeks
• 9+ months later: higher completion rate among students enrolled via the externship USP page

The stack behind it.