.jpg)
A data-driven test and learn framework optimizes a multi-million dollar creative investment, reducing wasted spend and maximizing campaign impact.
Creative strategy, often seen as a purely subjective endeavor, is transforming. By leveraging data and advanced analytics, we're empowering our clients to make creative decisions with the same rigor they apply to media strategies. This evolution in our approach, from a foundational boosted tree model to a sophisticated AI-driven system, has allowed us to move beyond simple performance metrics and truly understand the "why" behind creative success.
Our journey began with the Machine Learning Creative Model Score, a tool built on a boosted tree model. This initial model analyzed historical data to predict creative performance. It worked by processing several key factors:
- Date: When the creative was live.
- Tactic: The type of ad (e.g., banner, video).
- Targeting Type: The audience for the ad.
- Creative Version: The specific creative theme or variation.
- Spend: The budget allocated to the creative.
- Clicks: The number of user interactions.
The model's core function was to assess the feature importance of each variable, quantifying its impact on clicks. By analyzing these factors, it generated a numerical Creative Model Score between -1 and 1. A higher score indicated a greater likelihood of a creative being effective, providing a data-backed recommendation for future use. While effective, this model had a significant limitation: it relied on pre-defined parameters and human-tagged creative themes, which limited its ability to discover new insights and scale efficiently.
Phase 2: Leveling Up with AI: A New Frontier in Creative Analysis
To overcome the limitations of the original model, we've integrated custom-trained AI into our creative analysis process. This represents a significant leap forward, as the AI is trained to perform three core functions:
- Creative Evaluation: The model can quickly assess and score new creative assets based on projected performance.
- Creative Summarization: It can break down a video into quartiles, providing a detailed look at the visual and audio elements within each segment.
- Creative Taxonomy: This is a crucial function. The AI not only categorizes creative content based on themes we define but also identifies emergent themes that we may not have seen.
A new system operates as a powerful feedback loop. A creative is first run through the AI to get a potential performance forecast and a rich set of taxonomies. Then, new data is fed into an existing boosted tree model, which can further analyze which themes are most successful based on the new, more granular AI-generated data. The combined approach allows us to extract new, richer insights to feed back to the creative team, allowing for a more informed and refined creative strategy.
Phase 3: Testing Creative for Linear TV with a Data-Rich Environment
Before making a significant investment in creative and media placements, we launched a new creative test for Groundworks using OTT and YouTube. The goal was to identify top-performing creatives across four service lines by leveraging the data-rich environment of these platforms. This approach allowed us to reduce wasted media spend and maximize the impact of future media campaigns.
The test focused on two creative versions: a "Symptom" creative, which used a fear-driven tone to highlight a problem, and a "Combo" creative, which took a warmer, more balanced approach.
Methodology: A Multi-Layered Approach to Creative Analysis
A three-pronged methodology was used to analyze the creative across both OTT and YouTube, providing a deeper understanding of creative effectiveness.
- Brand Lift Studies: For brand lift studies, OTT was used to measure the direct impact of each creative on brand awareness. On YouTube, a two-phased approach maximized budget efficiency: a small initial campaign identified the top-performing creative, followed by a brand lift study on the winner to provide brand-level insights.
- View-Influenced Visits (ViV): ViV rates, which measure site visits that occurred after a user viewed an ad, quantitatively assessed which creative was most effective at driving direct response and converting viewers into visitors.
- AI-Powered Creative Analysis: A custom-trained AI model was used on YouTube to analyze creative assets at a granular level. The model broke down each video into quartiles, which helped determine drop-off rates and identified subtle creative themes, such as specific sound effects or branding timing, that were influencing performance.
Outcomes: Actionable Creative Intelligence
The real value of this case study lies in the outcomes that informed creative decision-making, not just the media performance metrics. By combining media testing with advanced creative analysis, we gained a new level of creative intelligence.
Quantitative Results showed the "Combo" creative drove the highest lift in brand awareness and site visits across most service lines, while the "Symptom" creative was more effective for a specific service.
AI-Driven Creative Insights provided the "why" behind these results. The AI identified that a lack of upfront branding (intro 10-15 seconds in) hurt brand recall. It also found that specific sound cues in the first half of the spot effectively captured attention, while the use of more brand-specific sonic elements could enhance recall. The analysis also confirmed that the "Symptom" spot's fear-driven tone was good for initial attention, but the "Combo" spot's warmer approach built stronger, more lasting brand recognition.
By integrating these data points, we provided the creative team with a clear, data-backed blueprint. We didn't just tell them what won; we told them why it won and what specific creative elements to change to improve future performance.
.png)
.png)
.png)
