Analysing Campaign Results

Hello traders,

This is Tim with you, once again, to continue our video tutorial series on trading strategy optimization, using Pine Optimizer.

In the last session, we saw how to create and launch a Pine Optimizer strategy optimization campaign.

In this session, we'll explore how to analyze the results of that campaign and extract from it some useful insights that will help you to optimize your strategy.

I have here, the same two browser tabs that we used in the previous session, still arranged side-by-side in separate windows.

But, since we no longer need to see what's happening in TradingView and Pine Optimizer, at the same time, we can merge these windows and gain a little more space to work with Pine Optimizer across the full width of the screen.

Note that I'm not closing the TradingView tab, since TradingView always needs to remain active when you're using Pine Optimizer for strategy optimization, but it's OK to move it into one of the hidden tabs in your browser.

As a reminder, this is the Pine Optimizer backtest results view. We arrived here, in the previous video, by running an optimization campaign and then clicking through to see its results.

This table lists the 9 backtests that our demo campaign executed. For each of the listed backtests, it shows a few key metrics that might help you to evaluate the performance of your strategy.

The backtests list is the centrepiece of this analysis view and its purpose is to allow comparison of backtest performance.

There's a lot going on in this table, and we'll work through the details progressively, but if you ever need help to understand how it works, just click its contextual help button, and you'll be navigated directly to the specific help topic for this feature.

In general, every Pine Optimizer feature has a contextual help button in the top right corner of its user interface, but you can also use the global help button in the header navbar to access the main help index, which (directly or indirectly) provides access to every available help topic.

We're working hard to keep the help content up-to-date with the evolution of Pine optimizer, but that doesn't necessarily mean that we've anticipated every user question that might arise. So, if you can't find the information you're looking for in the help system, please do reach out to us and we'll work with you to overcome any obstacles you might be facing.

So, returning to the backtests list, we see a table with 10 columns, each of which represents one of the performance metrics for the backtest. Actually, Pine Optimizer supports almost 70 different backtest metrics, so this is just a glimpse of how your backtests performed.

To customize this table, to show one of the other metrics, we can open its settings dialog and choose the combination of columns that we want to see. For example, if I want to compare strategy performance to buy-and-hold return, I can easily do so by adding that column to the table.

You can add as many columns as you like the to the table, but when you have a large number of columns, you'll be forced to scroll the table horizontally to see all of them.

Thankfully, there's an easier way. If you want to drill into the performance of a particular backtest, you can open the backtest metrics panel, using its launch button in the side navbar. Then, when you select any of your backtests, you'll see its metrics in much greater detail.

By default, this panel shows all the supported backtest metrics, but you can eliminate some of the noise by only showing the metrics that are relevant to your analysis.

To do this, just open the panel's settings dialog and deselect the metrics that are of no interest to you.

So, we have a lot of data available to analyze the performance of our campaign, but how do we use it?

Typically, when you're optimizing a strategy, you're looking for the best-performing backtest, which you may then use as the basis for further optimization experiments. This is called progressive optimization, and we'll cover the concept in more detail in an upcoming session.

For now, let's focus on finding the best-performing backtest from among this rather limited set of 9 examples that we're working with.

A simplistic approach is to choose one of the metrics - one that we believe reflects trading performance - and then find the winning backtest for that metric. For example, if we consider that net profit is the most relevant performance metric, we can reasonably easily identify the backtest with the highest net profit, which then qualifies as the winner.

This technique is fine when the list only contains 9 entries, but it becomes a chore when you have to search through hundreds of rows to find the highest value. To solve this, we can sort the table on any of the columns, simply by clicking its column header. Click once to apply an initial sort of the rows and click again to reverse the direction of sorting.

By the way, if you're unsure about the role of any given performance metric, you can hover over its column header to see its summary description. If that doesn't provide enough clarity, you can get more detail from the associated help topic, which you can directly access, from the backtest metrics panel, by clicking the help icon in the relevant row.

Now, I just mentioned that this is a simplistic approach to comparing backtest performance. And, that's because we typically need to consider multiple metrics, when choosing the best performer.

For example, although profitability is clearly an essential ingredient of a top-performing strategy, risk is also an important balancing factor. Most traders agree that it feels very uncomfortable to sit through long periods of drawdown, even if the eventual outcome is highly profitable, so it usually makes sense to balance profitability and risk, when evaluating backtests. And that means comparing multiple metrics.

One way to achieve this is by using one of the ready-made composite metrics, like the Sharpe Ratio or Sortino Ratio, which were created specifically to solve this problem. You can certainly take this approach, and Pine Optimizer already supports 3 metrics of this type, with more to be added in the future, if there's a demand for them.

But Pine Optimizer also has a feature that's designed to offer more flexibility than predefined ratios. It's called composite scoring, and you can see it here in the Score column. A composite score is your personalized scoring formula, applied to the other metrics in the backtest. A composite score can be based on any selection of metrics, combined in any ratio that makes sense to you.

For example, if you want to base your scoring on net profitability and drawdown, but you think that profitability is twice as important as drawdown, you can define a scoring formula that combines these two base metrics, in the ratio 2:1, and Pine Optimizer will then automatically score all of your backtests in this way. I'll show you how to set up your personal scoring formula in a later video.

But, for now, it's worth noting that a well-designed scoring formula can greatly simplify the way you compare backtest results and it will also keep your comparison process consistent.

At this point, you may be thinking that this table looks like an overwhelming mass of numbers, and you'd have a point. One of the ways that we can use, to get the useful insights to jump out from the data, is by applying some color.

If we open the backtest list settings dialog and switch to the options tab, we can activate color highlighting, which applies contextual shading to the table cells. Each cell is highlighted with a shade of green or red, which reflects the positive or negative implications of the cell value, relative to other cells in the same column. So, the best-performing cell in every column will be assigned a bright shade of green and the worst performer will get a bright shade of red, with all the intermediate values receiving a less intense shade that falls proportionally between those two extremes.

If we apply a column sort to the table, the header of the sorted column will also be color highlighted, to reflect whether the first cell is the best or worst performer in the column.

It's worth noting that Pine Optimizer correctly assigns color shading, based on the sentiment of a cell, not just its numeric value. So, the highest net profit will be assigned the brightest shade of green, while the highest drawdown will get the brightest shade of red, since these two metrics are opposite in sentiment.

When you evaluate a color highlighted backtests list, you should ideally be looking for a backtest that has all of its cells bright green, since this means that it is the winning performer across all of the visible metrics. But, to be realistic, this won't occur very often, since you're dealing with real-world data, so you should aim to find the backtests that are green across the greatest number of columns.

Keep in mind that this table only reflects a handful of the available performance metrics, so, even with color highlighting, you shouldn’t treat it as the complete picture. If you want to see a full visualization of an entire backtest, you can turn on color highlighting for the backtest metrics panel, again via the options tab on its settings dialog.

With color highlighting active, what you're seeing here is the performance of every metric, relative to the same metric for all the other backtests under comparison. Again, it's unlikely that you'll find a backtest that's all-green, but a high proportion of green cells is a good indicator that you're looking at one of the stronger-performing backtests in your campaign.

Returning to the backtests list, there are several different ways in which we can present this data, depending on our analysis goals. This table is currently configured in absolute mode, meaning that most of its columns contain absolute values, such as dollar amounts. For example, the gross and net profit columns currently show the actual amount of profit earned, in the account base currency which, in this case is US dollars.

But, very often it makes more sense to evaluate trading performance using relative values, such as percentages. We can use the presentation mode toggle button to switch the table to percentage mode, and all of the columns that support multiple modes will then be adjusted to reflect percentage values. The backtest metrics panel also supports multiple presentation modes and, again, when you switch modes, all of the metrics that support multiple modes will immediately be adjusted.

If you're curious about the mathematics behind a particular metric, you can always refer to its help topic, as I described previously. For metrics that support mode switching, you'll see a table that lists its supported modes and describes exactly how each of them is calculated.

You'll notice that this example shows 3 modes; the two that I just described and a 3rd mode, which we call risk-relative mode, indicated by the letter R. This 3rd mode represents metric values as a proportion of trade risk, and is especially useful for traders that set themselves performance goals in terms of risk/reward ratio. To make metrics like this feasible entails a minor modification of your Pine Script code, in order to capture the risk level of each of your trades. We're currently working with a group of test users to determine the best way to implement it, so this feature is not yet deployed. Therefore, although the mode toggle does include the R mode among its options, please be patient until we activate this feature, at some time in the near future.

One other way you may wish to adjust the backtests list is by using trade direction filters. In its current state, this table is showing the aggregate results of all trades in each backtest. But, by applying a direction filter, we can visualize how the backtests would differ if we executed only the short trades, or only the long trades. For some strategies (and the MACD strategy seems to be a good example of this) excluding one of the trade directions can have a dramatic effect on overall performance, which represents an instant optimization, regardless of which backtest you ultimately choose as the winner.

When you've identified a winning backtest, you'll either use it as the starting point for further optimization campaigns, or you may wish to consider it for active trading. Either way, you'll need to know the exact configuration of that backtest, in order move forward with it.

You can view the strategy properties that were assigned to a backtest, by opening the strategy properties panel, using its launch button in the side navbar. These values will be identical across an entire strategy version, regardless of which backtest is selected, and will typically be the same as the default strategy properties defined in your global settings (although they may differ if you assigned custom values when you created your strategy).

You can also view the unique parameter configuration of a backtest, by opening the backtest parameters panel, again using the appropriate launch button in the side navbar. The parameters listed in this view correspond to those defined in your Pine Script. And, for any selected backtest, this panel shows the actual parameter values assigned during backtest execution.

At this point, you've probably seen enough to effectively analyze your backtests and identify a winner. But, occasionally, you may need to drill deeper, by examining some individual trades.

To to this, we open the trades list panel, which presents all of the trades for the selected backtest. In its initial state, the table shows the trades in chronological order, meaning that the oldest trade is at the top.

But this table works in exactly the same way as the backtests list, so you you can click on any column header, to sort the trades in a different sequence ...

... you can switch the presentation mode to see the column data in either absolute or relative format ...

... you can add or remove columns, according to your preferences ...

... and you can enable color highlighting, to see which trades contributed the most or least to your backtest performance.

As with the backtests list, the trades list includes only a handful of columns that reflect the most important trade properties.

If you need to drill into the detail of a particular trade, you can open the trade properties panel, using its launch button in the side navbar. Then, when you select any of your trades, you'll see its metrics in much greater detail.

This panel works just like the backtest metrics panel. By default it shows all the available trade properties, but you can adjust this, to suit your preferences, using the settings dialog. This panel also supports color highlighting and allows you to switch between absolute and relative presentation modes, but since you've already seen these features, I won't show you again.

Before we wrap up, let's quickly look at two more views that can be helpful in comparing the performance of your backtests.

The equity curve view is very similar to the equity curve that you're already familiar with, from TradingView's Strategy Tester, which allows you to visualize the evolution of your trading account balance over the duration of the backtest period.

The difference, here, is that you can switch between any number of backtests, giving you a more immediate and meaningful understanding of the equity impact of minor changes in backtest configuration.

Finally, there's the trades timeline view, which is unique to Pine Optimizer, and shows the distribution of trades, along the timeline of the backtest period. This view gives you an intuitive visualization of how your trades collectively contributed, positively or negatively, to your account balance.

And so, we've reached the end of this tutorial on analysing Pine Optimizer campaign results.

In the next session, we'll be looking at how best to manage your Pine Optimizer workspace. Until then, thanks for watching.