FIA Insights – Computer Vision
From AI-assisted shape recognition and radio message tagging to simply having more eyes and ears on race incidents, the capability of the FIA’s Remote Operations Centre (ROC) has grown massively over the 2023 season, as the FIA’s Head of the Remote Operations Centre and Deputy Race Director Tim Malyon and Single Seater Head of Information Systems Strategy Chris Bentley explain…
How has the ROC developed since its launch? What have been the main areas of focus?
Tim Malyon: The ROC was launched at the beginning of the 2022 season, three weeks before the Bahrain Grand Prix and it we went through several phases. The first was getting the technology and the architecture right, so I would say that races 1-4 were an observational phase. Races five to eight, were about implementing greater levels of technology to allow the ROC to actively support Race Control and that involved strengthening the communication lines between the track and the ROC, as well as deploying new equipment and IT tools to give the ROC the same tools that exist in Race Control. Finally, I would say, probably the last third of 2023, it was actively contributing more to the identification of issues and the policing of the regulations in session.
Chris Bentley: This year the focus was on the development of the event management software platform, to enable more people and remote people to contribute in a much more efficient manner. If you look at how we did tracking of incidents last year, we were watching a video feed and if we saw an issue, we would then go on the intercom, verbally give a timestamp and a car and then Race Control would have to go and find it. What we’ve done this year is develop the systems to be a lot more automated so that from the ROC, we can pause the video, press a button, check the data is right and hit send to add the incident into the Race Control systems.
TM: Year two has been about developing the tool chain to work with people at remote sites and, to remove a lot of the verbal communication that we had in year one, as although it sounds strange, verbal communication tends to lead to miscommunication and errors.
Track limits continue to be a talking point in Formula 1. How has the ROC helped to improve the process of detection in 2023?
TM: As I think most people will know, track limits really came into focus at the Austrian Grand Prix where a number of drivers were penalised following the race and there were some 1200 potential track limits transgressions to examine. We’ve changed the approach significantly between then and now. As an example, at the recent Qatar Grand Prix we had eight people working on track limits instead of the four we had in Austria and between them they monitored 820 corner passes, which were then whittled own to 141 reports sent to Race Control and of those Race Control elected to delete 51 laps. Because of Austria and improvements made to the software, we can deal with those checks and turn them into 150 reports. Now it’s simple a case of clicking down a list of reports and saying yes or no.
When it comes to things like track limits, the ROC, at the moment, is a data crunching facility. Will it evolve in the future? Honestly, I think it might evolve slightly in the direction of VAR where we actually nominate people working remotely as a so-called judge of fact in the motor sport terminology. But for the moment, it is a data crunching resource designed to assist Race Control.
CB: It’s taking workload off Race Control’s desk, and it allows us to expand the total work done by Race Control during a race. It’s true, too, of things like time penalties
in the race. It used to be that would have to be looked at in Race Control. Now that’s taken offline. We have the ability, via software tools, to check multiple videos, mark the incident as analysed and create a composite video for race control or the stewards to look at. So again, it’s taking a lot of data, filtering that data, and then giving an easy package of data to Race Control, or to the stewards to enable them to do their work quicker.
What’s changed between Austria and now?
TM: If I was to boil it down to the main points in terms of looking at Austria and now then the first thing to say is that we made a state change. We used to have three competing data sources in terms of how a potential track limit would be identified, on-car detection, which was the car estimating its position relative to the track, you would get a loop detection, or you would get a manual detection via someone’s eyes. Essentially, what we concluded after Austria is that those three data sources were all sending potential reports and if you took out all the ones that were being erroneously reported by the loops, and then did the analysis to say all the ones that were correctly reported by the loops, were they also captured by the human, the answer was yes. So we basically concluded that the loops were insufficiently accurate. And that by far, our most accurate solution was having a data analyst looking at the video itself. In fact, that’s an interesting element of the story as currently, through loop positioning, through GPS positioning etc, the human still wins.
CB: We’ve turned off loops now for every circuit unless there’s a chicane, because it just gets in the way of what we’re trying to achieve. And ultimately the rule of thumb is that if it’s too close to call, then the benefit of the doubt goes with the driver.
What are the next steps in the ROC’s evolving suite of tools?
TM: What we’re trying to do for the future is improve all of that technology and deploy new ones. Car positioning continues to be developed to improve accuracy. We’re also planning to double the size of the ROC in terms of the number of people going from four to eight, next year, and we will double the connection bandwidth between the track and Geneva to facilitate more people working remotely.
CB: The next step is Computer Vision. This involves shape analysis, where we have a line that is the track edge and the software works out the number of pixels past that line.
TM: At the moment we’ve ‘brute forced’ the situation by saying ‘we need to make thousands of checks, how do we do that’? Well, we throw people at it, because that’s the most accurate solution. What we’re looking to do now is introduce a level above ROC, and that’s AI software.
Again, it might sound strange but the methodology with this AI has a lot of parallels with discussions going on in medicine at the moment and the use of Computer Vision, for example, to scan data from cancer screening. What they’ve concluded is they don’t want to use the Computer Vision to diagnose cancer, what they want to do is to use it to throw out the 80% of cases where there clearly is no cancer in order to give the well trained people more time to look at the 20%. And that’s what we are targeting.
So, as we said, currently it’s 800 down to 140, down to 50. What we’re targeting with the AI is to take that 800 down to 50 – to remove the ones that clearly don’t need a human review. So we have two layers of check now and we’ll add the extra Computer Vision layer upstream. And that will allow the expert users in the ROC to look at a smaller number of potential infringements, which should further reduce the number of reports that go to Race Control, and overall increase the speed of processing.
CB: And that’s happening this weekend in Abu Dhabi where we will parallel run Computer Vision. The parent company of SBG, the company that supplies the RaceWatch platform to the FIA, is called Catapult and they are the people that put the vests with tiny receivers on professional sportsmen and women, in NFL, football etc. There are examples in NFL where they can identify every player on the pitch, even if they’re in a big huddle. We can also use that technology on our live feeds. That will be the same as the new tool, and then we will be able to draw the ‘lines of interest’. And then the AI would learn as it goes along
It’s been a year of exponential development and improvement, so what are the key takeaways from 2023?
TM: I think the main one is to use the technology appropriately and make the technology work for you. That’s the big thing we’ve done over the course of this year. Secondly communication is still king. And as we expand that’s even more important, and that’s been a learning process, not only in terms of process and procedure, but also with people and how they interact with these new systems, with each other and with Race Control.
Take that forward to year three and the biggest imperative is to expand the facility and continue to invest in software, because that’s how we’ll make big strides. And the final takeaway for me is be open to new technologies and continue to evolve.
I’ve said repeatedly that the human is winning at the moment in certain areas. That might be the case now but we do feel that ultimately, real time automated policing systems are the way forward.